Working Notes: a commonplace notebook for recording & exploring ideas.
Home. Site Map. Subscribe. More at expLog. — Kunal

Challenges & Research Directions for LLM Inference Hardware

Notes

Inference basics

• Prefill: user prompt/query with a lot of tokens
• Decode: sequentially generate the output ('autoregressive')
• KV Cache: proportional to input + output seq length
• Generally run prefill & decode on different servers, making decode less memory bound

Decode challenge 1: memory

• inherently memory bound

• memory bandwidth moves more slowly than flops: 80x vs 17x

• normalized price of capacity & bandwidth are increasing over time

  • hbm and dram prices will diverge

• sram is insufficient, tried by groq

• ddr is becoming cheaper

• notes from gpt:

  • ddr double data rate dr, relies on flip flops
  • dr dynamic ram needs to be refreshed constantly, stored in capacitors + transistor
  • sr static ram stored in flip flop transistors, doesn't needto refresh constantly
    • used for cpu cache
    • fastest and most expensive
  • hbm stack -
    • stack dram dies vertically with microscopic wires
    • placed next to gpu on silicon interposer
    • bandwidth = wire * speed * edge; hbm pushes many wires slowly
    • modern gpus are: 80-192gb with 3-8TB/s bandwidth across all stacks

end2end latency

• measured by time to first token
• or by time to completion
  • slower with longer outputs
  • each decode iteration has higher memory access latency
• interconnect latency outweights bandwidth

Research opportunities

For modern systems design

• tco, total cost of ownership

• average power consumption

• carbon dioxide equivalent emmisions

• performance must be meaningful

• must be delivered within datacenter capacity, constrained by power, space, co2e

• power & co2e are first order targets

directions in the paper

• high bandwidth flash

  • limited writes, store weights in it
  • nees larger reads
  • mix types of memory

• processing near memory

  • augment memory dies with small processors
  • processing in memory (on the same die) is too hard to use
  • pnm allows for larger shards to break llms apart
  • pim may still make more sense for mobile

• 3d memory logic staciking

  • compute on hbm base
  • custom 3d designs enable bandwidth / higher bandwidth per watt with wider & denser memory interface
  • harder to manage heat
  • memory logic coupling needs some standardization

• low latency interconnect

  • better topology can improve latency
  • process in network
  • ai chip optimization for interconnect
  • improve reliability, allow far imperfect comms

• need a roofline based simulator to explore options

• + more sharding techniques

• current hardware design doesn't match what decoding nees

• aside: good llm design feels like it needs a very detailed & realistic constraint solver

• notes/cost_efficient
• books/parallel_programming
• notes/parallel_programming/lesson_3
• notes/transformers/inference
• notes/gpumode/002_bookch
• notes/speed
• notes/gpumode/068_gpulandscape
• notes/deepseekv3
• notes/memcached
• notes/buildingllms