Prefill, decode and KV cache: choosing hardware for an LLM (2026)
An LLM does not stress hardware the same way when it reads your request as when it writes its answer, and understanding the two phases, and the role of memory bandwidth, is what lets you choose the right card and the right architecture, from a workstation to a rack.
Updated on 2 September 2026: DGX Station brought back to the NVIDIA sheet values (252 GB HBM3e at 7.1 TB/s, 496 GB LPDDR5X at 396 GB/s, 748 GB coherent); the Qwen3.6 35B A3B example recomputed from the model's config.json (10 full-attention layers out of 40, 2 KV heads, head dimension 256, i.e. 20 KiB per token) and the actual size of NVIDIA's NVFP4 conversion (23 GB); H100 PCIe replaced by the current H100 NVL sheet; Spark cluster capacity aligned with the NVIDIA product page; unsourced efficiency and price ratios removed.
The two phases of an LLM: prefill and decode
A generation unfolds in two phases with opposite hardware demands, and prefill processes your entire request in one pass: system prompt, context and question. These are large matrix-by-matrix multiplications, perfectly parallel, that saturate the card's compute. This phase, being compute-bound, sets the delay before the first token, and a high-power GPU with native FP4 or FP8 support shortens it.
Decode writes the response one token at a time, and each token depends on the previous ones; these are matrix-by-vector operations that do little compute but reread a large amount of data. This phase, bound by memory bandwidth, sets the generation speed, that is the time between two tokens, while compute intensity drops to a few dozen operations per byte here, and GPU utilisation often falls below half. The resulting rule is simple: prefill demands FLOPS, decode demands bandwidth.
Why is decode bound by memory bandwidth?
Since generation rereads all the model's weights at every token, speed depends directly on how fast the card reads its memory, which makes it the decisive factor, and it varies by a factor of thirty depending on the solution.
One example puts a figure on the gap. On the same compact model, the RTX PRO 6000 server generates around 215 tokens per second where a DGX Spark delivers about 67, three times slower, even though both load the model with ease. Both figures are independent measurements taken by Alex Ziskind in the video cited in the references, on a 4-bit quantisation served by llama.cpp, rather than calculated ceilings. On a seventy-billion-parameter model, a single Spark drops to just a few tokens per second in generation: its 128 GB of memory loads the model, but its 273 GB/s of bandwidth cannot serve it fast. At the other end of the scale, an H200, B200 or B300 rack targets maximum throughput for long context and large-scale serving, and a GB300 NVL72 rack pushes coherent memory to 20.7 TB across seventy-two cards.
The DGX Station illustrates a trap, flagged in orange on the diagram. Its 252 GB of HBM3e serve at 7.1 TB/s, but the machine adds 496 GB of much slower LPDDR5X (NVIDIA sheet re-read on 2 September 2026), so that as long as the model and the KV cache fit within the 252 GB of HBM, generation is fast. As soon as they spill over into LPDDR5X, that portion is read at RAM speed, 396 GB/s per the same sheet, and decode slows down sharply on the overflowed part. The large capacity is there to load very large models, not to serve all of them at full speed.
The KV cache, what weighs down decode
To avoid recomputing everything at every token, the model stores the keys and values of attention in a KV cache. This cache grows with context length, the number of concurrent requests and the number of layers, to the point that over long responses and several users it sometimes exceeds the size of the model itself.
Since it is reread at every token, it worsens the decode bottleneck and takes up video memory.
Three levers keep it in check: grouping attention heads, or GQA, which reduces the number of keys and values to store; KV cache quantization, which compresses it; and a controlled context length, while a fourth route, linear attention, replaces the growing cache with a constant state: Qwen3.8-27B applies it on 48 of its 64 layers. This is also why very long context first demands memory bandwidth, before any other optimisation.
What has to fit on a card
"The model fits on the card" is not limited to the weights. Four items share video memory, and the last one is variable: the model's weights, the embedding model and the reranker in a RAG pipeline, and the KV cache, which grows with context and the number of requests.
Take a concrete example: Qwen3.6 35B A3B in NVFP4, 256K-token context, on a 96 GB RTX PRO 6000. The A3B label means 3 billion parameters are active per token, but all 35 billion reside in memory, since a mixture-of-experts model loads all of its experts.
| Item | Size (order of magnitude) |
|---|---|
| Qwen3.6 35B A3B weights, NVFP4 (4-bit) | 23 GB read from nvidia/Qwen3.6-35B-A3B-NVFP4 on 2 September 2026 (all 35 B reside) |
| Embedding model, ~4 B in FP8 | ~4 GB |
| Reranker, ~2 B in FP8 | ~2 GB |
| Framework and activations | ~4 GB |
| Fixed subtotal | ~33 GB |
| KV cache for 256K tokens, in FP16 | ~5.4 GB |
| KV cache for 256K tokens, in FP8 | ~2.7 GB |
The KV cache is the item that varies, since its size per token equals twice the number of layers, times the number of KV heads, times the head dimension. The config.json of Qwen3.6 35B A3B declares forty layers of which only ten use full attention (the other thirty use linear attention, with a constant state), two KV heads and a head dimension of 256: 2 × 10 × 2 × 256 × 2 bytes = 20,480 bytes, i.e. 20 KiB per token in FP16, so ~5.4 GB for 262,144 tokens; half that in FP8 (~2.7 GB), half again at 4-bit (~1.3 GB). A 256K-token request with an FP8 KV cache therefore needs roughly 33 + 2.7 ≈ 36 GB, which fits comfortably on a 96 GB card. A model with full attention on every layer and more KV heads would multiply this item by five to ten.
How much context fits, then? After the fixed subtotal (~33 GB), around 63 GB remain for the KV cache. In FP16, at 20 KiB per token, that takes a single request to about three million tokens (63 GB / 20,480 bytes), far beyond the declared 262,144-token window. In practice, this budget is shared across concurrent requests: at 256K tokens per request with an FP8 KV cache (2.7 GB each), the card serves around twenty in parallel (63 / 2.7 ≈ 23). Lowering KV cache precision or enabling GQA therefore buys context or concurrency directly, and co-locating embedding, reranker and LLM on a single card is the typical pattern for a small sovereign RAG platform; beyond that scale, one card is dedicated to inference and another to the retrieval models.
The interconnect: when several cards must talk to each other
As soon as a model is split across several cards in tensor parallelism, every layer triggers an all-reduce that synchronises the cards, so at every token, and the speed of this link then becomes the bottleneck, with orders of magnitude that leave no room for doubt.
| Solution | Multi-GPU interconnect | Note (with or without NVLink) |
|---|---|---|
| GB300 NVL72 (rack) | NVLink 5 and NVSwitch, 130 TB/s across the domain | 72 B300 GPUs seen as one, 20.7 TB of coherent memory |
| B300 and B200 SXM | NVLink 5, 1.8 TB/s per card, full mesh | Near-linear tensor parallelism across eight cards |
| H200 SXM and H100 SXM | NVLink 4 and NVSwitch, 900 GB/s per card | Full mesh across eight cards, with NVLink |
| H200 NVL (PCIe) | NVLink bridge up to four cards, 900 GB/s; otherwise PCIe | Same memory bandwidth as the SXM, 4.8 TB/s |
| H100 NVL (PCIe) | NVLink bridge in pairs, 600 GB/s; otherwise PCIe | 94 GB at 3.9 TB/s, versus 80 GB at 3.35 TB/s for the H100 SXM (NVIDIA H100 sheet, re-read on 2 September 2026) |
| Any single PCIe card, no bridge | PCIe Gen5 x16, ~128 GB/s | Seven times slower than NVLink 4, without NVLink |
| DGX Station (GB300) | Internal NVLink-C2C, 900 GB/s between CPU and GPU | Single-GPU workstation, 748 GB of coherent memory (NVIDIA sheet) |
| DGX Spark cluster (ConnectX-7 RoCE) | 200 Gb/s, i.e. ~25 GB/s | Network link between machines, no switch |
On PCIe, the tensor-parallelism all-reduce drops scaling efficiency by a proportion that depends on the model and the batch size, and that is measured on the target configuration rather than assumed, where NVLink keeps this exchange negligible.
One card, or several? PCIe, NVLink, SXM
The right choice follows a clear logic, dictated by what fits on a single card and by the cost of the interconnect as soon as a second one is needed.
- A single card. No interconnect bottleneck. This is the best option when the model fits: the RTX PRO 6000 and its 96 GB then dominates on performance per cost, ahead even of an H100 on single-card throughput.
- Two to eight PCIe cards, without NVLink. Tensor parallelism suffers from the all-reduce. It is better to split the model in a pipeline, which only exchanges once per token, or to run independent replicas to serve several users. PCIe is enough for compact models and large request batches.
- SXM cards with NVLink and NVSwitch. Tensor parallelism scales almost linearly. But in inference, this mesh often goes unused: you pay more (SXM cards sell on quote, in a dedicated chassis) and draw more power (up to 700 W configurable per H200 SXM versus 600 W per H200 NVL, NVIDIA sheet) for capacity that is mostly useful for training. To serve a model, PCIe is the economical choice, except for very large dense models split at low latency.
Modern engines handle this trade-off: vLLM detects the topology at startup and switches to the all-reduce optimised for NVLink when cards are bridged, otherwise it falls back to PCIe.
Stacking DGX Sparks: capacity, not speed
Linking two DGX Sparks through their ConnectX-7 port pools 256 GB of memory and makes it possible to load a model that is impossible on a single machine; NVIDIA advertises up to 700 billion parameters across four linked machines (product page re-read on 2 September 2026).
But the 200 Gb/s link remains slow compared with internal memory: in low-latency serving, every added machine adds its exchange to the path of every token, and in batch mode the pipeline clearly outperforms tensor parallelism over this network link. The message fits in one sentence: a Spark cluster unlocks models too large for a single machine, it does not speed up a model that already fits.
How do you size the hardware for an LLM?
Sizing therefore depends on the model, the context length and the number of users, not on one universal king card, and price ranges by tier are covered in our AI server pricing guide, and model choice in our open-source LLM comparison. QDNA integrates this hardware and operates it for you, from a workstation to a rack.
- The model fits on one card: single GPU, the simplest and best cost per token.
- Latency priority on a very large model: SXM and NVLink, tensor parallelism within the machine.
- Multi-user throughput priority: single-GPU replicas or pipeline, PCIe is enough.
- Very long context, large KV cache: target memory bandwidth, H200 or B200, with GQA and cache quantization.
- Maximum capacity under budget or noise constraints: unified memory, Mac Studio or DGX Spark, capacity comes first, not throughput.
- Several machines: tensor parallelism within the machine via NVLink, pipeline between machines over the network.
Frequently asked questions
Why does my local LLM generate slowly despite a large GPU?
Generation, or decode, is limited by memory bandwidth, not by compute power. For every token, the card rereads all the weights and the KV cache. A card with large capacity but low bandwidth, such as the DGX Spark at 273 GB/s, can load a large model but generates it slowly.
Do you need NVLink for LLM inference?
Rarely. NVLink matters when a model is split across several cards in tensor parallelism, where an all-reduce synchronises the cards at every token. For everyday inference, PCIe is enough, and NVLink mainly benefits multi-GPU training. For very large models split across cards at low latency, it becomes useful again.
SXM or PCIe to serve an LLM?
PCIe is more cost-effective and sufficient for most inference workloads, since the NVLink mesh on SXM cards often goes unused during generation. SXM is justified for multi-GPU training, where the all-reduce across eight cards requires the NVSwitch's 900 GB/s.
Is the DGX Spark fast for an LLM?
The DGX Spark offers 128 GB of unified memory but only 273 GB/s of bandwidth. It excels at prototyping and with compact models, but generates large dense models slowly. Linking Sparks unlocks capacity, up to 700 billion parameters across four machines according to NVIDIA, not per-token speed.
How do you reduce the size of the KV cache?
Three levers: grouping attention heads, or GQA, which reduces the number of keys and values to store; KV cache quantization, which compresses these activations; and a controlled context length, since the cache grows with conversation length and the number of concurrent requests.
Let's size your hardware to your real usage
A call to scope the card, the architecture and the runtime against your models, your context length and your number of users.
Book a call