LLM inference engines compared in 2026: vLLM, SGLang, TensorRT-LLM, llama.cpp and Dynamo

Choosing an LLM inference engine in 2026 shapes throughput, latency and memory use as much as the accelerator itself does, so this review compares vLLM, SGLang, TensorRT-LLM, NVIDIA Dynamo, llama.cpp and Ollama on published measurements from Jarvislabs, Spheron, Lambda, LMSYS, Red Hat and Tom's Hardware, and it covers multi-token prediction (MTP), prefix caching and prefill/decode disaggregation.

By QDNA · Published on 26 September 2026 · 16 min read

What does an LLM inference engine actually do?

An inference engine schedules the prefill and decode phases, manages the KV cache, groups requests into a continuous batch and optionally runs speculative decoding. The same model on the same GPU can serve several times more tokens per second (tok/s) depending on the engine, its quantization formats and its caching strategy.

Serving a prompt involves two distinct workloads, since prefill processes the whole input in one pass and dominates the time to first token, while decode generates one token per step and dominates the rest of the response. The engine decides how much memory the KV cache occupies, when requests join or leave the batch, and how attention kernels split across GPUs, while continuous batching, introduced by vLLM, keeps the accelerator busy by admitting new requests as soon as older ones finish, which raises aggregate concurrency without manual tuning.

Two families of techniques now separate engines further: speculative decoding drafts several tokens with a small model or an MTP head, verifies them in parallel and accepts the correct prefix, which cuts latency at low batch size, whereas a prefix cache stores the KV states of already computed prompt prefixes, so conversations and shared system prompts skip part of prefill. SGLang implements this through RadixAttention, while vLLM offers automatic prefix caching, and prefill/decode disaggregation goes one step further by running the two phases on separate pools of GPUs, which Dynamo orchestrates with KV-cache-aware routing.

Which inference engines are worth comparing in 2026?

Six inference engines matter in 2026: vLLM, SGLang, TensorRT-LLM, NVIDIA Dynamo, llama.cpp and Ollama. Five are Apache 2.0 or MIT licensed, and they differ mainly in speculative decoding support, prefix caching, prefill/decode disaggregation, accepted weight formats and how fast they serve the models of the moment.

Ollama remains a wrapper over llama.cpp: the Red Hat 2026 serving benchmark, as quoted by Towards AI, measured 793 output tok/s for vLLM against 41 for Ollama under load, and 920 against 155 tok/s at 50 users, with a P99 of 80 ms versus 673 ms. Dynamo does not execute models itself, since it splits prefill and decode across separate GPU pools and routes requests by KV cache content. TensorRT-LLM compiles engines per model and hardware, which Spheron measured at about 28 minutes of cold start versus 62 seconds for vLLM, and SemiAnalysis reports that a hidden size hard-coded at 4,096 corrupted DeepSeek V4 Pro outputs for nine days before a fix, while vLLM and SGLang served the model from its release.

Which 2026 features change the engine choice?

The 2026 features move the competition from kernels to variable-length speculation, recurrent state management and disaggregation: DSpark and DFlash2 on the drafter side, the unified radix tree and HiCache on the cache side, decode context parallelism and W4A4 on the Blackwell side. Each feature has a merge date and a published measurement.

• Speculation with variable verification. DSpark, published by DeepSeek in July 2026 and integrated into SGLang on 6 July and into vLLM for Kimi K3 on 27 July, sizes the verified window per request from the drafter's confidence: LMSYS measured on Kimi K3 a 68 percent gain at batch 256 on chat and 24 percent on math against full verification. Red Hat's FastMTP heads, since Speculators 0.6 and vLLM 0.24, retrain the MTP head for recursive drafting.
• Caching for hybrid architectures. vLLM rebuilt its prefix cache on 27 July 2026 for the recurrent state of Kimi K3 (69 KDA layers for 24 MLA layers), which also benefits Qwen3.8 and GLM 5.3 Flash; SGLang moves KDA state with copy-on-write and snapshots, and offers an opt-in unified memory where states and KV blocks draw from one pool.
• Unified radix tree and HiCache. SGLang 2026 release notes raise the cache hit rate from 43.8 to 60.8 percent and cut mean time to first token from 1.57 to 1.07 s, while HiCache tiers the KV cache across HBM, host memory and disk for long-running agent sessions.
• Blackwell in FP4. SGLang's W4A4 MegaMoE path, live since 2 June 2026 on DeepSeek V4, quantizes activations to MXFP4 as well; version 2 KV compression kernels and MHC path fusion cut DeepSeek V4 prefill overhead; decode context parallelism (DCP) breaks the tensor parallel ceiling at 128K input on eight B200 GPUs.
• Bubble-free scheduling. Breakable CUDA graphs on the DeepSeek V4 prefill path, SGLang's overlap scheduler and vLLM's V1 engine remove idle time between steps, and ReplaySSM shrinks the draft memory of state-space models from 11.5 to 1.8 GB according to SGLang notes.
• Workstations. llama.cpp merged multi-token prediction on 16 May 2026, then EAGLE-3, DFlash and DSpark drafters, and Google has published MTP drafters for the whole Gemma 4 family since 5 May 2026, announced at up to 3x and picked up by vLLM, SGLang, MLX and Ollama.

What do 2026 benchmarks say on Blackwell and on recent models?

The 2026 Blackwell benchmarks show that, at a fixed interactivity target, throughput per GPU depends first on the serving recipe, disaggregation, expert parallelism and speculation, and then on the engine: DeepSeek V4 Pro went from 2,200 to 11,200 tokens per second per GPU on GB300 NVL72 in two months with the same SGLang.

Three lessons follow from these measurements. First, the same engine gains a factor of five in eight weeks on a new model, so the date of a measurement matters as much as the engine name: the InferenceX GB300 lane moved from MTP to DSpark in September 2026 and the curves still move. Second, prompt structure decides the ranking, since Spheron measured less than 5 percent between vLLM, SGLang and TensorRT-LLM on unique prompts, while The AI Engineer measured 29 percent more throughput for SGLang, 16,200 against 12,500 tokens per second, once more than 60 percent of prefixes are shared, which is the regime of agents. Third, Hopper comparisons remain a useful but dated reference: on H100 in May 2026, Jarvislabs placed vLLM ahead on unique prompts (23,523 tokens per second against 16,787 for SGLang and 16,517 for TensorRT-LLM on Qwen2.5-7B) and SGLang ahead on per-token latency on RULER 16K (63.2 versus 96.7 ms), with output flat between 553 and 582 tokens per second from sixty requests upward, a sign that prefill saturates before the engine does.

On Blackwell servers, the Lambda run confirms the maturity of TensorRT-LLM on its vendor's hardware for Nemotron 3 Ultra, while the provisional vLLM 0.22 figure, 450 of 512 requests completed, shows how young the support for this hybrid Mamba model was at test time. The Nemotron 3 Ultra technical report claims up to six times the throughput of competing open models at NVFP4 precision on GB200, but it compares TensorRT-LLM for itself against vLLM for the others, which rules out reading it as an engine comparison at constant recipe.

How much does multi-token prediction change, per engine and per model?

Multi-token prediction speeds up single users by 1.2x to 3.1x depending on the engine, the model and the drafter, then decays as batch size and context length grow. Acceptance rate per position decides the gain, and published measurements recommend two or three speculative tokens rather than five.

Native MTP heads differ per model family, as Red Hat showed on 8 September 2026 with vLLM 0.24, Speculators 0.6 and two H200 GPUs by measuring per-position acceptance of 0.897, 0.719 and 0.476 on Qwen3-Next-80B, raised to 0.912, 0.776 and 0.616 by FastMTP heads, for up to 1.25x median inter-token latency. The Nemotron 3 Ultra technical report of 9 June 2026 reports a mean acceptance length of 4.387 at draft length 7 on SPEED-Bench, against 2.667 for DeepSeek V4 Flash.

Generic drafters now rival native heads, since DeepSeek's DSpark (arXiv 2607.05147, July 2026) reports 60 to 85 percent higher per-user throughput than MTP-1 in production, while SGLang measured 383.7 tok/s at batch 1 on DeepSeek V4 Pro TP8 B300 on 6 July 2026, and vLLM raised Kimi K3 from 118 to 370 tok/s per user, a 3.14x factor, on 16 GB300 GPUs, with accepted length 4.73 in code and 2.61 in creative writing. Kimi K3 ships no native MTP head, per its config.json, and on AMD MI300X and MI355X, the vLLM blog of 23 August 2026 measured up to 2.87x with DFlash on Gemma 4 26B-A4B, 2.83x with Gemma MTP and 2.20x with native MTP on Qwen3.5-122B-A10B on MATH500.

llama.cpp merged MTP on 16 May 2026 in PR 22673, and when measured on Qwen3.6 27B on an RTX PRO 6000, throughput went from 45.76 to 79.37 tok/s, a 1.73x gain, while a MoE 35B-A3B model gained only 1.17x per Jarvislabs. On a DGX Spark with Q4_K_M and five draft tokens, an NVIDIA forum measurement from May 2026 shows 28.3 tok/s versus 13.1 at one request, but 29.9 versus 41.5 at four requests, and issue 23752 tracks a regression up to 28 percent on Apple Metal. A Hugging Face discussion on Qwen3.8-27B-FP8 measured acceptance of 0.702, 0.298, 0.149, 0.085 and 0.085 across five speculative tokens, an accepted length of 2.32 and 23 tok/s on an RTX PRO 6000, hence the advice to stop at two or three tokens. VLLM issue 47602 documents acceptance decay with context length on Qwen3.6-27B.

How does throughput scale with context length and concurrent users?

Throughput first saturates during prefill as context grows, then hits memory capacity per request as concurrency rises, so engines add chunked pipelines, decode context parallelism and prefill/decode disaggregation. Time to first token, not raw tok/s, becomes the binding constraint for long inputs under load.

LMSYS showed the prefill wall on 15 January 2026 with chunked pipeline parallelism: DeepSeek V3.1 at one million tokens saw TTFT fall from 48.5 s with PP1 TP8 to 15.5 s with PP4 TP8, a 3.31x prefill throughput gain. Qwen3-235B-A22B FP8 needed 420.91 s at one million tokens in PP8 TP4 but 10.54 s at 128K. On GB300 NVL72 with DeepSeek R1 NVFP4 at 128K input and 8K output, LMSYS measured 226.2 tok/s per GPU on 19 February 2026, and MTP kept 224.2 tok/s per GPU while raising per-user speed from 23 to 43 tok/s. Capacity reached 40 requests per GPU with 288 GB versus 24 with 192 GB, at 35,136 bytes of KV cache per token, and TTFT held at 8.6 s for 128K inputs with 32K dynamic chunks.

Decode-side scaling has its own limit, since SGLang 2026 release notes report that at 128K input on eight B200 GPUs, plain tensor parallelism plateaus near 680 tok/s while decode context parallelism keeps scaling with more chips. Disaggregation composes with speculation: the PyTorch and SGLang post of 23 June 2026 measured about 11,200 tok/s per GPU on DeepSeek V4 Pro FP4 at 8K input and 1K output on GB300 NVL72 with Dynamo and MTP, versus 2,200 on day 0, a 5x gain at 50 tok/s per user. InferenceX counts that throughput per decode chip, which matters when you compare bids. Two cautions close the loop, since arXiv 2605.24217 shows that single-process Python benchmark clients inflate measured TTFT and TPOT under load and proposes NTPOT, and SiliconBench (arXiv 2609.19169) measured 4.3x to 7.7x scaling from concurrency 1 to 16 with flat TTFT for CUDA vLLM and SGLang on DGX Spark, while llama.cpp flattened on agent load.

Which engine for an RTX 5090, a DGX Spark or a Blackwell server?

For workstations, SGLang currently leads on a single RTX 5090 and vLLM wins on dual RTX 5090 at long context, while Blackwell servers favor TensorRT-LLM or a SGLang and Dynamo pair depending on the model release date. Measure on your own prompts before committing, because software bottlenecks outweigh memory capacity.

When Tom's Hardware benchmarked Qwen3.8 27B on 8 September 2026, one RTX 5090 under vLLM delivered 20 tok/s without MTP with context capped at 32K, while SGLang ran almost three times faster than vLLM on that single card. Two RTX 5090 cards under vLLM held 70 to 80 tok/s across the context sweep and 100 to 110 tok/s with MTP. Llama.cpp showed a time to first token of about 30 minutes at long context on an RTX 5090, and a DGX Spark reached about 20 tok/s with MTP but kept interactive TTFT up to its native 262,144-token context.

Quantization changes the arithmetic, as the SGLang v0.5.19 cookbook measured 202 configurations on RTX 5090, RTX PRO 6000 and DGX Spark: NVFP4 with EAGLE reached 152.9 tok/s per user on an RTX 5090 with FP32 state versus 144.5 with BF16, and weights dropped from 28.5 GB in FP8 to 16.5 GB in NVFP4. The vLLM recipe for two RTX 5090 cards reports 445,875 KV tokens of capacity at 262K context in NVFP4 with an MTP acceptance of 0.897. On Blackwell clusters, TensorRT-LLM led the Lambda run on Nemotron 3 Ultra, while SGLang with Dynamo delivered the PyTorch five-fold gain on DeepSeek V4 Pro, so align the engine with your model's day-0 support.

Official sources

• Jarvislabs, vLLM vs SGLang vs TensorRT-LLM comparison: jarvislabs.ai
• Spheron, vLLM vs TensorRT-LLM vs SGLang benchmarks: spheron.network
• Lambda, Nemotron 3 Ultra inference benchmarks: lambda.ai
• Red Hat, optimize vLLM speculative decoding with FastMTP heads: developers.redhat.com
• vLLM, speculative decoding on AMD GPUs: vllm.ai
• vLLM, Kimi K3 with DSpark drafters: vllm.ai
• LMSYS, chunked pipeline parallelism for long context: lmsys.org
• LMSYS, GB300 long-context serving: lmsys.org
• LMSYS, DSpark in SGLang: lmsys.org
• PyTorch, serving DeepSeek V4 on GB300 with SGLang: pytorch.org
• DeepSeek, DSpark (arXiv 2607.05147): arxiv.org
• arXiv 2605.24217, benchmark client bias and NTPOT: arxiv.org
• SiliconBench, arXiv 2609.19169: arxiv.org
• Tom's Hardware, Qwen3.8 27B on RTX 5090 and beyond: tomshardware.com
• SGLang cookbook, Qwen3.8 27B configurations: docs.sglang.io
• vLLM recipes, Qwen3.8 27B on dual RTX 5090: recipes.vllm.ai
• InferenceX (SemiAnalysis), Kimi K3 on B300: inferencex.semianalysis.com
• InferenceX (SemiAnalysis), GLM 5 and 5.1 on B200 and MI355X: inferencex.semianalysis.com
• InferenceX (SemiAnalysis), DeepSeek V4 from day 0 to day 43: inferencex.semianalysis.com
• NVIDIA, NIM optimizations on Nemotron 3 Ultra: developer.nvidia.com
• LMSYS, Kimi K3 day-0 support: lmsys.org
• Nebius, MLPerf Inference v6.0 results: nebius.com
• Google, MTP drafters for Gemma 4: blog.google
• SGLang releases: github.com
• NVIDIA, Nemotron 3 Ultra technical report: research.nvidia.com
• NVIDIA forum, MTP in llama.cpp on Qwen3.6 27B: forums.developer.nvidia.com