LLM throughput in tokens per second: the four conditions that change everything, measured on sixteen machines

An AI server is sold with one figure, so many tokens per second, and that figure depends as much on the software, the model format and the number of users as on the machine itself. Here is how to read it and what to ask before buying, with tables and sources in hand.

By QDNA · Published on 30 August 2026 · Updated on 7 September 2026 · 14 min read

Updated on 7 September 2026: rewritten in plain language with a glossary up front, eight sourced tables (llama.cpp, LMSYS, NVIDIA white paper, Hugging Face, fifteen sources of DGX Spark measurements) on top of the four videos of the original version; the professional card price remains the idealo.fr listing of 2 September 2026.

Reading this article: five words to know

A language model works in small pieces of words called tokens, and everything that follows is measured in tokens per second; here are the five notions the tables rely on, without which a throughput figure says nothing.

• Token. A piece of a word, about three quarters of a word in English. The model reads and writes token by token.
• Reading the input text (engineers say prefill; "pp512" in the tables means 512 tokens read at once). It is fast, because the machine processes the whole text in parallel rather than word by word.
• Writing the answer (generation; "tg128" means 128 tokens written one by one). This is what the user sees scrolling on screen, and it is therefore the figure that matters most in practice. For every token written, the machine re-reads the entire model in memory: speed therefore depends on memory speed, not on the number of cores.
• Model compression (quantisation). A model is stored with 16, 8 or 4 bits per value; the tables note these formats F16, Q8_0 and Q4_0, and other names exist (FP8, NVFP4, MXFP4). Fewer bits mean a smaller file that is re-read faster, at the cost of answers that are sometimes poorer.
• One user or several. A single stream is one person waiting for an answer; aggregate throughput adds up all the answers written at the same time for several people. The two figures are unrelated and often confused.

One last notion: the calculated maximum speed, obtained by dividing memory bandwidth (in gigabytes per second) by the size of the model file (in gigabytes). It is a theoretical ceiling, never reached, and our pages always name it a calculation, never a measurement.

What is the same model worth on sixteen machines?

From 14 to 290 tokens per second when writing an answer, for the same model file and the same test command, a factor of twenty between the M1 chip of a 2020 Mac and the RTX 5090 graphics card of 2025.

These figures come from the two public tables of llama.cpp, a widely used free program for running models locally, where hundreds of people measure the same file with the same command on their own machine; table 1 brings them together, with the memory bandwidth stated by the manufacturer whenever an official source gives it.

Table 1. The same model (Llama 2 7B, 3.79 GB file compressed to 4 bits) tested with the same command on sixteen machines. Sources: public llama.cpp tables, discussions 4167 and 15013, re-read on 6 September 2026. These are measurements.

Three things can be read in this table, starting with NVIDIA's DGX Spark and Apple's M4 Pro, which have the same memory bandwidth and write 57 and 51 tokens per second, a 12% gap that comes from the software and the driver rather than from memory. The RTX 5090 gaming card beats the RTX PRO 6000 professional card by 6% at equal bandwidth, which says what the professional card really sells: 96 GB of memory, not speed. Finally the H100, a data-centre chip, writes only 268 tokens per second on this small model, less than a gaming card: a 3.79 GB file does not keep a chip of that size busy, and a ranking made on a small model does not hold for a large one.

Can a machine's speed be predicted from its spec sheet?

A maximum can be calculated, never the real speed: the machines measured reach between 44% and 83% of their calculated maximum, depending on the chip.

The calculation is simple, since the machine re-reads the whole model for every token written: if its memory delivers 273 gigabytes per second and the file weighs 3.79 gigabytes, it cannot write more than 72 tokens per second, which is the DGX Spark's case, and it actually writes 57, or 79% of that maximum. Our article on how a model reads, writes and uses its working memory sets out that mechanism.

Table 2. Calculated maximum speed = memory bandwidth divided by 3.79 GB. Measured speed comes from table 1. The maximum is a calculation, the share reached a comparison.

The missing share has an explanation: on every token the machine also spends time launching its calculations, re-reading its working memory and coordinating its units. That share grows with the size of the chip: Apple's two Ultra chips, with 76 and 80 GPU cores, reach only 44% to 45% because a seven-billion-parameter model is not enough to keep them busy, while the small M1 chip reaches 79%. A calculated maximum therefore serves to spot an impossible figure or a badly tuned installation, not to rank two machines.

What is gained by compressing the model?

Speed multiplied by 2.3 to 3.2 when going from 16 to 4 bits, for a file 3.6 times smaller, on the six chips measured.

Table 3 takes six Apple chips from the same llama.cpp table and changes nothing but the file format: uncompressed (13.48 GB), compressed to 8 bits (7.16 GB) or to 4 bits (3.79 GB), sizes read on the Hugging Face repository that hosts these files.

Table 3. Same chip, same model, three compression levels; answer-writing speed in tokens per second (llama.cpp, discussion 4167). Measurements.

The gain follows the file size without matching it: the file is 3.6 times smaller, writing runs 2.3 to 3.2 times faster, because the fixed share of time per token does not shrink with the model. On a model ten times larger, Alex Ziskind observes the same thing: Llama 70B on one RTX PRO 6000 card writes 20 tokens per second at 8 bits and 60 at 4 bits. That gain is paid in answer quality, which no speed table shows and which must be checked on your own tasks.

Does the software count as much as the machine?

Yes, because a mere update of the same software gains up to 33% on one chip and one file, while changing software altogether gains another 14% for a single user.

Table 4 follows one M2 Ultra chip and one file through four successive versions of llama.cpp, the first without an optimisation called Flash Attention, the next three with it.

Table 4. Same chip (76-core M2 Ultra), same file, four successive versions of the same software; writing speed in tokens per second (llama.cpp, discussion 4167). Measurements.

Between the first and the last version, 4-bit writing goes from 94 to 125 tokens per second. The Tech-Practice channel measures the other side of the same phenomenon by changing software rather than version: on one Apple M5 Max machine and one Qwen3.6-35B-A3B model at 4 bits, MLX writes 98 tokens per second where llama.cpp writes 86. A speed figure without the software's name and version therefore does not say what it compares.

What happens when the model no longer fits in memory?

Speed collapses, and even a model that just fits into memory can remain unusable for a single person, because the machine then spends all its time re-reading a model too large for it.

Alex Ziskind sets two cards against each other on Llama 70B: the RTX 5090, with its 32 GB, cannot hold the model and falls to 1.8 tokens per second because part of the model lives off the card, while the RTX PRO 6000, which holds it entirely, writes 7; four times more, for a model that neither serves comfortably.

Fitting in memory is not enough either. LMSYS fits Llama 3.1 70B at 8 bits into the DGX Spark's 128 GB and measures 2.7 tokens per second for a single user: 273 GB/s divided by some 70 GB of model gives a calculated maximum close to 4, and the measurement reaches 69% of it. The machine does not overflow, it is simply too slow for that model with one person, which LMSYS sums up by reserving such uses for prototyping.

Is the most expensive card the fastest?

No, not on a model that fits in both cards, and two independent sources show it on two different models.

Table 5 takes the LMSYS review of the DGX Spark, which runs GPT-OSS 20B with the Ollama software on three NVIDIA machines.

Table 5. Three NVIDIA machines, tokens per second for a single user (LMSYS, DGX Spark review, 13 October 2025). Measurements.

The RTX PRO 6000 Blackwell, selling for more than €12,000 (idealo.fr, 2 September 2026), writes 215 tokens per second against 205 for the RTX 5090, and table 1 even reverses the order on Llama 2 7B with 274 against 290. Alex Ziskind reads the same thing on Gemma 3 4B: an RTX 5080 writes 119 tokens per second, an RTX 5090 writes 239 and the RTX PRO 6000 server writes 233. On a model that fits everywhere, memory bandwidth decides, and both Blackwell cards have the same. The professional card takes the lead back as soon as the model grows, which the same series shows on Gemma 3 12B, going from 50 to 88 tokens per second. Buying 96 GB to serve a four-billion-parameter model means paying for capacity that goes unused; the DGX Spark, at 49.7 tokens per second on the same model, pays for its slow memory.

How many users at the same time?

Serving several people at once multiplies the total of tokens written by 3 to 20 on a machine that does not change: it is the strongest factor, and the one most often left unsaid.

Table 6 gathers five independent sources on the DGX Spark, alone or in a pair, first with one user, then with several.

Table 6. One user, then several: five independent sources, tokens per second (re-read on 2 and 6 September 2026). Measurements; the gains are computed by QDNA.

The gain depends on the model and the software, since a conventional eight-billion-parameter model gains a factor of 18 between 1 and 32 requests, because writing one more token for another person costs almost nothing in re-reading the model. A 284-billion-parameter "expert" model, which activates only part of itself on each token, gains only a factor of 3 with 12 requests, because each request wakes different parts. The total for several serves to size a shared service; the speed for one person serves to judge a single user's comfort; a figure that does not say which it is serves no one.

More unsettling still, the ranking between machines inverts with the length of the requested answer. In Alex Ziskind's video, for 128-token answers the Spark leads, while for 1,024 tokens the Mac Studio takes the lead back with 396 tokens per second against 386, so that two honest articles can name two opposite winners without either of them lying. Since the author does not name the model precisely, we take the ratios, not the absolute values.

Where can reliable DGX Spark measurements be found?

In some fifteen public sources, five of which give the command and the versions needed to redo the measurement; on one model, their results range over a factor of four.

The DGX Spark is the most measured machine in its class, because its 128 GB of shared memory and its modest bandwidth make it a textbook case of the calculated maximum; table 7 gathers the sources we re-read, with one example each, from NVIDIA's official post to user repositories.

Table 7. Fifteen public sources of DGX Spark measurements, with one example each (reading / writing in tokens per second; "→" separates a single user from the total for several). All re-read on their original page on 6 September 2026.

Three families stand out among these sources, beginning with the reproducible ones (llama.cpp discussion 16578, the dgx-spark-field-notes repository, Kubesimplify, DandinPower, nabe2030, howtospark.com) give the command, the version and the text lengths. Launch-day reviews (ServeTheHome, StorageReview, Level1Techs, LMSYS) photograph October 2025 software that has improved a great deal since, while NVIDIA's post gives text lengths but no software version. Two community repositories even make the same maximum calculation as our table 2 and find the same share reached: DandinPower measures 10.7 tokens per second on Qwen3 32B against a maximum of 15, nabe2030 measures 64% and 78% of the maximum on two expert models.

Table 8 shows what that diversity does to a single model, gpt-oss 120B, the most measured file on this machine, for a single user.

Table 8. The same gpt-oss 120B model on a single DGX Spark, for a single user: eight readings from seven sources. Measurements by the sources; the ranking is QDNA's.

A factor of four on a machine and a model that have not changed, and every gap has a cause that can be read in the conditions. ServeTheHome measures with Ollama without any tuning on launch day; Level1Techs counts the wait before the first word inside its speed; the two llama.cpp readings of October 2025, ten days apart, go from 42.0 to 55.4 tokens per second through the software update alone; Dendro Logic asks a 1,500-token question and StorageReview uses another compression format; in August 2026, the same user measures 42.1 with Ollama and 60.7 with vLLM on the same question. A "DGX Spark speed" figure without those conditions therefore says something only about the person who produced it.

What to ask before believing a figure?

Six pieces of information, without which no one can tell whether the stated gap comes from the machine or from a mere setting; the tables above show that each of them moves the figure by at least 10%.

• The exact model, with its size in parameters and the file weight in gigabytes.
• The compression level, naming the format (Q4_0, Q4_K_M, FP8, NVFP4, MXFP4), not only the bit count.
• The software and its version, with the options that matter.
• The length of the answer, and of the input text if reading counts.
• The number of people served at the same time, and whether the figure is a total or the speed for one.
• The nature of the figure: a measurement with its procedure, or a maximum calculated from the spec sheet.

This is the discipline we apply to our own pages, where a calculated maximum is named a calculation and a measurement is published with its procedure. Our measurement pages carry only pairs that were actually measured, the others staying without a published value, and our memory calculator separates what is calculated from what is measured in the same way.

References

The llama.cpp tables, the LMSYS review, the NVIDIA white paper, the Hugging Face repository and the fifteen sources of table 7 (linked in the table itself) were re-read on 6 September 2026; the three videos and the three multi-user readings had been re-read on 2 September 2026. No value was reproduced by QDNA; the maximums, shares and gains are QDNA calculations from those sources. The RTX PRO 6000 Blackwell 96 GB price is the idealo.fr listing of 2 September 2026 (from €12,483 for the Max-Q, €14,314 for the Server Edition).

• llama.cpp, discussion 4167: measurements on Apple chips, Llama 2 7B at 16, 8 and 4 bits
• llama.cpp, discussion 15013: measurements on graphics cards (CUDA), Llama 2 7B at 4 bits
• LMSYS, in-depth review of the DGX Spark, 13 October 2025 (SGLang and Ollama, 1 to 32 requests)
• NVIDIA, RTX Blackwell GPU architecture white paper: bandwidths of the RTX 5090, 5080, 5070 and 3090
• NVIDIA, RTX PRO 6000 Blackwell Workstation Edition datasheet: 96 GB GDDR7, 1,792 GB/s
• Hugging Face, ggml-org/Llama-2-7B-GGUF: file sizes at 16, 8 and 4 bits
• Entrpi on the NVIDIA developer forum, DeepSeek V4 Flash on one DGX Spark, 1 to 12 requests
• Classmethod, DeepSeek V4 Flash 0731 on two DGX Spark, vLLM, 1 to 6 users
• MiaAI-Lab, DeepSeek V4 Flash on two DGX Spark, DSpark, results of 14 August 2026
• Tech-Practice, MLX against llama.cpp on Qwen3.6-35B-A3B, Apple M5 Max
• Alex Ziskind, measurements on a 96 GB RTX PRO 6000 server, several models and compressions
• Alex Ziskind, four machines compared, llama.cpp then vLLM with more and more users