Qwen3-Embedding-4B FP8 on RTX 5080
Real measurements from 31 July 2026, embedding model served by vLLM
| Model | Qwen/Qwen3-Embedding-4B (text embedding, no generation) |
|---|---|
| Format | FP8 |
| GPU | NVIDIA GeForce RTX 5080, 16 GB, Blackwell architecture (sm_120) |
| Runtime | vLLM 0.24.0, torch 2.11.0+cu130, CUDA 13.0, driver 595.84 |
| Sustained throughput | 46 requests/s and about 14,100 tok/s on 1,024-character text |
| Peak throughput | 96 requests/s on 256-character text, 19,075 tok/s on 4,096-character text |
| Latency | p50 of 21.5 ms and p90 of 21.7 ms at concurrency 4 on 1,024 characters |
| Measurement date | 31 July 2026 |
The seven measured configurations
Each row is a batch of requests sent to the production server, at the stated concurrency, with randomized text. Throughput is computed over the full batch duration and latencies are recorded request by request.
| Configuration | Concurrency | Requests | Text size | Total time | Throughput | Throughput | Latency p50 | Latency p90 |
|---|---|---|---|---|---|---|---|---|
| 1 | 1 | 1 request | 1,024 characters | 33 ms | 30 requests/s | 9,288 tok/s | 32.9 ms | 32.9 ms |
| 2 | 1 | 10 requests | 1,024 characters | 217 ms | 46 requests/s | 14,032 tok/s | 21.6 ms | 22.0 ms |
| 3 | 4 | 20 requests | 1,024 characters | 433 ms | 46 requests/s | 14,173 tok/s | 21.6 ms | 22.0 ms |
| 4 | 4 | 50 requests | 1,024 characters | 1,078 ms | 46 requests/s | 14,112 tok/s | 21.5 ms | 21.8 ms |
| 5 | 8 | 100 requests | 1,024 characters | 2,157 ms | 46 requests/s | 14,174 tok/s | 21.5 ms | 21.7 ms |
| 6 | 4 | 50 requests | 256 characters | 518 ms | 96 requests/s | 7,456 tok/s | 10.3 ms | 11.0 ms |
| 7 | 4 | 50 requests | 4,096 characters | 3,178 ms | 16 requests/s | 19,075 tok/s | 63.5 ms | 64.1 ms |
Reading the measurements
On 1,024-character text, the server saturates at 46 requests per second from configuration 2 onward, and this ceiling does not move when concurrency rises from 1 to 8 or when the batch grows to 100 requests. Token throughput stays flat around 14,100 tok/s across the whole plateau, with p50 latency holding at 21.5 ms under concurrency 8.
On 256-character text, the server reaches 96 requests per second but processes only 7,456 tok/s: the fixed cost of each request dominates the compute on short inputs. On 4,096-character text, throughput peaks at 19,075 tok/s, which confirms the card is not limited by the model weights. The limiting factor on short inputs is the per-request overhead, between the vLLM scheduler queue and the client round trip.
Verdict
This pair suits enterprise RAG indexing and large-scale document embedding: at 14,100 tok/s sustained, a 10 million token corpus indexes in under twelve minutes on a consumer 16 GB card. It does not suit text generation, since Qwen3-Embedding-4B only produces vectors. For higher retrieval accuracy, the Qwen3-Embedding-8B variant in the same format is the next planned measurement on this machine.
Exact measurement procedure
The measurement ran on 31 July 2026 against the production vLLM server, exposed as an OpenAI-compatible endpoint on /v1/embeddings. The bench-emb.py script, written in Python with asyncio and a concurrency semaphore, sends seven configurations in randomized order: texts of 256, 1,024 and 4,096 characters, concurrency of 1, 4 and 8, batches of 1 to 100 requests. Tokens are those counted by the server in each response, and latencies are measured client side, request by request.
The machine is an Ubuntu 24.04 server fitted with a 16 GB NVIDIA GeForce RTX 5080 (Blackwell, sm_120), driver 595.84, CUDA 13.0, torch 2.11.0+cu130, vLLM 0.24.0. The Qwen/Qwen3-Embedding-4B model is served in FP8 with GPU memory utilization set to 0.35. The procedure is reproducible on any equivalent machine with the same script and the same endpoint.
Known limits
During the measurement, the card also hosted a reranking service using about 4 GB of video memory: the published figures are therefore conservative compared with a dedicated card. The saturation at 46 requests per second on 1,024-character inputs is not yet explained: it may come from the vLLM scheduler queue, the PCIe link or the measurement client, and a follow-up run will decide. Finally, these figures date from 31 July 2026 and cover a single server; another card or another vLLM version may produce different results.
Going further
The Qwen family is presented in the Qwen 3.6 and Gemma 4 datasheet, the server used here in the vLLM datasheet, and the memory mechanics behind these throughputs in the hardware: prefill and decode guide. The sizing matrix shows which generative models fit on each machine in the catalogue, and the measurements section lists the other measured or planned pairs.