vLLM

Inference Runtime

vLLM delivers high-throughput serving for production. It relies on PagedAttention, tensor and pipeline parallelism, free prefix cache reuse, and FP8 or NVFP4 quantization. Its interface remains compatible with the OpenAI protocol.

Key points: Production · PagedAttention · Prefix cache · Multi-GPU · FP8 and NVFP4 · OpenAI API.

Role

vLLM delivers high-throughput serving for production. It serves open-weight models behind an interface compatible with the OpenAI protocol.

How it works

It relies on PagedAttention, tensor and pipeline parallelism, free prefix cache reuse, and FP8 or NVFP4 quantization.

Use cases

vLLM serves multi-user production workloads on GPU servers, where throughput and cost per token matter.

Integration

It is exposed to LiteLLM and to the platform, and is the default inference engine on RTX PRO 6000 and H200 servers.