Dynamo

Inference Runtime

Dynamo distributes inference for reasoning models across large GPU fleets. It separates the prefill and decode phases and routes requests based on KV cache state, which increases throughput at rack scale.

Key points: Disaggregated · GPU fleet · Prefill and decode separation · Cache-aware routing · Rack scale.

Role

Dynamo distributes inference for reasoning models across large GPU fleets, at rack scale.

How it works

It separates the prefill and decode phases and routes requests based on KV cache state, which increases overall throughput.

Use cases

Dynamo targets high-load deployments and demanding reasoning models, where a single machine is no longer enough.

Integration

It orchestrates multiple vLLM instances and sits above the inference layer for the largest fleets.