Architecture

Inference runtime

The engines that serve models, from high-throughput production to edge inference, with fine-tuning and the ninfer engine optimized for RTX 5090 and RTX PRO 6000 Blackwell.

vLLM

vLLM delivers high-throughput serving for production.

Triton Inference Server

Triton serves models from multiple frameworks on the same infrastructure.

Dynamo

Dynamo distributes inference for reasoning models across large GPU fleets.

llama.cpp

llama.cpp stays lightweight.

Unsloth

Unsloth speeds up fine-tuning by a factor of two, according to its publisher, while cutting the memory required, through the LoRA and QLoRA methods.

ninfer

ninfer is a C++/CUDA inference engine designed to maximize single-GPU throughput on NVIDIA GeForce RTX 5090 and RTX PRO 6000 (Blackwell sm_120).

Which engine should you choose?

The engine does not change what a model can do. It decides throughput, how many concurrent requests the machine sustains, and which weight formats are accepted. The same model served by two engines needs the same memory for its weights, but does not deliver the same service.

Engine	What it targets	Documented models
vLLM	production serving, throughput and concurrency	7
Triton Inference Server	industrial deployment across several models	7
Dynamo	distributed serving across several nodes	7
llama.cpp	workstations and modest hardware	1
Unsloth	fine-tuning and quantisation	5
ninfer	maximum single-GPU throughput (RTX 5090 & RTX PRO 6000)	1

How much memory per model?

The weight footprint depends on the model and the format, not on the engine. In NVFP4 it ranges from 16 GB for the most compact to 1,610 GB for the largest open model in the catalogue. The attention cache sits on top and grows with the context served.

Combination by combination in serving a model, machine by machine in sizing.

Documented combinations

See also the quantisation comparison, which decides memory more surely than the model itself.