llama.cpp

Inference Runtime

llama.cpp stays lightweight. It uses the GGUF format, combines CPU and GPU, and applies quantisation from Q4 to Q8. It suits DGX Spark, Mac Studio, single-workstation setups, and offline use.

Key points: Edge · GGUF format · CPU and GPU · Q4 to Q8 quantisation · Offline.

Role

llama.cpp stays lightweight. It serves models locally, including without a connection, on modest hardware.

How it works

It uses the GGUF format, combines CPU and GPU, and applies quantisation from Q4 to Q8 to fit in memory.

Use cases

llama.cpp suits DGX Spark, Mac Studio, single-workstation setups, and offline use.

Integration

It acts as the local engine for dwarfstar and is exposed to the platform via LiteLLM.