llama.cpp

Inference Runtime

By QDNA · Published on 5 July 2026 · Updated on 2 September 2026

llama.cpp stays lightweight. It uses the GGUF format, combines CPU and GPU, and applies integer quantisation from 1.5 to 8 bits. It suits DGX Spark, Mac Studio, single-workstation setups, and offline use.

Key points: Edge · GGUF format · CPU and GPU · 1.5 to 8-bit quantisation · Offline.

Role

llama.cpp stays lightweight. It serves models locally, including without a connection, on modest hardware.

How it works

It uses the GGUF format, combines CPU and GPU, and applies integer quantisation from 1.5 to 8 bits to fit in memory (repository README, re-read on 2 September 2026).

Use cases

llama.cpp suits DGX Spark, Mac Studio, single-workstation setups, and offline use.

Integration

It acts as the local engine for dwarfstar and is exposed to the platform via LiteLLM.

License and source code

llama.cpp is published under the MIT licence and implemented in C++. The source is public, which means the runtime can be audited, patched and pinned to a known commit rather than trusted on a vendor's word.

Repository: ggml-org/llama.cpp. Licence and implementation language are re-read from this source by the QDNA fact base. The version number is deliberately not published here: it would go stale between two upstream releases without anything catching it.