Coding with a local LLM: harness, vLLM and LiteLLM
A useful code assistant is not just a model. It combines a harness that acts on the repository, open-weight models served locally by vLLM, a LiteLLM gateway that exposes a single API, and a KV cache that makes long sessions affordable.
Why a harness changes everything
A language model alone produces text. A useful code assistant acts on the repository: it reads files, proposes a fix, runs the tests, checks that the build compiles, and iterates. This execution loop is called a harness.
OpenCode and Hermes fill this role. OpenCode orchestrates tool calls on the file system, applies approval rules to sensitive actions, and delegates to sub-agents to parallelise research, analysis or test tasks. Hermes adds a role-based orchestration layer: one model reasons, another codes, another verifies. Without a harness, the model remains a text generator; with one, it becomes a technical collaborator.
The architecture behind the code assistant
A typical local architecture assembles five building blocks. The OpenCode harness presents tools to the model and runs the editing loop. Hermes acts as a role-based orchestrator when the team needs specialisation. LiteLLM exposes a single OpenAI-compatible API to the harness, assigns a virtual key to each developer, and applies quotas per person or per team. The vLLM inference engine serves open-weight models on the organisation's hardware with fine-grained GPU memory management. The choice of model depends on the task profile: long-form reasoning, stable tool calling, or a quality-cost trade-off for everyday work. On the hardware side, one or more RTX PRO 6000 units are enough for a whole team, while one or more DGX Spark units are enough for a single developer.
Source code never leaves the infrastructure
The argument fits in one sentence: the code does not leave the internal network. The harness calls LiteLLM on a private address, LiteLLM routes to vLLM on the same machine or a neighbouring host, and the model processes the request without a single byte crossing the Internet.
This property solves several problems at once. Industrial secrecy stays within the certified perimeter. NIS2 compliance requirements find a straightforward technical answer. Auditors can verify that production code appears in no third-party vendor log. Legal teams no longer need to negotiate clauses on data sent to an external service. Sovereignty becomes a property of the system, not a contractual promise.
The local KV cache makes long sessions economical
A productive development session lasts several hours. The model rereads the same module, finds the same definitions, walks through the same dependencies. Recomputing attention over this shared prefix on every turn is expensive in both time and energy.
vLLM keeps a KV cache in GPU memory between turns of the same session. The shared prefix of the context is only recomputed once, and each subsequent turn only processes the new fragment. Over a four-hour session with a context of several hundred thousand tokens, this mechanism divides total cost by a factor that commonly exceeds ten. The model stays useful for a long time without spend spiralling, and the developer can let the assistant iterate without watching the bill.
The table below summarises the gain observed over a typical five-hour session with a useful context of 250,000 tokens.
| Regime | Tokens processed | Relative cost |
|---|---|---|
| Cloud, billed per token, no cache | 100% of tokens | Reference |
| Cloud, billed per token, shared cache enabled | 30% of tokens | Three times lower |
| vLLM on-premises, GPU KV cache | 12% of useful tokens | More than eight times lower |
Which model for which code task
The landscape of open-weight models for code is expanding fast. The right approach is to choose the model according to the task profile, not the other way round.
GLM 5.2 handles heavy refactoring and long-range reasoning: 744 billion parameters in a mixture-of-experts, with a fraction active per token, to absorb an entire code base without losing the thread. Kimi K2.7 Code stands out on long sessions with many successive tool calls: it keeps reasoning coherent across chains of a hundred actions or more. DeepSeek V4 covers two roles: the Pro variant for deep investigations, the Flash variant for the team's everyday work, combining speed with contained cost. Qwen 3.6 27B fits on a single card and suits a lone developer who wants an immediate answer without mobilising a cluster. Gemma 4 falls into the same category, with a different profile on niche languages.
Scaling up is then simply a matter of selecting the model per task, without touching the harness or LiteLLM.
A single API, virtual keys and per-developer quotas
LiteLLM acts as the single entry point. The harness talks to LiteLLM as it would talk to OpenAI. LiteLLM routes to vLLM, applies guardrails and logs every call in an audit trail. Developers only ever handle one key, which they exchange for a virtual key issued in their name.
This virtual key carries a quota per person and per day, sometimes per project. The administrator caps an intern's spend, opens a wide quota for a technical lead, and cuts off access to a specific model for a contractor. Provisioning takes a single configuration file, and restarting the service takes about ten seconds. Developers switch models without switching tools.
Hardware follows the same logic. A lone developer works on a DGX Spark with a single model. A team of five to ten developers relies on one or more RTX PRO 6000 units and serves several models in parallel, all routed through LiteLLM. The financial tipping point between self-hosting and an external API depends on monthly volume and task profile: the detailed analysis is in our study on the cost of on-premises versus API LLM.
A useful code assistant stays local by design: the code never leaves the infrastructure, the KV cache makes long sessions affordable, and LiteLLM enforces a single API, virtual keys and per-developer quotas.
Moving from proof of concept to daily use
Three conditions combined turn a prototype into a team practice. First, a harness configured for the repository, with explicit approval rules on sensitive files and a dedicated test sub-agent. Second, a vLLM engine sized for the reference model and a KV cache calibrated to the typical context size. Third, a LiteLLM policy that assigns each developer a virtual key, caps quotas and logs calls for audit purposes.
Once these three building blocks are in place, the code assistant moves from a curiosity to production infrastructure. The code stays within the organisation, long sessions stay affordable, and every developer consumes their fair share.
Frequently asked questions
Why run a code assistant locally rather than in the cloud?
Source code never leaves the infrastructure, which satisfies sovereignty requirements, industrial secrecy and NIS2 compliance. The local KV cache reduces the cost of long sessions, and latency stays predictable.
What role does LiteLLM play in this architecture?
LiteLLM presents a single OpenAI-compatible API to the harness. It assigns a virtual key to each developer, applies per-person quotas and routes the request to the chosen open-weight model. The application only has a single endpoint to maintain.
What hardware should you choose to get started?
A DGX Spark station is enough for a developer working with mid-size models. A team serving several models in parallel relies on an RTX PRO 6000 workstation or an equivalent cluster to absorb concurrent load.
Does the KV cache really change the economics of the project?
Over a session lasting several hours, the shared prefix of the context is only recomputed once. The marginal cost per turn becomes negligible, which makes long iterative work viable.
Does a local model write code of sufficient quality?
Recent open-weight models score highly on real bug-resolution benchmarks such as SWE-bench, on par with the best closed APIs. Served locally, the assistant keeps your code within your own walls, which removes intellectual property leakage to a third party and leaves you in control of access and logging.
Source code that stays with you
A no-commitment call to identify the model and hardware that fit your coding practice.
Book a call