Pitfalls of putting an on-premises LLM into production
Eight mistakes that cost dearly in operations, and the measure that prevents each one. Lessons from the field, not theory.
1. Images that change under your feet
Tags such as nightly or latest change content without warning. Pin every image by its digest. Deployment becomes reproducible again, and a silent update can no longer break the platform.
2. The corrupted compilation cache
On first start-up, the engine compiles routines tailored to the hardware. Interrupting a container during this compilation corrupts the cache and triggers cascading failures. Never kill a service during its initial compilation.
3. The assumed mount versus the real one
A volume's declared mount does not always match the actual mount inside the container. Check the real mount, from inside the service, not the declaration. A model cache that is not persistent forces a fresh download on every restart.
4. Optimisations enabled by default
Certain optimisations promise gains and often deliver a net loss depending on the model and the load. Test them case by case, never enable them by default. The right configuration is measured, not assumed.
5. Client caches left un-invalidated
After a change to the server configuration, particularly context length, invalidate client caches. Otherwise applications keep sending requests calibrated for the old configuration.
6. Unvalidated interconnect
On a multi-card server, parallelisation exchanges data between cards for every token generated. A poorly detected interconnect triggers a silent fallback and halves throughput. Validate it once, properly, before any deployment.
7. Misleading throughput measurements
Inference measurements are noisy, with significant variance over short windows. The reliable protocol comes down to four rules.
- Enable full generation to make lengths comparable.
- Repeat at least three times and take the median.
- Never measure cold, since the first batches compile routines.
- Vary the prompts, otherwise you measure the cache rather than actual compute.
A quantified reference report, established properly once, becomes the baseline for comparing all future changes.
8. Unmanaged network flows and data
An inference service left open on the network without an explicit policy lets data leak out. Three open-source building blocks close this door at the kernel level. Cilium applies micro-segmentation by identity rather than address, from the network layer to the application layer. Hubble makes every allowed or blocked flow visible. Tetragon stops an unauthorised process before it executes and logs every exit attempt. The default rule denies everything, then opens only the necessary flows, without forgetting name resolution.
On the data side, an open-source masking tool detects and masks personal information before transmission, using regular expressions and entity recognition. Prompt injection remains the leading vulnerability of applications built on language models, and its indirect form, hidden inside a document or a web page, targets autonomous agents. Input filtering and a confidentiality test of the system prompt close this vector. Telemetry stays sovereign: metrics and logs for the entire stack are collected on-premises, on infrastructure hosted in Europe, with no external dependency.
Frequently asked questions
Why pin images by digest?
Because floating tags change content without warning. The digest guarantees reproducibility.
How do you measure throughput without getting it wrong?
Full generation, at least three repetitions, median, never cold, varied prompts.
Should default optimisations be enabled?
No. Test them case by case; some degrade performance depending on the model.