Local AI platform and open-source LLM: the case for on-premise, analysed

A local AI platform built on an open-source LLM: this analysis examines what on-premise actually delivers, what it asks in return, and why the balance tips in its favour as soon as the data matters. Every claim is verified at the source.

Published on 5 September 2026 · Updated on 6 September 2026

What on-premise delivers, point by point

What on-premise delivers rests on four distinct points, better examined one at a time since they do not rest on the same guarantees:

• your data never leaves your infrastructure;
• nobody can withdraw the model you rely on;
• you keep control of the version and the tuning;
• your bill stops depending on a supplier.

The first three can be verified on your own installation, without taking anyone's word for it. The fourth depends on volume, and is the only one requiring a calculation.

Confidentiality, the point that depends on nothing else

Confidentiality is the most solid point of the analysis, because it depends neither on a licence, nor on a volume, nor on a third party's good faith. When the model runs on your hardware, requests do not leave your network, and that is observed in the outbound traffic.

The difference from a contractual guarantee is one of nature, not degree. A no-retention clause engages a supplier's liability; a local architecture removes the question, since there is nothing held elsewhere. That is what makes the approach acceptable for a medical record, a trade secret or material covered by professional privilege.

It also simplifies the record of processing activities: with no transfer outside the European Union, articles 44 to 49 of the GDPR no longer have to be documented for that processing, and the processor question does not arise.

Permanence: nobody can withdraw the model

Permanence is the second guarantee that only on-premise provides. A downloaded model stays usable if its publisher removes it from the catalogue, changes its terms, changes its pricing or disappears. The weights are on your disks, and they stay there.

The point looks theoretical until the day it is not. It is the argument that carries weight in regulated sectors, where a validated application must keep returning the same result for years, and we developed it in our article on the kill switch.

Control of the model, concretely

Control of the model is the third guarantee, and the most misunderstood, because it gets confused with the freedom to retrain. It is not that: it is control over everything surrounding the model that determines how it behaves.

• The version. You decide when to change model, and you can roll back. No update is forced on you overnight.
• The format. You choose the quantisation according to your own trade-off between quality, memory and throughput, and you can serve two formats side by side.
• The serving. The inference engine, the batch size, the context window and the key-value cache are your settings, not a supplier's.
• Adaptation. A light fine-tune on your own documents stays possible, which no pay-per-token interface allows.

A practitioner describes, in May 2026, the concrete form that control takes. His local platform assembles LiteLLM, Ollama, Qdrant, Redis and Langfuse, and application code knows only an alias, "coding" or "reasoning": changing model means editing one configuration line, without touching a single application, and every call is traced. It is the architecture this site describes in its sheets, and it fits in under a thousand lines.

That control has a practical consequence that often settles the matter: the system's behaviour is reproducible. At identical version, format and settings, the same question returns the same answer, which makes auditing possible.

It also has a reach that is underestimated, and a case published on 1 September 2026 measures it. Constrained by data-residency requirements, a company consolidated the traffic of over 200 internal applications onto a single self-hosted model, adapted from production error analysis. The result surpasses a baseline seven times larger on their internal arena, 69.6 to 65.8, as well as on instruction following and function calling, while absorbing half the platform's traffic, 116 million requests a month, at a fraction of the serving cost. Such tuning on one's own traffic is only possible when one holds the model.

Making the model progress: fine-tuning, harness and enterprise memory

Holding the weights opens three levers a closed API does not offer: fine-tuning the model on your activity with QLoRA on a single card, choosing the harness around it, and building it an enterprise memory. Published measurements show that each of these levers weighs as much as the choice of model.

Fine-tuning on your activity, on a single card

The QLoRA method, published in May 2023, fine-tunes a 65-billion-parameter model on a single 48 GB GPU while preserving full 16-bit fine-tuning performance (arXiv:2305.14314). The resulting Guanaco model reaches 99.3% of ChatGPT's level on the Vicuna benchmark after 24 hours on a single card, with a caveat the authors state themselves: the chatbot benchmarks of the time are not trustworthy for comparing assistants. Three years later, domain results speak more clearly:

• Radiology: LoRA fine-tuning of Qwen2.5-3B and Qwen3-4B on 162,000 examples and nine tasks improves accuracy by 53% to 89% depending on the task over the un-tuned model, and the models then fit in 1.8 to 2.4 GB on an ordinary CPU (arXiv:2605.00421).
• Biomedical claim verification: Mistral-7B fine-tuned with QLoRA on only 1,008 examples surpasses GPT-4o and GPT-5, by up to 12 F1 points, at a fraction of the cost; the author himself flags a dataset artifact that inflates in-domain scores (arXiv:2606.12854).
• A useful counter-example: on an Uzbek legal question-answering system, QLoRA fine-tuning of the generator yields a negative result, and the authors recommend fine-tuning the embedding model rather than the language model (arXiv:2608.29284). Fine-tuning is measured, not presumed.

What argues for on-premise here is simple: the fine-tuned weights contain your know-how, and they stay on your disks. A model behind a closed API does not retrain at your place. The single-card edge study already cited (arXiv:2607.02523) shows that the full cycle, data, fine-tuning, serving, fits inside the perimeter.

The harness weighs as much as the weights

An August 2026 study compares two configurations of the same coding-agent harness, with identical weights, on 169 SWE-bench Verified tasks in a 20,480-token window: the per-task pass rate rises from 28% to 49%, and complete solutions from 43 to 72 (arXiv:2608.26218). The authors conclude that model and harness must be evaluated together, and note that the gap narrows with a large window: the harness matters most under context constraints. Two other measurements point the same way:

• ARC-AGI-2: the refinement published by Poetiq takes Gemini 3 Pro from 31% at $0.81 per task to 54% at about $31 per task, without changing the model; ARC Prize notes that these refinement harnesses remain domain-specific (ARC Prize 2025 results analysis).
• SWE-bench Verified: coupling planning and episodic memory in the agent resolves on average 25 more cases, 5.0 points, than a harness-matched baseline, and reduces repeated failed actions and context-window exhaustion (arXiv:2608.06811).

The DeepSWE benchmark cited above in fact fixes the harness for all models, precisely because it changes the result. Locally, that harness is yours: the execution loop, the exposed tools, the approval rules and the context management are set on site, as our agent harness sheet describes.

An enterprise memory around the model

The LongMemEval benchmark measures a 30% accuracy drop for commercial assistants and long-context models when information must be retained across sustained interactions (arXiv:2410.10813). A memory layer corrects this, and published measurements give the order of magnitude:

• Agent memory: on the LOCOMO benchmark, Mem0 gains 26% relative on the LLM-as-a-judge metric over OpenAI's memory, with p95 latency cut by 91% and more than 90% of tokens saved against full context; it is a vendor paper, to be read as such (arXiv:2504.19413).
• Search matters more than structure: a plain agent-controlled lexical search over raw logs reaches a 58.2 mean on MemoryAgentBench, ahead of HippoRAG 2 at 53.2, with the same backbone (arXiv:2608.12888). Indexing your own logs, on site, is the first step.
• Practitioner account: "in plain RAG, a bad retrieval usually hurts one turn; in runtime-learned memory, a bad write changes many future turns", and memory is "often an economics and continuity win before it is an across-the-board recall win" (M. Lanham, Medium, April 2026). His second-hand figures are not repeated here.

This memory, vector store, relation graph, wiki of facts, is the most company-specific part of the whole platform, and therefore the one with the least reason to leave. Our three-layer memory sheet details an implementation where the embedder and the reranker run on the local GPU.

These four figures come from four different benchmarks and do not add up; they only say that each lever, taken alone, moves the result measurably.

First point to check: read the licence

The first thing to check is the licence, since "open source" covers rather different realities depending on the publisher. Querying the Hugging Face API on 5 September 2026 across nine of the most downloaded open-weight models, four carry a recognised free licence and five a licence written by their publisher.

The good news is that choice is not lacking: Apache 2.0 and MIT cover four of the nine models in the table, including the most downloaded ones, and both licences allow commercial use without particular conditions. It is a matter of checking before committing, as with any software dependency.

Bespoke licences are not closed for all that, they set conditions worth knowing. The Mistral Research License asks for permission for any use not expressly provided for. The Llama 4 licence applies freely below 700 million monthly active users, a threshold a European company rarely reaches, and requires displaying "Built with Llama". Google publishes its own terms for Gemma.

What the reference definition requires

The reference definition is worth knowing so as to use the right words. The Open Source Initiative, under the name OSAID 1.0, asks for three components: the parameters, the training and inference code, and data information allowing an equivalent system to be rebuilt.

Publishing the weights satisfies the first of those three. That is why the exact term is "open weights", and it is the one this site uses. It takes nothing away from the guarantees described above, which rest on the weights themselves rather than on publishing the training code.

Second point to check: the cost changes nature

The second thing to check is cost, which does not disappear but changes nature. Local replaces a usage-proportional expense with a fixed one, which becomes an advantage beyond a certain volume and stays one thereafter.

The assumptions published on this site give the order of magnitude, taken unchanged from our other articles: a DGX Spark workstation at €4,875 excl. VAT, a DGX Station at €95,000 excl. VAT, electricity at €0.1624 per kWh excl. VAT at the professional tariff.

The threshold we publish sits around 55 agentic users, or 309 occasional ones, at DeepSeek V4 Flash's peak rate. Beyond it the gap widens in favour of local, and becomes predictable once the hardware is written down. The full calculation appears in our article on on-premises versus API.

This threshold reasoning is not particular to this site. A cost-benefit analysis published on arXiv in August 2025 builds the same framework, setting the total cost of a local deployment of open models against the major providers' subscriptions, and concludes on a break-even point that depends on usage level and performance needs. The figures are ours; the method is shared.

One line comparisons forget is worth planning for: engineering time. A local model has to be updated, monitored and resized. It is the price of control, and it drops markedly when the platform is industrialised from the start.

What open models are worth, measured

What open models are worth is now measured on contamination-free benchmarks, and the result is clear: they hold the leading places. Datacurve's DeepSWE benchmark evaluates 113 original software engineering tasks across 91 repositories and five languages, with hand-written verifiers.

Two lessons follow. First, models you can download and run yourself rank sixth and seventh on a benchmark designed to separate frontier coding agents. The gap with closed models has narrowed to the point of no longer driving the choice.

Second, the result-to-cost ratio leans firmly towards open. GLM 5.3 Flash reaches 63% for $0.24 per task, sixteen times less than GLM 5.3 for six points fewer. That is precisely the profile wanted for high-volume internal use.

On which machine, concretely

On which machine to run these models is decided pair by pair, and this site publishes one page per model and per piece of hardware rather than a general estimate.

One example gives the order of magnitude: GLM, at 744 billion parameters, occupies about 428 GB in NVFP4, which fits within the 748 GB of a DGX Station and the 768 GB of an RTX PRO 6000 server, but not in a more precise format.

This table is a calculation, not a measurement: it sets each model's published weight against each machine's memory, "just fits" meaning the remaining headroom is consumed quickly as context grows. Each cell points to a sizing page detailing headroom and possible formats, and our memory calculator adds the key-value cache for your context and concurrency.

Compact models change the scale of the problem. An energy benchmark published in June 2026 measures nine open models from 1 to 7 billion parameters on a single consumer card, an RTX 4060 Ti with 16 GB, sampling power through nvidia-smi at 2 Hz. The most frugal draw 0.56 and 0.65 joules per output token, and the authors find that architecture and quantisation matter more than parameter count.

A second study, published in August 2026, evaluates three open models under five billion parameters locally on mathematical reasoning, with fixed settings, per-question resource measurement and paired significance tests. A third, in May 2026, shows fine-tuning a model on a single edge-class accelerator, for telecom network troubleshooting where data sovereignty and latency are operational requirements, studying the trade-offs of LoRA rank, sequence length and key-value cache. The literature documents what local makes possible, not only what it costs.

To choose your own model and machine pair, our sizing section confronts each model with each piece of hardware, memory footprint and headroom included.

Hybrid: do not choose, route

The "local or API" debate assumes a binary choice that need not exist, because a hybrid architecture lets both coexist. A semantic router classifies each request before any model call, then sends it to the local model or to an external one depending on what it contains.

The classification rests on embedding similarity rather than keywords, so on the meaning of the text. A request touching a customer file, a contract or health data goes to the sovereign model; a generic rewrite or a translation of public documentation can go to a pay-per-token interface, cheaper at low volume.

That shift changes the nature of the trade-off set out above. You no longer need to reach the threshold of 55 users to justify local, since local only handles what requires it. And you no longer expose sensitive data to get an API's price, since the sorting happens before the call.

Three conditions make the arrangement sound. The router must sit upstream of the gateway, not after it, otherwise the request has already left the perimeter. Its decision must be logged, failing which it cannot be audited. And the default behaviour must be local, so that an uncertain classification fails on the right side.

A practitioner measured, in July 2026, what happens when one does not route. He replayed 27 real tasks from his personal assistant, an agent with about 90 tools, on qwen3-coder:30b served by an RTX 3090, against Claude's historical answers on the same tasks. The cost, measured on the card's actual power draw, falls to $0.00015 per task against $0.763, a ratio of 5,150. But judged quality falls too, from 89.4 to 22.8 out of 100, and the same model, which had passed 100% of a scoped 17-task benchmark, degrades sharply in a 90-tool environment. His conclusion matches ours: do not ask the local model to do everything, but hand it the slices where it is close, keeping the frontier model where the tool surface matters. That is precisely what a router decides, request by request.

Our semantic router sheet details the embedding classification, the leak prevention and the placement upstream of LiteLLM.

How to decide, concretely

Deciding means answering four questions in order, and the first settles most cases on its own, which saves comparing costs for nothing:

1. Does your data forbid leaving the perimeter? If so, the debate is over: on-premise wins whatever the cost.
2. Does your volume exceed the hardware break-even, around 55 agentic users? If so, on-premise pays off as well as being controlled. Compare against the threshold, not your headcount.
3. Do your flows mix sensitive and generic? If not, everything is generic and below the threshold: a pay-per-token API suits, switching to local as soon as sensitivity or volume rises.
4. Do you have the engineering time for two paths and a router? If so, the hybrid with a semantic router takes the best of both. Otherwise, split by hand in the application: sensitive flows local, generic ones on the API, and the router comes later.

Whichever path, read the licence of the chosen model, or the API terms, for your precise use, commercial included: it is a model choice, not a hosting one, and it comes afterwards.

For choosing the model, our open-source LLM comparison covers sizes, memory footprints and uses. For implementation, see installing a local LLM in a company. For hardware, our AI server price guide.

Further reading on the blog

This article covers the trade-off; the other articles on the blog each cover a step, and are best read in this order depending on where you stand:

• Which open-source model to choose: sizes, memory footprints, licences and uses.
• Which AI server, and at what price: ranges by tier, from workstation to rack.
• On-premises versus API: the threshold calculation, its assumptions and capacity caveats.
• Cost comparator, pay-per-token API or on-site server: the same calculation with your own headcount, prices and machine.
• Memory calculator: weights, key-value cache and maximum context across the eight machines, with your own values.
• Installing a local LLM in a company: model, hardware, runtime, gateway, security.
• Prefill, decode and key-value cache: what actually sets the hardware you need.
• Enterprise RAG: connecting the model to your documents without exposing them.
• Generative AI and GDPR: what the French regulator recommends, and extraterritorial risk.
• Kill switch and local LLM: why permanence is a compliance argument.

Sources

Declared licences: Hugging Face API, read on 5 September 2026. Definition: Open Source AI Definition 1.0, Open Source Initiative. Mistral licence text: Mistral AI Research License. Llama 4 licence text: meta-llama repository. Coding benchmark: DeepSWE, Datacurve, leaderboard read on 6 September 2026. Local energy: arXiv:2608.00008. Compact local models: arXiv:2608.22048. Self-hosted model on production traffic: arXiv:2609.01572. Single-GPU fine-tuning: arXiv:2607.02523. Cost-benefit framework: arXiv:2509.18101. Practitioner accounts, cited as such: A. Apostolov, Medium, July 2026; A. Mahajan, Medium, May 2026. QLoRA fine-tuning: arXiv:2305.14314, arXiv:2605.00421, arXiv:2606.12854, negative result arXiv:2608.29284. Harness effect: arXiv:2608.26218, arXiv:2608.06811, ARC Prize 2025. Memory: arXiv:2410.10813, arXiv:2504.19413 (vendor paper), arXiv:2608.12888; M. Lanham, Medium, April 2026, cited for its analysis only. Cost assumptions: identical to those of our other articles, detailed in the price guide.