Enterprise chatbot: the internal ChatGPT use case (RAG, GDPR)
An internal chatbot does not need an American service to answer employees' questions. Hosted on your own machines and connected to your own sources, it stays within the enterprise's perimeter.
The use case: an internal ChatGPT that knows your business
A business unit asks for an assistant able to answer teams' questions about internal procedures, contracts, technical documentation or the knowledge base. The concrete expectation fits in one sentence: a ChatGPT that knows our company, and that does not talk to anyone else.
This expectation covers three requirements. First, the tool must cite its sources and rely on internal documents, not on an approximate general memory. Second, employees' questions, indexed documents and produced answers must stay within the enterprise's perimeter. Third, the tool must work without dependence on a third-party service that would log the exchanges or use them to train its models.
A sovereign chatbot meets these three requirements. The model runs on a company machine, the index of internal sources is rebuilt on-premises, and request filtering blocks any leakage to an outside provider. The conversation stays private, indexing stays private, inference stays private.
- Internal support: answering teams' questions on tools, procedures and technical documentation.
- Human resources: explaining leave, payroll, benefits and internal regulations.
- Legal and procurement: finding a clause in a contract, comparing two versions, checking an obligation.
- Customer service: preparing advisors' answers from the knowledge base, without exposing customer data.
- Business documentation: querying the company's manuals, standards and reference frameworks in natural language.
Why the public cloud does not fit this use case
Conversational SaaS offerings present several limitations for internal business use. The vendor's policy applies to submitted data: depending on the terms of service, inputs may be logged, retained, and sometimes reused to improve the models. European data protection regulation classifies this practice as a transfer to a processor located outside the Union, which requires an impact assessment and specific contractual clauses.
The leakage risk is not confined to the admin console. Prompts entered by employees travel to the vendor's data centres, pass through international points of presence, and are subject to the provider's local legal obligations. For sensitive content, such as customer files, HR data, contractual documents or trade secrets, this processing chain exposes the enterprise to a risk that no client-side technical measure can neutralise.
An open-weight model served on a company machine reverses this reasoning. Data does not leave the internal network. The operator remains sole master of encryption, logging, retention and purging. Compliance with European data protection regulation is demonstrated by design, not through an accumulation of contractual clauses.
RAG: connecting the chatbot to internal sources
The model alone does not know your enterprise. Retrieval-augmented generation, or RAG, completes the model's answer with passages extracted in real time from a document index. Indexing rebuilds this index from internal sources: file server, wiki, document base, electronic document management, archived mail.
| Step | Action | Location |
|---|---|---|
| Indexing | Chunking, vectorisation and storage of documents | Internal server |
| Query | Retrieval of relevant passages by similarity | Internal server |
| Composition | Injecting passages into the model's context | Inference machine |
| Answer | Drafting with citation of retrieved sources | Internal interface |
This chain offers a practical advantage for business units. The chatbot does not just answer, it cites the documents it draws on. An outdated internal procedure can thus be spotted by teams and updated, rather than propagated by a model with no temporal reference. Relevance improves with the quality of the index, and the index grows richer through ongoing contributions.
The architecture behind the chatbot
QDNA's architecture reuses the building blocks described on the site: open-weight models, the vLLM runtime or llama.cpp, the LiteLLM gateway and the semantic router. The hardware sheet details the configurations by segment.
The open-weight model serves inference. GLM 5.2, DeepSeek V4, Mistral Large and Qwen 3.6 cover everyday needs, and Mistral Magistral, Nemotron 3 Ultra or GLM 5.2 in reasoning mode handle complex questions. The vLLM runtime serves large models on NVIDIA GPUs with fine-grained KV cache management, and llama.cpp or MLX takes over on more compact configurations, notably Mac Studio and DGX Spark.
The LiteLLM gateway presents a single interface to the chat application. Model aliases, per-user virtual keys and cost-based routing are configured once and for all, and the application does not need to know the list of available models. The semantic router classifies the request before LiteLLM: through embedding similarity, it distinguishes confidential content from public content, and directs the request to a local sovereign model or to an external model when the topic allows it. This filtering prevents data leakage by design, without user intervention.
The whole forms a reproducible chain: source indexing, request classification, model selection, inference, source citation, internal logging. Each link stays on the enterprise's infrastructure, and the chain is supervised like any production service.
Which hardware for which organisation size
Hardware choice follows from the model selected and the volume of concurrent requests. The hardware sheets detail the configurations validated by segment.
| Segment | Workstation or server | Models served |
|---|---|---|
| Micro-business, team of five to twenty people | DGX Spark or Mac Studio | Qwen 3.6 27B, Gemma 4, Mistral Small |
| SME, twenty to two hundred people | RTX PRO 6000 server | GLM 5.2 quantised, Mistral Large, DeepSeek V4 |
| Large enterprise | B300 or H200 cluster | GLM 5.2, DeepSeek V4 Pro, Nemotron 3 Ultra |
DGX Spark holds particular appeal for very small structures: the GB10 superchip delivers 1 PFLOP in FP4 within a 150 mm desktop form factor, with 128 GB of unified memory. A cluster of two to four nodes linked by ConnectX-7 pushes capacity beyond 400 billion parameters. The Mac Studio M3 Ultra, with up to 512 GB of unified memory, accommodates quantised mixture-of-experts models and stays silent in an office. For an SME, the RTX PRO 6000 server hosts models from 70 to 200 billion parameters and handles several simultaneous conversations without congestion.
GDPR compliance and data hosting
Compliance with European data protection regulation results from the combination of three elements. The model processes questions without transmitting data to a third party. The vector index stores embeddings and source passages on infrastructure controlled by the enterprise. Audit logs, kept on-premises, document each request for traceability purposes.
The semantic router reinforces this compliance through a simple mechanism. Any request identified as potentially sensitive, for example through the presence of an identifier, a customer name or a contractual reference, is automatically directed to a local model. No configuration exception can let this type of content out, which removes human error at the source. For health data, hosting on HDS-certified infrastructure becomes a prerequisite, and the platform adapts through the choice of data centre.
The enterprise's technical documentation also benefits from running through this pipeline. Product sheets, internal procedures and knowledge bases are indexed once, refreshed at regular intervals, and become queryable by authorised employees. Sovereignty is not limited to inference: it extends to the organisation's documentary memory.
From chatbot to agentic platform
A sovereign chatbot covers the question-answering use case over internal sources. The agentic platform goes further: it orchestrates several agents, delegates sub-tasks to them, capitalises repeated procedures into reusable skills and retains context across sessions. The chatbot is the entry point, and the platform is the target.
This trajectory is built in stages. An internal chatbot validates source indexing and pipeline compliance. A second stage adds specialised agents, for example an agent for drafting meeting summaries and an agent for summarising long documents. A third stage installs the agent harness, three-layer memory and human oversight of at-risk actions. Each stage builds on blocks already in place, and the sovereign chatbot forms the operational foundation.
The useful decision is not to bolt a chatbot onto an existing stack. It is to identify the two or three internal tasks that would benefit from a model, and to equip them with a pipeline that is compliant, observable and reversible.
Frequently asked questions
Why a sovereign chatbot rather than a SaaS service?
SaaS subjects requests to the vendor's policy and transfers data outside the Union. The sovereign chatbot keeps inference, indexing and logging within the enterprise's own perimeter.
How does the chatbot access internal sources?
A vector and lexical index, rebuilt on-premises from the documents, serves as the basis for RAG. The model composes its answer using the retrieved passages, and cites its sources.
Does the chatbot work entirely offline?
Yes, on DGX Spark or Mac Studio for compact models, and on an RTX PRO 6000 server or B300 cluster for large models. The client workstation only communicates with this internal server.
Which open-weight models does the platform use?
GLM 5.2, DeepSeek V4, Mistral Large, Qwen 3.6 and Gemma 4 depending on the need. The LiteLLM gateway routes each request to the least costly model capable of handling it.
Does the model need to be trained on our data?
Rarely at the start. Document retrieval, or RAG, is enough to ground answers in your knowledge base without training: it stays current and cites its sources. Fine-tuning then sets a tone, a format or business vocabulary, once the need has stabilised. Details on grounding are covered in our guide to enterprise RAG.
Evaluate your sovereign chatbot
A no-obligation call to scope your use case, your internal sources and the appropriate hardware segment.
Book a call