Sovereign AI inference in the enterprise: 8 × RTX PRO 6000 Blackwell vs 8 × H200 SXM

To process confidential enterprise document streams in September 2026, an organisation deploys an on-premises redundant inference cluster across eight accelerator cards, eliminating third-party data transfers through a calculated trade-off between professional PCIe workstation cards and datacenter modules.

Prices: RTX PRO 6000 Max-Q card at LDLC and NVIDIA marketplace (read on 26 September 2026), BIZON and Supermicro configurators for four-card machines (on quote), HGX H200 server at CTO Servers (2 September 2026), ECB rate of 25 September 2026. Throughputs are QDNA working assumptions, not measured.

By QDNA · Published on 16 September 2026 · Updated on 26 September 2026 · 10 min read

Updated on 26 September 2026: costs redone: RTX PRO 6000 investment brought back to the prices read (card at €14,758 before VAT at LDLC, four-card machines on integrator quotes, €105,000 to €140,000 for two nodes instead of €384,000), H200 server at the ECB rate of 25 September, and correction of a thousandfold error on the cost per million tokens (about €165 to €200 per million at 27 million tokens a month, not €0.45); throughputs are now presented as unmeasured assumptions.

Why size two separate servers for on-premises inference?

Deploying two identical physical servers guarantees service continuity through active load distribution and hardware fault tolerance. Should any compute node experience an unplanned outage, the secondary machine immediately absorbs the entire inference workload without interrupting ongoing operations.

In an enterprise production environment, a single point of failure exposes mission-critical software to severe outages. Distributing four accelerators per chassis across twin server chassis allows engineering teams to perform kernel upgrades and runtime maintenance without downtime.

Dual-node topologies also distribute electrical load and thermal dissipation across independent rack circuits. Each machine operates on a dedicated electrical feed absorbing approximately 3,200 W without requiring the complex facility plumbing or high-density liquid loops demanded by monolithic multi-rack installations.

What are the architectural differences between RTX PRO 6000 and H200 SXM?

The RTX PRO 6000 Blackwell card operates over a standard PCIe bus with ninety-six gigabytes of fast memory, whereas the H200 SXM module integrates one hundred and forty-one gigabytes of high-bandwidth memory connected through dedicated NVLink interconnects.

The H200 SXM datacenter module is designed for raw compute throughput across distributed model weights. Equipped with HBM3e delivering 4.8 TB/s per GPU and bidirectional NVLink fabrics running at 900 GB/s, it synchronises model activations across accelerators at speeds unmatched by PCIe buses.

Conversely, the server-grade RTX PRO 6000 Blackwell accelerator is rated for an operating thermal design envelope reaching up to 600 W per card under maximum compute loads. Delivering 1,792 GB/s over GDDR7 memory within reinforced PCIe Gen 5 chassis slots, it operates on high-velocity forced-air cooling without requiring dedicated liquid cooling loops.

Which open-weight models run on each cluster?

A cluster powered by RTX PRO 6000 accelerators serves fast and compact open-weight models such as Qwen 3.8 27B or DeepSeek V4 Flash, while the H200 architecture comfortably runs massive frontier networks like GLM 5.3 Deep without quantization compromises.

With 768 GB of total VRAM across both servers, an RTX PRO 6000 deployment hosts multiple specialised model instances simultaneously. An engineering team can concurrently run a thirty-billion-parameter reasoning model, dense vector embedding models and structured extraction microservices.

The H200 SXM modules, aggregating 1,128 GB of ultra-fast HBM3e memory, become essential when serving colossal dense models in native half-precision floating point. However, this expenditure must align with actual enterprise reasoning complexity requirements.

What inference throughput can you expect with vLLM?

The vLLM engine optimises inference throughput by combining continuous batching of incoming requests with key-value cache reuse in graphics memory. Across the eight RTX PRO 6000 cards, aggregated memory bandwidth maintains smooth response rates under heavy concurrent workloads.

Language model generation decomposes into two phases: prompt prefill and token decoding. Prefill saturation stresses tensor compute cores, while sequential decoding performance is strictly governed by memory bandwidth.

In enterprise document analysis featuring long shared prompts, vLLM prefix caching retains precomputed attention layers directly in GPU memory. This mechanism reduces redundant computations across recurring documents, stabilising request latency during peak business hours.

What is the total cost of ownership over three years?

Total cost of ownership includes initial hardware capital expenditure amortised over thirty-six months, continuous power consumption and secure datacenter hosting. The RTX PRO 6000 cluster delivers a lower initial entry barrier than provisioning an infrastructure equipped with H200 SXM modules.

Examining financial commitments over a 36-month operational horizon reveals a stark economic distinction tied to physical node redundancy. Two complete servers equipped with four RTX PRO 6000 cards each represent approximately €105,000 to €140,000 in capital expense: the eight cards alone cost about €118,000 excl. VAT (€14,758 excl. VAT per Max-Q card at LDLC on 26 September 2026, $16,000 on the NVIDIA marketplace according to Thunder Compute), and integrators such as BIZON, Exxact or Supermicro quote a complete four-card machine between $60,000 and $80,000 on request, i.e. about €53,000 to €70,000 excl. VAT per node at the ECB rate of 25 September 2026 (€1 = $1.1403). That split provides native two-node fault tolerance. For the 8 × H200 SXM5 cluster, a single 8-GPU HGX server chassis costs approximately €451,000 (£387,785 at CTO Servers on 2 September 2026, ECB rate of 25 September). If the enterprise deploys two servers to achieve identical dual-chassis physical redundancy without a single point of failure, the initial hardware outlay reaches approximately €902,000.

Factoring electricity costs at €0.1624/kWh for continuous 6.4 kW aggregate draw and private datacenter colocation, the RTX PRO 6000 cluster totals approximately €4,480 to €5,450 per month: €2,917 to €3,889 of amortisation over 36 months, €759 of electricity and €800 of colocation. When serving the dense Qwen 3.8 27B model under vLLM in FP8 precision, we take as a working assumption, not measured, 1,500 tokens/s peak throughput and a smoothed average of 500 tokens/s, i.e. around 1.3 billion tokens monthly at full load. The cost per million tokens is read by dividing the monthly cost by the volume served: about €165 to €200 per million for a typical enterprise volume of 27 million tokens monthly, and about €3.4 to €4.2 per million at continuous full saturation. The previous version of this article published €0.45 and €0.010 per million, a thousandfold error on that division, corrected on 26 September 2026.

How to secure the LiteLLM gateway and the private network?

The LiteLLM gateway secures internal access using isolated virtual keys and departmental quotas without requiring any external internet connectivity. All inference requests remain confined within the enterprise local network, preventing any confidential data leakage.

Placing an orchestration gateway in front of distributed vLLM nodes streamlines enterprise administration. LiteLLM assigns virtual API keys to internal teams, enforces rate limits and logs every transaction to local immutable audit trails for compliance reporting.

Complete network isolation guarantees trade secret protection and alignment with European cybersecurity standards. No operational data, legal briefs or internal records ever leave the perimeter of the corporate datacenter.

Sovereign inference infrastructure rests on a pragmatic balance: hardware tailored to actual production model profiles, strictly air-gapped access gateways, and predictable long-term operating costs.

Official sources

Sources: official vLLM distributed inference documentation, accelerated computing specifications from NVIDIA enterprise compute, and cybersecurity guidance from ANSSI.