2x DGX Spark RDMA cluster + Qwen 3.8 27B mono-spark

Case study : January 2026, in production since February 2026.

Goal

Stand up a local platform that runs DeepSeek V4 Flash 284B (13B active) in FP4 and Qwen 3.8 27B in FP8, with two distinct contexts: 1 million tokens for DeepSeek on a 2-Spark cluster, 262,144 tokens for Qwen on a mono-spark. The platform must accept concurrent calls from several internal agents through LiteLLM, and route each request to the most suitable model via a semantic router.

Hardware

Item	Reference	Quantity
GPU server	NVIDIA DGX Spark : 128 GB unified memory, ConnectX-7 200 Gb/s	3
RDMA switch	NVIDIA Spectrum-3 200 GbE, 8 ports	1
Cabling	QSFP56 DAC 200 Gb/s, 1 m	4
UPS	APC Smart-UPS 3000 VA, double conversion	1
12 U rack	APC NetShelter, 19-inch, 600 x 1000 mm	1

Total cost

Item	Detail	Amount ex. tax
DGX Spark	3 x €3,800	€11,400
Spectrum-3 switch + DAC cables	€1,800 + €200	€2,000
UPS + rack	€1,100 + €300	€1,400
QDNA engineering	6 weeks of integration and production rollout	€6,800
Total investment		€21,600

Estimated recurring annual cost: €1,200 in energy (the 3 Sparks draw 1,100 W under average load), €600 in UPS maintenance, €400 in internal bandwidth.

Steps and timeline

Week 1 : Scoping and hardware selection. Half-day workshop: use cases, expected maximal context, target throughput, network constraints. Decision: three DGX Sparks rather than one DGX Station, because the single-node Spark latency does not cover DeepSeek FP4 1M context. Validation of the NVFP4 format on DeepSeek weights.
Week 2 : Delivery and rack-mount. The three Sparks landed in a 12 U rack, UPS wired in, ventilation checked. The Spectrum-3 switch was configured in RoCE v2 mode, two links between each Spark and the switch for redundancy (four links used, two in reserve).
Week 3 : RDMA cluster bring-up. NCCL configuration pointed at the ConnectX-7 interfaces, bandwidth validation through nccl-tests all_reduce_perf: 185 Gb/s measured between the two Sparks, 92 % of nominal. First run of a DeepSeek model sharded across two nodes, 1M-context prefill measured at 28 seconds.
Week 4 : DeepSeek harness. The orchestration script that pushes weights to the two nodes, configures the sparse KV cache, monitors inter-node latency. Throughput validated at 68 tokens/s per user, 45 tokens/s under moderate load (10 users).
Week 5 : LiteLLM orchestration and semantic router. LiteLLM deployed as single entry point, branch « deepseek » on the Spark cluster, branch « qwen » on the mono-spark. The semantic router analyzes each request and chooses the branch based on content: short factual queries to Qwen, long generation or reasoning to DeepSeek.
Week 6 : Load testing and production rollout. Locust tests: 200 virtual users, peak throughput of 1,800 requests per minute, p95 latency at 1.8 s. Production cutover, SLA validation, debugging of a retry loop on the semantic router that doubled requests.

Context sizes and measured throughput

Model	Hardware	Quantization	Max context	Prefill 1 M (Qwen 262 K)	Single-user throughput
DeepSeek V4 Flash 284B (13B active)	2 x Spark RDMA cluster	NVFP4	1,048,576 tokens	28 s	68 tok/s
Qwen 3.8 27B	DGX Spark mono	FP8	262,144 tokens	3 s	120 tok/s

DeepSeek harness on the two nodes

The harness rests on three building blocks:

vLLM in tensor-parallel mode sharded over 2 GPU: each Spark exposes one Grace-Blackwell GPU, NCCL bridges them over RDMA. The DeepSeek model is split into two shards, attention and experts distributed across the two nodes.
Sparse KV cache with segmented prefill: to handle 1 M context, the KV cache is organized in 128 K segments, mounted on demand according to the active window. The decode never walks inactive segments, which brings memory consumption back to 64 KiB per token instead of 256 KiB.
Inter-node latency monitoring: a watchdog continuously measures the RDMA RTT. If the moving average exceeds 12 µs, the cluster triggers an alert and cuts inter-node traffic to switch to degraded mode (context limited to 512 K).

LiteLLM ecosystem and semantic router

LiteLLM exposes two routes: /v1/chat/deepseek (Spark cluster) and /v1/chat/qwen (mono-spark). The external API is OpenAI-compatible, one client for both models. Logs are unified in a local SQLite database, purged after 30 days.
Semantic router: a lightweight embedding (bge-small) computes cosine similarity between the query and a hand-labelled reference set of 200 prototypes. Above 0.72 similarity with a « long reasoning » prototype, the request goes to DeepSeek. Otherwise, to Qwen. Added latency: 18 ms.
OpenCode: the internal agents (four of them: refactor, review, documentation, migration) consume the LiteLLM API. The agent/model split is explicit: Qwen for fast and cheap agents, DeepSeek for long-file refactoring and architecture review.
OpenRouter Router: a second, silent router replicates requests to OpenRouter (Claude Opus 5, GPT-5.5) for quality comparisons. Activated on only 1 % of traffic, with sampling and divergence logs.
Mem0 + QDNA semantic layer: agent successes and failures are stored as (task, model, latency, quality score). The semantic layer enriches each request with a project-context summary and learned user preferences.
Hermes Agent: top-level orchestrator, triggers sub-agents, accumulates results, handles fallbacks. Hermes never touches the RDMA paths: it calls LiteLLM over HTTP, LiteLLM is what pushes to the cluster.

Diagram

                 ┌───────────────────────────────┐
                 │  Hermes Agent                  │
                 │  (orchestrator, HTTP)         │
                 └───────────────┬───────────────┘
                                 │
                 ┌───────────────▼───────────────┐
                 │  Semantic router (bge-small)  │
                 │  + Mem0 memory layer         │
                 └───────────────┬───────────────┘
                                 │
                 ┌───────────────▼───────────────┐
                 │  LiteLLM (entry point)        │
                 │  /v1/chat/deepseek  /v1/chat/qwen │
                 └───┬───────────────────────┬───┘
                     │ RDMA over RoCE v2     │ PCIe
                     │                       │
       ┌─────────────▼──────────┐ ┌─────────▼─────────┐
       │  Spark #1 + Spark #2    │ │  Spark #3        │
       │  cluster NVFP4 2x128GB │ │  Qwen 3.8 27B FP8 │
       │  DeepSeek V4 Flash 284B│ │  mono-node        │
       │  NCCL all_reduce 185G/s│ │  128 GB unified  │
       └────────────────────────┘ └───────────────────┘
                        │
                 ┌──────▼──────┐
                 │  Switch    │
                 │ Spectrum-3 │
                 │ 200 GbE    │
                 └────────────┘

Operational metrics (first six months)

Indicator	Value
Requests served per day	14,500 (average), 31,800 peak
Latency p50	0.42 s
Latency p95	1.8 s
Routing rate to DeepSeek	34 %
Routing rate to Qwen	66 %
Marginal cost per request	€0.001 (amortization + energy)
Availability over six months	99.97 % (one 2 h 30 outage on UPS)

Want to replicate this architecture?

Half-day scoping offered, on your use cases and your volumes.

Book a call

FAQ

Why two DGX Sparks instead of one?

To serve DeepSeek V4 Flash 284B (13B active) in FP4, which exceeds the memory of one unit (128 GB). The ConnectX-7 RDMA between the two Sparks forms a 256 GB unified domain, sufficient in NVFP4.

What maximal context in practice?

DeepSeek: 1 M tokens with sparse KV cache over 2 nodes. Qwen 3.8 27B mono-spark: 262,144 tokens (262 K) in FP8, which fit in the 128 GB of the Spark.

How long to production?

Six weeks from signature to the first validated query. Three weeks on hardware integration and the cluster, one week on the DeepSeek harness, two weeks on LiteLLM orchestration and the semantic router.

Total investment cost?

Three Sparks at €3,800 each, RDMA switch, UPS, racks and cabling: €14,800 hardware. Plus €6,800 of QDNA engineering, totalling €21,600 excluding energy and bandwidth.