On-Premises LLM Training and RFT: From QLoRA to Production Serving

Fine-tuning an open-weight model on premises guarantees parameter ownership and precise adherence to internal tool interfaces. Here is the operational engineering method from trajectory curation to vLLM serving.

By QDNA · Published on September 12, 2026 · Updated on September 12, 2026 · 14 min read

Why specialise an open model on site rather than calling a remote API?

Specialising an open model on site guarantees complete confidentiality of training data and eliminates dependence on unpredictable quotas and arbitrary terms of remote cloud APIs. Organisations operating under strict regulatory compliance or safeguarding proprietary software assets cannot transmit their internal operational logic to external platforms.

Local fine-tuning grounds open weights such as detailed on the Hugging Face model card for Qwen3.8-27B into structured tool-calling protocols including the Model Context Protocol standard. Commercial models evaluated across broad tasks frequently produce malformed arguments or redundant calls when presented with complex enterprise APIs. Updating specific weight matrices on curated domain trajectories enforces rigid schema compliance and reduces repeated execution failures.

Direct infrastructure ownership also establishes predictable operating expenditures. Whereas per-token billing scales linearly with agentic reasoning volume, deploying dedicated accelerators caps operational costs while granting engineering teams permanent sovereignty over checkpoints and runtime configurations.

How to construct a reinforcement training dataset through rejection sampling (RFT)?

Constructing a reinforcement training dataset relies on running autonomous agent workflows inside isolated sandboxes, where only trajectories verified by deterministic success criteria are preserved. This methodology prevents distribution contamination by selecting training traces solely upon objective functional proof.

The technical harness employs the save agent rs (save-agent-rs) orchestrator written in Rust linked to containerised Linux execution sandboxes via the Docker daemon and LiteLLM gateway. The agent receives technical assignments such as vulnerability patching or repository refactoring, exercising over fourteen hundred system tools. Each trial produces a comprehensive execution log recording natural language thoughts, structured tool parameters, and raw shell outputs.

When base exploration encounters deadlocks, an automated teacher wave queries a secondary reasoning model to unlock the obstacle. Once execution completes and satisfies an unambiguous binary test suite, the entire interaction trace is formatted as structured JSON Lines. This rejection sampling mechanism ensures that every training example demonstrates an orderly sequence of validated actions.

Which hyperparameters and masking rules stabilise QLoRA on H100 and H200 accelerators?

Stabilising QLoRA adaptation requires a rank of 64, an alpha multiplier of 128, a learning rate of 2e-4, and strict assistant-only loss masking. These empirical parameters, building upon the foundational QLoRA paper on arXiv, maintain steady gradient norms on modern compute hardware such as the H200 SXM server without risking numerical divergence.

Compute resources are provisioned on the NVIDIA Brev acceleration platform, offering on-demand H100 80 GB PCIe or SXM and H200 141 GB SXM instances at accessible hourly rates between $1.50 and $3.50. This consumption-based model completes a full fifty-step adaptation cycle for under fifteen dollars of direct expenditure, eliminating capital allocation for dedicated physical appliances.

Leveraging these enterprise accelerators is technically mandatory compared to consumer cards or legacy A100 units constrained to 40 GB of VRAM. At a sequence length of 16,384 tokens with full backpropagation across all linear layers, activation tensors alone require over 32 GB of memory buffer. On a 40 GB A100 unit, transient attention spikes trigger immediate CUDA out-of-memory terminations. Modern H100 and H200 hardware delivers the necessary headroom along with HBM3 and HBM3e memory bandwidth of 3.35 to 4.8 TB/s, accelerating forward and backward passes.

A decisive architectural asset of the Brev environment is the persistent /workspace NVMe volume. When halting compute instances between experimental runs, the fifty-five gigabyte base weights and Python virtual environments remain preserved on disk. This persistent caching eliminates recurring transfers of raw tensor files over the network and guarantees deterministic startup script execution.

The decisive architectural control is assistant-only loss masking. Across multi-turn agent interactions, evaluating loss on static system instructions or verbose tool outputs disrupts language modelling representations. Gradient backpropagation is therefore restricted strictly to tokens generated directly by the assistant during reasoning and tool invocation phases by injecting generation boundary markers into the chat template and leveraging native Transformers masking.

How to supervise training telemetry and prevent gradient instabilities automatically?

Supervising training telemetry relies on tracking L2 gradient norms, monitoring loss curves for sudden divergences, and retrieving credentials securely from an identity vault. A background telemetry worker parses log records at every step to halt execution when abnormal parameter fluctuations emerge.

The supervisory monitor confirms that the L2 gradient norm remains between 0.05 and 1.50. Spikes exceeding this boundary indicate imminent gradient explosion, whereas drops toward zero denote vanishing signals. Upon detecting anomalies, the supervisor adjusts learning rates or scales gradient accumulation steps dynamically.

Credential provisioning and registry tokens are managed through an OpenBao secrets engine. Secrets are injected directly into environment memory during process initialization, preventing cleartext exposure inside deployment files or version control repositories.

How to merge low-rank adapters and deploy the specialised model under vLLM?

Deploying the specialised model requires merging LoRA adapter weights directly into the base BF16 layers, followed by initialization under vLLM with an engine-level tool parser. Merging eliminates runtime computational overhead during high-concurrency token generation.

The merging routine, aligned with the vLLM engine project, reconstructs full precision tensors by adding low-rank product matrices to original weights, writing the unified artifact to local storage and private registries as the save-rs-merged-v7 checkpoint.

Post-training artifact governance relies on private Hugging Face Pro repository infrastructure. The professional tier provides unlimited private model and dataset repositories accompanied by elevated Git LFS bandwidth to transfer fifty-five gigabyte unified checkpoints without transfer bottlenecks. Upon completing parameter fusion, an automated script uploads the LoRA adapters and the consolidated save-rs-merged-v7 checkpoint to the secure hub, indexed by commit hash and semantic version tags.

Authentication against private repositories uses fine-grained programmatic access tokens provisioned directly from the OpenBao identity vault. This pattern shields proprietary weights from public disclosure while enabling local vLLM inference nodes to synchronize verified checkpoints automatically during continuous deployment pipelines.

The vLLM engine (version 0.15 or later) is then started with automated function calling flags (--tool-call-parser qwen3_coder), a production context of 65,536 tokens, and a 92% video memory allocation quota.

For deployment on workstation workstations hosting cards such as the RTX PRO 6000, the consolidated model can undergo subsequent NVFP4 quantisation via NVIDIA ModelOpt. This format leverages specialised hardware tensor cores, reducing memory footprints by half while preserving baseline task performance.

Which objective benchmarks confirm non-regression and operational stability?

Confirming operational stability demands a zero-failure completion rate on specialized tool tasks without degrading general natural language reasoning benchmarks. Automated verification suites must execute immediately after checkpoint conversion to validate model integrity.

The qualification harness executes synthetic evaluation suites testing code refactoring, network inspection routines, and system exception management. Generated outputs are evaluated against rigid structural assertions to verify that domain specialization has not induced catastrophic forgetting.

Production latencies, time-to-first-token, and throughput metrics are documented under sustained concurrent workloads. Passing these functional gates permits rolling deployment across internal networks, establishing a dependable and sovereign production pipeline.