ninfer
Inference Runtime
ninfer is a from-scratch C++/CUDA inference engine designed to maximize single-GPU throughput. Optimized for NVIDIA GeForce RTX 5090 and RTX PRO 6000 hardware based on the Blackwell architecture (sm_120), it incorporates speculative decoding, native NVFP4 and INT8 quantization, and exposes OpenAI and Anthropic compatible HTTP interfaces.
Key points: Single-GPU throughput · GeForce RTX 5090 · RTX PRO 6000 Blackwell sm_120 · C++/CUDA · Speculative decoding · NVFP4 quantization · OpenAI and Anthropic APIs.
Role
ninfer maximizes inference throughput on individual workstations or single-GPU servers. It targets scenarios where a resident model must execute with minimal per-token latency, without the overhead of a distributed multi-GPU scheduler.
Architecture and operation
The engine is written in C++ with custom CUDA kernels, removing interpreter overhead during computational graph execution. It targets Blackwell compute capabilities (sm_120), notably on the GeForce RTX 5090 (32 GB) and RTX PRO 6000 (96 GB). ninfer uses self-contained artifacts packaging encoded weights, configuration, tokenizer, and speculative decoding heads (MTP, DFlash). KV cache management supports FP8 and NVFP4 formats to preserve memory headroom.
Use case
ninfer suits engineering workstations and dedicated servers running decoding-heavy workloads such as code assistants and interactive agents. It provides high generation rates for an individual user or focused team.
Integration
ninfer exposes HTTP endpoints compatible with OpenAI and Anthropic API protocols, alongside a local CLI. It connects directly to the LiteLLM gateway and multi-agent orchestration frameworks.
Licence and source code
ninfer is open-source software written in C++ and CUDA. Source code is available on GitHub: Neroued/ninfer.