If you have been running DeepSeek models through a hosted API or a generic vLLM fork, the interesting question this week is narrower: can you run their production-shaped stack on hardware you control? DeepInfer is DeepSeek’s answer — open-sourced under Apache 2.0, documented on GitHub as the framework that “powers DeepSeek’s large-scale production services,” with this release focused on offline inference and the core pieces needed to serve at scale.
That license matters for companies that hesitate on research-only weights or non-commercial clauses. Apache 2.0 covers use, modification, and redistribution of the framework itself. It does not invent missing product features: the README is explicit about what is not in the box.
What ships today
The only supported model in the open release is DeepSeek-V4.1-Flash. The default checkpoint is deepseek-ai/DeepSeek-V4.1-Flash-1017, pulled into the Hugging Face cache on first run (or pointed at with DEEPINFER_HF_MODEL_DIR). Environment baseline from the README:
- Linux, Python 3.12, GCC ≥ 13
- NVIDIA SM100 GPUs with CUDA 13 (
DEEPINFER_ACCELERATOR=nvidia) - Huawei Ascend 950 NPUs with CANN 9.2 (Ascend build path)
Build is from source via ./build.sh after creating an empty Python 3.12 env. Offline entry point: python examples/offline_inference.py.
Feature set listed by DeepSeek includes optimized kernels (FlashMLA, DeepGEMM, DeepEP, DeepSelect, TileKernels / TileLang), Attention DP / balanced CP, MoE EP + EPLB, DSpark speculative decoding with confidence-scheduled verification (default max draft length 5), SWA bounded replay, asynchronous scheduling, batch-invariant inference, deterministic sampling (NVIDIA-only for the reproducibility knobs), and structured outputs via regex / JSON Schema / EBNF.
Illustration: dual-stack self-host path — NVIDIA SM100 and Huawei Ascend 950 as first-class targets, not a dark cyber-infographic template.
What you still have to build
DeepInfer is not a drop-in “HTTP inference platform.” The README lists components required for a complete inference service that are not included in this release:
- A KV cache system
- Full VE / Prefill / Decode disaggregation service wiring (much engine functionality exists; external service glue is still needed)
- Load balancing across data-parallel replicas for both prefill and decode
There is also no full online HTTP API, auth layer, or request-management product in the open tree. Compile successfully and you still own caching, front-door APIs, and multi-replica routing — the expensive ops work that turns a benchmark harness into a billable endpoint.
Hardware reality check
Official performance charts are measured on 4 SM100 GPUs for prefill and 64 GPUs for decode, each within one scale-up domain, with a reproducible sweep script. That is useful for labs that already sit near that class of interconnect — and a poor proxy for a four-GPU SMB box. Unwire’s reporting also notes that public materials do not give a clean same-condition NVIDIA-vs-Ascend bake-off, so dual-stack buyers should budget their own porting time.
For the model itself, secondary coverage (Unwire, attributing DeepSeek’s September V4.1-Flash materials) describes a MoE with roughly 552B total parameters, about 8B active on input and 16B active on output, plus claims of much smaller KV / SSD footprint versus the prior generation. Those model-architecture savings are not “DeepInfer magic”; treat them as model facts, separate from the new framework.
Clone github.com/deepseek-ai/DeepInfer, build on Linux with Python 3.12 / GCC 13+, and run offline inference against DeepSeek-V4.1-Flash-1017 if you have SM100 + CUDA 13 or Ascend 950 + CANN 9.2. Use Apache 2.0 freedom to fork and integrate. Plan separately for KV cache, disaggregation wiring, DP load balancing, and any HTTP/auth front end — those are explicitly out of scope for this release. Stack DNA is primarily vLLM, with inspiration from SGLang and TensorRT-LLM, so existing vLLM operators will recognize patterns even if the pin is DeepSeek-only for now.
Business-model angle
DeepSeek already sells tokens; open-sourcing the offline engine changes the self-host calculus more than the API price list. Teams in jurisdictions that prefer on-prem DeepSeek weights, or that mix NVIDIA and Ascend fleets, get a documented dual accelerator story under a permissive license. Teams that only wanted a managed OpenAI-compatible URL get little new — there is still no batteries-included online service in the repo.
Ignore unverified social claims about enormous daily token volumes unless DeepSeek publishes them; they are secondary chatter, not README facts. The checkable story is smaller and more useful: Apache 2.0 offline inference for V4.1-Flash on SM100 and Ascend 950, with clear gaps for anyone shipping a production front door.