What changed
DeepSeek published DeepInfer as open source on GitHub in October 2026. The project is the high performance inference framework that powers DeepSeek production services. The first public drop focuses on offline inference. It supports DeepSeek-V4.1-Flash and runs on NVIDIA SM100 GPUs or Huawei Ascend 950 NPUs.
The README lists core features. They include custom kernels built with FlashMLA, DeepGEMM, DeepEP, and TileLang helpers. They also include attention data parallel paths, MoE expert parallelism with load balancing, and confidence scheduled verification for draft tokens. Async scheduling overlaps CPU batch prep with GPU work. Structured outputs accept regex, JSON schema, or EBNF grammar through SamplingParams. Batch invariant mode on NVIDIA keeps outputs stable when batch size changes. Operators still need to bring their own KV cache layer and load balancing for a full service.
DeepSeek ships benchmark plots in the repo. Prefill tests use four SM100 GPUs inside one scale up domain. Decode tests use sixty four GPUs in another domain. Those charts are a starting point for your own sweeps, not a guarantee on your network or your prompt mix.
Why it matters
Many teams want DeepSeek class speed without guessing what happens inside a closed stack. A source release lets you measure prefill and decode on your own hardware. That data feeds capex plans and vendor bake offs. It also gives legal and security teams a repo they can scan before they allow a model on private data.
How this differs from a generic vLLM fork
DeepSeek notes that DeepInfer started from vLLM and borrowed ideas from SGLang and TensorRT-LLM. The open drop is not a full hosted product. It is the engine slice with hooks for MoE routing, draft verification, and sliding window attention replay that match DeepSeek V4.1 defaults. Teams already on stock vLLM should plan a side by side eval rather than a hot swap.
Who is affected
Inference platform owners and ML ops leads carry the first review. Applied AI teams that run agents on MoE models will feel latency changes first. Chip buyers comparing SM100 against Ascend should use the bundled sweep scripts as a starting point, not as a final contract metric.
What to do next
Build from source with Python 3.12, GCC 13 or newer, and CUDA 13 for NVIDIA builds. Export DEEPINFER_ACCELERATOR=nvidia or ascend before you run ./build.sh. Run examples/offline_inference.py with the default Hugging Face checkpoint or point DEEPINFER_HF_MODEL_DIR at a local copy. Capture prefill and decode charts from examples/sweep/run_sweep.sh and compare them to your current serving stack before you change production images.
If you need strict repeatability for evals, enable batch invariant mode and disable draft verification as the README describes. Pass a fixed seq_seed in SamplingParams and record driver versions alongside results. Security teams should treat the repo like any other inference engine and scan containers before they reach prod VLANs.
What to watch
The README states that KV cache systems and prefill or decode disaggregation are not in this release. Watch the GitHub issue tracker for tags that add them. Also watch whether Ascend and NVIDIA paths stay feature parity on structured output and deterministic sampling flags.
Limits to keep in mind
The license is Apache 2.0 for code, but model weights stay under their own terms on Hugging Face. Ascend builds need CANN 9.2 and may lag NVIDIA feature flags. Nothing in this release replaces observability, rate limits, or tenant isolation in your API gateway. Treat DeepInfer as a performance lab until your team wires cache, auth, and quota layers around it. Budget time for a rebuild when CUDA or CANN versions shift, because kernel paths are tied to specific driver stacks.
Photo: RDNE Stock project / Pexels
Sources
- Primary — DeepSeek-AI, DeepInfer repository (October 2026). Feature list, build steps, and license.
- Primary — DeepSeek-AI, DeepSeek-V4.1-Flash-1017 on Hugging Face (2026). Default weights for examples.
- Secondary — GetStream, awesome-ai-news October 2026 (October 2026). Launch pointer with link to DeepInfer.
Related in AI
Continue reading