DeepSeek Open-Sourced DeepInfer. The Deployment Q&A Is the Part to Read.

DeepSeek put DeepInfer on GitHub this week under Apache 2.0. The README calls it "the high-performance inference framework that powers DeepSeek's large-scale production services." It ships with one supported model, DeepSeek-V4.1-Flash-1017, and two supported targets: NVIDIA SM100 GPUs on CUDA 13 and Huawei Ascend 950 NPUs on CANN 9.2.

Most of the coverage stops at that sentence. I think the more useful file is doc/inference_deployment_qa.md, a long Q&A about how DeepSeek actually runs V4.1 Flash. It's rare to get this much operating detail from a lab: storage ratios, load-balancer tiers, expert re-placement intervals, KV cache TTLs. It reads like a production design review. Below is what I took from it as someone who has to size and run clusters, plus the limits of what was released.

What the repo is, and what it isn't

Be clear about scope before planning anything around it. The public release is an offline engine. The README lists three things a full online service needs that are not included:

• A KV cache system
• Vision-encoder / prefill / decode disaggregation ("most required engine functionality is already implemented")
• Load balancing across data-parallel replicas for prefill and decode

The Q&A says the KV cache and load-balancing pieces are "tightly coupled to our internal infrastructure." So you get the engine and its kernels, and DeepSeek's design for the rest is written down in prose rather than shipped as code.

What is in the engine:

• Kernels from FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang ops
• Attention DP with balanced context parallelism, MoE expert parallelism with EPLB
• DSpark speculative decoding with confidence-scheduled verification
• "SWA bounded replay" for sliding-window attention (more on this below)
• An asynchronous, overlapped CPU scheduler
• Batch-invariant inference and deterministic sampling, both NVIDIA-only
• Structured outputs (regex, JSON schema, EBNF)

The README credits vLLM as the original base, with ideas taken from SGLang and TensorRT-LLM. The build needs Linux, GCC 13 or newer, and Python 3.12. Nothing older than SM100 is listed, so H100 and H200 fleets are out for this release.

There is also deepseek-recipe, a Rust library that converts Chat Completions, Responses, and Messages API requests into V4.1 token sequences. One detail is worth copying: strings like <｜User｜> inside user input or tool results are encoded as ordinary text, never as control tokens, and tool-call markers are matched by token ID, so text can't inject a tool call.

The topology DeepSeek recommends

The Q&A is specific about parallelism:

• Prefill: EP2–8 / DP2–8, using the smallest EP/DP that HBM allows. In other words, ordinary 8-GPU boxes.
• Decode: EP64–128 / DP64–128, using the largest EP that your GPUs allow. That means a 64- to 128-GPU scale-up domain.
• Vision encoder: one GPU per instance, and DeepSeek recommends cheaper GPUs for it.
• No tensor parallelism and no context parallelism in production "for now."

The sizing line I'd underline: "1P + 1D is deployable," the prefill-to-decode GPU ratio is roughly 1:1, and because decode wants EP64 or more, "the smallest efficient deployment unit is 64 GPUs of prefill plus 64 GPUs of decode." So if you were hoping to try DeepSeek's production recipe on a single node, you can run it, but you won't be measuring what DeepSeek measures.

The published benchmark charts match that. Prefill was measured on 4 SM100 GPUs and decode on 64, each within one scale-up domain. The charts are NVIDIA-only. There are no Ascend 950 throughput numbers in the repo, so don't quote any.

Storage: the number that surprised me

DeepSeek retains KV cache for days, not minutes. The design goal is more than 72 hours of retention. The storage details:

• Block KV cache (main KV plus indexer K) lives on a dedicated 3FS cluster, not in host memory. Any prefill machine can hit any prefix, so they don't need prefix-affinity routing.
• About 14.4 TB of logical KV storage per online GPU. Physical capacity has to be larger than that.
• Recommended KV storage server: 20 × 30.73 TB SSDs, at least 40 GB/s read and 5 GB/s write, enough capacity for 24 hours of data.
• 3FS runs 10+2 erasure coding. A read goes to all 11 data nodes in the group, and any 10 responses are enough to decode.
• Roughly 100 PiB of SSD in total. The TTL is 3 days for agent traffic and 7 days for chatbot traffic, because chatbot hits have a long tail. A 3-day TTL cost chatbot traffic 0.5–1% of hit rate.

They give a reason for using a separate storage tier: compute nodes fail "more than an order of magnitude" more often than storage nodes, so keeping cache off the GPU hosts keeps it alive through failures. Swap-in from host shared memory to HBM uses GPUDirect RDMA instead of the CUDA copy engine. The copy engine does poorly on small KV block copies, and an RDMA NIC can saturate PCIe without using SMs.

If you plan capacity for a cache-heavy agent workload, 14.4 TB per GPU is the figure to bring to your storage team. It is one company's number on one model, but it's a real measured number, and those are rare.

Load balancing, in enough detail to copy

The prefill load balancer splits requests by total context length into four tiers: under 16K, 16–64K, 64–256K, and 256K or more. Engines are reassigned across tiers based on each tier's share of compute demand in the previous round. Inside a GPU, the chunk size isn't fixed. Each step gets a time budget in nanoseconds, a fitted quadratic cost model cost(q, c) predicts step time from new tokens and already-computed prefix, and the scheduler solves for the largest q that fits under a 32,768-token cap.

On decode, the balancer weighs two things for each GPU: how many requests it's running and how much KV it holds. Short requests go to GPUs with few requests but a lot of KV, and long requests go to GPUs with little KV. DeepSeek says the whole system reaches about 80% of a perfectly balanced offline benchmark.

Expert load balancing has measured numbers too. Prefill starts from 4 redundant experts per GPU, uses a 100-step statistics window, and re-assigns experts every 1,000 steps, giving moe_lb (slowest rank over mean) of 1.074 at EP4/DP4. Decode starts from 1 redundant expert per GPU and re-assigns every 10,000 steps, giving moe_lb 1.090 at EP64. Expert weights move within the group over NVLink symmetric memory.

Failover is blunt. In wide EP, one bad GPU takes down the whole instance's communication domain. On restart, the instance takes however many healthy GPUs it gets as its new EP size. A draining decode engine gets a one-minute grace period, then forwards unfinished requests and their KV to other engines. Scale-in is blocked while quota utilization is above 0.8.

DSpark: modest gains, and they depend on load

DSpark's draft model proposes 5 tokens, and a confidence head predicts how likely each one is to be accepted. The scheduler then verifies only the prefix with the best expected payoff. DeepSeek's EP64/DP64 table is honest about where that helps:

At light load it's slightly worse. At DeepSeek's production point of about 60 sequences per rank, it's 4–7% better. Accepted length barely changes (3.29 vs 3.20 on average). The gain comes from skipping verification work on low-confidence tokens. These are estimates computed from benchmark data, not end-to-end measurements, and the Q&A says so.

SWA bounded replay and FP4 placement

Two smaller details are worth knowing.

V4.1 Flash mixes sliding-window and compressed attention. When a prefix hits in the block cache but its SWA KV is missing, DeepInfer recomputes only the last 128 tokens and limits attention to that replay segment. That makes the recompute cost a constant. On the decoder side, prefill recomputes only the last 128 prompt tokens instead of running 2,560 tokens through 20 decoder layers, which DeepSeek says "nearly halv[es] total prefill computation." The README's default settings are DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140.

FP4 shows up in three places: main KV (per-16 e4m3, NVFP4-like but without the global scale), indexer KV (MXFP4), and routed-expert weights (MXFP4).

On kernels, DeepSeek admits the public FlashMLA is missing one hand-tuned SASS optimization, which makes the decoding kernel 5–10% slower than their internal one. They put the end-to-end gap at most 5%.

What I'd do with this

If you run SM100 hardware and serve V4.1 Flash, the engine is worth benchmarking against your current vLLM or SGLang build with examples/sweep/run_sweep.sh. Watch two things the benchmark README calls out. Decode throughput is reported under "full-acceptance accounting," meaning 1 + nextn tokens per step whatever the real acceptance rate, so scale it by your own acceptance rate before comparing. And don't trust step times from PROFILE=1 runs, which the docs say can be inflated by up to nearly 50% at batch size 1.

If you don't run this model, read the Q&A anyway. The storage ratio, TTL policy, tiered prefill balancing, and EP failover behavior apply to any wide-EP MoE deployment. DeepSeek's headline claim is that one cluster serves 155 trillion tokens per day at US$38 per GPU-hour in revenue "at valley pricing." That's their number about their own operation, and I can't verify it. The engineering choices in the document are much more useful than the headline anyway.

Anpoo Sivanadi, Staff Software Engineer