DeepInfer Benchmarks: 2× Serving Throughput, 0.59 s TTFT and Hopper Support from Community PRs

DeepSeek open-sourced DeepInfer on October 20, 2026. It is the inference engine behind DeepSeek’s production services, released as an offline runtime for DeepSeek-V4.1-Flash (see our DeepInfer explainer). By October 23 the repository had about 20,500 stars and 191 pull requests. [1] We read every PR and its discussion to see what the community changed, what it measured, and where DeepInfer looks likely to go next.

The short version: turning off one default doubled serving throughput, a prefix cache cut time to first token from 8.9 s to 0.59 s, and the engine already runs on Hopper and AMD. Almost none of this is merged.

Figure 1. DeepSeek’s production serving architecture, from the repository’s doc/images/architecture_overview.png. The open release covers the engine; the load balancers and KV cache store are not fully released. [2]

Official scope: two chip families, one merged PR

DeepInfer officially supports NVIDIA Blackwell (SM100) and Huawei Ascend 950. The contributing guide has a separate ci_npu test marker, so Ascend goes through the same process as GPUs. [3]

Two lines in the deployment Q&A sum up the repository’s role. DeepSeek says inference code and performance work for future DeepSeek models will keep being open-sourced in DeepInfer. It also says KV cache management and load balancing are tightly coupled to internal infrastructure and “not yet fully open-sourced.” [2]

Of the 191 PRs, the one merged is #157 from maintainer wx-csy, which bumps DeepEP. [4] Among outside PRs, #92 got a reply from maintainer beginlner:

Good job! But we won’t be merging it — you can maintain the code yourself.

PR #92 ports DeepInfer to Hopper. The message is clear: DeepSeek values the work, and the author maintains it. [5] Issues are disabled on the repository, so RFCs also arrive as PRs.

What the community is building

Figure 2. DeepInfer PRs by topic, grouped by title keywords, as of October 23, 2026. [4]

Grouped by title, 32 PRs port hardware, 21 fix evaluation, 19 add KV caching or prefill/decode (PD) disaggregation, and 8 add online serving. Those are the gaps the official release left open.

Hardware ports: Hopper and AMD run end to end

Hopper (#92) is the most complete port. The author ran DeepSeek-V4.1-Flash on 8× H20-3e (141 GB) and reports: [5]

The PR changes 56 files and adds 9,159 lines. Weights are repacked at load time, and the author checked all 47,232 expert matrices bit for bit. A reviewer testing on H200 found an alignment bug in the pinned Humming 0.1.12 dependency, fixed upstream in 0.1.13. [5]

AMD (#201) targets MI350X and MI355X with a full ROCm backend. The author split it into five smaller PRs, #202 to #206, for review, then reopened the main PR. [6]

Ascend 910B (#152) is an RFC with measurements. Each 910B3 card has 60.96 GiB usable. The 510 GB checkpoint grows to about 770 GB after W8A8 quantization, so the author estimates at least two 8-card machines. [7]

Apple Silicon (#168) adds the first native Metal kernels. The author notes that a Mac needs roughly 128 GB of unified memory or more before local inference is more than a demo. [8]

Online serving: EPLB was eating half the time

The release is an offline engine. PR #147 wraps it in a minimal OpenAI-compatible /v1/chat/completions server. The PD split (#197) and the benchmark reproductions (#196) build on that server. [9]

In #147’s comments, user whn09 ran a one-hour multi-turn agent trace on 8× B200 and found the server hard-coded EPLB, the expert load-balancing rearrangement, to on. Rearrangement ran every 30 steps and after every prefill. Online, nearly every request has a prefill, so it ran almost constantly. Each rearrangement took about 0.34 s, while 30 decode steps took about 0.28 s. [9]

He reran the same trace with EPLB off:

Throughput doubled. With EPLB off, the server matched vLLM’s published InferenceX B200 point for the same workload (0.97× throughput, 0.96× interactivity). The PR author, kzjeef, confirmed the result the same day and made EPLB off by default. [9]

Figure 3. Three community A/B tests. Numbers are self-reported in PRs #200, #147 and #144. [9][10][11]

Prefix cache and PD split

#144 adds a host-memory prefix cache in the style of SGLang’s HiCache. On 4× GB300 with a single-concurrency multi-turn agent trace, median time to first token fell from 8.9 s to 0.59 s, and the server finished 224 requests in an hour instead of 99. [10]

#197 splits prefill and decode across two GB300 trays. From concurrency 8 to 256, p90 inter-token latency stayed between 8.6 and 15 ms, where a single tray ranged from 12 to 98 ms. [12] #214 adds Mooncake as the KV transport. On one host its outputs were byte-identical to the TCP path at the same latency. The author’s point is that cross-host transfer can then use RDMA directly. [13]

Reproducing DeepSeek’s curves

The README publishes prefill and decode Pareto curves at 131,072 context.

Figure 4. DeepSeek’s official prefill curve, from the DeepInfer README. [14]

#196 reproduced those sweeps and reached 64–81% of the official prefill curve on 4× GB300 and 59–69% on 4× B200. [15]

#200 found one cause. The Engram table fetch relies on GPU-initiated RDMA. On many cloud hosts the driver rejects the required memory registration, and DOCA falls back to a CPU proxy mode. Nothing is logged at warning level, and the fetch runs 5–10× slower. In one profiled prefill step, the fetch took 394 ms of 1,133 ms (35%). [11]

The author switched to reading the table from shared host memory (UVA). On 8× B200, prefill and decode came back close to the official curves:

Figure 5. Uploaded by the PR #200 author. Blue dashed: official curve. Red: #200. [11]

kzjeef retested the change on a GB300 tray whose ConnectX-8 links were down, leaving one 100 Gb/s bond. Prefill came in 10–20% above DeepSeek’s published curve at every batch size: [11]

Figure 6. From PR #200’s comments, uploaded by kzjeef. [11]

Where DeepInfer is heading

DeepSeek’s statements and the first week of PRs point to a few likely directions:

What to use on your hardware

• B200, GB300 or Ascend 950: run the official offline engine. To serve it, start from #147 and #197, and keep EPLB off. If prefill is far below the curve, check for the CPU proxy fallback in #200.
• H20, H100 or H200: use the #92 branch and upgrade Humming to 0.1.13.
• AMD, Ascend 910, RTX 5090 or Mac: these ports are early. Follow and test them, but don’t plan production on them yet.
• Production online serving today: vLLM and SGLang still have the more complete stack.

DeepInfer community FAQ

Does DeepInfer run on H100, H200 or H20?

Not officially. DeepSeek supports NVIDIA Blackwell SM100 and Huawei Ascend 950. Community PR #92 runs DeepSeek-V4.1-Flash end to end on 8× and 4× H20-3e, and the author maintains it outside the main branch. On H200, upgrade its Humming dependency to 0.1.13.

Can DeepInfer serve an OpenAI-compatible API?

Not in the official release, which is offline only. Community PR #147 adds a minimal /v1/chat/completions server, and #197 adds a prefill/decode split on top of it. Neither is merged.

Why is DeepInfer slower than DeepSeek’s published benchmarks?

Reproductions on GB300 and B200 reached 59–81% of the official prefill curve. PR #200 found that GPU-initiated RDMA for the Engram table can silently fall back to a CPU proxy and run 5–10× slower. Reading the table from host memory restored the published numbers.

Should I turn off EPLB when serving DeepInfer?

For online serving, community tests say yes. On 8× B200 at 128 concurrency, EPLB rebalancing used about half of wall time and turning it off doubled throughput. PR #147 now defaults EPLB to off.

References

1. deepseek-ai/DeepInfer. Repository: about 20,500 stars and 2,200 forks as of 23 October 2026.
2. deepseek-ai/DeepInfer. doc/inference_deployment_qa.md and architecture diagram. GitHub.
3. deepseek-ai/DeepInfer. CONTRIBUTING.md. GitHub.
4. deepseek-ai/DeepInfer. Pull requests, including #157. Accessed 23 October 2026.
5. DarkraiHL. #92: SM90 (Hopper) support, with comments. GitHub.
6. vivienfanghuagood. #201: AMD ROCm backend for MI350X/MI355X. GitHub.
7. wangzhao-11a. #152: RFC, Ascend 910B (A2) support. GitHub.
8. lucifer-holly. #168: Apple Silicon (MPS) support. GitHub.
9. kzjeef. #147: minimal OpenAI-compatible chat server, including whn09’s EPLB measurements in the comments. GitHub.
10. kzjeef. #144: host-memory prefix cache. GitHub.
11. JohnQin-llm. #200: opt-in UVA Engram fetch, with kzjeef’s GB300 retest in the comments. GitHub.
12. kzjeef. #197: prefill/decode split over two hosts. GitHub.
13. whn09. #214: Mooncake KV transport. GitHub.
14. deepseek-ai/DeepInfer. README and prefill_pareto.png. GitHub.
15. kzjeef. #196: README benchmark reproductions on 4× GB300 and 4× B200. GitHub.