DeepInfer and DeepSeek Recipe: Open Models, Open Kernels, Open Inference

DeepSeek put out a four-part thread on X on October 20 that ended with the line "Open models. Open kernels. Open inference." Attached were two GitHub repositories. DeepInfer is the inference framework DeepSeek uses to serve DeepSeek-V4.1-Flash in production. deepseek-recipe is the piece that turns an API request into a model prompt and turns the model's output back into an API response.

The model is brand new. DeepSeek-V4.1-Flash-1017 went up on Hugging Face under the MIT license on October 18, two days before the thread. Now DeepSeek has opened the code that sits between those weights and a working API too, and most labs never publish that part.

What's in DeepInfer

DeepInfer is mostly built on vLLM, and the README credits SGLang and TensorRT-LLM for some of its ideas. It pulls in kernels DeepSeek had already open-sourced (FlashMLA, DeepGEMM, DeepEP) and adds custom kernels written in TileLang.

For parallelism it supports attention data parallelism, balanced context parallelism, and MoE expert parallelism with expert load balancing (EPLB). Speculative decoding is built in: the model drafts up to five tokens at a time by default, and a confidence-based scheduler decides which of them to keep. CPU batch scheduling runs asynchronously alongside device execution. Structured output can be constrained by regex, JSON schema or an EBNF grammar. On NVIDIA hardware it also offers batch-invariant inference and deterministic sampling, so a request gets the same result no matter what else happens to be in the batch.

The feature we found most interesting is what DeepSeek calls SWA bounded replay. During prefill, the decoder replays only a short suffix of the input instead of the whole thing. For V4.1 the default is 140 tokens, which has to cover the sliding-window attention span plus the maximum backtracking length. That cuts the recomputation each request needs.

DeepInfer runs on two kinds of hardware, NVIDIA SM100 GPUs (the Blackwell generation) and Huawei Ascend 950 NPUs, and builds against CUDA 13 or CANN 9.2. It is licensed under Apache 2.0.

DeepSeek is also clear about what's missing. There is no KV cache system, no prefill/decode disaggregation, and no load balancing across data-parallel replicas, and the README says a full service needs all of them. Only one model is supported so far, DeepSeek-V4.1-Flash. The performance charts were measured on 4 GPUs for prefill and 64 for decode, with no direct comparison against vLLM or SGLang. So what we got is the engine at the center of DeepSeek's serving stack, and the pieces around it stay in house for now. The thread says inference code for future DeepSeek models will go into the same repo.

What's in deepseek-recipe

deepseek-recipe is a set of Rust libraries with Python bindings, released under the MIT license. It takes requests in OpenAI Chat Completions, OpenAI Responses or Anthropic Messages format, converts them into one internal conversation format, and renders that as a DeepSeek V4 or V4.1 prompt. On the way back it parses reasoning content, tool calls, JSON output and stop sequences into a response in whatever format the request came in, streamed or complete.

It handles the details that self-hosted deployments tend to get wrong: thinking mode and reasoning effort (with reasoning_tokens reported separately in usage), function tools and V4.1's deferred tool loading, json_schema and regex structured output, image input for V4.1, and log probabilities. Default output limits are 8,192 tokens, 65,536 in thinking mode, and 131,072 in thinking mode at max effort. The library does no inference itself. You bring the model backend, the tool execution and the HTTP layer.

If you've ever hosted an open model, you know downloading the weights is the easy part. Self-hosted deployments usually drift from the official API somewhere in the chat template, the tool-call parsing or the way reasoning output comes back. DeepSeek says deepseek-recipe is "aligned with the processing logic behind the official DeepSeek API," so anyone hosting the model can now check their setup against DeepSeek's own code.

Why DeepSeek would give this away

Start with the business. DeepSeek makes its money from the model, and the serving software exists to support it. The weights are already MIT-licensed, so every third-party host that serves DeepSeek well spreads the model further. A host with broken tool calls or the wrong template makes the model look worse than it is. Publishing deepseek-recipe protects the model's reputation on APIs DeepSeek doesn't run.

Compute is also tight. DeepSeek's API was down for nearly five hours on October 15. The company apologized and issued credits, and some experts quoted by Bloomberg blamed a shortage of computing capacity. When your own capacity is limited, it helps a lot if other operators can serve your model efficiently on their own hardware.

The release also makes DeepInfer the reference implementation. Future DeepSeek models will ship their inference code in this repo, so engine vendors and cloud providers that want day-one support will have working code to follow instead of a paper to interpret.

Why we admire it

DeepSeek is our favorite frontier AI lab, and this release is a good example of why. Over the years they have opened the weights, published detailed technical papers, released the kernels underneath (FlashMLA, DeepGEMM, DeepEP), and now shared the serving engine and the API layer. Every one of those releases saved other teams a lot of work. The whole industry moves faster because one lab keeps showing how it builds things, and every AI company, us included, could learn from that.

Ascend gets the same support as NVIDIA

DeepInfer has exactly two hardware targets, NVIDIA SM100 and Huawei Ascend 950, and CANN is a first-class build option next to CUDA. Most inference frameworks add non-NVIDIA hardware later as a port. Here Ascend was supported on day one, in the production framework of one of the most widely used open model families.

Much of NVIDIA's advantage comes from CUDA and the nearly two decades of software built on top of it. Each serious workload that runs well without CUDA chips away at that advantage, and a top lab's production inference stack running on Ascend counts as one.

It's one repo, though. The README gives no Ascend performance numbers, and batch-invariant inference and deterministic sampling still only work on NVIDIA. Still, we read it as a clear signal that Ascend is on its way to becoming general-purpose compute for serving large models. If that holds, the effect on NVIDIA's position will get easier to see over time.

What this means for private deployment

For us the biggest takeaway is that private deployment of open models just got easier. Infrastructure companies and data center operators can start from the code DeepSeek runs in production instead of reverse engineering it. Compute centers built on Ascend now have a reference stack for one of the strongest open models out there. The more operators that can serve a strong model efficiently, the lower the cost per token gets, and the more people can use capable models at a low price.

deepseek-recipe is useful even if you never host a model. Any gateway that serves several API formats, ours included, has to map reasoning content and tool calls between OpenAI-style and Anthropic-style responses. DeepSeek's own mapping is now public, which gives everyone something to compare against.