How Does DeepSeek's DeepInfra Serve 155 Trillion Tokens a Day?

How does DeepSeek's DeepInfra serve a model to millions of users?

DeepInfra splits each request into two jobs and runs them on different machines. Prefill reads your prompt in one parallel pass on 8-GPU nodes. Decode writes the reply one token at a time on large GPU groups. A disk-backed KV cache skips most prompt reading, expert replication keeps GPUs evenly loaded, and speculative decoding speeds up generation. DeepSeek says one cluster running this stack serves 155 trillion tokens per day. The open release is partial, though.

Why can't you just run a normal engine on many GPUs?

Locally, one user sends one prompt and one GPU does everything. At scale, requests arrive at once with wildly different sizes. One user sends a ten-word question. An agent dumps a whole codebase a few milliseconds later.

Every request hides two opposite workloads:

If every GPU does both jobs, they interfere. Somebody's reply stalls whenever a big prompt shows up. DeepInfra exists to answer what it takes to serve a model to millions of people, beyond merely running it.

What happens to a request, step by step?

1. Formatting with DeepSeek Recipe. This separately open-sourced component converts requests from OpenAI or Anthropic format into the exact tokens the model expects, and converts output back. It also closes a prompt injection hole. Models use special tokens to mark where a user turn ends and the assistant turn begins. A naive server can turn a user-typed marker into a real control token. Recipe keeps whatever you type as plain text. In the creator's test, naive tokenization produced two assistant turns, while Recipe produced one.

2. Cache lookup. Before reading anything, the system checks the KV cache, the saved memory of prompt text already read. DeepSeek says it holds this on a dedicated cluster with about 100 petabytes of SSD. That is why the API charges much less for cached input.

3. Prefill. Only the new part of the prompt gets processed. A load balancer sorts requests into four length buckets, so short prompts do not queue behind a request of a quarter million tokens. Each GPU gets a time budget per step, and the scheduler predicts how much prompt fits inside it.

4. Decode. The KV cache moves to the decode machines, and tokens stream back to you.

Why does the KV cache last for days?

Most providers keep a prompt cache alive for around five minutes. DeepSeek keeps it for 3 days on agent traffic and 7 days on chat. It reports that 96% of input tokens are cache hits, so the model reads only about four of every hundred input tokens for real.

For builders the lesson is practical. Agents resend the same long context every turn. Keeping your prompt prefix stable is the cheapest optimization available, because it is what lets you hit the cache at all.

How does prefill avoid redundant work?

Part of the model uses sliding window attention, where certain layers only attend to the most recent 128 tokens. Instead of recomputing that window through every layer from scratch, DeepInfra replays only the short window at the end of the prompt. DeepSeek says this nearly halves total prefill work.

Why do prefill and decode run on different machines?

Prefill is compute-bound. It runs on regular nodes with eight GPUs each. Decode is memory-bound, since every token requires reading the model weights and conversation memory. It runs on super nodes with hundreds or more GPUs.

Separate pools can be tuned separately, which gives DeepSeek independent control over time to first token and time to the final response. In production, the report says it needs roughly one prefill GPU per decode GPU.

How do expert replication and speculative decoding speed up decode?

DeepSeek V4.1 Flash is a mixture-of-experts model. Each token activates only a subset of specialized sub-networks. In decode, DeepSeek spreads experts across 64 or more GPUs, so each GPU holds a handful and can serve a very large batch.

The catch is uneven popularity. Some experts get far more traffic, and the slowest GPU sets the pace for the group. So each GPU keeps a spare slot holding a copy of a popular expert. The system reshuffles those copies every few thousand steps based on current traffic.

Decode is still sequential, so speculative decoding provides the last big gain. A small draft head guesses the next five tokens. The big model verifies all five in one step. DeepSeek adds a confidence head that predicts how likely each guess is to be accepted, so compute goes only to guesses likely to survive. DeepSeek puts the gain at about 6% throughput. At this scale, that is a lot of GPUs.

Why can temperature zero still give different answers?

At temperature zero you expect identical output for identical input. On a busy server you often do not get it. GPUs choose different ways to sum numbers depending on how many requests share a batch. Tiny rounding differences can flip which token wins. Someone else's prompt in your batch can change your answer.

DeepInfra includes a batch-invariant mode that forces the math to come out the same regardless of batch contents. It is off by default because it is slower. Turn it on when you need reproducible evals or a debuggable agent.

The creator reproduced the effect on a Mac with a small open model. The same prompt at temperature zero gave one answer alone and a different one, diverging at token 14, once batched with another prompt. Every run repeated exactly, so it was not sampling noise. The test used a tiny model in a simple library, where padding also changes, so treat it as an illustration of the problem rather than a measurement of DeepInfra.

What do the production numbers look like?

DeepSeek reports these figures:

• 155 trillion tokens per day from a single cluster.
• About 14,000 tokens per second per decode GPU across all its users.
• Each user still sees a couple hundred tokens per second.
• Roughly $38 of revenue per GPU hour, at off-peak prices.

Be careful with the revenue figure. It assumes busy GPUs all year and depends on the cache system that has not been released. Extrapolations like revenue per gigawatt inherit those assumptions.

Can I run DeepInfra myself?

Probably not as released. Three limits matter:

1. Offline inference only. There is no API server in the box.
2. Key infrastructure is missing. The KV cache system and load balancers are tied to DeepSeek's internal infrastructure. You get a description, not code.
3. Hardware. The kernels are written for Nvidia's data center Blackwell chips such as the B200. The creator checked the source: the attention library only builds for those chips, and the matrix multiplication library throws an unsupported architecture error elsewhere. That includes the different Blackwell chip in a DGX Spark. Recompiling will not fix it, so check the kernel target architecture before renting hardware.

If you just want to run DeepSeek V4.1 Flash, it already works on standard engines such as vLLM, which DeepInfra was originally built on. What you can reuse today is DeepSeek Recipe, if you build on DeepSeek models, and the habit of keeping prompt prefixes stable.