Why Does Temperature 0 Still Give Different Answers in LLMs?
Temperature 0 isn't always deterministic. Batch composition changes GPU math and flips tokens. Here's why, and when to use batch-invariant inference.
Why does temperature 0 still give different answers?
Temperature 0 removes sampling randomness, but it does not remove numeric variation. On a busy server, your prompt is processed in a GPU batch with other users' requests. The batch size and makeup can change how the GPU adds up floating-point numbers. Tiny rounding differences can change which token scores highest, and once one token flips, the rest of the reply diverges. Batch-invariant inference forces the same math regardless of batch.
TL;DR
- Temperature 0 is not a determinism guarantee. It picks the highest-scoring token, but the scores themselves can shift slightly between runs.
- Other people's prompts can change your output. The GPU may choose a different summation order depending on how many requests share the batch.
- A small rounding difference can flip one token, and every token after it then conditions on that new choice.
- DeepSeek's open-sourced inference engine includes a batch-invariant mode that makes results independent of batch contents. It is off by default because it is slower.
- Turn it on for evals and agent debugging, where you need to tell a real regression from noise. Leave it off for ordinary production traffic where speed and cost matter more.
- You can see the effect on a laptop. A creator's test with a small open model changed its answer at token 14 when the same prompt ran inside a batch.
What does temperature 0 actually promise?
Temperature controls how the model samples from its output probabilities. At 0, the model skips sampling and takes the top-scoring token (greedy decoding). If the scores were identical on every run, the output would be too.
That "if" is the problem. The scores come out of a long chain of matrix multiplications and reductions on a GPU. Those operations are deterministic in principle. In practice, the order in which numbers get added can vary, and that order is not always fixed by your prompt alone.
How does batch composition change floating-point math?
Floating-point addition is not associative. Adding the same numbers in a different order can give a slightly different result, because each step rounds. The differences are tiny, often in the last few bits.
GPU kernels split work across many threads and then combine partial sums. The way they split and combine often depends on the shapes involved, including how many requests are in the batch. Serving a model at this scale is mostly about not doing work twice and never letting a GPU sit idle. Your request might share a batch with one other prompt at one moment and fifty at the next. The kernel can pick a different reduction strategy each time, and the same inputs yield slightly different logits.
Most of the time this changes nothing visible. But when two candidate tokens are nearly tied, a rounding nudge decides the winner. Language models generate one token at a time, with each token feeding the next. One flipped token sends the whole continuation down a different path.
What did the laptop experiment show?
A creator reproduced the effect on a Mac with a small open model. The setup was simple: send the same prompt at temperature 0 once on its own, then again inside batches with other prompts.
- On its own, the model said blue light gets scattered more than other colors, then claimed the sky is actually green or violet.
- In a batch with just one other prompt, the answer diverged at token 14 and said the sky is mostly blue.
- Every run repeated exactly. Same condition, same output, every time.
That last point matters. It rules out random sampling. The output depended on what else was in the batch, not on chance.
The creator was upfront about a caveat. The model was tiny and the library simple, so batching also changed the padding. That muddies the exact mechanism. It still shows the same class of problem that batch-invariant mode is designed to remove, and it shows the problem is not limited to giant clusters.
What is batch-invariant mode?
DeepSeek open-sourced the inference engine behind its API, with a write-up covering parts it did not release. One feature is a batch-invariant mode. It forces the math to come out the same no matter what else is in the batch. Without it, a busy server often does not treat your request the same way every time, and tiny rounding differences can flip which token wins.
The tradeoff is speed. The fastest kernel for a given batch shape is often not the one that gives identical results across shapes. The mode is off by default because doing the math the same way every time is slower. DeepSeek's write-up frames it as a setting you enable when you need reproducibility, not as the standard path.
When should you turn batch-invariant mode on?
Turn it on when a changed output would cost you time or trust.
| Situation | Enable it? | Why |
|---|---|---|
| Running evals and comparing model or prompt versions | Yes | You need score differences to reflect real changes, not batch noise. |
| Debugging an agent that behaves inconsistently | Yes | It removes one source of variation so you can isolate prompt, tool, or logic bugs. |
| Regression tests on fixed prompts | Yes | Reproducible outputs make diffs meaningful. |
| High-volume production chat | Usually no | The speed cost rarely buys much when users never compare runs. |
| Creative or exploratory generation | No | Variation is acceptable or even wanted. |
A practical pattern: run evals and debugging sessions in invariant mode, then confirm behavior in your normal serving configuration before shipping. If results differ materially between the two, your prompt sits near a decision boundary and deserves a closer look.
How can you make LLM inference more reproducible?
Batch invariance is one lever. A few general habits help too.
- Set temperature to 0 (and fix any seed the API supports) so sampling is not adding noise on top.
- Control the batch. For local runs, send one request at a time during evals so batch makeup cannot vary.
- Pin the stack. Keep the model version, inference engine version, and hardware the same between comparisons.
- Use a batch-invariant mode where your engine offers one. In DeepSeek's case, that is the setting to enable for eval and debugging runs.
- Log outputs and diff them. If two runs diverge, find the first differing token. A late divergence on a near-tie points to numeric noise rather than a prompt problem.
Note the limits of what DeepSeek released. The open version covers offline inference only, with no API server in the box. Its kernels target Nvidia's data-center Blackwell chips such as the B200, and the creator found the attention and matrix-multiplication libraries would not build or run on other chips, including the Blackwell variant in a DGX box. Check the target architecture before renting hardware. The model already runs on regular engines like vLLM, which Deep Infer was originally built on.
Is the cost of invariance worth it?
For most developers, the question is where nondeterminism hurts. If you ship a chatbot, a slightly different phrasing on rerun is harmless. If you run an eval suite and see a two-point swing between runs of the same model, you cannot trust the comparison. If you debug an agent whose tool calls change with no code change, you lose hours chasing a ghost.
In those cases, slower but repeatable inference is cheap compared with the cost of unreliable measurement. Treat it like a debug build: use it when you need certainty, and drop it when you need throughput.
Frequently Asked Questions
Is temperature 0 supposed to be deterministic?
Conceptually yes, since it always picks the top token. In practice the underlying scores can vary slightly between runs because of floating-point rounding and batch-dependent GPU kernels. So identical settings do not guarantee identical text on shared infrastructure.
Can another user's prompt really change my output?
On a server that batches requests, yes. Your prompt's results can depend on how many requests share the batch, because that can change how the GPU sums numbers. The effect is usually tiny, but a near-tied token can flip and change the rest of the reply.
Does batch-invariant mode make inference slower?
Yes, which is why DeepSeek's engine ships with it off by default. Doing the math the same way every time gives up some of the speed from picking the best kernel for each batch shape. Enable it when reproducibility matters more than throughput.
Can I see this problem on my own machine?
Yes. A creator ran the same prompt at temperature 0 alone and inside a batch on a Mac with a small open model, and the answer changed at token 14. Repeated runs of each setup matched exactly, so the cause was batch contents, not sampling. Padding differences in the simple library may have contributed.
How we source, write, and correct our articles. Editorial standards
