--- title: Why Temperature 0 Isn't Deterministic in LLM Inference description: Temperature 0 picks the highest-logit token every time, but that doesn't make LLM output deterministic. Thinking Machines sent 'Tell me about Richard Feynman'... canonical: https://daily.dev/posts/why-temperature-0-isn-t-deterministic-in-llm-inference-embg7tbdo twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Why Temperature 0 Isn't Deterministic in LLM Inference | daily.dev og:description: Temperature 0 picks the highest-logit token every time, but that doesn't make LLM output deterministic. Thinking Machines sent 'Tell me about Richard Feynman'... og:url: https://daily.dev/posts/why-temperature-0-isn-t-deterministic-in-llm-inference-embg7tbdo og:image: https://api.daily.dev/og/posts/EMBg7TBDo.png og:image:alt: Why Temperature 0 Isn't Deterministic in LLM Inference og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Why Temperature 0 Isn't Deterministic in LLM Inference **[Daily Dose of Data Science \| Avi Chawla \| Substack](https://daily.dev/sources/dailydoseofds)** · 7 min read · 12 upvotes · 2 comments ## Summary Temperature 0 picks the highest-logit token every time, but that doesn't make LLM output deterministic. Thinking Machines sent 'Tell me about Richard Feynman' to Qwen3-235B 1,000 times at temperature 0 and got 80 different completions. The cause isn't random GPU thread timing. It's that batch size changes how kernels split their additions, and floating-point addition isn't associative, so logits shift slightly and argmax can flip at near ties. In the Feynman test, all runs matched until token 103, then diverged. Batch-invariant kernels fixed this but the improved version took 42 seconds versus 26 seconds for default vLLM on Qwen3-8B; vLLM and SGLang now ship flags for it. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does my LLM give different outputs at temperature 0 even though sampling is disabled? Temperature 0 makes token selection deterministic (always picking the highest logit), but the logits themselves vary slightly between runs because floating-point addition is not associative. Inference servers batch requests together, and the batch size determines how kernels split their summation work, changing the order of addition and the resulting logit values enough to flip argmax at near ties. _Developers debugging flaky eval reruns can track inference engine fixes like batch invariance on daily.dev._ ### What is batch invariance in LLM inference servers? Batch invariance means a single request's computed numbers should not depend on what other requests are batched alongside it. Thinking Machines coined the term after finding that kernels for matmuls, RMSNorm, and attention split their additions differently depending on batch size, producing slightly different logits and non-deterministic outputs at temperature 0, even though each individual forward pass is run-to-run deterministic. _Teams building reproducible evals can follow batch-invariance support in vLLM and SGLang via daily.dev._ ### What is the performance cost of enabling deterministic inference in vLLM? On Qwen3-8B with 1,000 requests, default vLLM took 26 seconds, while the first deterministic (batch-invariant) version took 55 seconds, later improved to 42 seconds with a better attention kernel. vLLM exposes this via the VLLM_BATCH_INVARIANT=1 environment variable (beta), and SGLang offers an --enable-deterministic-inference flag with a reported average slowdown of 34.35%. _Weighing reproducibility against latency before enabling these flags is easier when tracking vLLM and SGLang updates on daily.dev._ ## Community discussion Top comments from developers on daily.dev. **@soapbravo** · 0 upvotes > For deterministic work, quantization matters as well. **@kartiknvj** · 0 upvotes > I'd happily take 26 to 42 seconds on Qwen3-8B for eval runs. Replaying a bad production output is trickier, because the replay only matches if prod was also on the batch-invariant kernels when it happened. I'd probably keep a small invariant pool just for the traffic I expect to debug later. ## Similar posts on daily.dev - [Why does the same prompt at temperature 0 return different answers?](https://daily.dev/posts/why-does-the-same-prompt-at-temperature-0-return-different-answers--crimw9jlj) · Medium · 1 upvotes · 0 comments - [How to Decode the Temperature Parameter in LLMs](https://daily.dev/posts/how-to-decode-the-temperature-parameter-in-llms-dwt1ejyx8) · Towards Data Science · 3 upvotes · 0 comments - [How LLMs Choose Their Words: A Practical Walk-Through of Logits, Softmax and Sampling](https://daily.dev/posts/how-llms-choose-their-words-a-practical-walk-through-of-logits-softmax-and-sampling-cu0ugeeus) · Machine Learning Mastery · 2 upvotes · 0 comments --- Tags: [#gpu](https://daily.dev/tags/gpu), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/why-temperature-0-isn-t-deterministic-in-llm-inference-embg7tbdo) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Why Temperature 0 Isn't Deterministic in LLM Inference","url":"https://daily.dev/posts/why-temperature-0-isn-t-deterministic-in-llm-inference-embg7tbdo","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/why-temperature-0-isn-t-deterministic-in-llm-inference-embg7tbdo"},"datePublished":"2026-10-26T23:17:13.311Z","dateModified":"2026-10-26T23:53:21.003Z","description":"Temperature 0 picks the highest-logit token every time, but that doesn't make LLM output deterministic. Thinking Machines sent 'Tell me about Richard Feynman'...","image":"https://media.daily.dev/image/fetch/s--y3ws_x0e--/f_auto,pg_1,q_auto/https://substackcdn.com/image/fetch/%24s_%21Hj99%21%2Cw_1200%2Ch_675%2Cc_fill%2Cf_jpg%2Cq_auto:good%2Cfl_progressive:steep%2Cg_auto/https%253A%252F%252Fsubstack-post-media.s3.amazonaws.com%252Fpublic%252Fimages%252Fdf8439c9-d228-4f03-849d-6ec88d67a471_1200x1200.gif?_a=BAMAMiiw0","thumbnailUrl":"https://media.daily.dev/image/fetch/s--y3ws_x0e--/f_auto,pg_1,q_auto/https://substackcdn.com/image/fetch/%24s_%21Hj99%21%2Cw_1200%2Ch_675%2Cc_fill%2Cf_jpg%2Cq_auto:good%2Cfl_progressive:steep%2Cg_auto/https%253A%252F%252Fsubstack-post-media.s3.amazonaws.com%252Fpublic%252Fimages%252Fdf8439c9-d228-4f03-849d-6ec88d67a471_1200x1200.gif?_a=BAMAMiiw0","isAccessibleForFree":true,"articleSection":"Daily Dose of Data Science | Avi Chawla | Substack","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Daily Dose of Data Science | Avi Chawla | Substack","logo":"https://media.daily.dev/image/upload/s--4IHQgTOw--/f_auto/v1710503712/logos/dailydoseofds","url":"https://daily.dev/sources/dailydoseofds"},"commentCount":2,"discussionUrl":"https://daily.dev/posts/why-temperature-0-isn-t-deterministic-in-llm-inference-embg7tbdo","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":12},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":2}],"keywords":"gpu,ai-inference,vllm","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Daily Dose of Data Science | Avi Chawla | Substack","item":"https://daily.dev/sources/dailydoseofds"},{"@type":"ListItem","position":3,"name":"Why Temperature 0 Isn't Deterministic in LLM Inference"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/why-temperature-0-isn-t-deterministic-in-llm-inference-embg7tbdo","comment":[{"@type":"Comment","text":"For deterministic work, quantization matters as well.","datePublished":"2026-10-28T12:44:27.761Z","url":"https://daily.dev/posts/EMBg7TBDo#c-qJAY9yUOd","author":{"@type":"Person","name":"Soap Bravo","url":"https://daily.dev/soapbravo","image":"https://media.daily.dev/image/upload/s--rkKDWshV--/f_auto/v1783071551/avatars/avatar_1D30kcJ92FZKAXwAPCvAl?_a=BAMAMicg0"}},{"@type":"Comment","text":"I’d happily take 26 to 42 seconds on Qwen3-8B for eval runs. Replaying a bad production output is trickier, because the replay only matches if prod was also on the batch-invariant kernels when it happened. I’d probably keep a small invariant pool just for the traffic I expect to debug later.","datePublished":"2026-10-27T12:38:43.295Z","url":"https://daily.dev/posts/EMBg7TBDo#c-IDJoYXx0v","author":{"@type":"Person","name":"kartik-nvjk","url":"https://daily.dev/kartiknvj","image":"https://media.daily.dev/image/upload/s--3gGgsVCw--/f_auto/v1781456774/avatars/avatar_TvTVeiMdkRCqWUDullFmy?_a=BAMAMiWQ0"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/why-temperature-0-isn-t-deterministic-in-llm-inference-embg7tbdo#faq","mainEntity":[{"@type":"Question","name":"Why does my LLM give different outputs at temperature 0 even though sampling is disabled?","acceptedAnswer":{"@type":"Answer","text":"Temperature 0 makes token selection deterministic (always picking the highest logit), but the logits themselves vary slightly between runs because floating-point addition is not associative. Inference servers batch requests together, and the batch size determines how kernels split their summation work, changing the order of addition and the resulting logit values enough to flip argmax at near ties. Developers debugging flaky eval reruns can track inference engine fixes like batch invariance on daily.dev."}},{"@type":"Question","name":"What is batch invariance in LLM inference servers?","acceptedAnswer":{"@type":"Answer","text":"Batch invariance means a single request's computed numbers should not depend on what other requests are batched alongside it. Thinking Machines coined the term after finding that kernels for matmuls, RMSNorm, and attention split their additions differently depending on batch size, producing slightly different logits and non-deterministic outputs at temperature 0, even though each individual forward pass is run-to-run deterministic. Teams building reproducible evals can follow batch-invariance support in vLLM and SGLang via daily.dev."}},{"@type":"Question","name":"What is the performance cost of enabling deterministic inference in vLLM?","acceptedAnswer":{"@type":"Answer","text":"On Qwen3-8B with 1,000 requests, default vLLM took 26 seconds, while the first deterministic (batch-invariant) version took 55 seconds, later improved to 42 seconds with a better attention kernel. vLLM exposes this via the VLLM_BATCH_INVARIANT=1 environment variable (beta), and SGLang offers an --enable-deterministic-inference flag with a reported average slowdown of 34.35%. Weighing reproducibility against latency before enabling these flags is easier when tracking vLLM and SGLang updates on daily.dev."}}]} ```