--- title: Why Other Requests Change Your LLM’s Output | daily.dev description: LLM outputs at temperature 0 aren't actually deterministic because other requests sharing your GPU batch change the result. Thinking Machines Lab sampled 1,000... canonical: https://daily.dev/posts/why-other-requests-change-your-llm-s-output-jy97euvqo twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Why Other Requests Change Your LLM’s Output | daily.dev og:description: LLM outputs at temperature 0 aren't actually deterministic because other requests sharing your GPU batch change the result. Thinking Machines Lab sampled 1,000... og:url: https://daily.dev/posts/why-other-requests-change-your-llm-s-output-jy97euvqo og:image: https://api.daily.dev/og/posts/JY97EUVQO.png og:image:alt: Why Other Requests Change Your LLM’s Output og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Why Other Requests Change Your LLM’s Output **[Daily Dose of Data Science \| Avi Chawla \| Substack](https://daily.dev/sources/dailydoseofds)** · 9 min read · 0 upvotes · 0 comments ## Summary LLM outputs at temperature 0 aren't actually deterministic because other requests sharing your GPU batch change the result. Thinking Machines Lab sampled 1,000 completions of the same prompt from Qwen3-235B and got 80 unique outputs. The cause is that GPU kernels reorder floating-point additions differently depending on batch size, and floating-point addition isn't associative, so nearly tied logits can flip. vLLM, SGLang, and DeepSeek's DeepInfer now offer batch-invariant kernel flags to fix this, though SGLang's most slowdowns are between 25% and 45% in LMSYS benchmarks. With batch invariance on, all 1,000 completions came out identical. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why does my LLM give different outputs for the same prompt even at temperature 0? Output depends on which other requests share your GPU batch, not just your prompt. GPU kernels reorder floating-point additions differently depending on batch size, and since floating-point addition isn't associative, logits shift slightly. Thinking Machines Lab found 1,000 identical requests to Qwen3-235B at temperature 0 produced 80 unique completions because of this batch-dependent rounding. _Anyone debugging flaky eval results can follow daily.dev for deep dives into LLM inference reproducibility._ ### How do I make vLLM inference deterministic across different batch sizes? Set the environment variable VLLM_BATCH_INVARIANT=1 when serving a model with vLLM, which is still in beta. This forces every reduction to use a fixed order regardless of batch size, trading some throughput for bitwise-reproducible outputs. SGLang offers an equivalent --enable-deterministic-inference flag, with throughput costs between 25% and 45% in LMSYS benchmarks. _Teams tuning reproducible LLM evals can track these inference flags and tradeoffs on daily.dev._ ### How accurate is retriever-free RAG with decision models compared to standard embedding-based RAG? On Natural Questions, a decision-model-based navigation approach put the correct answer in its top pick 65.4% of the time, versus 48.0% for embedding search plus an 8B reranker and 60.8% for GPT-5 reranking the dense top 20. The gain comes from better ranking: when both methods retrieve the answer, navigation ranks it first 70.1% of the time versus 61.9% for embedding search. _Developers evaluating RAG architectures can compare retrieval strategies like this through daily.dev._ ## Similar posts on daily.dev - [Why Temperature 0 Isn't Deterministic in LLM Inference](https://daily.dev/posts/why-temperature-0-isn-t-deterministic-in-llm-inference-embg7tbdo) · Daily Dose of Data Science \| Avi Chawla \| Substack · 12 upvotes · 2 comments - [Why does the same prompt at temperature 0 return different answers?](https://daily.dev/posts/why-does-the-same-prompt-at-temperature-0-return-different-answers--crimw9jlj) · Medium · 1 upvotes · 0 comments - [Catching poor LLM performance and accuracy before deployment](https://daily.dev/posts/catching-poor-llm-performance-and-accuracy-before-deployment-i9x2r50jf) · Red Hat Developer · 0 upvotes · 0 comments --- Tags: [#gpu](https://daily.dev/tags/gpu), [#rag](https://daily.dev/tags/rag), [#ai-inference](https://daily.dev/tags/ai-inference), [#vllm](https://daily.dev/tags/vllm) [View this post on daily.dev](https://daily.dev/posts/why-other-requests-change-your-llm-s-output-jy97euvqo) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Why Other Requests Change Your LLM’s Output","url":"https://daily.dev/posts/why-other-requests-change-your-llm-s-output-jy97euvqo","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/why-other-requests-change-your-llm-s-output-jy97euvqo"},"datePublished":"2026-10-31T20:28:56.129Z","dateModified":"2026-10-31T20:29:25.668Z","description":"LLM outputs at temperature 0 aren't actually deterministic because other requests sharing your GPU batch change the result. Thinking Machines Lab sampled 1,000...","image":"https://media.daily.dev/image/fetch/s--KSOYXZol--/f_auto,pg_1,q_auto/https://substackcdn.com/image/fetch/%24s_%21fxG4%21%2Cw_1200%2Ch_675%2Cc_fill%2Cf_jpg%2Cq_auto:good%2Cfl_progressive:steep%2Cg_auto/https%253A%252F%252Fsubstack-post-media.s3.amazonaws.com%252Fpublic%252Fimages%252F5a8f8bb1-b631-4984-89b2-5f2d80b6df03_900x900.gif?_a=BAMAMiiw0","thumbnailUrl":"https://media.daily.dev/image/fetch/s--KSOYXZol--/f_auto,pg_1,q_auto/https://substackcdn.com/image/fetch/%24s_%21fxG4%21%2Cw_1200%2Ch_675%2Cc_fill%2Cf_jpg%2Cq_auto:good%2Cfl_progressive:steep%2Cg_auto/https%253A%252F%252Fsubstack-post-media.s3.amazonaws.com%252Fpublic%252Fimages%252F5a8f8bb1-b631-4984-89b2-5f2d80b6df03_900x900.gif?_a=BAMAMiiw0","isAccessibleForFree":true,"articleSection":"Daily Dose of Data Science | Avi Chawla | Substack","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Daily Dose of Data Science | Avi Chawla | Substack","logo":"https://media.daily.dev/image/upload/s--4IHQgTOw--/f_auto/v1710503712/logos/dailydoseofds","url":"https://daily.dev/sources/dailydoseofds"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/why-other-requests-change-your-llm-s-output-jy97euvqo","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"gpu,rag,ai-inference,vllm","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Daily Dose of Data Science | Avi Chawla | Substack","item":"https://daily.dev/sources/dailydoseofds"},{"@type":"ListItem","position":3,"name":"Why Other Requests Change Your LLM’s Output"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/why-other-requests-change-your-llm-s-output-jy97euvqo#faq","mainEntity":[{"@type":"Question","name":"Why does my LLM give different outputs for the same prompt even at temperature 0?","acceptedAnswer":{"@type":"Answer","text":"Output depends on which other requests share your GPU batch, not just your prompt. GPU kernels reorder floating-point additions differently depending on batch size, and since floating-point addition isn't associative, logits shift slightly. Thinking Machines Lab found 1,000 identical requests to Qwen3-235B at temperature 0 produced 80 unique completions because of this batch-dependent rounding. Anyone debugging flaky eval results can follow daily.dev for deep dives into LLM inference reproducibility."}},{"@type":"Question","name":"How do I make vLLM inference deterministic across different batch sizes?","acceptedAnswer":{"@type":"Answer","text":"Set the environment variable VLLM_BATCH_INVARIANT=1 when serving a model with vLLM, which is still in beta. This forces every reduction to use a fixed order regardless of batch size, trading some throughput for bitwise-reproducible outputs. SGLang offers an equivalent --enable-deterministic-inference flag, with throughput costs between 25% and 45% in LMSYS benchmarks. Teams tuning reproducible LLM evals can track these inference flags and tradeoffs on daily.dev."}},{"@type":"Question","name":"How accurate is retriever-free RAG with decision models compared to standard embedding-based RAG?","acceptedAnswer":{"@type":"Answer","text":"On Natural Questions, a decision-model-based navigation approach put the correct answer in its top pick 65.4% of the time, versus 48.0% for embedding search plus an 8B reranker and 60.8% for GPT-5 reranking the dense top 20. The gain comes from better ranking: when both methods retrieve the answer, navigation ranks it first 70.1% of the time versus 61.9% for embedding search. Developers evaluating RAG architectures can compare retrieval strategies like this through daily.dev."}}]} ```