--- title: Cross-request KV cache exposure in the GLM-5.3 Workers... description: A misconfigured KV cache pool in Cloudflare's GLM 5.3 Workers AI deployment let 155 responses contain data from other customers' requests between August 27 and... canonical: https://daily.dev/posts/cross-request-kv-cache-exposure-in-the-glm-5-3-workers-ai-model-and-our-remediation-with-sglang-qkref7qhu twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang | daily.dev og:description: A misconfigured KV cache pool in Cloudflare's GLM 5.3 Workers AI deployment let 155 responses contain data from other customers' requests between August 27 and... og:url: https://daily.dev/posts/cross-request-kv-cache-exposure-in-the-glm-5-3-workers-ai-model-and-our-remediation-with-sglang-qkref7qhu og:image: https://api.daily.dev/og/posts/qkRef7qHU.png og:image:alt: Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang **[Cloudflare](https://daily.dev/sources/cloudflare)** · 19 min read · 0 upvotes · 0 comments ## Summary A misconfigured KV cache pool in Cloudflare's GLM 5.3 Workers AI deployment let 155 responses contain data from other customers' requests between August 27 and September 9, 2026. The prefill GPUs only had room for 681,280 tokens despite a 1,048,576-token context window, and an SGLang bug caused over-length prompts to be sent through anyway, reading stale KV cache pages from earlier requests. Cloudflare took the model offline within 40 minutes of declaring an incident, fixed the pool size, patched SGLang, and restored service the same day. SGLang released fixes in v0.5.20 through v0.5.22, and Cloudflare recommends all SGLang operators upgrade. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What caused the cross-request KV cache exposure in Cloudflare's GLM 5.3 Workers AI model? A misconfigured KV cache pool combined with an SGLang bug. Cloudflare deployed GLM 5.3 with a 1,048,576-token context window but a prefill GPU pool sized for only 681,280 tokens. When prompts exceeded the pool, SGLang's rejection path replaced the prompt with a placeholder instead of aborting, so decode nodes read stale cache pages from earlier requests, exposing their data in 155 responses between August 27 and September 9, 2026. _Teams running multi-tenant LLM inference can follow incident writeups like this on daily.dev to catch similar KV cache risks early._ ### Which SGLang version fixes the KV cache cross-request exposure bug found in disaggregated prefill/decode mode? The fix landed in SGLang v0.5.20 with follow-ups in v0.5.21 and v0.5.22, which also added per-transfer checksums, abort-ACK preservation, and deferred KV release enabled by default. Operators running disaggregated SGLang are advised to update to v0.5.22 or later, or backport the relevant patches, since the bug only affects deployments where the KV pool is smaller than the model's context length. _Developers upgrading SGLang deployments can track fixes like this on daily.dev before shipping to production._ ### How can I prevent SGLang from exposing another request's KV cache data when a prompt exceeds the context window? Measure the KV pool size against the context length on every prefill and decode node from the engine's startup log after any change to weights, quantization, parallelism, or memory fraction, and lower context length if the pool is smaller. Also upgrade to SGLang builds including PRs #36988 and #38935 so over-length requests are rejected before handoff, run deferred KV release on both roles, and bound Mooncake transfer batches. _Operators tuning disaggregated inference setups can compare hardening checklists like this on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 1 discussion and 46 comments across hackernews (as of 2026-10-27). **TL;DR:** Most of the discussion is a dispute over whether the post's title unfairly implicates the GLM-5.3 model for what commenters agree was actually a Cloudflare configuration bug plus an SGLang bug; a smaller thread digs into how the context-window/KV-cache-pool mismatch went undetected in testing. **Sentiment:** 10% positive · 20% mixed · 70% skeptical **The case for** - Several commenters praised the transparency and speed of the incident response and remediation. - Some found the root-cause explanation (prefill pool smaller than advertised context window) clear and well laid out. **The pushback** - Many argued the title misleadingly implicates GLM-5.3 when the bug is actually in Cloudflare's configuration and SGLang. - Commenters questioned how Cloudflare advertised a 1M-token context window while only provisioning 681k tokens of KV cache and still passed testing. - Some see this as evidence of inadequate tenant isolation in shared GPU/model hosting generally, beyond this one incident. - A few recalled similar unexplained cross-session leakage experiences with other providers (e.g., ChatGPT), suggesting this class of bug may be more widespread. **By community** - hackernews (heated): A large share of the thread argues the post's title unfairly blames the GLM-5.3 model for what is really a Cloudflare/SGLang bug, while others defend the title as accurate and non-misleading. **Hottest debate:** Whether the post's title misleadingly implies the vulnerability lies in the GLM-5.3 model itself rather than Cloudflare's own infrastructure and SGLang. **Open questions** - Why didn't pre-launch testing catch the mismatch between the advertised 1M-token context window and the smaller KV cache pool? - Is this kind of cross-tenant KV cache leakage a broader risk across other inference providers that hasn't been publicly disclosed? **Highlights** > This doesn't really have anything to do with GLM-5.3. This is what happens when you share models between users and don't do your bookkeeping properly, the actual model doesn't matter. What it shows is that there isn't enough isolation between the user sessions and that memory can overlap when that should never be possible. This is an SGLANG and a Cloudflare bug long before it is a Z.AI model bug, I don't see what they could have done about it. Cloudflare should own it instead of naming other (imo innocent) parties in the title. > — [jacquesm on hackernews · 3 comments](https://news.ycombinator.com/item?id=50253748) > At no point do we claim that it is a bug with the model. We specifically outline how it was a mis-configuration on Cloudflare's part and a bug in SGLang. We name the model in the title because our customers need to know which of our endpoints was affected. In fact, we say "the GLM-5.3 Workers AI model", not "GLM-5.3". > — [kflansburg on hackernews · 4 comments](https://news.ycombinator.com/item?id=50253832) > "We deployed GLM 5.3 with a 1,048,576-token context window, but the KV cache pool on our prefill GPUs only had room for 681,280 tokens. We were aware of this mismatch before launch. " Did they advertise it as 1,048,576 token when they only had room for 681,280 tokens? > — [segmondy on hackernews · 3 comments](https://news.ycombinator.com/item?id=50252927) > Sounds like they are saying they tested with a long prompt and just tested for a failure, but the test likely would have seen that same KV cache leakage and just interpreted it as success. > — [Graziano\_M on hackernews · 1 comments](https://news.ycombinator.com/item?id=50253902) > Reminds me of a bug I found in a very popular browser testing service that spun up iOS vm's/devices for on-device testing. I copied the clipboard from the device, and my clipboard was full of another users credentials that they were obviously trying to paste into their own VM. Their devices were leaking clipboard data to each other. I reported the vulnerability and asked if there was any bounty program (I wasn't looking for vulnerabilities but didn't hurt to ask), to which they said no, but they could send me a signed letter thanking me of my contributions. I said, great, thanks - and never heard back. All pretty disappointing. > — [reassess\_blind on hackernews](https://news.ycombinator.com/item?id=50254223) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=50252159) · 131 points · 46 comments ## Similar posts on daily.dev - [Smaller, faster, safer: running Kimi and GLM at scale](https://daily.dev/posts/smaller-faster-safer-running-kimi-and-glm-at-scale-wsdraipr3) · Cloudflare · 0 upvotes · 0 comments - [Building the foundation for running extra-large language models](https://daily.dev/posts/building-the-foundation-for-running-extra-large-language-models-h2xp2rdat) · Cloudflare · 0 upvotes · 0 comments - [Cloudflare Workers AI: run LLMs without API keys](https://daily.dev/posts/cloudflare-workers-ai-run-llms-without-api-keys-sxc7e3lma) · Flavio Copes · 1 upvotes · 0 comments - [Prompt Caching is GA on DigitalOcean Serverless Inference: Up to 90% Off the Context Your Agents Resend](https://daily.dev/posts/prompt-caching-is-ga-on-digitalocean-serverless-inference-up-to-90-off-the-context-your-agents-res-wwec2jh2k) · DigitalOcean · 0 upvotes · 0 comments - [vLLM vs. SGLang: Choosing the right inference engine for your workload](https://daily.dev/posts/vllm-vs-sglang-choosing-the-right-inference-engine-for-your-workload-lirwkfyqx) · Red Hat Developer · 0 upvotes · 0 comments --- Tags: [#ai-inference](https://daily.dev/tags/ai-inference) [View this post on daily.dev](https://daily.dev/posts/cross-request-kv-cache-exposure-in-the-glm-5-3-workers-ai-model-and-our-remediation-with-sglang-qkref7qhu) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang","url":"https://daily.dev/posts/cross-request-kv-cache-exposure-in-the-glm-5-3-workers-ai-model-and-our-remediation-with-sglang-qkref7qhu","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/cross-request-kv-cache-exposure-in-the-glm-5-3-workers-ai-model-and-our-remediation-with-sglang-qkref7qhu"},"datePublished":"2026-10-26T22:05:17.934Z","dateModified":"2026-10-27T09:23:16.601Z","description":"A misconfigured KV cache pool in Cloudflare's GLM 5.3 Workers AI deployment let 155 responses contain data from other customers' requests between August 27 and...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/a02d9d4a05b65644c4a5b5e931cdc79f?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/a02d9d4a05b65644c4a5b5e931cdc79f?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Cloudflare","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Cloudflare","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/38522e1d11354cd6b7af66f9d4316735","url":"https://daily.dev/sources/cloudflare"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/cross-request-kv-cache-exposure-in-the-glm-5-3-workers-ai-model-and-our-remediation-with-sglang-qkref7qhu","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-inference","timeRequired":"PT19M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Cloudflare","item":"https://daily.dev/sources/cloudflare"},{"@type":"ListItem","position":3,"name":"Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/cross-request-kv-cache-exposure-in-the-glm-5-3-workers-ai-model-and-our-remediation-with-sglang-qkref7qhu#faq","mainEntity":[{"@type":"Question","name":"What caused the cross-request KV cache exposure in Cloudflare's GLM 5.3 Workers AI model?","acceptedAnswer":{"@type":"Answer","text":"A misconfigured KV cache pool combined with an SGLang bug. Cloudflare deployed GLM 5.3 with a 1,048,576-token context window but a prefill GPU pool sized for only 681,280 tokens. When prompts exceeded the pool, SGLang's rejection path replaced the prompt with a placeholder instead of aborting, so decode nodes read stale cache pages from earlier requests, exposing their data in 155 responses between August 27 and September 9, 2026. Teams running multi-tenant LLM inference can follow incident writeups like this on daily.dev to catch similar KV cache risks early."}},{"@type":"Question","name":"Which SGLang version fixes the KV cache cross-request exposure bug found in disaggregated prefill/decode mode?","acceptedAnswer":{"@type":"Answer","text":"The fix landed in SGLang v0.5.20 with follow-ups in v0.5.21 and v0.5.22, which also added per-transfer checksums, abort-ACK preservation, and deferred KV release enabled by default. Operators running disaggregated SGLang are advised to update to v0.5.22 or later, or backport the relevant patches, since the bug only affects deployments where the KV pool is smaller than the model's context length. Developers upgrading SGLang deployments can track fixes like this on daily.dev before shipping to production."}},{"@type":"Question","name":"How can I prevent SGLang from exposing another request's KV cache data when a prompt exceeds the context window?","acceptedAnswer":{"@type":"Answer","text":"Measure the KV pool size against the context length on every prefill and decode node from the engine's startup log after any change to weights, quantization, parallelism, or memory fraction, and lower context length if the pool is smaller. Also upgrade to SGLang builds including PRs #36988 and #38935 so over-length requests are rejected before handoff, run deferred KV release on both roles, and bound Mooncake transfer batches. Operators tuning disaggregated inference setups can compare hardening checklists like this on daily.dev."}}]} ```