--- title: Smaller, faster, safer: running Kimi and GLM at scale description: Cloudflare's Workers AI team details three techniques used to serve large mixture-of-experts models (Kimi K2.6 and GLM 5.2) efficiently at scale. First, KV... canonical: https://daily.dev/posts/smaller-faster-safer-running-kimi-and-glm-at-scale-wsdraipr3 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Smaller, faster, safer: running Kimi and GLM at scale | daily.dev og:description: Cloudflare's Workers AI team details three techniques used to serve large mixture-of-experts models (Kimi K2.6 and GLM 5.2) efficiently at scale. First, KV... og:url: https://daily.dev/posts/smaller-faster-safer-running-kimi-and-glm-at-scale-wsdraipr3 og:image: https://api.daily.dev/og/posts/wSdRaIPR3.png og:image:alt: Smaller, faster, safer: running Kimi and GLM at scale og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Smaller, faster, safer: running Kimi and GLM at scale **[Cloudflare](https://daily.dev/sources/cloudflare)** · 7 min read · 0 upvotes · 0 comments ## Summary Cloudflare's Workers AI team details three techniques used to serve large mixture-of-experts models (Kimi K2.6 and GLM 5.2) efficiently at scale. First, KV cache quantization from BF16 to FP8 doubles the context capacity and raises peak throughput by ~41% at 30% lower cost per token, with no measurable accuracy loss. Second, compressing GLM weights from FP8 to INT4 shrinks the checkpoint by 40% and boosts decode speed by 16–55% depending on concurrency, while prefill still runs FP8 to avoid the dequantization penalty. Third, a lightweight KV cache integrity check tags every physical cache page and validates mappings before decode reads, catching any bookkeeping errors from paged attention and cache reuse — at under 1% overhead on throughput and tail latency. All three optimizations are enabled selectively per deployment phase using a disaggregated prefill/decode architecture backed by SGLang. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Community take How the wider developer community reacted, aggregated from 1 discussion and 35 comments across hackernews (as of 2026-08-03). **TL;DR:** The HN thread is dominated by complaints that the blog post appears AI-generated, with only a minority engaging with the actual technical content; those who do raise legitimate concerns about the rigor of the accuracy benchmarks and the lack of public pricing. **Sentiment:** 15% positive · 30% mixed · 55% skeptical **The case for** - KV cache quantization transparency is appreciated, as some providers do this silently without disclosure. - The KV cache integrity check mechanism is noted as a genuinely neat engineering detail. - vLLM's independent study corroborates that FP8 KV cache quantization delivers meaningful gains with negligible accuracy loss, including on coding benchmarks. **The pushback** - The blog post is widely suspected to be AI-generated, with commenters citing telltale prose patterns and calling it 'word-expanded slop'. - The accuracy evaluation suite is criticized for lacking coding benchmarks, which matters because tool-call errors compound in long-running agentic tasks. - The claim that FP8 KV quantization doesn't change model answers is seen as too strong — commenters suggest KL divergence or token-probability comparisons would be more rigorous. - Cloudflare hides its pricing behind a login, making it impossible to evaluate the service without signing up. - Cloudflare is noted to not yet support Kimi K3, making the offering less compelling given how cheap K2.7 Code already is elsewhere. - Commenters lament that Cloudflare's engineering blog, once considered high quality, has declined into verbose, low-information posts. **By community** - hackernews (skeptical): The thread is dominated by AI-slop complaints and meta-discussion about detecting AI-generated content, with only a handful of comments engaging the technical substance — and those raise concerns about benchmark rigor and missing pricing. **Hottest debate:** Whether the blog post is AI-generated and whether that makes it unworthy of engagement, versus accepting AI-polished writing if the underlying ideas are sound. **Open questions** - Why does Cloudflare hide pricing behind a dashboard login rather than publishing it publicly? - How does FP8 KV cache quantization affect token probability distributions (e.g., KL divergence) rather than just downstream benchmark scores? - Are there plans to support newer models like Kimi K3? - Which model families are most sensitive to KV cache quantization, and has Cloudflare tested beyond Kimi K2.6? **Highlights** > Nice to see a provider being transparent about KV cache quantisation. I've been suspecting that some providers do this silently whilst heavily promoting their unquantised weights, even though KV quantisation can degrade quality more than weight quantisation. However, I wish their testing were more detailed. Firstly, some model families are more sensitive to KV quantisation than others (only Kimi K2.6 was tested). Secondly, the evaluation suite they use to claim that FP8 KV quantisation is indistinguishable is noticeably lacking coding benchmarks; in long-running tasks, minor tool call errors compound over time. > — [scrlk on hackernews · 2 comments](https://news.ycombinator.com/item?id=49160216) > They made an extremely strong claim: > None of this would matter if it changed the model's answers If they want to assert that the answers don’t change, then perhaps they should calculate the statistical distance between the token probability outputs or something to that effect. I doubt the results would indicate that the answers don’t change by any reasonable interpretation. Maybe the results are still good enough. > — [amluto on hackernews · 1 comments](https://news.ycombinator.com/item?id=49161181) > I don't disagree, but at some point in the last year they ended up severely word-expanded. So in a revealed sense, they are no longer meant for human consumption except for those who don't significantly value their own time. There is very little information in the post that an agent can't pull for you: * they use quantized models * they quantize KV cache * they have a cache tagging mechanism to prevent cache misuse (neat) The agent can extract numbers without filler prose as well. > — [arjie on hackernews](https://news.ycombinator.com/item?id=49161214) > Cloudflare blogs are not meant to be human-read, AFAIK. They're raw material meant to be fed into an agent to be filtered down. I rarely read the contents because they are usually word-expanded to a greater degree than an article from The Atlantic. > — [arjie on hackernews · 1 comments](https://news.ycombinator.com/item?id=49160512) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=49158581) · 122 points · 35 comments ## Similar posts on daily.dev - [Building the foundation for running extra-large language models](https://daily.dev/posts/building-the-foundation-for-running-extra-large-language-models-h2xp2rdat) · Cloudflare · 0 upvotes · 0 comments - [Cloudflare Builds High-Performance Infrastructure for Running LLMs](https://daily.dev/posts/cloudflare-builds-high-performance-infrastructure-for-running-llms-gig99bgnb) · InfoQ · 1 upvotes · 0 comments - [Powering the agents: Workers AI now runs large models, starting with Kimi K2.5](https://daily.dev/posts/powering-the-agents-workers-ai-now-runs-large-models-starting-with-kimi-k2-5-i4gjlknn5) · Cloudflare · 1 upvotes · 0 comments --- Tags: [#data-science](https://daily.dev/tags/data-science), [#ai-inference](https://daily.dev/tags/ai-inference) [View this post on daily.dev](https://daily.dev/posts/smaller-faster-safer-running-kimi-and-glm-at-scale-wsdraipr3) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Smaller, faster, safer: running Kimi and GLM at scale","url":"https://daily.dev/posts/smaller-faster-safer-running-kimi-and-glm-at-scale-wsdraipr3","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/smaller-faster-safer-running-kimi-and-glm-at-scale-wsdraipr3"},"datePublished":"2026-08-03T13:03:31.059Z","dateModified":"2026-08-03T23:57:15.488Z","description":"Cloudflare's Workers AI team details three techniques used to serve large mixture-of-experts models (Kimi K2.6 and GLM 5.2) efficiently at scale. First, KV...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/1af36f4424e9b68b2300a16de2ec971f?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/1af36f4424e9b68b2300a16de2ec971f?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Cloudflare","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Cloudflare","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/38522e1d11354cd6b7af66f9d4316735","url":"https://daily.dev/sources/cloudflare"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/smaller-faster-safer-running-kimi-and-glm-at-scale-wsdraipr3","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-science,ai-inference","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Cloudflare","item":"https://daily.dev/sources/cloudflare"},{"@type":"ListItem","position":3,"name":"Smaller, faster, safer: running Kimi and GLM at scale"}]} ```