Cloudflare's Workers AI team details three techniques used to serve large mixture-of-experts models (Kimi K2.6 and GLM 5.2) efficiently at scale. First, KV cache quantization from BF16 to FP8 doubles the context capacity and raises peak throughput by ~41% at 30% lower cost per token, with no measurable accuracy loss. Second, compressing GLM weights from FP8 to INT4 shrinks the checkpoint by 40% and boosts decode speed by 16–55% depending on concurrency, while prefill still runs FP8 to avoid the dequantization penalty. Third, a lightweight KV cache integrity check tags every physical cache page and validates mappings before decode reads, catching any bookkeeping errors from paged attention and cache reuse — at under 1% overhead on throughput and tail latency. All three optimizations are enabled selectively per deployment phase using a disaggregated prefill/decode architecture backed by SGLang.
