Kimi and GLM's Context Window Just Doubled — Cloudflare's Shrinking Trick Is Wild

Hey everyone, it's me! Today I found a genuinely technical post from Cloudflare's engineering blog about how to run AI models "lighter, faster, and safer."

Cloudflare Blog blog.cloudflare.com

What was announced?

Cloudflare's official blog walked through the optimizations it uses to serve huge open-weight models like Moonshot AI's Kimi and Z.ai's GLM at scale on its own infrastructure. These models are large Mixture of Experts (MoE) architectures with long context windows, which means the real bottleneck isn't compute — it's GPU memory.

Cloudflare's approach breaks down into three main levers.

• KV cache quantization (compressing from 16-bit down to 8-bit to save memory)
• Model weight compression (shrinking from 8-bit down to 4-bit to reduce checkpoint size)
• KV cache integrity checks (a safety measure that prevents corrupted data from being returned even under heavy concurrent load)

The story so far

Until now, the KV cache was typically kept in BF16 (16-bit). For Kimi K2.6, that meant the amount of context you could hold in memory topped out at roughly 686,000 tokens. For long-context models, the KV cache tends to fill up GPU memory before the model weights even get a chance to.

So Cloudflare converted the KV cache to FP8 (8-bit floating point, e4m3 format). That alone roughly halves its size, pushing the amount of context that fits in memory up to about 1.37 million tokens — twice as much. What's neat is that memory pressure really only bites during decode, so Cloudflare keeps the compute-heavy prefill phase in BF16 for accuracy, applying FP8 only where it matters.

Single-request throughput dips slightly, from 137 tokens/second on BF16 to 125 tokens/second on FP8, but the real payoff shows up at higher concurrency. At 64 concurrent requests, BF16 already runs out of memory by the time you hit 32 requests, while FP8 keeps humming along at 2,192 tokens/second. Overall, that works out to a 41% throughput improvement at roughly 30% lower cost per token.

As for accuracy, the benchmarks barely move: GSM8K goes from 94.24% to 94.09%, ARC-Challenge from 66.72% to 67.49%, and MMLU from 89.11% to 89.04%.

What changes

Thanks to this optimization, Cloudflare's infrastructure can handle far more concurrent requests on the same GPUs when serving huge models like Kimi and GLM. For developers running long-context inference, that should translate into more stable latency and better cost efficiency.

The integrity checks described below also add a safety net: even under heavy parallel load, the system is far less likely to silently return corrupted data.

Dive Deep

Beyond the KV cache, Cloudflare also tackled compressing the model weights themselves for GLM 5.2. Converting the weights from FP8 (8-bit) to INT4 (4-bit integers) produced these results.

• Checkpoint size: 705 GB down to 421 GB (about a 40% reduction)
• Per-GPU memory under 8-way tensor parallelism: 88 GB down to 52 GB
• The freed-up memory lets the same hardware hold roughly 1.18 million more tokens of KV cache

Decode throughput improved too — a single request goes from 60 tokens/second to 92 tokens/second, a 55% gain. At 64 concurrent requests, that gap narrows to 16% (1,672 versus 1,933 tokens/second).

Prefill (the step that processes the entire prompt up front) is a different story: INT4 actually slows it down, since INT4 weights need to be decompressed before computation. FP8 hits 10,160 tokens/second at prefill versus just 8,660 for INT4. Cloudflare's answer is a disaggregated architecture — FP8 for prefill, INT4 for decode — getting the best of both. Accuracy holds up too, staying within 0.8 points of FP8 across every benchmark (MMLU: 86.60% for FP8 versus 86.54% for INT4).

The other pillar is KV cache integrity checking. Once hundreds of concurrent requests are sharing physical cache pages, even extremely rare bookkeeping bugs start showing up regularly at that scale. Cloudflare tags every physical cache page with a value that changes whenever the page gets reallocated, and the server records which pages and tags each request is supposed to use. If a tag doesn't match, the request gets aborted instead of returning corrupted data.

The overhead for this check stays under 1% for both throughput and p95 latency across concurrency levels from 1 to 8. It's also implemented as a separate batch check rather than being fused into the attention kernel, specifically to avoid race conditions between GPU threads.

These benchmarks were all run using SGLang, an open-source inference engine, and Cloudflare says it upstreams its patches back to the project. Looking ahead, the team mentioned a few directions.

• Rolling out FP8 KV caches fleet-wide
• Validating NVFP4-format weights on Blackwell GPUs
• Further reducing the overhead of the integrity checks

Wrap-up

• Huge MoE models like Kimi and GLM tend to be GPU-memory bound rather than compute bound
• Switching the KV cache from BF16 to FP8 (e4m3) roughly doubled the context that fits in memory (686,000 to 1.37 million tokens)
• Compressing GLM 5.2's weights from FP8 to INT4 cut checkpoint size by 40% while keeping accuracy within 0.8 points
• Using FP8 for prefill and INT4 for decode balances speed and accuracy
• New KV cache integrity checks abort requests before they can return corrupted data

If you run your own LLM inference stack or spend your days wrestling with GPU memory, this one's for you.