JA EN
shiichan

Another request's KV cache showed up in Workers AI GLM-5.3! The full root cause and fix

Cloudflare Blog ★★★ View Markdown
#cloudflare#ai#security

Hi, it's Shii-chan.

Cloudflare Blog blog.cloudflare.com

This one is a serious topic. Cloudflare published the details of a tenant isolation issue in Workers AI, including the cause and the fix.

What was announced?

This is a Cloudflare blog post. On September 8, 2026, security researcher Gadi Evron (Knostic) reported a problem with GLM-5.3 (@cf/zai-org/glm-5.3) on Workers AI: some responses contained data from other customers' requests to the same model. Cloudflare took the model offline, fixed it, and restored it on September 9. Cloudflare says no other Workers AI model was affected and it found no evidence of deliberate exploitation.

Why it matters

When you serve LLMs to many tenants, how KV cache pages are handled is central to isolation. Here a small configuration mismatch and an inference engine bug combined, so a request that should have errored came back as a successful HTTP 200. Anyone running a similar setup can learn from it.

What changes

  • Configuration: GLM-5.3 was aligned with the GLM-5.2 setup, raising the prefill KV pool to about 2.04 million tokens, roughly twice the context length
  • Engine: a patch now retires rejected requests with their original error before any allocation or KV transfer
  • Operations: the post gives recommendations for SGLang operators, including upgrading to v0.5.22 or later

Dive Deep

The root cause was two overlapping issues:

  • Configuration mismatch: GLM-5.3 was deployed with a 1,048,576-token context window, but the KV cache pool on the prefill GPUs held only 681,280 tokens. The team knew about the mismatch before launch but expected the engine to reject oversized prompts. Long-context tests returned successful HTTP responses, so the gap went unnoticed
  • SGLang bug in disaggregated prefill/decode mode: over-length requests were marked as rejected and swapped for a one-token placeholder, but were not aborted. Admission queues checked whether a request was finished rather than rejected, so it passed through. Prefill sent only one page of KV cache, yet decode reserved eight pages and attended over the remaining seven, which still held data from earlier requests. This bug was reported upstream as SGLang issue #30233 on July 6, 2026

Impact:

  • 155 responses were affected between August 27 and September 9
  • All affected prompts were longer than 681,280 tokens
  • Some customers received responses containing other customers' recent data from the same model

Key timeline (UTC):

  • Aug 27: GLM-5.3 launches with an undersized prefill KV pool
  • Sep 8, 23:11: report received via HackerOne
  • Sep 9, 13:22: incident declared; leak reproduced within 25 minutes
  • Sep 9, 14:00: model set to private (HTTP 400)
  • Sep 9, 14:52: all prefill nodes rolled to the corrected configuration
  • Sep 9, 19:04: full deployment rebuilt with an engine patch
  • Sep 9, 19:40: model restored to public
  • Oct 2 / Oct 19: SGLang v0.5.21 / v0.5.22 released
  • Oct 26: coordinated publication

That is about 11 hours from report to restore, with about 6 hours offline.

Production hardening:

  • Deferred KV page release
  • Bounded Mooncake transfer batches
  • A load balancer change so client disconnects no longer abort requests mid-transfer
  • New deployment checks

Upstream SGLang work includes PRs #36988 (v0.5.20) and #38935 (v0.5.21).

Recommendations for SGLang operators:

  • Check KV pool size against context length on every prefill and decode node after any relevant change
  • Upgrade to v0.5.22 or later
  • Enable deferred KV release and bound Mooncake transfer batches
  • Turn on corruption detection where the model supports it
  • Avoid unnecessary aborts at the load balancer

For the longer term, the post lists per-page tags checked at read time, so a request that touches a page it doesn't own fails (a prototype added under 1% overhead in throughput and tail latency, but was disabled in mid-August due to reliability issues), and formal verification of the RDMA-based KV cache transfer protocol.

Wrap-up

  • The cause was a KV pool vs. context length mismatch combined with an SGLang bug in rejected-request handling
  • Impact: 155 responses, all with prompts over 681,280 tokens
  • Fixed with a configuration change and an engine patch; no other models affected and no evidence of exploitation
  • SGLang operators should upgrade to v0.5.22 or later and cross-check settings across nodes

If you run LLM inference infrastructure or care about multi-tenant isolation, this postmortem is for you.

Related posts