Cloudflare Built the Foundation for Extra-Large LLMs — and Made Kimi K2.5 3x Faster!

Hey everyone, it's Shiichan! Today I found a post where Cloudflare pulls back the curtain on how they run gigantic LLMs fast — I can't stop being excited!

Cloudflare Blog blog.cloudflare.com

What was announced?

Cloudflare's Blog announced that they built a custom technology stack to run extra-large language models (LLMs) efficiently on their own infrastructure. It's tuned especially for agentic use cases, and they made the Kimi K2.5 they covered before a full 3x faster!

Kimi K2.5 has over 1 trillion parameters, and its weights alone are about 560 GB — an enormous model. Running it needs at least 8 H100 GPUs (80 GB of VRAM each), and even then you're left with 30 GB+ per GPU for the KV cache. For comparison, Llama 4 Scout runs on just 2 H200 GPUs and can handle 1.2M+ tokens.

Why it matters

As models keep getting bigger, they simply won't fit on a single GPU anymore. And in the age of agents, speed really matters for tool calls and long contexts. It's not just about running the model — it's about keeping latency low and stable while holding down cost, and that foundation is what this post is all about.

What changes

The clearest win is speed. The time per output token (p90) dropped from roughly 100 ms to 20-30 ms — a 3x improvement. Time to First Token (TTFT) also became much less variable, so it feels snappier.

On top of that, the prompt cache hit ratio rose from 60% to 80% even during peak times. Cached tokens get discounted pricing, so it's gentle on your wallet too.

Dive Deep

Here's a quick tour of the tricks Cloudflare is using.

Prefill-Decode disaggregation Inference splits into two stages: prefill, which reads the input and is compute-bound, and decode, which generates output one token at a time and is memory-bound. They run these on separate servers, each tuned independently, and add token-aware load balancing to keep latency stable.

Prompt caching With prompt caching, an x-session-affinity header routes requests to the region where the input tensors are cached. Hit the cache and you get cheaper, faster responses.

Sharing the KV cache Using Moonshot AI's Mooncake (Transfer Engine and Store), they share the KV cache across multiple GPUs. RDMA transports like NVLink and NVMe over Fabric move memory without going through the CPU, extending the cache beyond GPU VRAM onto NVMe storage.

Speculative decoding A small draft model (NVIDIA's EAGLE-3) proposes candidate tokens first, and the target model validates them together in a single forward pass. It's especially effective for tool calls and structured outputs.

Infire The finishing touch is Infire, their own Rust-based inference engine. It combines pipeline-parallel and tensor-parallel modes with expert-parallelism, keeps memory overhead very low, and achieves cold-start times under 20 seconds even for the largest models. On unconstrained systems, they claim 20% higher throughput in tokens/second.

Wrap-up

• Cloudflare revealed the stack it uses to run extra-large LLMs fast for agents, making Kimi K2.5 3x faster
• It combines PD disaggregation, prompt caching, KV-cache sharing, speculative decoding, and its own Infire engine
• Time per token (p90) went from ~100 ms to 20-30 ms, and cache hit ratio improved from 60% to 80%
• Kimi K2.5 (~560 GB) runs on as few as 8 H100 GPUs

This one really lands for anyone who wants to run agents in production, or engineers who love digging into large-model inference optimization!