# Unweight: Cloudflare shrinks an LLM by 22% with zero quality loss!

Hey everyone, it's Shii-chan! Today I got excited about a slightly greedy dream: making a model smaller while keeping its accuracy exactly the same.

## What was announced?

The Cloudflare Blog announced **Unweight**, a new compression system that can make LLM weights up to 15-22% smaller. The best part: the outputs stay 100% bit-exact lossless, so the model behaves exactly the same and only the size shrinks, and it needs no special hardware.

On Llama-3.1-8B, their initial results show about 30% compression of the MLP (Multi-Layer Perceptron) weights alone, which adds up to a 15-22% reduction in overall model size and roughly 3 GB of VRAM savings.

Cloudflare also published a detailed [technical paper](https://research.cloudflare.com/nikulin2026) and open sourced the [GPU kernels](https://github.com/cloudflareresearch/unweight-kernels).

## Why it matters

Every time an LLM generates a single token, it has to read every weight out of GPU memory, and that's the bottleneck. On the NVIDIA H100 GPUs Cloudflare uses in its datacenters, the tensor cores can crunch data about 600 times faster than memory can deliver it.

So you get stuck on memory bandwidth, not compute. That means the smaller the weights, the fewer bytes you read, the faster you go, and the more headroom your GPU gets.

## What changes

When the weights get smaller, you can pack more models onto the same GPU. For someone running inference all over the world, like Cloudflare's [Workers AI](https://www.cloudflare.com/developer-platform/products/workers-ai/), saving memory per GPU translates directly into "running more models, in more places, more cheaply."

And because it's lossless, you don't have to worry about the accuracy drop that comes with quantization.

## Dive Deep

The trick is pretty simple and clever. A BF16 (16-bit brain floating point) weight is made of a 1-bit sign, an 8-bit exponent, and a 7-bit mantissa. The sign and mantissa scatter almost randomly, so they're hard to compress. But the exponent is a different story.

Research shows that out of 256 possible exponent values, the top 16 alone cover over 99% of the weights in a typical layer. Information theory says you only need about 2.6 bits to represent that skew, far less than the 8 bits that are allocated.

So Unweight leaves the sign and mantissa alone and compresses only the exponent byte with [Huffman coding](https://en.wikipedia.org/wiki/Huffman_coding), the classic technique that gives short codes to common values and long codes to rare ones.

But the real challenge isn't the compression itself, it's decompressing fast enough that it doesn't slow inference down. Unweight's key move is decompressing the weights inside the GPU's fast on-chip shared memory and feeding them straight to the tensor cores, without a round-trip through slower main memory. The decompression is integrated with Cloudflare's Rust-based [inference engine](https://blog.cloudflare.com/cloudflares-most-efficient-ai-inference-engine/) (Infire).

Of course it isn't free. Right now there's a 30-40% throughput overhead end-to-end, measured on an H100 SXM5: largest at batch size 1 (about 41%) and narrowing to about 30% at batch 1024. There are three main sources (small-batch fixed costs, redundant weight-tile reconstruction, and the excluded down projection), and they're still being optimized. Extrapolated to Llama 70B, this could save roughly 18-28 GB depending on configuration.

## Wrap-up

- Unweight is a bit-exact lossless compression system that makes LLM weights up to 15-22% smaller
- It Huffman-codes only the exponent bytes of BF16; the top 16 exponents cover over 99%, needing only about 2.6 bits
- It decompresses in on-chip shared memory and feeds the tensor cores directly, with no special hardware required
- The cost is a 30-40% throughput overhead (about 41% at batch 1, about 30% at batch 1024), still being improved
- A technical paper is published and the GPU kernels are open source

If you wrestle with GPU memory on an inference platform, or you just love low-level GPU optimization, this one's a real treat!