R2 Data Catalog Now Sweeps Away Unreferenced Data Files On Its Own!
Hi everyone, Shii-chan here! Today I found a small but genuinely nice update that makes storage cleanup a little smarter, so let me tell you about it.
Cloudflare Changelog
developers.cloudflare.com
What was announced?
This comes from Cloudflare's Changelog. R2 Data Catalog, the managed Apache Iceberg catalog built into R2, now removes unreferenced data files during automatic snapshot expiration. This reduces storage costs and means you no longer need manual maintenance jobs to reclaim space from deleted data.
The story so far
Until now, snapshot expiration only cleaned up Iceberg metadata files like manifests and manifest lists. The actual data files that were no longer referenced by active snapshots stayed in R2 storage until you ran remove_orphan_files or expire_snapshots yourself through an engine like Spark. That meant extra operational overhead, and stale data files kept eating up storage.
What changes
From now on, both metadata and data files get cleaned up automatically. When a snapshot is expired, any data files that are no longer referenced by retained snapshots are removed from R2 storage. Even if you leave things alone, extra files won't pile up. Nice, right?
Dive Deep
To enable snapshot expiration at the catalog level, you use a Wrangler command like this:
# Enable catalog-level snapshot expiration
npx wrangler r2 bucket catalog snapshot-expiration enable my-bucket \
--older-than-days 7 \
--retain-last 10
--older-than-days sets how old a snapshot has to be to qualify, and --retain-last sets how many to keep. For more configuration details and other automatic maintenance operations, check out the table maintenance documentation.
Wrap-up
- R2 Data Catalog's snapshot expiration now automatically deletes unreferenced data files too
- Previously it only handled Iceberg metadata, and the data itself needed manual cleanup via Spark and friends
- Enable it with the
wrangler r2 bucket catalog snapshot-expiration enablecommand - A welcome improvement for anyone running Iceberg tables on R2 who wants to cut storage costs and cleanup chores!