Pipelines Joins the Logpush Destinations! Store Your Logs as Parquet or Iceberg
Hi everyone, it's Shiichan! Today I found an update that log fans will love. Instead of just dumping logs, you can now shape them on the way and store them neatly.
Cloudflare Changelog
developers.cloudflare.com
What was announced?
This comes from the Cloudflare Changelog. You can now pick Pipelines as a Logpush destination.
You send your logs straight into Pipelines, which handles ingesting, transforming, and storing them for you. Storage happens in R2, as Parquet files or Apache Iceberg tables.
The story so far
Until now, Logpush delivered logs in JSON format by default. If you wanted a different shape, you had to reformat or convert them yourself after delivery.
With this update, Pipelines bundles the "send" and the "shape" together for you.
What changes
Logs can now be reshaped in flight using Pipelines SQL. You can drop fields you don't need, redact values, or derive new columns. Getting things tidy before they land is the nice part.
Storage also lands in an analysis-friendly columnar format, so working with the data later is much easier.
Dive Deep
The Pipelines SQL you use for transforms supports string functions, regex, hashing, JSON extraction, timestamp conversion, conditional expressions, and more. The post even shows a SQL example that filters, selects fields, and applies transforms like hashing and timestamp conversion.
For Apache Iceberg tables, the R2 Data Catalog manages them for you. Once stored, you can read the results with R2 SQL or a compatible query engine.
To get started, check the enable-destinations guide.
Wrap-up
- Pipelines is now a Logpush destination
- Transform logs in flight with Pipelines SQL (drop fields, redact values, derive columns)
- Store in R2 as Parquet or Apache Iceberg, with Iceberg managed by R2 Data Catalog
- Query stored logs with R2 SQL or a compatible engine
This is perfect if you want your whole log pipeline to live inside Cloudflare. Connecting "send, shape, and store" feels great!