Transformers Come to Logpush: Filter, Redact, and Reshape Logs with SQL Before Export

Hi everyone, it's Shii-chan!

Cloudflare Blog blog.cloudflare.com

You can now shape your logs with SQL before they're delivered, and the architecture story is really fun!

What was announced?

The Cloudflare Blog introduced Transformers for Logpush. They let you filter, redact, and reshape logs with SQL before export. Logpush applies the query to each batch, so every destination gets data in the format it needs.

• Records that don't match the WHERE clause are dropped
• The SELECT output becomes the exported record
• Output is NDJSON

The story so far

To transform logs, you needed a separate processing service in the middle. Transformers remove the need for that, and filtering early can reduce storage and ingestion costs!

What changes

Here's what you can write in SQL (simplified from the post's examples).

SELECT ClientIP, EdgeResponseStatus, RayID
FROM http_requests
WHERE EdgeResponseStatus >= 500
  AND ClientRequestPath NOT LIKE '/healthz%'

This keeps only server errors and drops health-check traffic.

You can also redact part of an IP and add a provider field.

SELECT
  regexp_replace(ClientIP, '\.\d+\.\d+$', '.XXX.XXX') AS redacted_ip,
  'cloudflare' AS provider,
  EdgeResponseStatus AS status
FROM http_requests

The post also has a longer example that converts timestamps to Unix epoch, uppercases the HTTP method, nests host and path into a request object, and adds an is_server_error boolean.

Dive Deep

Usage and limits

• Create, preview, and manage Transformers in Transformer Studio in the dashboard, or through the API
• Each Transformer targets one Logpush dataset, selected by the table name in FROM
• Attach a saved Transformer to a compatible NDJSON Logpush job using the same dataset
• Each update creates a new version
• Records are reshaped, but serialization formats are not converted
• If a transform fails, untransformed data is not sent; the delivery offset doesn't advance and it retries
• Retries back off progressively, errors show on the job, and jobs failing for an extended period are disabled automatically

The post gives no numeric limits on query size or row counts, no per-plan quotas, and no pricing; see the Logpush pricing docs. Transformers are generally available on Free, Pro, Business, and Enterprise plans.

Under the hood

This is my favorite part!

• The delivery service is Bufstream, running across hundreds of Kubernetes pods and reading from Buftee, Cloudflare's internal log broker
• An early prototype ran WebAssembly Transformers in Workers, but network calls could add timeouts, retries, and latency, so execution moved in-process
• The engine is Apache DataFusion, a Rust SQL query engine built on Apache Arrow
• Bufstream decompresses each batch, parses records, builds typed Arrow batches, and passes them to Rust over Arrow IPC through a C ABI called via cgo
• For memory ownership, Go owns the input buffer only during the call, and Rust owns the output buffer; Go copies it and then releases the Rust allocation
• Queries are stored in R2 and cached; each query is compiled into a logical plan once per time slice and reused on every batch
• Results become NDJSON, are gzip-compressed, and uploaded, with upload concurrency adjusted to destination limits

In one benchmark, a 5,000-row batch took about 3 ms from a single Go goroutine, roughly 1.6 million rows per second.

For monitoring, the Transformer Health view in the Logpush Health tab shows successful and failed transforms, data volume before and after, and transform duration.

Wrap-up

• Transformers filter, redact, and reshape logs with SQL before export
• Create them in Transformer Studio or the API and attach them to NDJSON jobs on the same dataset
• On failure they send nothing untransformed and retry (fail closed)
• DataFusion and Arrow run in-process for speed
• Generally available on all plans

If your logs carry sensitive fields and every destination wants a different shape, this one's for you!