Pipelines and R2 Data Catalog Can Now Be Written in Terraform!

Hi there, I'm Shiichan! Today I found an update that infrastructure fans are going to love, so let me tell you about it.

Cloudflare Changelog developers.cloudflare.com

What was announced?

The Cloudflare Changelog announced that Pipelines and R2 Data Catalog can now be managed through Terraform. From the Cloudflare Terraform provider v5.19.0, four new resources have been added.

Just to recap, Pipelines ingests streaming data via Workers or HTTP endpoints, transforms it with SQL, and writes it to R2 as Apache Iceberg tables. R2 Data Catalog manages those Iceberg tables, compaction, and compatibility with query engines.

The story so far

Until now, you had to configure Pipelines and R2 Data Catalog individually through the dashboard or API. But if you want to run them as a team or reproduce the same setup, you really want to manage them as Infrastructure-as-Code. With this update, you can now define a complete data pipeline as Terraform code.

What changes

Here are the four resources that were added:

• cloudflare_r2_data_catalog — enable Data Catalog on an R2 bucket
• cloudflare_pipeline_stream — create a stream that receives events via HTTP or Worker bindings
• cloudflare_pipeline_sink — create a sink that writes to R2 Data Catalog or R2
• cloudflare_pipeline — create a pipeline with SQL connecting a stream to a sink

Receive with a stream, transform with SQL, write out to a sink. Being able to declare this whole flow as code makes reproducing environments, reviewing changes, and version control so much easier.

Dive Deep

The detailed arguments for each resource are documented in the Terraform registry. Check out cloudflare_pipeline, cloudflare_pipeline_stream, cloudflare_pipeline_sink, and cloudflare_r2_data_catalog.

The overall workflow is also written up in the Pipelines Terraform documentation, so that's a great place to start when you actually build one.

Wrap-up

• Cloudflare Terraform provider v5.19.0 lets you manage Pipelines and R2 Data Catalog as IaC
• The added resources are cloudflare_r2_data_catalog, cloudflare_pipeline_stream, cloudflare_pipeline_sink, and cloudflare_pipeline
• You can define a whole data pipeline as code, from stream to SQL transform to writing out to a sink

This update is perfect for anyone already using Pipelines or R2 Data Catalog who wants to manage their setup as code!