JA EN
shiichan

A Billion Events a Second: Cloudflare's Town Lake and Skipper Reshaped Its Internal Data

Cloudflare Blog ★★★ View Markdown
#cloudflare#ai#infrastructure

Hey everyone, it's Shiichan! Today's a treat for data lovers. This is the story of how a company that pushes over a billion events per second made all that data something anyone could actually reach.

Cloudflare Blog blog.cloudflare.com

What was announced?

On the Cloudflare Blog, the team introduced two tools they built for internal use. One is Town Lake, a unified data analytics platform, and the other is Skipper, an AI data agent that runs on top of it.

Town Lake is "a single SQL interface to everything Cloudflare knows," and Skipper is the piece that lets anyone ask a question in plain English and get a correct, auditable answer back in seconds. The name comes from a lake in Austin, Texas.

The story so far

Cloudflare's network spans 330+ cities across 120+ countries, and its data lived all over the place: production databases, ClickHouse, Kafka, Google Cloud buckets, BigQuery, and a long tail of pipelines.

Even a simple question like "how many domains that signed up today are in the Top 100 by traffic?" meant you had to know which system to ask, what credentials to use, what query language to write, and whether the data was sampled, fresh, or stale.

On top of that, the analytics pipeline downsamples to handle 700M+ events per second. Great for dashboards, wrong for things like billing where accuracy is everything. Parts of the internal reporting stack even depended on external vendors and another cloud.

What changes

Now anyone with the right permissions and a need to know can get answers to company questions in one place.

You can also pick the data that fits the job: fresh, unsampled data for billing and security investigations, and fast, downsampled data for dashboards and exploration. PII is detected automatically, sensitive tables are locked by default, and every access is audited with time-bound grants. Best of all, it's built entirely on Cloudflare's own products (R2, Workers, Access, Workflows).

Dive Deep

At its core, Town Lake is a data lakehouse: a query engine reading object storage, with a metadata layer that makes it behave like a database.

  • The query engine is Apache Trino. One SQL statement can join a Postgres table, a ClickHouse table, and an Iceberg table on R2 without materializing intermediate results.
  • Storage is R2 Data Catalog (managed Apache Iceberg), with schema evolution, time travel, and compaction that makes older data cheaper while keeping it queryable.
  • Metadata lives in DataHub, access control in Lifeguard (rules stored in D1), and PII detection in Skimmer (two-pass column classification with Workers AI).
  • ELT is handled by Transformer, and ingestion by an orchestrator running on Kubernetes.

Governance is "default-closed": new tables are discoverable but not queryable until reviewed. When you hit one, the error isn't "permission denied" but "this table needs review, click here to request one."

The fun Skipper detail is Code Mode. Instead of exposing 30 separate tools, they expose just search and execute, and let the model write JavaScript that calls the whole toolset at once, so a five-step workflow isn't five round-trips.

const datasets = await skipper.search_datasets({ query: "billing product revenue" })
const queryId = await skipper.start_query({ sql: "SELECT ..." })
const results = await skipper.fetch_results({ queryId, mode: "inject" })
return skipper.create_chart({ chartType: "bar", data: results.rows })

Everything Skipper does runs as the calling user, so it can't query a table you don't have access to.

The numbers are wild too: billing-related queries are 53% of everything Town Lake serves, with 91,760 queries from 324 employees in a recent period. "Top 100 customers by revenue" now returns in about 3 seconds, and 200-300 line legacy SQL queries are now five lines.

My favorite lesson from the team was this one.

Less prompting is more.

They dropped the step-by-step prompts, gave the model high-level guidance, and quality went up. They also close with the idea that "the boring infrastructure is the hard part" - row-level access control, auditing, PII detection.

Wrap-up

  • Town Lake is Cloudflare's internal unified data platform, built on Trino + Iceberg + R2
  • Skipper is the AI agent on top that goes from natural language to validated SQL to an answer in seconds
  • Safety is engineered in: default-closed governance, automatic PII detection, and running as the calling user
  • It's genuinely used internally: billing is 53% of queries, with 324 people running 91,760 of them

If you love reading about data-platform design built entirely on a company's own products, or you want to democratize internal data with AI, this one's for you!

Related posts