Under the hood of D1: how its control plane scales to millions of databases!

Hi, it's Shii-chan! D1's design for supporting millions of databases has been revealed!

Cloudflare Blog blog.cloudflare.com

What was announced?

The Cloudflare Blog explains the architecture of the D1 control plane. D1 is a serverless database built on SQLite, with one SQLite Durable Object backing each database. The post is about managing millions of databases while keeping query operations at the edge.

Why it matters

As of October 8, 2026, live non-deleted databases grew 15x over the past year and 65x over two years. Daily active databases (those that ran at least one query that day) grew 18x in a year. Millions have been created, with hundreds of thousands active at a time.

What changes

The post offers guiding principles for building at scale on the Developer Platform. Keeping centralized components off the hot path applies well beyond D1.

Dive Deep

Operations fall into three categories:

• Data plane: query requests. As fast as possible and always at the edge
• Control plane: HTTP API operations and internal flows from non-query requests. These may depend on central, single-region components
• Background asynchronous plane: analytics and billing. Eventually consistent and can run anywhere

The main components:

• Edge API Worker: handles API requests (dashboard, Wrangler, CLI) with authentication, authorization, rate limiting, validation, and entitlement checks
• Eyeball Worker: proxy for requests to D1 Durable Objects, centralizing observability, retries, and error handling
• Durable Objects: one per database, storing metadata and state, and executing SQL
• Postgres (Core): the source of truth for database metadata, reached through Hyperdrive
• Workers KV: routing metadata such as the database ID to Durable Object mapping
• Queues: smooth writes from Durable Objects to Postgres
• ClickHouse: analytics, historical storage snapshots, and row-usage events for billing
• Quicksilver: an internal globally replicated key-value store used for limit-enforcement flags

The post lists seven design principles:

• Design every process for failure and idempotency
• Keep centralized, single-region components out of the data plane hot path, especially queries
• Use Workers KV for critical routing metadata so cache misses rarely reach central databases
• Use Hyperdrive for connection pooling and caching, preferring read replicas
• Precompute aggregations to avoid scans during request handling
• Use eventual consistency and queues where strong consistency isn't needed
• Load-test with realistic workloads and fix the bottlenecks they reveal first

Database creation goes like this:

• Check the account's database count against its free tier or account limits
• Generate a UUID and a new Durable Object ID with newUniqueId(), then initialize the Durable Object
• Write the metadata and DO mapping to Postgres. This is the creation checkpoint: the database counts as created even if the final response never reaches the user
• Write the mapping to Workers KV on a best-effort basis

Retries use new IDs each time, because reusing a UUID with a fresh DO ID would let multiple DOs claim the same database. The DO is initialized before the Postgres write so queries never reach an uninitialized object.

Each Durable Object heartbeats the control plane every 30 seconds plus per-database jitter. This lets an orphaned DO learn that its database no longer exists, and it also repairs Workers KV state.

On the query hot path, the Edge API Worker looks up the mapping in Workers KV, falling back to Postgres replicas through Hyperdrive on a rare miss. It then goes through the Eyeball Worker to the Durable Object. User Workers with a D1 binding skip the lookup and go straight to the Eyeball Worker.

Storage limits are 5 GB per account on Free and 1 TB on Workers Paid (can be raised on request). Usage updates flow through a queue, and a scheduled job runs every second to aggregate deltas per account. When the total exceeds the limit, the DO blocks writes that would increase size. For the Free daily row limit, a Limiter finds over-limit accounts from ClickHouse usage events, emails them, and sets a Quicksilver flag so DOs reject queries at the edge.

Wrap-up

• Queries stay at the edge, management is central, and aggregation is asynchronous
• Postgres, KV, ClickHouse, and Quicksilver each have their own job
• Single-region parts are kept off the hot path, and heartbeats provide self-correction
• Limits are enforced with pre-aggregation and flags, without scans

This one is for anyone interested in large multi-tenant design and serverless platforms!