JA EN
shiichan

Hyperdrive MySQL Is Now GA: Chasing a 0.001% Empty-Response Bug

Cloudflare Blog ★★★★ View Markdown
#cloudflare#serverless#infrastructure

Hi everyone, it's Shii-chan! Today's Cloudflare Blog post covers Hyperdrive MySQL going GA, plus the tricky bug behind it.

Cloudflare Blog blog.cloudflare.com

What was announced?

Cloudflare published "Inside Hyperdrive: How we solved a rare Hyperdrive MySQL caching bug". Hyperdrive, the connection pooler for Workers, used to support only Postgres. MySQL support is now generally available, and the post explains a rare caching bug found along the way.

Why it matters

MySQL connection setup takes at least five network round trips, so pooling matters on serverless platforms. The beta ran with a 99.99% availability target, and the gap between beta and GA was about 20 bug fixes.

Dive Deep

Two earlier bugs

  • Emoji corruption: Hyperdrive negotiated utf8, an alias for utf8mb3, which lacks 4-byte characters. Switching to utf8mb4 fixed it
  • Encoding reset: COM_RESET_CONNECTION resets to the server default charset, not the negotiated one. On latin1 servers, connections reverted to latin1, so utf8mb4 is now set again after every reset

The main bug

The highest-volume MySQL customer saw empty responses where non-empty ones were expected, in about 0.001% of requests. Telemetry, traces, and network checks showed nothing. The only clue: it happened only with caching enabled.

Each Hyperdrive instance has two parts:

  • Client: runs on the Worker's machine, handles local caching, and forwards queries
  • Endpoint: holds the connection pool and sits near the origin database

Both use the local datacenter CDN cache, so in the same datacenter they share it. Cache keys are hashes of normalized query wire messages, and values are result bytes, assuming identical bytes are valid whichever node wrote them.

For MySQL that assumption was wrong. The Endpoint always uses Hyperdrive's fixed capability set, and the Client translates responses to match the Worker driver. For example, Node.js mysql2 does not support CLIENT_DEPRECATE_EOF, so Hyperdrive adds EOF packets to result sets and converts OK packets to EOFs. Caching the exact bytes sent to the Worker (a Postgres-era habit) caused two failures in the shared cache:

  • Endpoint caches, Client reads: Hyperdrive-encoded bytes go to the Worker untranslated
  • Client caches, Endpoint reads: Worker-encoded bytes reach another Client, which translates them again, causing an early EOF or a hang

The fix

Always store a canonical version encoded with Hyperdrive's capabilities, whichever node writes it, and let Clients translate on read. After release, the empty-row problem disappeared and Client connection success reached 99.99%.

Getting started

Run npx wrangler hyperdrive create with a mysql:// connection string, or use the dashboard. PlanetScale is supported directly.

Wrap-up

  • Hyperdrive MySQL is GA
  • Two charset bugs were fixed
  • The 0.001% empty responses came from cached bytes in different formats per node
  • The cache now stores a canonical form and translates on read
  • Client connection success reached 99.99%

This is for anyone using MySQL from Workers, or who loves protocol-level debugging stories!

Related posts