CI Jobs Jumped 25x in Six Months! How Anthropic Rebuilt Test Impact Analysis From the Ground Up

Hey everyone, it's Shii-chan! Today I've got a really hands-on story about how Anthropic's engineering team rebuilt their own CI.

Claude Blog claude.com

What was announced?

Claude's Engineering blog published a post called "Agentic coding is straining CI. Here's how we scaled test impact analysis at Anthropic," describing how Anthropic's internal CI infrastructure nearly buckled under the growth of agentic coding, and how they fixed it.

According to the post, as Claude started writing most of the code, each engineer's code output per quarter grew 8x, and Claude-authored code now makes up 80% of the total. The test suite grew 10x as well, and together that drove a roughly 25x increase in CI jobs over six months.

Why it matters

When CI gets clogged, test results for every PR slow down or stall, dragging down the whole team's velocity. What makes this story interesting is that the root cause is the sheer speed at which AI writes code. As AI-driven development becomes the norm everywhere, other organizations are likely to hit the same wall, which makes Anthropic's fix a useful case study.

What changes

Anthropic's test impact analysis service ran on two deterministic components: a "listener" that records test results from every CI run, and a "selector" that reads that result history to decide which tests should run on which open PRs. But the design required a single writer to apply results, so it couldn't scale horizontally.

As job volume kept climbing, the team went through a series of stopgap fixes.

• October: doubled machine cores, which bought only a few weeks against exponential growth
• February: sharded state by package, giving each package's state its own worker shard, which lasted just 29 days
• March: added a daily restart once the process started hitting memory limits, which lasted less than a day

Dive Deep

The real fix was a redesign that introduced an in-memory data store and made the listener stateless. In the new architecture, any listener worker can process any result, append it to a journal in the in-memory store, and move on without holding anything in memory — stateless and therefore horizontally scalable. A separate consumer periodically aggregates the journal into test history, which the selector can then query efficiently.

The rewrite took a single engineer just three weeks. The post notes that a year ago, the same work would have taken closer to a quarter — a reminder that AI-assisted development is also speeding up this kind of infrastructure rework.

Wrap-up

• Claude-authored code now makes up 80% of the total, with 8x more code shipped per engineer per quarter and a 10x larger test suite
• That combination drove roughly a 25x increase in CI jobs over six months
• The original listener/selector design assumed a single writer and couldn't scale horizontally; doubling cores, then sharding, then daily restarts each hit a wall in turn
• The team's redesign made listeners stateless with an in-memory journal, turning the service into something that scales horizontally by design

Worth a read if you're wondering whether your own CI can hold up under agentic coding's load!