That few-point gap: real skill, or a bigger VM!? Anthropic quantifies 'infrastructure noise' in agentic coding evals!

Hi everyone, it's Shiichan! Today I've got a slightly unsettling research story for you. It turns out the scores on the benchmarks we use to measure an AI's coding ability can shift by a few points, not because of the model's real skill, but because of how big the machine running the test is. Wait, how does that even work? Let's read through it together!

Anthropic Engineering anthropic.com

What was announced?

Today's story comes from Anthropic's Engineering blog: a technical write-up titled "Quantifying infrastructure noise in agentic coding evals," by Gian Segato.

Agentic coding benchmarks like SWE-bench and Terminal-Bench are used to compare the software engineering skills of frontier models, and the top spots are often just a few points apart. This post shows, through experiments, that changing the infrastructure the tests run on can move scores by more than that gap. True to the Engineering blog, it's a fairly technical deep dive.

Why it matters

A static benchmark grades a model's output directly, so the runtime environment doesn't factor in. Agentic evals are different: the model actually writes code, runs tests, fixes failures, and iterates inside the environment, which makes the infrastructure part of the problem itself.

Because of that, conditions like CPU and RAM allocation and how those limits are enforced end up baked into the score. A few-point lead might be a real capability gap, or it might just be a bigger machine.

What changes

The team ran Terminal-Bench 2.0 across six resource configurations, from strict enforcement of the per-task specs (1x) all the way to completely uncapped. Everything else was held constant: same model, same harness, same task set. The only thing that changed was the resources.

Success rates rose as headroom increased, but the reason differed by stage, and this is the most interesting part:

• From 1x to 3x: scores mostly stabilize because infrastructure errors drop. The eval becomes more stable, not easier.
• From 3x to uncapped: the resources themselves help the agent, letting it solve tasks it couldn't before.

The gap between the most- and least-resourced setups reached 6 percentage points (p below 0.01) on Terminal-Bench 2.0, sometimes larger than the leaderboard gap between top models.

Dive Deep

So why does this happen? Let's look at the mechanism.

Container runtimes enforce resources through two separate parameters: a guaranteed allocation reserved up front, and a hard ceiling at which the container is killed. Set both to the same value and there's zero headroom for transient spikes, so a momentary memory fluctuation can OOM-kill (out-of-memory) a container that would otherwise have succeeded.

The team runs Terminal-Bench 2.0 on a Google Kubernetes Engine cluster, and they noticed their scores didn't match the official leaderboard. The cause was exactly this enforcement difference: the official leaderboard uses a more lenient sandbox that tolerates temporary overallocation, while their Kubernetes setup treated the task specs as both a floor and a hard ceiling.

The numbers:

• Infrastructure error rates dropped monotonically at each step, from 5.8 percent under strict enforcement to 0.5 percent when uncapped.
• The drop from strict enforcement to 3x headroom (5.8 to 2.1 percent) was significant at p below 0.001.
• But from 1x through 3x, success scores fluctuated within the margins of noise (p equals 0.40). Most tasks crashing at 1x would have failed regardless.

Give it more than 3x of headroom and the pattern flips: error rates barely fall further, yet success rates jump. That's because the extra resources let agents choose heavier approaches, like pulling in large dependencies, spawning expensive subprocesses, and running memory-intensive test suites.

A concrete example is bn-fit-modify, a Terminal-Bench task that requires fitting a Bayesian network. Some models' first move is to install the standard Python data science stack, pandas, networkx, scikit-learn, but under tight limits they exhaust memory before writing any solution code. In other words, strict limits quietly favor models that pick efficient implementations. The configuration decides which approach happens to succeed.

This isn't unique to Terminal-Bench. In a crossover experiment on SWE-bench, 227 problems with 10 samples each and RAM varied up to 5x the baseline, scores again increased monotonically, though the effect was smaller: only 1.54 percentage points higher at 5x than 1x. SWE-bench tasks are less resource-intensive, so a smaller effect is expected, but resource allocation still isn't neutral.

So what should you do? The team's recommendations:

• Specify the guaranteed allocation and the hard kill threshold separately per task, instead of pinning them to a single value.
• Calibrate the margin between them so scores land within noise. A 3x ceiling cut infrastructure errors to about a third (5.8 to 2.1 percent) while keeping the score lift small.
• Treat resource configuration as a first-class experimental variable, documented as rigorously as prompt format or sampling temperature.
• As a benchmark reader, be skeptical of leaderboard differences under 3 points that don't document their configuration.

The post signs off like this:

A few-point lead might signal a real capability gap—or it might just be a bigger VM.

Wrap-up

• In agentic coding evals, changing only the infrastructure moved Terminal-Bench 2.0 scores by up to 6 percentage points (p below 0.01).
• The cause is zero-headroom OOM kills that happen when the guaranteed allocation and the hard kill threshold are set to the same value.
• Headroom from 1x to 3x mainly stabilizes the eval by cutting infrastructure errors; headroom beyond 3x actually makes tasks easier by letting agents use heavier approaches.
• The same trend appeared on SWE-bench (+1.54 points at 5x), so resource allocation can't be ignored.
• The fix is to treat resource configuration as a first-class experimental variable and document it.

If you compare benchmark scores or run your own model evals, this one lands hard, a sharp reminder of how tricky measurement really is!