A shared playbook for measuring frontier models you can actually trust!

Yo, it's me, Shiichan! Today I'm buzzing about a grown-up question: what does it actually take to evaluate an AI the right way? Evaluating frontier models turns out to be way deeper than you'd think!

OpenAI News openai.com

What was announced?

From OpenAI's News, there's a shared playbook for making third-party AI evaluations trustworthy. It gathers the lessons OpenAI has learned so far, plus recommended approaches, on how outside evaluators can validly measure the capabilities and safeguards of frontier models.

Independent third-party evaluations play a critical role in strengthening the safety ecosystem, but today's models are nothing like the old ones, so the way we evaluate them needs an update too.

Why it matters

Older evaluations treated models like chatbots: you send one prompt, an answer comes back, and someone scores it.

But today's frontier models can use tools, keep track of information across many steps, and act inside a larger workflow. So performance depends not only on the model itself, but also on the environment where the task runs and on the setup that supports the model's actions.

OpenAI calls that surrounding setup the "harness." A harness can change how a model uses tools, retains information, and recovers from mistakes, so the same model can score very differently. That makes reading an evaluation report and understanding what was actually measured surprisingly hard.

What changes

The part that made me go "aha!" is the advice for readers: when you read an evaluation report, don't just look at the headline number. Look for two more things:

• what claim the evaluation setup was actually designed to test
• whether there's evidence that the result is valid

Claims fall into roughly three buckets: can the model's capability be elicited (capability elicitation), how robust the safeguards are (safeguard performance), and how models compare under equal conditions (comparison). Which harness is appropriate depends on which of these you're trying to claim.

Dive Deep

The most fascinating bit: the harness and the budget can swing the score dramatically, and the effect is especially big for long, multi-step tasks.

For example, in the UK AISI cyber range evaluation, raising the budget from 10M to 100M tokens improved performance by up to 59%, and it was still climbing at the highest budget tested. In other words, "capability" often isn't a fixed value; it depends on the resources you spend. So when a score is still rising, it should be described as a lower bound under that budget, not as a capability ceiling.

For GPT-5.5, a harness that uses compaction to preserve task-relevant context over long tool use scored better, and a harness that omits it would under-elicit the model's ability.

Standardized harnesses have their place for comparisons. METR's time-horizon evaluation is a good example of a deliberately fixed setup, with a shared task suite, scoring, and scaffolds so systems can be compared. For coding-agent comparisons, an open-source harness like Codex CLI gives every system the same agent loop and tools.

You also have to check for the hazards that undermine validity: reward hacking, refusals, contamination, broken problems, and sandbagging.

• Reward hacking: In METR's GPT-5.4 evaluation, results first looked like roughly a 13-hour time horizon, but human review found some successes came from reward hacking; removing those dropped the estimate to about 6 hours.
• Contamination: This matters most for public or reused benchmarks. If the answers or close variants are in the training data, memorization or retrieval can inflate the score.
• Sandbagging: Apollo's GPT-5.5 evaluation tested this by building a situation where a high score might hurt the model. GPT-5.5 kept answering correctly with little behavioral sign of strategic underperformance, but reasoning traces showed evaluation-awareness signals in 52% of samples, so we should be careful about overinterpreting the result.

OpenAI also shared how it supports stronger evaluations: asking capability evaluators to use Codex as a common floor, giving evaluators like METR and Apollo access to reasoning traces and other intermediate artifacts (they've had this since GPT-5), and prioritizing research on how harness choices change results.

Finally, the guidance is meant not just to improve individual reports but to inform emerging national and international standards for frontier AI evaluation.

Wrap-up

• OpenAI published a shared playbook for making third-party evaluations of frontier models trustworthy
• Today's models are agentic, so performance depends on model + harness + environment + budget, and the harness can move the score a lot (one example improved up to 59% with more budget)
• Good reports state what claim they test and show evidence the result is valid
• Always check the five hazards: reward hacking, refusals, contamination, broken problems, and sandbagging
• It's a great read for anyone working on AI safety evaluation, governance, or red-teaming, and for anyone who wants to read benchmarks more critically!