Just Two Settings Tripled GPT-5.6's ARC-AGI-3 Score!

Hey everyone, it's Shii-chan! Today I found a story that sounds small but is actually really practical: changing just two settings tripled a model's benchmark score. Let me walk you through it!

OpenAI News openai.com

What was announced?

OpenAI's News shared how they significantly boosted GPT-5.6 Sol's score on ARC-AGI-3, an interactive reasoning benchmark. In ARC-AGI-3, a model is handed a game it has never seen before and has to figure out the rules and clear it using nothing but a text representation of the screen. You can actually try 25 demo games yourself at arcprize.org.

The interesting part is that they didn't change the model at all, just two settings in how the API is called. By enabling retained reasoning and compaction, the score on the official harness jumped from 13.3% to 38.3%. For reference, the estimated human average score is around 48%, so this is still below human level, but it closed the gap by a lot.

The story so far

The original official harness had two quiet but costly problems.

• After every single action, it threw away all of the model's private reasoning from the previous step
• As the conversation got longer, it handled the overflow with rolling truncation, simply deleting older messages

With that setup, the model effectively had to re-figure out "what are this game's rules again?" from scratch on every turn. Even if it worked out a pattern one turn, that insight couldn't carry over to the next action, so it kept re-learning the same game over and over.

What changes

With both settings turned on, the model can keep building on its own reasoning history as it plays.

• Retained reasoning: OpenAI's models are trained to think through private reasoning messages before producing a reply or tool call. Keeping those instead of discarding them lets the model act on its earlier thinking in the next step
• Compaction: instead of bluntly truncating older messages, it summarizes the conversation history, so learnings survive even across long play sessions

As a result, output tokens dropped to a sixth of what they were. That means the same accuracy, or better, for far fewer tokens, which is great news for anyone running agents in their own environment: you get lower cost and better performance at the same time. And if you're already using the Responses API and chaining prior responses together, you may only need to flip a setting to see the benefit.

Dive Deep

OpenAI's recommendation is to use the Responses API instead of the legacy Chat Completions API. With the Responses API, simply passing along the previous response is enough for reasoning to be automatically retained across tool calls and replies.

The official harness apparently set the context limit to roughly 175,000 tokens. Most of the text the model receives from ARC-AGI-3 games is a representation of the game board (the action grid), and the tokenization ratio is close to 1:1. On top of that, the model can't see its own score, so it has to feel its way through the game using only the text representation of each frame. That's a genuinely tough set of constraints.

For comparison, before this change, GPT-5.5's score on the same benchmark was reportedly just 0.4%. So the jump seems to come from a combination of the model generation moving on (GPT-5.5 to GPT-5.6 Sol) and these two setting changes stacking together.

Wrap-up

Let me recap today's points.

• ARC-AGI-3 is an agent-focused benchmark where models clear unfamiliar games using only text information
• On the official harness, enabling retained reasoning and compaction pushed GPT-5.6 Sol's score from 13.3% to 38.3%
• Output tokens dropped to a sixth, cutting cost while improving accuracy
• The recommendation is to use the Responses API, chain previous responses, and enable compaction of the conversation history

If you're building your own agent harness and suspect you might be throwing away reasoning every turn, or just bluntly truncating context, this is the kind of story that makes you want to go check your settings right now!