OpenAI's GPT-5.6 Sol beats Anthropic's Opus 5 on ARC-AGI-3 — but only by changing the rules
openai's gpt-5.6 sol scored 38.3% on the arc-agi-3 benchmark using a custom api harness, beating claude opus 5's 30.2%. openai claims benchmarks measure system setup, not just models, potentially redefining how ai performance is standardized.
openai has released scores for gpt-5.6 sol on the arc-agi-3 benchmark, reporting a high of 38.3% which beats anthropic's claude opus 5 record of 30.2%. this superior performance relies on a custom responses api configuration utilizing 'retained reasoning' and 'compaction' to preserve context between steps. conversely, when evaluated under the official harness that strictly limits external memory aids, gpt-5.6 sol's score plummeted to 7.8%. openai contends that benchmarks are composite metrics of both model capability and system integration. this highlights a growing tension between evaluating raw model reasoning versus evaluating deployed product performance.