On July 29, OpenAI reported that scores on ARC-AGI-3 — a benchmark that measures how agents learn by exploring unfamiliar 2D games — shifted substantially with the settings used to run them1. By the company’s account, GPT-5.6 Sol scored 13.3% on the public set with the official harness, and 38.3% once two API settings it uses in ChatGPT and Codex — retained reasoning and compaction — were enabled, with 6x fewer output tokens1. Scores use RHAE, a metric relative to a human baseline; from official gameplay logs, the company estimates the average human tester scored 48%1.
The causes it identifies are that the official harness discarded all private reasoning after each game action, and that it used rolling truncation, dropping the oldest messages once context exceeded 175,000 characters1. ARC’s rationale for that design was that a simple, generic harness makes model shortcomings more visible and comparisons fairer1. OpenAI’s stated conclusion is that evaluations rarely measure models in isolation — they also measure a bundle of less visible choices about API settings, harness design, and prompting1. This is the company’s own experiment, not an independent reproduction.
Sources
- How enabling two settings tripled our scores on the ARC-AGI-3 benchmark - OpenAI official (Ilan Bigio, Ted Sanders, July 29, 2026)
- OpenAI has successfully tripled its ARC-AGI-3 score by improving the GPT-5.6 harness - GIGAZINE (July 31, 2026)