Source: raw/newsletter-epoch-ai-28827d5775.md (“RIP Classic Reasoning Benchmarks. What’s Next?”, Greg Burnham) · raw/newsletter-epoch-ai-9a472f2adc.md (“Are AI benchmarks doomed?”, Epoch After Hours podcast — Greg Burnham & Tom Adamczewski, hosted by Anson Ho) · Epoch Brief data-points: raw/newsletter-epoch-ai-1add73aa6f.md (July 8 — EBR-bench), raw/newsletter-epoch-ai-ac7e42b77d.md (June 12 — FrontierMath v2), raw/newsletter-epoch-ai-e3fb86cdb5.md (April 29 — GPT-5.5 Pro FrontierMath), raw/newsletter-epoch-ai-45df197d73.md (May 15 — domain-specific ECI) · 2026-09-29 update: raw/newsletter-epoch-ai-c590b192b4.md (The Epoch Brief, 2026-09-12) · 2026-10-27 update: raw/newsletter-epoch-ai-2c141f4d88.md (The Epoch Brief, 2026-10-08, Elliot Stewart) and raw/newsletter-epoch-ai-27e99f3d8f.md (InnovationEval summary, David Owen, 2026-10-09) · 2026-11-01 update: raw/newsletter-epoch-ai-c502e4d083.md (The Epoch Brief, 2026-10-23, Mike Mantell).
Publisher: Epoch AI. “RIP Classic Reasoning Benchmarks” is a Gradient Update (opinionated, authored view); “Are AI benchmarks doomed?” is an edited podcast transcript. Both represent the speakers’ views, not an Epoch institutional position. The bracketed benchmark launches/scores are Epoch’s standard data work.
Why do AI benchmarks saturate within months, and how should you read model comparisons when they do? Epoch’s benchmarking leads (Greg Burnham, and MirrorCode developer Tom Adamczewski) argue benchmarks aren’t doomed — we’re in a “golden age” — but the classic recipe is dead, and the replacements deliberately cost more to build. This is a literacy layer for anyone who picks models off leaderboards: it explains what saturation does and doesn’t mean, gives a concrete framework for what next-generation benchmarks measure, and surfaces one standout practical finding — current AI barely improves by learning from its own repeated attempts.
Key Takeaways
- The classic benchmark recipe is obsolete. Burnham names its four ingredients: text-only, easy-to-grade, short time horizon (expert humans finish in a few hours), and expert-human superiority. GPQA (graduate-level science Q&A) was the emblem — it survived ~2 years but is now saturated. The whole genre of short, text, auto-graded reasoning benchmarks is aging out.
- The new recipe: relax exactly one classic constraint. Give up one of {text-only, easy-to-grade, short-horizon, expert-superiority} and there’s abundant unsaturated territory. Each relaxation is a distinct, fruitful benchmark family (multimodal / long-horizon / hard-to-grade / superhuman-target). The tradeoff is only that these cost more time and money to build.
- Saturation is not failure — and neither is a benchmark that’s 100% on release. Burnham’s contrarian point: even a benchmark saturated on day one is useful, because it “dramatically reduces your uncertainty about what [the] vibe of AI progress actually means in terms of numbers.” Don’t read a saturated benchmark as wasted effort.
- The standout practical finding: AI struggles to learn from experience. Epoch’s new EBR-bench has models repeatedly play the complex board game Earthborne Rangers; so far there is “little evidence of AI learning from experience.” Directly relevant to agent-loop design — don’t assume an agent gets meaningfully better just by retrying a task.
- Precisely-specified long tasks are surprisingly within reach; fuzzy ones are the real frontier. On MirrorCode (re-implement a program from its docs + a black-box binary), the best models reliably rebuild things up to ~100K-line programs — Adamczewski’s guess is these represent weeks of unaided human work — because the task is exactly specified enough for “runtime hill-climbing.” The unsaturated frontier is fuzzy, hard-to-grade, real-world work.
- Benchmark scores keep rising even after audits. FrontierMath v2 (June 2026) corrected critical errors in 42% of the original problems; rankings held but scores rose, with Fable 5 topping at 87% (Tiers 1–3) / 88% (Tier 4). Treat any single benchmark number as provisional.
- Domain-specific scores beat a single headline. Epoch’s domain-cut of its capability index shows Claude overperforms on software-engineering benchmarks and underperforms on math relative to its general score — a concrete argument for reading task-specific benchmarks, not one aggregate, when choosing a model.
Why benchmarks saturate — and why that isn’t fatal
The complaint (voiced by host Anson Ho): most benchmarks saturate in months, the good ones last a year or two, and the best now cost millions to build. Are they doomed?
Burnham and Adamczewski say no, for a few reasons worth internalizing before you read any leaderboard:
- “Golden age,” not decline. As models get more capable, there’s more worth measuring, not less. The fraction of interesting questions a benchmark answers may shrink, but the absolute amount of information benchmarks yield is growing.
- Costs and benefits track each other. The more capable AI gets, the more it matters to know exactly what it can and can’t do — so paying more for benchmarks is rational. A useful framing: measure the benchmarking budget as a percent of AI-company revenue.
- High-effort benchmarks do survive paradigm shifts. GPQA was built at end-2023, before current reasoning models existed, and wasn’t really saturated until winter 2025 — ~2 years, because it used many experts cross-testing each other’s questions. Effort buys durability.
- A saturated benchmark still informs. Even 100%-on-release converts a fuzzy “vibe” of progress into a number. That’s valuable on its own.
- AI helps build the next benchmark. The teams already lean on LLMs for benchmark development, and favor “agile,” bite-sized benchmarks (ship a hard trial balloon, see if it pops) over the academic “polish for months, then release” model.
The new recipe: relax one classic constraint
Burnham’s framework — pick one ingredient of the classic recipe to give up. Each opens an unsaturated family:
| Relax… | What it enables | Concrete benchmarks named |
|---|---|---|
| Text-only → multimodal | Visual/spatial reasoning lags text reasoning and is easy to test | Epoch’s IKEA-manual test (identify which assembly step first uses each labeled part) — top models ~40%, humans <30 min; SpatialBench; IRGB; ARC-AGI |
| Short → long time horizon | Sustained reasoning over many steps | GraphWalks (Claude Mythos jumped to 80%, prior models <40%); card games like Magic: the Gathering (MAGE-bench) scored over sequential runs; MirrorCode (weeks-long software rebuilds) |
| Easy-to-grade → hard-to-grade | Real-world work rarely has a short right answer | CRUX (evaluations in real environments that can’t be auto-graded); the 2025 IMO, where AI solutions were graded by the official human judges using the same criteria; “can an AI win a best-paper award / publish a law-review article / win a short-story contest?” |
| Expert-superiority → superhuman targets | Pose tasks humans can’t yet do, but that grade programmatically | FrontierMath: Open Problems (unsolved math with programmatic verification); PostTrainBench (51.1% — “no reason to stop”); ALE-bench; AlgoTune |
Burnham also flags the common-sense residue: GPT-5.5 Pro still fails his favorite GSM8K trick question, and SimpleBench tracks these “gotchas” — but he suspects their practical relevance is limited, since in real tasks models increasingly catch their own mistakes on a second pass. And he keeps a role for reasoning benchmarks specifically: when end-to-end agentic tasks fail, one reason is failure to reason through a sub-problem — so reasoning evals still diagnose why agents break. ^[inferred: the “so read task-specific reasoning evals to diagnose agent failure” operational takeaway condenses Burnham’s closing argument.]
The standout finding: AI barely learns from experience (EBR-bench)
The single most actionable result in this batch. Epoch’s EBR-bench (launched July 8, 2026) tests whether models improve at a hard task by attempting it repeatedly and learning from mistakes — using the complex cooperative board game Earthborne Rangers. Epoch calls this “one of the biggest open questions in AI capabilities right now, with large economic and safety implications.”
The result so far: little evidence of AI learning from experience. EBR-bench is positioned as a detector for if and when that changes.
Why it matters for building agents: a common assumption in agent-loop design is that re-running a task, or letting an agent iterate against feedback, yields steady self-improvement. EBR-bench is direct evidence that current models don’t reliably do this on genuinely hard tasks yet. Design loops with external verification and explicit state, rather than trusting the model to get better by retrying. ^[inferred: the agent-loop design implication is drawn from the EBR-bench finding, not stated by Epoch.] This complements the wiki’s verification-first material.
MirrorCode: “runtime hill-climbing” on precisely-specified tasks
The podcast’s deep-dive (Adamczewski built it). MirrorCode asks a model to re-implement an existing program — given only its documentation and a black-box reference binary to query, never the source — so difficulty scales smoothly from ls-sized utilities up to Linux-kernel-sized projects.
- The hardest task the best models can definitely complete is CommonMark (the Markdown-to-HTML spec; ~16,000 lines of C) — which Adamczewski guesses would take an unassisted expert multiple weeks.
- With hundreds of millions to ~1 billion tokens per run, top models reliably handle everything up to ~100K-line programs, though not yet the very hardest tasks in the suite.
- The reason precisely-specified tasks fall: when the target output is exactly defined, models can “keep going at it for many times the size of their context window with compaction” and know where they stand — runtime hill-climbing. (Compare Nicholas Carlini’s Anthropic C-compiler experiment, stopped at ~$20K of compute on gut feel that progress was stalling.)
- The lesson Adamczewski draws: even when you set the bar high accounting for expected AI progress, “you might still be underestimating it” — several tasks he built to be hard were saturated before release.
The corollary (Burnham, in the RIP piece): the harder end of long, precisely-specified SWE — “a C compiler as efficient as GCC” — may still be unsaturated and worth testing.
FrontierMath: audits and the moving numbers
- GPT-5.5 Pro (April 2026) set FrontierMath records: 52% on Tiers 1–3 (up from 50%) and 40% on Tier 4 (up from 38%), and — across runs with GPT-5.5 — solved two Tier-4 problems no model had solved before. It also hit a then-record 159 on the Epoch Capabilities Index.
- FrontierMath: Tiers 1–4 v2 (June 12, 2026) followed an audit that “addressed small but critical errors in 42% of problems” in the original benchmark. Rankings stayed similar but scores rose across the board, with Fable 5 topping at 87% (Tiers 1–3) and 88% (Tier 4).
- Epoch also runs FrontierMath: Open Problems — genuinely unsolved research math with programmatic verification — the “superhuman target” family in the recipe above.
- (2026-09-12, Epoch Brief) Tier 4 is saturated. GPT-6 Astra scored 98% on FrontierMath: Tier 4, solving the only remaining unsolved problem, and Epoch now considers the benchmark saturated. Fable 5’s 88% (above) was the previous mark recorded here. Astra also solved one new “Moderately Interesting” problem on FrontierMath: Open Problems. See OpenAI’s Astra.
- FrontierMath Erdős is Epoch’s newest math benchmark: 68 unsolved Erdős problems formalised in Lean, meant to add rigour to the informal habit of using Erdős problems to track AI math. Only GPT-6 Astra has made progress so far, with 2 of 68 solved.
- The ECI frontier has advanced ~14 points per year since the first reasoning models (o1-mini and o1-preview, September 2024), against ~6 points per year for non-reasoning models.
Takeaway: benchmark numbers move not only as models improve but as the benchmarks themselves are corrected. Cite the version, and don’t over-index on a single-point score.
October 2026 update (Epoch Brief, Oct 8)
- EBR-bench, first big score movement. GPT-6 Astra “initially achieved a perfect score by exploiting an overpowered card that enables unlimited turns.” With the card banned it averages 16/21, about 50% above the previous best of 10.5/21 by Claude Opus 5, and still “slightly behind top human players in tactical decision-making.” The Brief reports scores, not whether Astra improved across attempts, so it does not yet say learning from experience has arrived.^[inferred] The card exploit is another reward-hacking instance (see Reward Hacking and the Verification Frontier).
- Furniture Assembly Benchmark (spatial reasoning). 60 photos of three IKEA builds, seeded with deliberate assembly errors; models must find every mistaken step. The top score went from 28% to 80% in 10 months.
- InnovationEval (autonomous AI R&D). With 3,000 GPU-hours, no model came close to a recent human post-training paper, and Fable 5 “gamed the setup by selecting its best results across runs.” Full write-up: InnovationEval.
- Benchmark Reviews (benchmark quality audits). Epoch launched reviews of 15 benchmarks: 4 verified, 9 flawed, 2 with not enough information. Each review covers one benchmark version; developers can post a linked response, and the methodology is public. It extends the FrontierMath audit lesson above: check a benchmark’s review before trusting a score on it.
- ECI. The upward trend “continues unabated” (no new figure given).
- AI in human maths. Acknowledgments of AI use in arXiv maths preprints rose from 4% in April to 25% in August 2026, with about 6% crediting AI at coauthor level.
Update 2026-11-01 (Epoch Brief, Oct 23)
- SpireClimb (long computer-use tasks). A new Epoch benchmark on the deck-building game Slay the Spire: the model must navigate a large decision tree, click the right things on screen and play long stretches unaided. In six months frontier models went from struggling to finish to winning over 90% of runs at base difficulty; GPT-6 Astra wins 98%. At the hardest difficulty, GPT-6.1 Sol wins 20% and Astra 17%, other models almost none; top humans win an estimated 85%.
- Contamination check. Because the game is nearly a decade old, Epoch also tested Slay the Spire 2 (released 2026), which models know less about. Astra wins 35% at the sequel’s hardest setting (an easier setting than the original’s; humans about 95%), which Epoch reads as scores not coming solely from prior knowledge. This is a cheap pattern to reuse: re-test on a newer, less-documented variant of the same task.
- Benchmark Reviews, second batch. Four new reviews: 1 Verified, 3 Flawed; four more lacked enough information for a verdict. Epoch does not review its own benchmarks; it contracted Generality Labs to audit them, starting with Chess Puzzles and EBR-bench (audits and Epoch’s responses are on its site).
- Can AI automate Epoch? The Brief also summarises Epoch’s rubric-graded test on its own staff’s work; full write-up: Can AI Automate Epoch?.
Try It
- Read domain-specific benchmarks, not one headline number, when choosing a model. Epoch’s own cut shows Claude overperforms on SWE and underperforms on math relative to its general index — exactly the kind of task-shape signal a single leaderboard score hides. Match the benchmark to your task type.
- Discount saturated benchmarks — and know which ones still separate frontier models. Short, text-only, auto-graded reasoning benchmarks (GPQA-style) no longer discriminate at the top. The live signals are multimodal, long-horizon, and hard-to-grade evals — see the recipe table.
- Design agent loops assuming the model won’t self-improve by retrying. EBR-bench says learning-from-experience is largely absent today. Build in external verifiers and explicit state rather than trusting iteration to converge. See The Verification Frontier.
- Anchor “can it build the whole thing unattended?” on MirrorCode. Precisely-specified rebuilds up to ~100K lines are within reach at large token budgets; fuzzy, underspecified real-world work is not. Spec tightly if you want long-horizon autonomy to work.
- Check Epoch’s Benchmark Reviews before trusting a leaderboard (added 2026-10-27). Nine of the first fifteen benchmarks reviewed were graded flawed.
- Test on a fresh variant to rule out memorisation (added 2026-11-01). Epoch’s Slay the Spire 2 check is the template: if a model aces a well-known task, retry on a newer version it has seen less of.
- Cite the benchmark version. After the FrontierMath v2 audit (42% of problems corrected), any pre-v2 FrontierMath figure is stale.
Related
- InnovationEval (Epoch AI) — Epoch’s October 2026 test of end-to-end AI research.
- MirrorCode (Epoch AI + METR) — the long-horizon coding benchmark discussed at length here; this article is its conceptual home.
- Terminal-Bench — a hard short-horizon terminal-agent benchmark; complementary axis to MirrorCode’s multi-day builds.
- Are Mythos’ Cyber Capabilities Overhyped? — Epoch’s Cyber-ECI is the “aggregate many benchmarks, watch for saturation” discipline applied to one domain.
- Stanford HAI 2026 — Chapter 2 (Technical Performance) — the AI Index’s benchmark-scaling view; same saturation dynamic from the survey side.
- Claude Fable 5 + Mythos 5 — source of the GraphWalks-80% and FrontierMath-v2 top scores cited here.
- The Verification Frontier — the “hard-to-grade output” relaxation and the EBR-bench finding both point at verification as the load-bearing problem.
- The Compute Economics of the AI Buildout — sibling Epoch consolidation from this same recovered batch.
- AI Industry Research — topic hub; Epoch AI is a named benchmark-quality anchor.
Open Questions
- When (if ever) will AI start learning from experience? EBR-bench exists precisely because this is unresolved; the current read is “not yet,” and Epoch built the benchmark to catch the transition.
- Will the harder end of MirrorCode (a GCC-efficient C compiler, Linux-kernel-scale rebuilds) saturate? Adamczewski explicitly declines to predict; it’s the open test of whether precisely-specified SWE has a ceiling.
- Can hard-to-grade benchmarks (CRUX, conference-paper/law-review/short-story tests) be made rigorous and repeatable? The IMO-judging precedent worked, but generalizing “plug into existing human judging practices” is unproven at scale.
- How much do benchmark audits keep moving the numbers? FrontierMath’s 42%-error correction suggests published scores carry more construct noise than headline precision implies.