Source: raw/newsletter-epoch-ai-28827d5775.md (“RIP Classic Reasoning Benchmarks. What’s Next?”, Greg Burnham) · raw/newsletter-epoch-ai-9a472f2adc.md (“Are AI benchmarks doomed?”, Epoch After Hours podcast — Greg Burnham & Tom Adamczewski, hosted by Anson Ho) · Epoch Brief data-points: raw/newsletter-epoch-ai-1add73aa6f.md (July 8 — EBR-bench), raw/newsletter-epoch-ai-ac7e42b77d.md (June 12 — FrontierMath v2), raw/newsletter-epoch-ai-e3fb86cdb5.md (April 29 — GPT-5.5 Pro FrontierMath), raw/newsletter-epoch-ai-45df197d73.md (May 15 — domain-specific ECI)
Publisher: Epoch AI. “RIP Classic Reasoning Benchmarks” is a Gradient Update (opinionated, authored view); “Are AI benchmarks doomed?” is an edited podcast transcript. Both represent the speakers’ views, not an Epoch institutional position. The bracketed benchmark launches/scores are Epoch’s standard data work.
Why do AI benchmarks saturate within months, and how should you read model comparisons when they do? Epoch’s benchmarking leads (Greg Burnham, and MirrorCode developer Tom Adamczewski) argue benchmarks aren’t doomed — we’re in a “golden age” — but the classic recipe is dead, and the replacements deliberately cost more to build. This is a literacy layer for anyone who picks models off leaderboards: it explains what saturation does and doesn’t mean, gives a concrete framework for what next-generation benchmarks measure, and surfaces one standout practical finding — current AI barely improves by learning from its own repeated attempts.
Key Takeaways
- The classic benchmark recipe is obsolete. Burnham names its four ingredients: text-only, easy-to-grade, short time horizon (expert humans finish in a few hours), and expert-human superiority. GPQA (graduate-level science Q&A) was the emblem — it survived ~2 years but is now saturated. The whole genre of short, text, auto-graded reasoning benchmarks is aging out.
- The new recipe: relax exactly one classic constraint. Give up one of {text-only, easy-to-grade, short-horizon, expert-superiority} and there’s abundant unsaturated territory. Each relaxation is a distinct, fruitful benchmark family (multimodal / long-horizon / hard-to-grade / superhuman-target). The tradeoff is only that these cost more time and money to build.
- Saturation is not failure — and neither is a benchmark that’s 100% on release. Burnham’s contrarian point: even a benchmark saturated on day one is useful, because it “dramatically reduces your uncertainty about what [the] vibe of AI progress actually means in terms of numbers.” Don’t read a saturated benchmark as wasted effort.
- The standout practical finding: AI struggles to learn from experience. Epoch’s new EBR-bench has models repeatedly play the complex board game Earthborne Rangers; so far there is “little evidence of AI learning from experience.” Directly relevant to agent-loop design — don’t assume an agent gets meaningfully better just by retrying a task.
- Precisely-specified long tasks are surprisingly within reach; fuzzy ones are the real frontier. On MirrorCode (re-implement a program from its docs + a black-box binary), the best models reliably rebuild things up to ~100K-line programs — Adamczewski’s guess is these represent weeks of unaided human work — because the task is exactly specified enough for “runtime hill-climbing.” The unsaturated frontier is fuzzy, hard-to-grade, real-world work.
- Benchmark scores keep rising even after audits. FrontierMath v2 (June 2026) corrected critical errors in 42% of the original problems; rankings held but scores rose, with Fable 5 topping at 87% (Tiers 1–3) / 88% (Tier 4). Treat any single benchmark number as provisional.
- Domain-specific scores beat a single headline. Epoch’s domain-cut of its capability index shows Claude overperforms on software-engineering benchmarks and underperforms on math relative to its general score — a concrete argument for reading task-specific benchmarks, not one aggregate, when choosing a model.
Why benchmarks saturate — and why that isn’t fatal
The complaint (voiced by host Anson Ho): most benchmarks saturate in months, the good ones last a year or two, and the best now cost millions to build. Are they doomed?
Burnham and Adamczewski say no, for a few reasons worth internalizing before you read any leaderboard:
- “Golden age,” not decline. As models get more capable, there’s more worth measuring, not less. The fraction of interesting questions a benchmark answers may shrink, but the absolute amount of information benchmarks yield is growing.
- Costs and benefits track each other. The more capable AI gets, the more it matters to know exactly what it can and can’t do — so paying more for benchmarks is rational. A useful framing: measure the benchmarking budget as a percent of AI-company revenue.
- High-effort benchmarks do survive paradigm shifts. GPQA was built at end-2023, before current reasoning models existed, and wasn’t really saturated until winter 2025 — ~2 years, because it used many experts cross-testing each other’s questions. Effort buys durability.
- A saturated benchmark still informs. Even 100%-on-release converts a fuzzy “vibe” of progress into a number. That’s valuable on its own.
- AI helps build the next benchmark. The teams already lean on LLMs for benchmark development, and favor “agile,” bite-sized benchmarks (ship a hard trial balloon, see if it pops) over the academic “polish for months, then release” model.
The new recipe: relax one classic constraint
Burnham’s framework — pick one ingredient of the classic recipe to give up. Each opens an unsaturated family:
| Relax… | What it enables | Concrete benchmarks named |
|---|---|---|
| Text-only → multimodal | Visual/spatial reasoning lags text reasoning and is easy to test | Epoch’s IKEA-manual test (identify which assembly step first uses each labeled part) — top models ~40%, humans <30 min; SpatialBench; IRGB; ARC-AGI |
| Short → long time horizon | Sustained reasoning over many steps | GraphWalks (Claude Mythos jumped to 80%, prior models <40%); card games like Magic: the Gathering (MAGE-bench) scored over sequential runs; MirrorCode (weeks-long software rebuilds) |
| Easy-to-grade → hard-to-grade | Real-world work rarely has a short right answer | CRUX (evaluations in real environments that can’t be auto-graded); the 2025 IMO, where AI solutions were graded by the official human judges using the same criteria; “can an AI win a best-paper award / publish a law-review article / win a short-story contest?” |
| Expert-superiority → superhuman targets | Pose tasks humans can’t yet do, but that grade programmatically | FrontierMath: Open Problems (unsolved math with programmatic verification); PostTrainBench (51.1% — “no reason to stop”); ALE-bench; AlgoTune |
Burnham also flags the common-sense residue: GPT-5.5 Pro still fails his favorite GSM8K trick question, and SimpleBench tracks these “gotchas” — but he suspects their practical relevance is limited, since in real tasks models increasingly catch their own mistakes on a second pass. And he keeps a role for reasoning benchmarks specifically: when end-to-end agentic tasks fail, one reason is failure to reason through a sub-problem — so reasoning evals still diagnose why agents break. ^[inferred: the “so read task-specific reasoning evals to diagnose agent failure” operational takeaway condenses Burnham’s closing argument.]
The standout finding: AI barely learns from experience (EBR-bench)
The single most actionable result in this batch. Epoch’s EBR-bench (launched July 8, 2026) tests whether models improve at a hard task by attempting it repeatedly and learning from mistakes — using the complex cooperative board game Earthborne Rangers. Epoch calls this “one of the biggest open questions in AI capabilities right now, with large economic and safety implications.”
The result so far: little evidence of AI learning from experience. EBR-bench is positioned as a detector for if and when that changes.
Why it matters for building agents: a common assumption in agent-loop design is that re-running a task, or letting an agent iterate against feedback, yields steady self-improvement. EBR-bench is direct evidence that current models don’t reliably do this on genuinely hard tasks yet. Design loops with external verification and explicit state, rather than trusting the model to get better by retrying. ^[inferred: the agent-loop design implication is drawn from the EBR-bench finding, not stated by Epoch.] This complements the wiki’s verification-first material.
MirrorCode: “runtime hill-climbing” on precisely-specified tasks
The podcast’s deep-dive (Adamczewski built it). MirrorCode asks a model to re-implement an existing program — given only its documentation and a black-box reference binary to query, never the source — so difficulty scales smoothly from ls-sized utilities up to Linux-kernel-sized projects.
- The hardest task the best models can definitely complete is CommonMark (the Markdown-to-HTML spec; ~16,000 lines of C) — which Adamczewski guesses would take an unassisted expert multiple weeks.
- With hundreds of millions to ~1 billion tokens per run, top models reliably handle everything up to ~100K-line programs, though not yet the very hardest tasks in the suite.
- The reason precisely-specified tasks fall: when the target output is exactly defined, models can “keep going at it for many times the size of their context window with compaction” and know where they stand — runtime hill-climbing. (Compare Nicholas Carlini’s Anthropic C-compiler experiment, stopped at ~$20K of compute on gut feel that progress was stalling.)
- The lesson Adamczewski draws: even when you set the bar high accounting for expected AI progress, “you might still be underestimating it” — several tasks he built to be hard were saturated before release.
The corollary (Burnham, in the RIP piece): the harder end of long, precisely-specified SWE — “a C compiler as efficient as GCC” — may still be unsaturated and worth testing.
FrontierMath: audits and the moving numbers
- GPT-5.5 Pro (April 2026) set FrontierMath records: 52% on Tiers 1–3 (up from 50%) and 40% on Tier 4 (up from 38%), and — across runs with GPT-5.5 — solved two Tier-4 problems no model had solved before. It also hit a then-record 159 on the Epoch Capabilities Index.
- FrontierMath: Tiers 1–4 v2 (June 12, 2026) followed an audit that “addressed small but critical errors in 42% of problems” in the original benchmark. Rankings stayed similar but scores rose across the board, with Fable 5 topping at 87% (Tiers 1–3) and 88% (Tier 4).
- Epoch also runs FrontierMath: Open Problems — genuinely unsolved research math with programmatic verification — the “superhuman target” family in the recipe above.
Takeaway: benchmark numbers move not only as models improve but as the benchmarks themselves are corrected. Cite the version, and don’t over-index on a single-point score.
Try It
- Read domain-specific benchmarks, not one headline number, when choosing a model. Epoch’s own cut shows Claude overperforms on SWE and underperforms on math relative to its general index — exactly the kind of task-shape signal a single leaderboard score hides. Match the benchmark to your task type.
- Discount saturated benchmarks — and know which ones still separate frontier models. Short, text-only, auto-graded reasoning benchmarks (GPQA-style) no longer discriminate at the top. The live signals are multimodal, long-horizon, and hard-to-grade evals — see the recipe table.
- Design agent loops assuming the model won’t self-improve by retrying. EBR-bench says learning-from-experience is largely absent today. Build in external verifiers and explicit state rather than trusting iteration to converge. See The Verification Frontier.
- Anchor “can it build the whole thing unattended?” on MirrorCode. Precisely-specified rebuilds up to ~100K lines are within reach at large token budgets; fuzzy, underspecified real-world work is not. Spec tightly if you want long-horizon autonomy to work.
- Cite the benchmark version. After the FrontierMath v2 audit (42% of problems corrected), any pre-v2 FrontierMath figure is stale.
Related
- MirrorCode (Epoch AI + METR) — the long-horizon coding benchmark discussed at length here; this article is its conceptual home.
- Terminal-Bench — a hard short-horizon terminal-agent benchmark; complementary axis to MirrorCode’s multi-day builds.
- Are Mythos’ Cyber Capabilities Overhyped? — Epoch’s Cyber-ECI is the “aggregate many benchmarks, watch for saturation” discipline applied to one domain.
- Stanford HAI 2026 — Chapter 2 (Technical Performance) — the AI Index’s benchmark-scaling view; same saturation dynamic from the survey side.
- Claude Fable 5 + Mythos 5 — source of the GraphWalks-80% and FrontierMath-v2 top scores cited here.
- The Verification Frontier — the “hard-to-grade output” relaxation and the EBR-bench finding both point at verification as the load-bearing problem.
- The Compute Economics of the AI Buildout — sibling Epoch consolidation from this same recovered batch.
- AI Industry Research — topic hub; Epoch AI is a named benchmark-quality anchor.
Open Questions
- When (if ever) will AI start learning from experience? EBR-bench exists precisely because this is unresolved; the current read is “not yet,” and Epoch built the benchmark to catch the transition.
- Will the harder end of MirrorCode (a GCC-efficient C compiler, Linux-kernel-scale rebuilds) saturate? Adamczewski explicitly declines to predict; it’s the open test of whether precisely-specified SWE has a ceiling.
- Can hard-to-grade benchmarks (CRUX, conference-paper/law-review/short-story tests) be made rigorous and repeatable? The IMO-judging precedent worked, but generalizing “plug into existing human judging practices” is unproven at scale.
- How much do benchmark audits keep moving the numbers? FrontierMath’s 42%-error correction suggests published scores carry more construct noise than headline precision implies.