Source: raw/newsletter-epoch-ai-d6ff09ad19.md (“We tried to automate Epoch”, Kelly Hong, 2026-10-26; summary of https://epoch.ai/publications/can-ai-automate-epoch, not fetched) · raw/newsletter-epoch-ai-c502e4d083.md (The Epoch Brief, October 23, 2026)

Creator: Epoch AI (researchers Kelly Hong and Greg Burnham) · URL: https://epochai.substack.com/p/can-ai-automate-epoch · Platform: Epoch AI Substack

Epoch AI gave six leading models 11 tasks taken from its own staff’s work and graded the output against the rubric it uses for human employees. The best models, Claude Fable 5.1 and GPT-6 Astra, met 65% of rubric items on average, and no model reliably met Epoch’s bar; in the researchers’ words, current models “cannot yet replace workers, at least not at Epoch.” The useful part for anyone deciding what to hand to AI is the method: score a task against the standard you hold people to, and automate it only when it passes.

Key Takeaways

  • Setup. 11 tasks across five areas: graphic design, short research reports, interactive data tools, AI data-centre research and research design. Each model got roughly the context a new hire would, its own workspace and Google account, one attempt per task and no human help.
  • Result. Fable 5.1 and GPT-6 Astra roughly tied at 65%. Frontier closed models did best on well-defined work (coding, data analysis) and worst on open-ended judgment (designing informative experiments, interpreting results, picking up a house style from examples). Open-weight models struggled even on well-defined tasks.
  • 65% is a failed task. Epoch stresses that 65% of rubric items does not mean 65% of the way to automation: the passed items may be the easy ones, and every output “would be rejected or revised if a human had submitted it.”
  • Failure examples. Running a pilot experiment, GPT-6 Astra set the token budget too low for the models it was testing, so 22% of runs returned nothing, then reported those failures as a finding about model capability rather than a setup bug. Asked to design a chart in Epoch’s style from many examples, Fable 5.1 produced “a cluttered graph crammed with too much information”; an Epoch designer produced a clean, simple chart.
  • One task crossed the line. Epoch dropped a basic chart-generation task from the set because Fable 5.1 produces an acceptable chart in one try most of the time, and now partly automates that step in daily work.
  • Why it matters for the “productivity paradox.” Epoch’s explanation for benchmarks rising faster than economic impact: benchmarks measure progress against past models, favour easily graded tasks and get trained on. Partial-quality output still needs a person to find and fix every mistake, so impact arrives task by task as each one crosses 100%.
  • Earlier baseline. In February 2026 Epoch’s Anson Ho tried to automate his own job and the models failed every task; he forecast that would change by the end of the decade.

Try It

Run a small version of Epoch’s test on your own team’s recurring work.

  1. Pick 5-10 recurring tasks you would give a new hire (a weekly report, a client chart, a research brief).
  2. Write the rubric you already use to judge a person’s version. Make each item pass/fail.
  3. Give the model new-hire context only: the brief, examples of past work, access to the same files. One attempt, no coaching.
  4. Score it item by item. Anything below 100% stays a draft that a person fixes; anything at 100% on repeated runs is a candidate to automate.
  5. Rerun when a new model ships. Epoch’s chart task is the pattern: one task at a time crosses the threshold.
  6. Watch for self-reported findings. Astra’s token-budget error shows a model can turn its own mistake into a confident conclusion; check setup before believing a result.

Open Questions

  • The full report’s per-task scores, the four other models tested and the rubric items are not in the newsletter summary.
  • One attempt per task with no feedback understates what a model does with a human reviewing drafts; Epoch does not report that setting here.