Source: ai-research/claude-blog-cognition-devin-overnight.md — Anthropic first-party “Working at the Frontier” customer case study, claude.com/blog/working-at-the-frontier-how-cognition-trusts-claude-fable-5-to-work-through-the-night (published July 10, 2026; fetched 2026-07-16). Byline source: Silas Alberti, SVP of Research at Cognition. Part of the same case-study rollout as the sibling posts on Hebbia and Thomson Reuters.
Cognition builds Devin, an autonomous AI software engineer, and Silas Alberti’s team has run nearly every Claude model generation inside it since Devin’s build in early 2024. His verdict on Claude Fable 5: the first model his team would trust to run completely unattended overnight. This article covers Cognition’s “trust no eval” philosophy, its own “Frontier Code” anti-slop benchmark (where Fable 5 roughly tripled the prior Opus model’s score on the hardest subset), and the concrete eight-hour unattended session that changed the team’s mind.
Key Takeaways
- “We trust no eval.” Cognition has watched models ace public benchmarks and then fall apart the moment engineers actually used them. Its adoption bar is dogfooding: highest-taste developers put each new model through a real day of work, and the test is whether the resulting code is something they’d actually keep.
- Frontier Code, Cognition’s own “anti-slop” benchmark, exists because public benchmarks kept rewarding code that passes tests but wouldn’t survive a real codebase. On its hardest subset, the prior Opus model scored around 10%; Claude Fable 5 scored about 30% — roughly a 3x jump the team initially suspected was a bug.
- The real story is task horizon, not raw benchmark score. “Before Fable, you could delegate agents that could stay on-task for a couple of minutes, maybe an hour” before sessions drifted or introduced subtle bugs. With Fable 5, Alberti told it to keep working overnight and “it’s been working for eight hours straight and actually making real progress. I hadn’t seen that before.”
- Fable 5 was the first model to properly use Cognition’s internal debugging tools — paging through logs in the browser and drawing conclusions despite noise, stating invariants before executing against them on a migration that had tripped up earlier models, and on incident triage, pinning down the root cause while explicitly saying what it didn’t know.
- Devin’s founding bet — agents running in the cloud for hours at a time — is now viable. Devin already watches Slack channels and jumps into issues unprompted, or monitors production and triages spikes autonomously. Alberti expects 90% of agent sessions to be proactive (find a problem, scan the codebase, message the fix) within a year or two.
Where earlier models hit their limit
Cognition built Devin in early 2024, when “the basic mechanics of an agent barely held together.” Devin takes on the work engineers never quite get to — codebase migrations, bug backlogs, features that keep slipping — for customers ranging from high-growth startups to Fortune 500 companies, where a small bug introduced quietly can cause real downstream problems.
Alberti traces the first real capability jump to Claude 3.6 Sonnet in late 2024 — the first model that could reliably chain tools and hold a multi-step task, which tripled Devin’s internal usage when the team plugged it in. That history is exactly what makes him hard to impress: “We’ve been burned like this a bunch of times” by models that ace a benchmark and then fall apart in practice.
Even with that progress, one ceiling remained: how long an agent could run before losing the thread. Give an earlier model five ideas to weigh at once and it would lose track and get confused; on one database migration, a prior Opus model technically finished the job but introduced a series of subtle bugs along the way. Incident triage showed the same shape — earlier models stayed at the surface of the logs instead of digging for the relevant line, and were trained to give an answer no matter what, so they’d “confidently claim the first plausible thing they discover and then stop.” Engineers learned to tune them out.
Claude Fable 5 clears Cognition’s own bar
Cognition grades models on Frontier Code, built specifically because existing benchmarks kept rewarding code that passed tests but wouldn’t survive a real codebase — Alberti calls it an “anti-slop” standard. On its hardest subset, the prior Opus model scored around 10%; Claude Fable 5 scored about 30%.
The team’s first reaction was suspicion — “Is there a bug? This can’t be true.” Normally a benchmark jump triggers weeks of engineer debate over whether the model is actually better in practice. This time the dogfooding agreed with the numbers immediately: “It was kind of a shocker, honestly.”
“The biggest thing we noticed was the horizon, how long it can be self-sufficient,” Alberti says. “There have been tasks where I was about to go to bed and I was like, ‘Okay, just please keep working on this and don’t stop until I wake up.’ And then I wake up, and it’s been working for eight hours straight and actually making real progress.” The horizon held because Fable 5 stayed clear-headed in messy context: it was the first model to properly use Cognition’s internal debugging tools (paging through browser logs despite noise), stated the invariants it would hold itself to on a migration that had tripped up earlier models before executing against them, and on triage pinned down the root cause while saying what it didn’t know — which Alberti says is what actually rebuilds engineer trust. He places the jump in a small class of true step changes that come roughly once a year.
What’s next
Cognition’s founding bet was that agents should run in the cloud for hours at a time — a bet the models weren’t ready for during the company’s first year. Alberti says Fable 5 makes the full version of that bet viable, and some of it already ships: Devin can watch a Slack channel and jump into an issue without being tagged, or monitor production and triage a spike on its own. When it gets one of those right, “it feels like a real engineer on the team.” He expects this to become the default for engineering teams — in a year or two, 90% of agent sessions will be proactive ones that find a problem, scan the codebase, and message a fix. “A lot of these things we’ve always wanted to build at the company are now possible.”
Try It
- Build your own “anti-slop” benchmark on your hardest, most representative subset of real work rather than relying solely on public leaderboards — Cognition’s Frontier Code is a concrete template: reward code that survives a real codebase, not code that merely passes tests.
- Test long-horizon trust incrementally. Alberti’s team didn’t jump straight to unattended overnight runs — the trust built from watching the model hold invariants and self-report uncertainty on smaller tasks first.
- Treat “says what it doesn’t know” as a trust signal, not a weakness. Cognition explicitly contrasts this with earlier models that were trained to always give an answer — the model that admits uncertainty on incident triage is the one engineers learn to actually rely on.
Open Questions
- No specific customer names or case counts are given for Devin’s proactive Slack-monitoring / production-triage features — unclear how widely deployed vs. still emerging.
- The “90% of agent sessions will be proactive within a year or two” figure is Alberti’s own forward-looking estimate, not a measured statistic — treat as a prediction, not a current metric.
- Frontier Code’s full methodology (task count, scoring rubric, how “anti-slop” is operationalized beyond the general description) is not published in this post.
Related
- Claude Fable 5 and Claude Mythos 5 — the main model-release article this case study substantiates (the topic index’s brief “Cognition FrontierCode” benchmark mention this article expands into full detail, including the named 10%→30% jump and the overnight-session anecdote).
- Working at the Frontier: How Hebbia Builds AI for Financial Diligence — sibling case study in the same series: another domain expert running its own benchmark against a new Claude release.
- Working at the Frontier: How Thomson Reuters Builds AI for High-Stakes Professional Work — sibling case study; Thomson Reuters’ “hold steady across long chains of tool calls” requirement is the same long-horizon-reliability theme from a legal-AI angle.
- Dynamic Workflows — the wiki’s other major long-horizon-autonomous-agent proof point (Jarred Sumner’s Bun Zig→Rust rewrite, ~50 workflows / 64 parallel Claudes over 11 days); Devin’s single-agent overnight run is the complementary “one agent, many hours” shape against that piece’s “many agents, one goal” shape.
- The Verification Frontier — “we trust no eval” and “said what it didn’t know” are both direct, named instances of this thesis: invest in your own verification, don’t trust the benchmark score at face value.
- Picking the Right Model — Cognition’s dogfooding-over-leaderboard methodology is a concrete real-world example of this article’s general framework.
- Cursor — AI-Native Code Editor (Anysphere) — already documents that Cognition (Devin’s maker) acquired Windsurf in July 2025, placing Cognition in the wiki’s agentic-IDE competitive landscape; no reciprocal link back to this new article exists there yet (see summary for suggested backlink).