Harness Design for Long-Running Agents — Anthropic's Two-Part Engineering Series

Source: ai-research/anthropic-engineering-effective-harnesses-long-running-agents.md (anthropic.com/engineering/effective-harnesses-for-long-running-agents, published Nov 26 2025, written by Justin Young) + ai-research/anthropic-engineering-harness-design-long-running-apps.md (anthropic.com/engineering/harness-design-long-running-apps, published Mar 24 2026, written by Prithvi Rajasekaran, Anthropic Labs) — a connected two-part series; the second post explicitly builds on and extends the first. Surfaced via claude-ai/_research-agenda.md: building-effective-human-agent-teams named a “Doer-Verifier” harness as Anthropic’s trust mechanism for long-running agents, but direct inspection of that article’s own cited primary source found no actual outbound link — the claim was corrected there. Neither post uses the term “Doer-Verifier.” The second post’s own term is generator-evaluator, explicitly inspired by GANs. “Doer-Verifier” is best read as a plain-language gloss on the same underlying pattern, not a quotation from either post.

Two Anthropic engineering posts, four months apart, telling one evolving story: how do you keep an AI agent making real progress on a task too large for one context window, across many hours, without a human checking every step? The first post solves the memory problem — an initializer agent scaffolds the environment once, a coding agent picks up where the last one left off using git history and a progress file. The second solves the judgment problem the first one didn’t touch: agents evaluating their own work “confidently praise” it even when mediocre, so a separate evaluator agent — skeptical by design, tuned with few-shot examples — checks the work instead. Together they’re the closest thing Anthropic has published to a reusable engineering blueprint for the loop-engineering pattern this topic otherwise covers from the outside.

Key Takeaways

• The core problem (Nov 2025): every new context window starts with zero memory of the last one — like an engineer starting a shift with no handoff notes. Two failure modes: trying to one-shot the whole project (running out of context mid-feature, undocumented), and later sessions looking around, seeing partial progress, and declaring the job prematurely done.
• The fix: an initializer agent (once) + a coding agent (every session after). The initializer writes init.sh, a claude-progress.txt log, an initial git commit, and — the load-bearing artifact — a feature list as JSON (200+ granular features for a claude.ai clone, each starting "passes": false). JSON specifically because “the model is less likely to inappropriately change or overwrite JSON files compared to Markdown files.” The coding agent works one feature at a time, commits to git with descriptive messages, and self-verifies before marking anything “passing.”
• Testing was its own separate failure mode. Claude marked features complete after unit tests or curl checks without verifying end-to-end. Fix: browser automation (Puppeteer MCP), testing “the way a human user would.” Named residual gap: Claude can’t see native browser alert/modal dialogs through Puppeteer MCP, so features relying on them stayed buggier.
• The second post’s real discovery: self-evaluation doesn’t work, even for models otherwise good at the task. “When asked to evaluate work they’ve produced, agents tend to respond by confidently praising the work — even when, to a human observer, the quality is obviously mediocre.” Worst on subjective judgment calls (design taste), but present even on objectively-checkable work.
• The fix isn’t eliminating leniency — it’s relocating it. A standalone evaluator LLM is still inclined to be generous toward LLM-generated output, but “tuning a standalone evaluator to be skeptical turns out to be far more tractable than making a generator critical of its own work.” Few-shot calibration examples with detailed score breakdowns reduced score drift across iterations.
• GAN-inspired generator/evaluator, proven first on frontend design (where “is this good?” resists a binary check) — the Dutch-art-museum example is the sharpest evidence: iteration nine produced a competent, expected landing page; iteration ten scrapped it for a CSS-perspective 3D gallery room, a creative leap the author hadn’t seen from single-pass generation.
• Scaled to a three-role planner → generator → evaluator architecture for full-stack coding. The Playwright-MCP-driven evaluator caught bugs at the level of exact code lines (LevelEditor.tsx:892 — a delete handler requiring two conditions when only one gets set on click). A sprint contract negotiated before any code is written bridges the gap between an intentionally high-level spec and a testable unit of work.
• The concrete cost/quality tradeoff, with real dollar figures. Solo run (20 min, $9):entitiesonscreendid{n}^{\prime }trespondtoinput,withthebrokenwiringundiagnosablefromtheUI.Fullharness(6hr,$200 — over 20x the cost): a mostly-working game with granular, directly-actionable Playwright bug reports.
• Newer, more capable models change which harness pieces are load-bearing — they don’t make the exercise obsolete. Opus 4.6’s improvements let the author drop the sprint-decomposition and context-reset machinery entirely (Opus 4.5 needed both to fight “context anxiety” — wrapping up work prematurely near the perceived context limit). The evaluator became conditional, not fixed: valuable exactly where a task sits past what the model does reliably solo, unnecessary overhead where it doesn’t. Rebuilt V2 harness: 4 hours, $124.70, broken into a real per-phase cost table.
• The closing thesis, worth carrying to any harness a reader builds: “the space of interesting harness combinations doesn’t shrink as models improve. Instead, it moves” — re-examine a harness at every new model release, stripping pieces no longer load-bearing and adding new ones for capability that wasn’t there before.

Part 1 (Nov 2025) — Solving for Memory

The Claude Agent SDK’s compaction keeps a single session going, but it doesn’t solve the cross-session handoff problem — and even Opus 4.5, given only a high-level prompt, would fail to build a production-quality app across multiple context windows. The fix decomposes into exactly two roles, formalized in a four-row failure-mode table: a feature-list JSON file (stops premature “done” declarations), a progress-notes-file-plus-git-log reading ritual at the start of every session (stops undocumented bug accumulation), self-verification against the feature list before marking anything passing (stops premature completion claims), and an init.sh script (stops wasted time re-deriving how to run the app). Every coding agent session runs the same three-step orientation ritual: pwd, read git logs + progress file, read the feature list and pick the highest-priority undone item — described as a token-saving convention, since the agent doesn’t have to rediscover how to test the code from scratch.

Part 2 (Mar 2026) — Solving for Judgment

Two lessons carried forward directly: decompose into tractable chunks, hand off context via structured artifacts. Two failure modes get new treatment. Context anxiety — a Sonnet-4.5-specific tendency to wrap up prematurely near the perceived context limit — is solved with context resets: a deliberate full teardown-and-rebuild-from-handoff-file, explicitly distinguished from compaction (which preserves continuity but not a clean slate, so anxiety can persist through it). Self-evaluation leniency is the post’s real contribution: agents grading their own work skew positive regardless of actual quality, and the fix is architectural, not a better self-review prompt — split the doer from the judge.

Applied first to frontend design (four grading criteria weighted toward design/originality over craft/functionality, since Claude already scored well on the latter by default; a Playwright-MCP-equipped evaluator navigating and screenshotting the live page before scoring), then scaled to full-stack coding via a three-agent planner → generator → evaluator architecture with sprint contracts negotiated up front. The V1 harness (Opus 4.5, sprint-decomposed, context-reset-dependent) cost $200over6hoursagainsta$9/20-minute solo baseline — 20x the cost for a game that actually worked versus one that didn’t. When Opus 4.6 shipped, the author methodically stripped the harness down one component at a time rather than guessing: the sprint construct and context resets came out entirely, the evaluator moved from per-sprint to a single end-of-run pass, and its value became conditional on how close the task sits to the model’s own reliable-solo boundary. The rebuilt V2 harness ran a Digital Audio Workstation build in 3hr50min for $124.70,withcostbrokendownbyexactagent-and-phase(Planner$0.46 → Build Round 1 $71.08\to QARound1$3.24 → …).

Try It

• Start with the Nov 2025 four-row table before reaching for a multi-agent architecture. A feature-list JSON + progress file + git-commit discipline solves a large share of “the agent lost the plot” problems on a single-agent harness, no evaluator required.
• Reach for a separate evaluator specifically when the check is subjective (“does this look good?”) or requires interacting with a running system a static code read can’t verify (UI bugs, API wiring) — not as a default bolt-on to every harness.
• If you’re on a newer, more capable model than whatever your harness was tuned against, re-examine which pieces are still load-bearing before assuming you need the full stack. Per the Mar 2026 post’s own experience, a model upgrade can retire context resets and demote the evaluator from mandatory to conditional.
• Cross-reference Verifier-First Loops for a third-party, checklist-style companion to this same “don’t let the agent grade its own homework” principle — useful as a quick-reference alongside this article’s deeper engineering account of why it fails and how to fix it.

Open Questions

• Whether “Doer-Verifier” is used verbally or internally at Anthropic distinct from these two posts’ own “generator-evaluator” terminology is unconfirmed — flagged, not resolved, since neither published post uses the phrase.
• The Mar 2026 post doesn’t specify whether the sprint-contract mechanism or the specific JSON grading-criteria format are available in any packaged/reusable form (a skill, a template repo). The Nov 2025 post at least links a public quickstart repo (github.com/anthropics/claude-quickstarts/tree/main/autonomous-coding); no equivalent is given for the newer three-agent harness.
• Cost figures ($200fortheV1fullharness,$124.70 for V2) are for Opus-tier models at the time of writing — no indication of what the same harness costs against Sonnet- or Haiku-tier models, or against Fable 5 / Opus 4.8.

Related

• Building Effective Human-Agent Teams — the article whose “Doer-Verifier” reference (now corrected — no outbound link existed in its actual source) led to this ingest.
• Verifier-First Loops — a third-party (alphabatcher/omarsar0), checklist-shaped take on the same “verification outside the agent” principle; this article is the deeper first-party engineering account of the same discipline.
• Loop Engineering: Getting Started with Loops — Anthropic’s own turn-based/goal-based/time-based/proactive taxonomy; the initializer+coding-agent and planner+generator+evaluator patterns here are worked examples of what a goal-based or proactive loop looks like in engineering practice.
• Claude Agent SDK — How the Agent Loop Works — the SDK mechanics (compaction, message lifecycle) both harnesses in this article are built on top of.
• Headroom — a different, complementary answer to the same context-window-exhaustion problem this article’s context-reset-versus-compaction section addresses.
• Should You Build a Loop? — the cost-math companion; this article’s $9-vs-$200-vs-$124.70 real dollar figures are a concrete data point for that article’s abstract cost-tradeoff framing.
• Dynamic Workflows — the parallel-worktree-plus-judge mechanism is a sibling pattern to this article’s sequential generator-evaluator loop.