Self-Improving Agent Loops — GAPA, ACE, and the Question of Who Judges

Source: wiki synthesis: Hermes GAPA — The Self-Improvement Loop, Fable 5 Memory Loop — ACE Pattern, Verifier-First Loops, When AI Builds Itself — Recursive Self-Improvement

Independently built self-improvement architectures keep converging on one shape: the agent generates work, assesses its own trajectory, persists what it learned as an editable artifact, and applies that artifact on the next run.^[inferred — the generate → assess → persist → apply naming is this article’s synthesis; each stage traces to the sources below] Nous Research’s GAPA loop in Hermes, the ACE pattern productionized in Claude Fable 5’s file-based memory, and the loop Anthropic runs on its own engineering org are three instances of that shape — and the verifier-first discipline from the agent-loops topic turns out to be the answer to the question that most separates them: who judges what gets persisted? This article compares the three, maps where each fails, and orders what a practitioner should copy first.

Key Takeaways

• One shape, independently converged. GAPA: trajectory capture → self-review every ~15 tool calls → autonomous SKILL.md creation → memory update → compounding reuse. ACE (Stanford/SambaNova/Berkeley, ICLR 2026): Generator → Reflector → Curator → permanent playbook. Anthropic internal: Claude writes >80% of merged code, a Claude judge rules success (76% on the most open-ended tasks, May 2026), and an automated Claude reviewer runs on every change. Same loop, three stacks.^[inferred — the alignment across the three is this article’s mapping]
• All three bet on weight-free improvement. Nothing here touches model weights: GAPA is “like back propagation but for prompts instead of model weights”; ACE keeps “frozen weights + evolving notes.” The improvement lives in human-readable, versionable, deletable artifacts — a bad lesson costs a delete, not a fine-tuning run. (Hermes is the partial exception: its trajectories also feed offline RL training via the ML Research Pipeline, so weights re-enter downstream.)
• The architectures differ most on what persists. Hermes persists executable artifacts — skill files, cron wiring, a Honcho user model, FTS5-indexed trajectories. ACE/Fable 5 persists natural-language lessons in one curated file. Anthropic’s org-level loop persists reviewed, merged code plus a growing automated-review layer.
• The judge is the fork in the road. GAPA’s assess stage is self-review — the same agent reads its own trajectory and writes to disk autonomously. ACE separates Reflector from Curator and its operator playbook forbids letting the generator also be the curator. Verifier-first goes furthest: proof must sit outside the agent entirely (tests, screenshots, artifacts), ideally checked by a different model family. A self-report-only judge is “two optimists agreeing.”
• The failure modes concentrate at the assess/persist boundary.^[inferred] Memory accumulation alone cut refusal rates 70-86% (Misevolution, ICLR 2026); one bad ACE curation step collapsed a playbook from 18k to 122 tokens and accuracy from 66.7% to 57.1%; Hermes produces half-working duplicate skills (morning-briefing and morning-briefing-1) and LLM-generated skills that fail in cron; unverified loops reward-hack and declare “done” early.
• Copy order for practitioners: a curated lessons file with human curation first, the four-line verifier checklist before any autonomous persistence, role separation (generator ≠ curator ≠ verifier), then the packaged full-stack version (Hermes) if you want it turnkey.

The convergent shape, mapped

Stage	Hermes GAPA	ACE / Fable 5 memory	Anthropic internal (RSI essay)
Generate	every API call, tool decision, and output recorded to sessions.json + state.db	a run produces an outcome	Claude authors >80% of merged production code (May 2026)
Assess	GAPA review every ~15 tool calls: what worked, what failed	Reflector evaluates what worked and why	a Claude judge rules success without corrections (76% on most open-ended tasks); automated Claude reviewer on every change — retrospectively would have caught ~1/3 of bugs behind past claude.ai incidents
Persist	writes SKILL.md to ~/.hermes/skills/, updates MEMORY.md, USER.md, user model	Curator decides which lessons enter the permanent playbook file	reviewed code merges; the review/eval layer accretes
Apply	next similar request runs the existing skill and refines it on new feedback	future run reads the file before acting	engineers merged 8× as much code per day in Q2 2026 vs 2024, directing and reviewing rather than writing

The table’s column alignment is this article’s construction^[inferred]; every cell traces to its source article. Two corroborating details: both agent-level loops fire selectively — GAPA only engages on complex tasks (trivial requests bypass it), and ACE’s measured gains (+10.6% agent tasks, +8.6% finance reasoning, zero weight updates) came from repeated non-trivial work. And capability amplifies the loop: Anthropic’s internal eval showed memory helped Fable 5 ~3× more than Opus 4.8 — stronger base models extract more from the same notes (single vendor-run eval on Slay the Spire, not independently replicated).

Who judges — the load-bearing difference

• GAPA: the generator grades itself. Stage 2 has the agent pause, read back its own trajectory, and decide what to persist — then write prompts, memory, and skills to disk autonomously. The guards are human and after-the-fact: memory nudges you approve or reject, hermes skills inspect <name> before trusting a skill in cron, and Docker (not local + /yolo) as the production backend. Notably, the strongest public GAPA proof — the v0.8.0 self-diagnosis that patched 5 tool-calling failure modes — worked “through automated behavioral benchmarking,” i.e., it leaned on an objective signal, not pure self-report.
• ACE: roles are split by design. Generator produces, Reflector evaluates, Curator admits lessons — and the operator playbook makes the split explicit: ground-truth feedback in the loop (without a reliable success signal “confident drift follows”) and human judgment in curation (“don’t let the generator also be the curator”). The curator role is load-bearing: the paper’s own collapse case came from a single bad curation step.
• Verifier-first: the judge lives outside the agent. Write the verifier before the loop runs — what counts as done, which checks run every pass, which artifact gets saved, which failure retries. Proof is tests, screenshots, benchmark curves, changed files — never the agent’s own explanation. @omarsar0’s practitioner formula even splits model families (one plans, one executes, a third evaluates) because “a separate evaluator model is harder for the maker to fool.” Karpathy’s rule is the ceiling on all of this: “If you can’t evaluate then you can’t auto research it.”
• Anthropic at org scale: LLM-as-judge plus a human bottleneck. Success on open-ended tasks is ruled by a Claude judge; humans still supply goals (“humans supply the goal, not the method”) and review — and human review has already become the Amdahl’s-law bottleneck as generation outpaces checking.

The synthesis: the assess stage is where every named failure mode in these sources concentrates — memory poisoning and notes rot are curation failures, reward hacking is a gamed judge, Hermes’s broken-skill and duplicate-skill issues are unreviewed persistence.^[inferred] The architecture you pick matters less than where you put the judge.

Failure modes, by family

• Memory-poisoning family (ACE/Fable 5): Misevolution — memory accumulation alone, no adversary, reduced refusal rates 70-86% across top models; alignment tipping — small deviations written into memory become compounding precedent; notes rot — a more capable model writes more persuasive bad notes; curation collapse — 18k → 122 tokens from one bad step.
• Autonomous-persistence family (Hermes): LLM-generated skills can fail (inspect before cron); skill naming conflicts produce overlapping half-working skills; a recurring stuck-loop bug; context bloat in long sessions; and a real security surface — GAPA edits prompts and writes to disk without a human in the loop.
• Self-grading family (verifier-first’s targets): models “pause the work early,” take “weird shortcuts (reward hacking),” and self-report success — the Verifier Theater configuration where the loop checks itself with no external standard. See Reward-Hacking and the Verification Frontier for the case that this worsens as models get stronger.
• Scale family (Anthropic): the loop works — and then review becomes the bottleneck (Amdahl’s law), while the remaining gap is goal-choice judgment, which none of these loops persist their way around.

What to copy first

1. Start with the ACE-style file, not the full stack. A writable Markdown lessons.md — one lesson per entry, one-line summary at top, delete-rather-than-duplicate — is the smallest version of the pattern (the Prompting Claude Fable 5 guide operationalizes exactly this setup).
2. Write the judge before you allow autonomous persistence. The four-line verifier-first checklist (done condition, per-pass checks, saved artifact, failure-retry path) applied to the persist step: no lesson or skill is committed until a verifier with access to real outcomes approves it.^[inferred — verifier-first states this for loop output; extending it to memory writes is this article’s bridge, matching the ACE article’s own Try It advice]
3. Separate the roles. The generator writes candidate lessons; a verifier (different model family, real outcomes, no access to the maker’s reasoning) decides which live. “The notes are the product now. Curate them like it.”
4. Audit on a cadence. Review the lessons file after 3-5 runs and delete any entry you can’t trace to an outcome; run hermes skills audit periodically if you’re on Hermes; expect the first 7 days to be rough — the loop needs runs to learn from.
5. Go turnkey only after the discipline is in place. Hermes ships the whole shape (capture, review, skills, memory, user model) preconfigured — valuable, but its judge is the generator, so the inspection habits above are not optional there.

Try It

• Add a lessons.md to any long-running agent today; after 3-5 runs, delete every lesson you cannot trace to a concrete outcome.
• Before enabling any autonomous skill/memory write, fill in the four-line checklist — if you can’t name the check and the artifact, keep persistence manual.
• On Hermes: run a repeating workflow for 7+ days, watch /skills grow, and hermes skills inspect <name> before any skill enters cron.
• Read ACE (arXiv 2510.04618) §4 if you’re building a production memory system — the curation failure mode is the most important section, per the Fable 5 memory article.

Related

• Hermes GAPA — The Self-Improvement Loop — the packaged, autonomous end of the spectrum: self-review, autonomous disk writes, human inspection as guard.
• Fable 5 Memory Loop — ACE Pattern — the curated-notes end: Generator/Reflector/Curator, misevolution and notes-rot risks, the operator playbook.
• Verifier-First Loops — the discipline that answers “who judges”: proof outside the agent, separate evaluator model, checklist before autonomy.
• When AI Builds Itself — the lab-scale evidence the loop compounds, and where it bottlenecks (review, goal choice).
• The Verification Frontier — why the assess stage gates everything: self-improvement compounds only where verification is cheap.
• Reward-Hacking and the Verification Frontier — the caveat: a cheap judge is also a gameable judge, and it worsens with capability.
• Reflexio — an external harness applying the same harvest-from-trajectories pattern to any agent.
• Prompting Claude Fable 5 — the official scaffolding for the memory-file setup this article recommends starting with.

Open Questions

• No head-to-head exists. Nobody has benchmarked GAPA-style executable-skill persistence against ACE-style lesson files on the same task suite — the sources measure each in isolation.^[inferred]
• Does Hermes’s self-review suffer measurable misevolution? The 70-86% refusal-rate finding covers top models generally; no Hermes-specific memory-poisoning measurement appears in these sources.
• Fable 5’s ~3× memory advantage is a single vendor-run eval (Slay the Spire) and the ACE article’s key risk paper was not independently verified at ingest — both flagged medium-confidence in the source.