Source: wiki synthesis: Hermes GAPA — The Self-Improvement Loop, Fable 5 Memory Loop — ACE Pattern, Verifier-First Loops, When AI Builds Itself — Recursive Self-Improvement

Refreshed 2026-10-27 from: raw/Last_Week_in_AI_258_-_Opus_5.5_Sol_and_Luna_Muse_DeepSeek-V4.1-Flash_Xi.md (https://www.youtube.com/watch?v=ZhNGG5-Brb0) and raw/Last_Week_in_AI_259_-_Dots_Sonnet_Self-Safety_Rogue_AI.md (https://www.youtube.com/watch?v=1gwiZ0EfhM4), Last Week in AI podcast hosts summarising two papers. The papers themselves were not read.

Independently built self-improvement architectures keep converging on one shape: the agent generates work, assesses its own trajectory, persists what it learned as an editable artifact, and applies that artifact on the next run.^[inferred — the generate → assess → persist → apply naming is this article’s synthesis; each stage traces to the sources below] Nous Research’s GAPA loop in Hermes, the ACE pattern productionized in Claude Fable 5’s file-based memory, and the loop Anthropic runs on its own engineering org are three instances of that shape — and the verifier-first discipline from the agent-loops topic turns out to be the answer to the question that most separates them: who judges what gets persisted? This article compares the three, maps where each fails, and orders what a practitioner should copy first.

Key Takeaways

  • One shape, independently converged. GAPA: trajectory capture → self-review every ~15 tool calls → autonomous SKILL.md creation → memory update → compounding reuse. ACE (Stanford/SambaNova/Berkeley, ICLR 2026): Generator → Reflector → Curator → permanent playbook. Anthropic internal: Claude writes >80% of merged code, a Claude judge rules success (76% on the most open-ended tasks, May 2026), and an automated Claude reviewer runs on every change. Same loop, three stacks.^[inferred — the alignment across the three is this article’s mapping]
  • All three bet on weight-free improvement. Nothing here touches model weights: GAPA is “like back propagation but for prompts instead of model weights”; ACE keeps “frozen weights + evolving notes.” The improvement lives in human-readable, versionable, deletable artifacts — a bad lesson costs a delete, not a fine-tuning run. (Hermes is the partial exception: its trajectories also feed offline RL training via the ML Research Pipeline, so weights re-enter downstream.)
  • The architectures differ most on what persists. Hermes persists executable artifacts — skill files, cron wiring, a Honcho user model, FTS5-indexed trajectories. ACE/Fable 5 persists natural-language lessons in one curated file. Anthropic’s org-level loop persists reviewed, merged code plus a growing automated-review layer.
  • The judge is the fork in the road. GAPA’s assess stage is self-review — the same agent reads its own trajectory and writes to disk autonomously. ACE separates Reflector from Curator and its operator playbook forbids letting the generator also be the curator. Verifier-first goes furthest: proof must sit outside the agent entirely (tests, screenshots, artifacts), ideally checked by a different model family. A self-report-only judge is “two optimists agreeing.”
  • The failure modes concentrate at the assess/persist boundary.^[inferred] Memory accumulation alone cut refusal rates 70-86% (Misevolution, ICLR 2026); one bad ACE curation step collapsed a playbook from 18k to 122 tokens and accuracy from 66.7% to 57.1%; Hermes produces half-working duplicate skills (morning-briefing and morning-briefing-1) and LLM-generated skills that fail in cron; unverified loops reward-hack and declare “done” early.
  • Copy order for practitioners: a curated lessons file with human curation first, the four-line verifier checklist before any autonomous persistence, role separation (generator ≠ curator ≠ verifier), then the packaged full-stack version (Hermes) if you want it turnkey.

The convergent shape, mapped

StageHermes GAPAACE / Fable 5 memoryAnthropic internal (RSI essay)
Generateevery API call, tool decision, and output recorded to sessions.json + state.dba run produces an outcomeClaude authors >80% of merged production code (May 2026)
AssessGAPA review every ~15 tool calls: what worked, what failedReflector evaluates what worked and whya Claude judge rules success without corrections (76% on most open-ended tasks); automated Claude reviewer on every change — retrospectively would have caught ~1/3 of bugs behind past claude.ai incidents
Persistwrites SKILL.md to ~/.hermes/skills/, updates MEMORY.md, USER.md, user modelCurator decides which lessons enter the permanent playbook filereviewed code merges; the review/eval layer accretes
Applynext similar request runs the existing skill and refines it on new feedbackfuture run reads the file before actingengineers merged 8× as much code per day in Q2 2026 vs 2024, directing and reviewing rather than writing

The table’s column alignment is this article’s construction^[inferred]; every cell traces to its source article. Two corroborating details: both agent-level loops fire selectively — GAPA only engages on complex tasks (trivial requests bypass it), and ACE’s measured gains (+10.6% agent tasks, +8.6% finance reasoning, zero weight updates) came from repeated non-trivial work. And capability amplifies the loop: Anthropic’s internal eval showed memory helped Fable 5 ~3× more than Opus 4.8 — stronger base models extract more from the same notes (single vendor-run eval on Slay the Spire, not independently replicated).

Who judges — the load-bearing difference

  • GAPA: the generator grades itself. Stage 2 has the agent pause, read back its own trajectory, and decide what to persist — then write prompts, memory, and skills to disk autonomously. The guards are human and after-the-fact: memory nudges you approve or reject, hermes skills inspect <name> before trusting a skill in cron, and Docker (not local + /yolo) as the production backend. Notably, the strongest public GAPA proof — the v0.8.0 self-diagnosis that patched 5 tool-calling failure modes — worked “through automated behavioral benchmarking,” i.e., it leaned on an objective signal, not pure self-report.
  • ACE: roles are split by design. Generator produces, Reflector evaluates, Curator admits lessons — and the operator playbook makes the split explicit: ground-truth feedback in the loop (without a reliable success signal “confident drift follows”) and human judgment in curation (“don’t let the generator also be the curator”). The curator role is load-bearing: the paper’s own collapse case came from a single bad curation step.
  • Verifier-first: the judge lives outside the agent. Write the verifier before the loop runs — what counts as done, which checks run every pass, which artifact gets saved, which failure retries. Proof is tests, screenshots, benchmark curves, changed files — never the agent’s own explanation. @omarsar0’s practitioner formula even splits model families (one plans, one executes, a third evaluates) because “a separate evaluator model is harder for the maker to fool.” Karpathy’s rule is the ceiling on all of this: “If you can’t evaluate then you can’t auto research it.”
  • Anthropic at org scale: LLM-as-judge plus a human bottleneck. Success on open-ended tasks is ruled by a Claude judge; humans still supply goals (“humans supply the goal, not the method”) and review — and human review has already become the Amdahl’s-law bottleneck as generation outpaces checking.

The synthesis: the assess stage is where every named failure mode in these sources concentrates — memory poisoning and notes rot are curation failures, reward hacking is a gamed judge, Hermes’s broken-skill and duplicate-skill issues are unreviewed persistence.^[inferred] The architecture you pick matters less than where you put the judge.

October 2026: evidence that self-improved harnesses overfit

Two papers, as summarised on Last Week in AI. Hosts’ descriptions only; no numbers were read out beyond those below.

  • RRSI (Regularized Recursive Self-Improvement of agent harnesses; Google Cloud AI Research with university collaborators). The question: when an agent improves its own harness, does it get better at new tasks or only the ones it optimised for? They find that “a lot of these recent published methods do in fact overfit,” with “a negative gain on out of distribution tasks, or… very small gain, if anything.”
    • The fix is regularisation: limits on update size (“update sparsity”), screening that blocks certain edits, and other constraints on what the harness may change. With these, their method generalised out of distribution, while four prior techniques “either degrade performance or barely show an improvement.”
    • A host’s intuition: with infinite retries at tweaking the scaffold, you overfit the harness the way you would overfit a model’s hyperparameters. The discipline is to pre-commit: “I’m giving myself 10 tweaks and I’m only gonna tweak within this range.”
  • DreamRSI (Recursive Self-Improvement Through Evolving Worlds). An explorer policy builds a tree of experiments and simulators. Instead of paying to explore new branches, alternative policies are re-evaluated only inside the already-explored tree, which the authors call “dreaming.” It is cheaper than new exploration and less fragile than fine-tuning on known answers.
  • What this means for GAPA- and ACE-style loops here ^[inferred]: the “who judges” question above has a second half, “on which tasks.” Hold out tasks the loop never optimised on, cap how many self-edits it may make per cycle, and check the held-out score before accepting an edit.

Failure modes, by family

  • Memory-poisoning family (ACE/Fable 5): Misevolution — memory accumulation alone, no adversary, reduced refusal rates 70-86% across top models; alignment tipping — small deviations written into memory become compounding precedent; notes rot — a more capable model writes more persuasive bad notes; curation collapse — 18k → 122 tokens from one bad step.
  • Autonomous-persistence family (Hermes): LLM-generated skills can fail (inspect before cron); skill naming conflicts produce overlapping half-working skills; a recurring stuck-loop bug; context bloat in long sessions; and a real security surface — GAPA edits prompts and writes to disk without a human in the loop.
  • Self-grading family (verifier-first’s targets): models “pause the work early,” take “weird shortcuts (reward hacking),” and self-report success — the Verifier Theater configuration where the loop checks itself with no external standard. See Reward-Hacking and the Verification Frontier for the case that this worsens as models get stronger.
  • Scale family (Anthropic): the loop works — and then review becomes the bottleneck (Amdahl’s law), while the remaining gap is goal-choice judgment, which none of these loops persist their way around.

What to copy first

  1. Start with the ACE-style file, not the full stack. A writable Markdown lessons.md — one lesson per entry, one-line summary at top, delete-rather-than-duplicate — is the smallest version of the pattern (the Prompting Claude Fable 5 guide operationalizes exactly this setup).
  2. Write the judge before you allow autonomous persistence. The four-line verifier-first checklist (done condition, per-pass checks, saved artifact, failure-retry path) applied to the persist step: no lesson or skill is committed until a verifier with access to real outcomes approves it.^[inferred — verifier-first states this for loop output; extending it to memory writes is this article’s bridge, matching the ACE article’s own Try It advice]
  3. Separate the roles. The generator writes candidate lessons; a verifier (different model family, real outcomes, no access to the maker’s reasoning) decides which live. “The notes are the product now. Curate them like it.”
  4. Audit on a cadence. Review the lessons file after 3-5 runs and delete any entry you can’t trace to an outcome; run hermes skills audit periodically if you’re on Hermes; expect the first 7 days to be rough — the loop needs runs to learn from.
  5. Go turnkey only after the discipline is in place. Hermes ships the whole shape (capture, review, skills, memory, user model) preconfigured — valuable, but its judge is the generator, so the inspection habits above are not optional there.

Try It

  • Keep a held-out task set the self-improvement loop never sees, and cap edits per cycle, per the RRSI result (2026-10-27).
  • Add a lessons.md to any long-running agent today; after 3-5 runs, delete every lesson you cannot trace to a concrete outcome.
  • Before enabling any autonomous skill/memory write, fill in the four-line checklist — if you can’t name the check and the artifact, keep persistence manual.
  • On Hermes: run a repeating workflow for 7+ days, watch /skills grow, and hermes skills inspect <name> before any skill enters cron.
  • Read ACE (arXiv 2510.04618) §4 if you’re building a production memory system — the curation failure mode is the most important section, per the Fable 5 memory article.

Open Questions

  • No head-to-head exists. Nobody has benchmarked GAPA-style executable-skill persistence against ACE-style lesson files on the same task suite — the sources measure each in isolation.
  • Does Hermes’s self-review suffer measurable misevolution? The 70-86% refusal-rate finding covers top models generally; no Hermes-specific memory-poisoning measurement appears in these sources.
  • Fable 5’s ~3× memory advantage is a single vendor-run eval (Slay the Spire) and the ACE article’s key risk paper was not independently verified at ingest — both flagged medium-confidence in the source.