Source: raw/How_to_use_Karpathy_s_Autoresearch_to_10x_Claude.md

Creator: Ben AI · URL: https://www.youtube.com/watch?v=bc4NrE0cOE0 · Duration: 19:46 · Platform: YouTube · Uploaded: 2026-04-07

Ben AI adapts Andrej Karpathy’s AutoResearch loop, built for machine-learning experiments, to improve Claude skills and CLAUDE.md files against pass/fail criteria. A test-runner subagent scores the current skill, the main agent proposes one change, a fresh subagent re-runs the test, a script or an LLM-judge subagent scores it, and the change is kept only if the score improves. His team packaged this as a plugin for Cowork and Claude Code and is extending it to scheduled weekly loops that optimize LinkedIn posts, email open rates and landing-page click-through against real metrics. The reusable parts are the loop shape and his rules for writing criteria; the plugin itself is behind his paid community.

The loop

  1. Define the criteria. Each must resolve to true or false, for example “the hook is under 136 characters.”
  2. Baseline. A test-runner subagent runs the current skill several times (he uses five) and scores it. His first example scored 80% (4 of 5 runs passed both criteria).
  3. Hypothesis. The main agent proposes one change, such as “add an explicit 136-character limit for hooks in step four.”
  4. Test. A separate subagent runs the edited skill, so the test is not done by the agent that wrote the change.
  5. Evaluate. Either a deterministic script (character counts, formats) or an LLM-judge subagent for criteria a script cannot check (does the hook match one of my templates?).
  6. Keep or discard. The main agent keeps the change if the score beats the baseline, discards it otherwise, and moves on to the next hypothesis.
  7. Stop at a set number of iterations or when a target score is reached. The run ends with the optimized skill and a dashboard of every iteration, kept or discarded.

This is the ratchet pattern in Karpathy’s AutoResearch, with a skill file in place of model code.

His results (single runs, self-reported)

TargetCriteriaIterationsResult
LinkedIn writer skillHook under 136 characters; bullets use hyphens580% baseline to 100%
LinkedIn writer skillHook matches a template in his hook reference file; nuance words on bold claims; follows a named framework (PAS, AIDA); includes a personal story from his background file1068% baseline, +27%
Second-brain CLAUDE.mdKnowledge routing to the right folders; wikilink creation; retrieval5+9.9%

Writing criteria that work

  • State the exact condition, not the goal. Not “make sure the hook is short enough” but “the first line must be under 136 characters.” A number, a format or a pattern you can test.
  • One variable per criterion. “If you need the word and to describe what you’re testing,” split it into two. You can still run several criteria in one loop.
  • Mine criteria from data. Give Claude your best and worst performers with their metrics and ask it to find patterns. He compared top and bottom LinkedIn posts by hook type, CTA and writing framework. Without data, give it good examples, or use industry best practice and other people’s posts.
  • Three levels for creative skills:
    1. Hard rules a script can check: hook under 136 characters, sentences of 5 to 12 words, paragraphs of three sentences or fewer.
    2. Your patterns, checked by an LLM judge: nuance words (“I think”, “I believe”, “I predict”) on bold claims, because AI states claims too confidently; a named writing framework.
    3. Real-world results, checked against metrics over time (below).
  • Deterministic skills may only need evals. For a process skill like his accounting skill (Stripe and invoices into a spreadsheet), the skill-creator’s built-in evals are enough. AutoResearch earns its cost on skills with some subjectivity, which he says covers most marketing, sales and support work. See Building Skills for skill-creator evals.
  • Difference from skill-creator evals: with evals you judge each result and feed back yourself; AutoResearch runs the judging and the next hypothesis on its own until it meets the criteria.

Running against real metrics

Ben has set up three scheduled tasks, none of which had run when he recorded: a LinkedIn optimizer, an email open-rate optimizer and a landing-page optimizer (testing H1 copy until a 3% click-through rate). The weekly LinkedIn task as he describes it:

  1. Read the folder that holds the active experiment and the allowed hook types, frameworks and CTAs.
  2. Scrape last week’s posts and their engagement.
  3. Compare average likes with the baseline: keep the change if up, revert if down.
  4. Set the next hypothesis, changing one variable (hook type, framework or CTA), until the target of 250 average likes.

He notes that other variables affect real-world results, that the feedback loop is slow, and that you can add a human approval step if you do not want it running alone. Optimize the skill against fixed criteria first, before handing it to a live loop.

Try It

  1. Pick one skill you run every week and write two true/false criteria for it, each with a number or a pattern.
  2. Ask Claude to build the loop from Karpathy’s repo. Ben’s route: download Karpathy’s AutoResearch repository, give Claude the folder, and ask it to adapt the loop to optimizing skills, with separate subagents for test-running, scoring and judging. Expect some back and forth.
  3. Start at five iterations. He finds results get worse after 10 to 15, and every iteration costs tokens; use ten only when you have many criteria.
  4. Read the criteria Claude restates before you approve the run.
  5. Mine your next criteria from data. Export your top and bottom ten posts with metrics and ask Claude which hook, CTA and structure patterns separate them.
  6. Only then schedule a weekly loop against live metrics, with one variable changed per week and a human check if the output is public.

Key Takeaways

  • The loop is simple: baseline, one-change hypothesis, independent test, scripted or judged score, keep or discard, repeat.
  • Criteria quality sets the ceiling. True/false, one variable each, stated as an exact condition.
  • Separate agents keep the scoring honest: the agent that proposes a change does not test or judge it.
  • Cap the iterations. Five is his default; more than 10 to 15 made results worse for him and costs more.
  • Most of a “subjective” skill can be made testable, through hard rules plus judge-checked patterns, before you touch live metrics.

Open Questions

  • The plugin is not public. Its skill, judge and test-runner prompts are only in his paid community, so the results above cannot be reproduced exactly.
  • No cost figures. He warns hours-long runs get expensive but gives no token or dollar numbers for any run.
  • Overfitting to the judge. A skill tuned to pass an LLM judge may learn the judge’s taste rather than the reader’s. He does not test for this.
  • Live-metric loops are untested. None of his three scheduled optimizers had run at recording, so there is no evidence yet that weekly metric-driven loops improve results.
  • “Under 136 characters” vs “under 130” — he uses both numbers for the same hook rule in the video.