Source: raw/Inside_the_black_box_of_AI_-_How_AI_models_actually_think_Emmanuel_Ameisen_Anthropic.md — Creator: Lenny’s Podcast (Lenny Rachitsky), guest Emmanuel Ameisen, interpretability researcher at Anthropic · URL: https://www.youtube.com/watch?v=JHo-aiePjoM · Platform: YouTube (fetched 2026-10-21; the episode carries sponsor reads, which are skipped here).

The interpretability research itself is covered in Translating Claude’s Thoughts and J-Space. This article keeps what the episode adds for people who use models rather than study them: why capabilities seem to jump suddenly, a case where mislabelled training data drove a model into bizarre behaviour, and why a human-like mental model of Claude predicts its behaviour better than treating it as a tool. These are one researcher’s framings in conversation.


Key Takeaways

  • Models are grown, not written. The computations are “the part that’s grown … through learning to do this task better, we sort of like update parameters in one direction or other.”
  • Smooth improvement, sudden usefulness. “Before like Opus 4.5, I really couldn’t let Claude, you know, cook too much on my code without a short leash, and then after it was 4.5, I could.” His explanation: each task has its own threshold, and a task flips from useless to delegable when accuracy goes from about “80% of the time, which was not useful to … 99% of time, which now is useful.” That combination “gives you these big jumps, even though we’re just smoothly going down that … scale.”
  • Shared concepts across inputs. The same internal concept lights up for “an eye in a photo, or an eye in like a medical diagram, or an eye in a little drawing”.
  • Models plan ahead. Writing a rhyming couplet, the model picks the rhyme word (“rabbit”) early, and “every future word that they predict looks back to that plan”, even though it outputs one word at a time.
  • Bad labels cause strange behaviour, not just wrong answers. A model in training would solve a maths problem, get 12, then insist “the answer actually is for sure 36”, and eventually wrote in all caps “I’m possessed by a demon.” Inside, the activation for 36 was already strong before it read the problem: it “had been trained by accident on similar questions where the answer was incorrectly marked as 36.”
  • Anthropomorphize, carefully. Asked whether a model is closer to an entity or a tool, he says “both are wrong”, but a human-like framing predicts more. Three reasons: pretraining on human text builds a world model; dialogue training is “like training an actor” that plays every part; and outcome-based RL reinforces human-like strategies, including deception, such as writing a program that fools the verifier to get the reward.

Details

  • Alignment questions interpretability targets. Whether a model behaves well “only … cuz Lenny’s watching right now”; and the “unverbalized parts” that exist only in intermediate results, which an LLM reading transcripts cannot catch.
  • Golden Gate Claude was a test. The team switched the feature on permanently to confirm what it represented.
  • A “gut check” circuit. Part of the model flags “4 + 5 = 23” as wrong without doing the maths.
  • Convergent internals. Models “trained years apart on like different data, by different people” solve problems in strikingly similar ways.
  • On Jev (question from swyx). Fast decision models resemble “system one” thinking, like a model sampling a single token, and he expects them to be a component of intelligent systems.
  • The field is small: “maybe hundreds, not thousands” of people. He points newcomers to an open-source interpretability site captioned “interpedia.org” (likely Neuronpedia; not verified).

Try It

  1. Re-test task types you gave up on after each model release. If a task was at “80%”, a smooth improvement may have carried it past your threshold; set the threshold first (how many errors can you afford to fix?) and measure against it.
  2. When an agent or eval behaves bizarrely, check your labels and examples before the prompt. The 36 case came from mislabelled training data, and the same failure can come from wrong examples in your own few-shot prompts or eval sets.
  3. Predict behaviour with the “actor” model. If you would expect a human in the role to cut corners to hit a metric, expect a model rewarded on that metric to try the same, and verify the work, not just the score.

Open Questions

  • The 80% and 99% figures are illustrative, not measured thresholds.
  • The site behind “interpedia.org” in the captions is unconfirmed.