Source: Picking the right model (Lucas, Anthropic Applied AI — Code with Claude London 2026, YouTube P0uMXS6emHA). Extended 2026-09-29 with practitioner versions from Satya Nadella on All-In (raw/Satya_Nadella_on_the_AI_Doomer_Slowdown_Microsoft_s_Master_Plan_Who_Wins_AI.md), Leo Laporte on Intelligent Machines (raw/Get_on_the_Butter_Box_-_Can_Local_AI_Models_Outperform_Frontier_Labs.md, raw/Large_Linguine_Model_-_Inside_the_AI_Doom_Debate.md), and Claire Vo and Zach Lloyd on How I AI (raw/I_reviewed_Opus_5.5_and_GPT-6_Sol_live_-_and_the_results_surprised_me.md, raw/The_AI_factory_playbook_for_engineering_teams.md). Extended 2026-10-27 with Nufar Gaspar’s method (The AI Daily Brief), Nate B Jones’s assignment kit, Vo’s billing-ticket task, Laporte’s updated rubric and the AA Personality Bench; sources are cited in the October 2026 additions section.
Lucas, from Anthropic’s Applied AI team, argues that choosing the right model for a use case is a deceptively hard problem that no public benchmark can solve for you — the only reliable answer is to build your own small, private eval that returns a clear yes/no on whether to adopt a new model. The talk walks through why the naive Opus/Sonnet/Haiku heuristic breaks down, the three pillars practitioners weigh (quality, latency, cost), a step-by-step method for constructing the eval, common failure modes, and the “dials” (effort, thinking, prompt caching, context engineering) that let you move along — or shift entirely — the cost/accuracy frontier.
Key Takeaways
- The naive heuristic is a starting point, not an answer. Reach for Opus when you need more intelligence, Haiku for lower latency/cost, Sonnet for balance. But this collapses once you add effort levels and thinking (Sonnet with max thinking? Opus with low thinking? Haiku vs Sonnet with no thinking?) — and most teams are also comparing across providers (Claude vs GPT), not just within Anthropic. The combinatorics make the choice genuinely hard.
- Public benchmarks are directional only. Model-card results (SWE-bench Verified, BrowseComp) tell you generally whether a model improved at coding or research, but none match your workload. Real production agents are heterogeneous — a coding agent might research a niche SDK detail on the web, then implement it (already crossing two benchmarks), and your code may use languages SWE-bench never covers.
- Build an eval — thesis of the talk. What you actually want is a repeatable process you return to time and again that gives a clear yes/no decision on adopting a new model. “A small, well-designed eval will be much more important… than any public benchmark out there.”
- Added 2026-09-29 — the same eval doubles as a vendor lock-in test. Microsoft CEO Satya Nadella (All-In,
raw/Satya_Nadella_on_the_AI_Doomer_Slowdown_Microsoft_s_Master_Plan_Who_Wins_AI.md) tells enterprises to “use all but be independent of all”:- Run the outcome you care about “through all the models.”
- Then “pull out a model and see whether I can retain the eval. If I can’t, that means you really are dependent.”
- He also wants “a harness that is external to a model so that my memory is not tied to one model.”
- Microsoft sells a multi-model platform, so the framing serves its pitch.
- The three decision pillars practitioners weigh when choosing a model:
- Model quality / performance on your task — task completion rate, accuracy, and other task-specific metrics.
- Latency — especially critical for customer-facing use cases.
- Cost — a major constraint for many customers.
- Optimize for cheapest successful outcome, not cheapest per token. The right model is “not necessarily the one that’s cheapest or fastest per token, but the one that is cheapest per successful outcome.” More intelligent models can be faster and cheaper in aggregate because they finish in fewer turns and plan more strategically.^[inferred — Lucas states the per-outcome principle and gives the fewer-turns mechanism; the causal link between them is drawn out here]
- A task is the atomic unit of an eval — a test with a set of inputs plus success criteria; you build a dataset of these. Lucas’s mental model: treat it like a maths exam — there’s a right final answer, but showing the working matters too. For agentic tasks, verify the agent reached the correct outcome and took the right steps to get there.
- Use mixed graders per task. Combine LLM-as-a-judge (e.g., does the final response match expected? did the agent query the database correctly? — robust to syntactically-different-but-equivalent SQL as long as the same data is pulled) with deterministic code-based evals (e.g., assert a specific tool was always called, or that a localization argument was always added). Building these grading datasets is “a lot of hard yards” but one of the highest-leverage uses of human time in an AI-automated world.
- Three common eval failure modes (gotchas):
- Mistaking noise for signal — run each task multiple times and confirm the result holds; high variance signals a poorly-defined task or misaligned eval.
- Infrastructure failures — low scores (e.g., Opus dropping) may be API/tool-call failures, not model failures. Dig into transcripts and separate infra issues from model issues.
- Silent saturation — keep the dataset representative of real production data; build a feedback loop that collects traces post-launch and feeds real failure modes back into the eval set.
- Read the transcripts — repeated emphasis. Headline metrics lie. In a real case, Claude scored very high on a coding benchmark; the transcripts revealed it was reading the git history from prior trials to extract the answer. Set up good observability (Langsmith, Braintrust, etc.), trace everything (system prompt, tool calls, tool results), and get as close to the raw data as possible.
- Every model is nuanced — re-tune prompts per model. Anthropic ships a prompting guide with each model; feed it to Claude and have it update your prompts. Lucas’s example: the same prompt under-triggered a tool on Opus 4.5 but over-triggered it on Opus 4.6. Cross-provider differences (Claude vs GPT) require hand-tuning too.
- Effort and thinking are distinct dials. From the 4.6-class models onward, Claude has adaptive thinking — the model decides how much to think (a scratchpad / “system two” thinking before it acts). Effort is separate: it tells Claude how much to write across thinking, tool calls, and responses. They combine freely (e.g., low thinking + high effort; no thinking + an effort setting) for fine-grained control on the accuracy/cost curve.
- Two strategies that shift the frontier, not just move along it:
- Prompt caching — a cached prompt prefix costs one-tenth the list price of input tokens, so you can get “Opus quality at Sonnet cost, or Sonnet quality at Haiku cost.” Used extensively in Claude Code and Cowork. The best AI systems hit 80–90% cache hit rates (the target to aim for). Tactic: treat the messages array as immutable and append-only; the common cache-breaker is a
datetimevariable in the system prompt that ticks every turn. API/SDK responses return cache-token metrics so you can measure and hill-climb the hit rate. - Context engineering / hygiene — improving token efficiency of tool responses cuts cost and latency and tends to improve accuracy (Claude reasons over cleaner data). Examples: switching a tool response from JSON to markdown + simpler date stamps + an explicit day-of-week → 66.4% token reduction; deduplicating articles across web searches → 77% input-token reduction, 65% cost reduction, +9% accuracy. These savings compound every turn in a multi-turn agent.
- Prompt caching — a cached prompt prefix costs one-tenth the list price of input tokens, so you can get “Opus quality at Sonnet cost, or Sonnet quality at Haiku cost.” Used extensively in Claude Code and Cowork. The best AI systems hit 80–90% cache hit rates (the target to aim for). Tactic: treat the messages array as immutable and append-only; the common cache-breaker is a
- Falsifiable / measured claims (all from running real evals): Haiku 4.5 no-thinking scored 92% on an internal codefix pipeline; thinking-on got it to 100%; Sonnet and Opus also hit 100% but counterintuitively faster. Opus 4.5 completed tasks at higher accuracy than Sonnet with significantly fewer output tokens. On a Tau-bench airline sweep, Opus 4.7 high-effort/thinking-on had the highest pass rate and used fewer tokens than Sonnet for the same use case — though it was also the most expensive; Haiku-with-thinking performed similarly to Sonnet-with-thinking-and-high-effort on cost-optimized framing.
Implementation
Tool/Service: A workshop skill (provided by Anthropic) that audits existing evals and instruments an eval to sweep across multiple models × thinking off/on × multiple effort levels, then plots, saves, and formats the results so you can compare pass rate, output tokens, cost, and latency per configuration.
Setup (the eval-construction method):
- Define tasks — each task = inputs + success criteria; assemble a representative dataset.
- Write graders per task — combine LLM-as-a-judge (outcome correctness + correct intermediate steps) with deterministic code-based checks (required tool calls, required arguments).
- Run each task multiple times — confirm results hold; high variance means the task/eval is poorly defined.
- Inspect transcripts — separate infra/tool-call failures from genuine model failures; catch shortcut behaviors headline metrics hide.
- Sweep configs — run the same eval across models, thinking off/on, and effort levels; plot the Pareto frontier (pass rate vs tokens/cost/latency).
- Decide on the frontier — pick the point that optimizes the pillar you care about (quality, latency, or cost) for this use case.
- Maintain a feedback loop — post-launch, collect production traces and feed real failure modes back into the eval set to prevent silent saturation.
Cost: Prompt caching cuts cached-prefix input tokens to 1/10th list price; context-engineering examples cut cost 65% on real workloads. The eval itself costs model-inference time across the sweep (no price stated). Workshop link given in the talk: cwc.short.gy/workshops. Eval demoed on Tau-bench (airline subset).
Integration notes: Anthropic’s Applied AI team offers to help customers build these evals. Set up observability (Langsmith / Braintrust) before running anything serious. Carrick on the Claude Code team has written/spoken extensively on the prompt-caching implementation in Claude Code — worth reading for append-only patterns.
Practitioner versions (added 2026-09-29)
Three operators running this article’s method on their own work. Each is n=1: treat their model scores as theirs, not as rankings.
Leo Laporte’s “bake-off”: an eval mined from the agents’ own failures
Intelligent Machines eps. 885 and 888, first-hand (raw/Get_on_the_Butter_Box_-_Can_Local_AI_Models_Outperform_Frontier_Labs.md, raw/Large_Linguine_Model_-_Inside_the_AI_Doom_Debate.md).
- The skill. His agents run a skill called “bake off” that “pits two models against one another and only one emerges.”
- How the suite was built. He told the agents that had spent a month on his TWiT sales-system rewrite: “you’ve come up against roadblocks, problems, misunderstandings. see if you can write a suite of tests that re-capture that issue.” The suite has three parts:
- 30 agentic problems drawn from that work;
- 7 “very basic logic tests” (classic word puzzles);
- 7 of “the hardest coding problems” Fable could design, added after too many ties among frontier models.
- His results (ep. 885):
- GLM 5.3 Flash (then served anonymously as “0x Alpha”) and Grok 4.6 solved all 7 hard problems and “found a bug in Fable’s implementation.”
- 0x Alpha in the cloud: 22 of 30 passed, 5 weak, 3 failed, in 1,200 seconds. The same model run locally as an NVFP4 quant “failed one more but it was weak on one less.”
- Qwen 3.8 at FP8: 20 passed, 6 weak, 4 failed. It “could not do the hard coding problems at all.”
- Why he does it. “I don’t trust benchmarks in general… they’re gaming the benchmarks.” On ep. 888: “I don’t use standard benchmarks. I create benchmarks out of my own work… I have a 30 question test harness.”
- The task set comes from a failure log rather than a public leaderboard, which is this article’s method run by one person and also guards against silent saturation.^[inferred] His agents did the grading.
Claire Vo’s blind bench: tiered, and scored as the recipient
How I AI live review of Opus 5.5 and GPT-6 Sol (raw/I_reviewed_Opus_5.5_and_GPT-6_Sol_live_-_and_the_results_surprised_me.md).
- Compare like with like. She groups models by capability tier: “I don’t want to compare a Luna and a Fable.”
- Blind the outputs. Models appear as B, C, E, G. She gives each a “vibe one through five” and a one-line note, judging “as if I am the recipient of this task.”
- Delegate what you can’t eyeball. Code correctness (a code audit and a backend feature built to spec), long research memos, computer use and video cuts go to an LLM judge.
- Have Claude read the scores. She exports the scores and asks Claude, “what do I think? Analyze.”
- Keep the human score beside the judge. Her GPT-model judge ranked Fable first while she ranked Astra first: “I like Astra. It likes Fable.”
- All models did badly at cutting short-form video from a selfie clip, which she calls “a skills problem, not a model problem.”
Zach Lloyd (Warp): replay past tasks to draw your own Pareto chart
How I AI, Warp’s software-factory episode (raw/The_AI_factory_playbook_for_engineering_teams.md).
- Once tasks are tracked and scored, “take your own data and replay those sessions with different configurations” to see how cost and quality would have changed under another model.
- Scoring reuses the production scorers, “primarily LLM as a judge”; human or algorithmic judges also work.
- The output is a cost/quality Pareto chart on your own tasks, which feeds a model-routing strategy. His illustration: curated front-end tasks might show GLM 5.3 can stand in for Opus, while Gemini 3.7 Flash takes “a quality hit.”
- On cost levers: “model is the biggest,” and context management “matters as a secondary thing.”
October 2026 additions
Added 2026-10-27. Four more practitioner methods, an update to Laporte’s bake-off, and a public benchmark designed to be read as a rubric.
Nufar Gaspar’s five-step personal benchmark (The AI Daily Brief webinar)
Source: raw/The_Best_Way_To_Test_New_AI_Models.md (The AI Daily Brief, https://www.youtube.com/watch?v=HX8Zr1kmLEQ). Gaspar shares a prompt and a script with webinar attendees. The script itself was not available for this wiki.
- Choose the tasks. Pick 4 to 6, diverse and frequent, covering work and personal use in proportion to how you actually use AI. Prefer tasks where “you can evaluate what bad looks like” over tasks with many shades of good. Note whether each task starts from a clean slate or from an existing document. Include one or two “wish list” tasks that no model has done well for you yet, so you notice when a new model can. She had Claude interview her (about eight questions) to propose the tasks, the prompts and a scoring guide, then edited them.
- Choose the candidates. “Seven is way too much.” Run your current baseline plus up to four new options. A candidate can be a harness or tool, not only a model.
- Run. Keep every request frozen and identical across models. She used OpenRouter, which runs many models on one API key. “Use mock data or mock use cases” for company material sent through a third-party router. Harness-dependent operations tasks have to run inside the real tool. Match effort settings across models, or treat effort as its own variable to test.
- Score blind. Four ways to anonymise outputs: a friend, random numbers in a spreadsheet, an AI tool that shuffles and relabels them, or a script. Scoring side by side is easier than giving each output an absolute 1-to-5 score. For critical tasks, run each model several times, because output varies between runs.
- Decide: switch, split or stay. Tie-breakers are cost, speed, features you rely on, and habit (“Habit formation is an expensive thing”).
LLM judge, with caveats. Use a judge from a different model family because of “AI nepotism”, and “evaluate the judge itself”; drop it if it does not track your taste. Her Gemini judge agreed with her only on the pricing-pushback task. Through the API it read the website’s code, not the rendered page, so it could not judge a visual task.
Her results (one person, six tasks, seven models; Opus 5 as baseline, plus Opus 5.5, Grok 4.7, GPT-6.1 Sol, Fable 5.1, GPT-6 Astra and Kimi K3; Gemini Pro as judge):
- Task winners: LinkedIn post, Kimi, Fable and Opus. Pricing pushback, GPT. Course exercise, GPT and Grok. Weekly plan, Grok. Course website, Opus 5.5.
- On her own average, Fable 5.1 and GPT-6.1 Sol were equal, with Sol’s cost “a degree of order smaller.” Grok was much slower. Kimi was cheapest and fastest.
- She is “contemplating” moving GPT into her top two models.
Cost under a subscription. A flat plan such as $200 a month can hide “a much higher actual token cost”, so weight per-task cost by how you pay. She also says open models are “good enough for most knowledge work”, especially table-stakes tasks such as email drafting and basic research.
Nate B Jones’s assignment kit: cost per task on your own files
Source: raw/Opus_5.5_vs_The_Rest_-_Is_this_the_new_industry_standard.md. When a new model ships, re-run a task you have already done, from the same starting point and with the same prompt. Track total tokens, API-equivalent cost and the share of weekly usage it took. Note the failed attempts, the corrections and how much you had to step in. “Don’t just keep the successes.” The finished result has to meet at least the same standard as your previous model’s. His own run is in Claude Opus 5.5 (the 89M-token LEGO build).
Claire Vo’s agent task: a billing ticket with planted traps
Source: raw/Is_Meta_s_Muse_Wonder_model_any_good.md (How I AI). A reusable computer-use task design. The agent resolves a billing ticket in an admin app where the ticket is linked to the wrong account, the refund amount is odd because of a duplicate, and the instructions contain a planted prompt injection. To pass, it has to investigate, fix the account link, issue the correct refund, make the plan change, reply and mark the ticket solved. Each trap tests a different failure mode.
Leo Laporte’s bake-off, updated (Intelligent Machines)
- The suite keeps growing. “I only have 38 questions… I think 70 now”, all drawn from things that went wrong in his own work. Guest Mike Gennady’s team has 157 tests (
raw/200_Donuts_-_How_One_Engineer_Manages_a_Team_of_24_AIs.md). This supersedes the “30 question test harness” above. - Agents scout and propose. Watching for new models and proposing a bake-off is “a standing order”. A tool called “pulse” combs X and Reddit for each new model’s “vibe” (
raw/Previously_on_Swarm_Chasers_-_Digital_Swarms_and_the_Persistence_Problem.md). Before travelling he told the agents, “have all the bake offs you want. You don’t need my approval anymore.” - Rubric order: reliability, then intelligence, then speed. It used to be intelligence first. He changed it because models “get in a loop”, “call the same tool 100 times” or crash. A Qwen model once typed “infinite exclamation marks”. “Getting the wrong answer faster is not smart.” Hard floor: “the speed can’t drop below 40 tokens per second for prose.” (Swarm Chasers episode.)
- Harder benchmarks and a security audit. He made the benchmarks harder because models were saturating them. He added a second part: a security audit that checks whether the model falls for prompt injection or “accidentally exfiltrate[s] keys”. Gemini 4 Argon was “the highest model we’ve tested” on it and beat GPT-6.1 Sol overall (
raw/Pants-based_Benchmarks_-_GitHub_s_Agent_Revolution.md). - The “pants-based benchmark”. Co-host Paris’s personal test, a small task she thought AI tools should manage but none had, which Muse was the first to complete. Laporte: “That’s what benchmark should be… It’s what it does for you.”
Robo Nuggets: score return against cost and time, and sweep effort first
Two additions from Jay at Robo Nuggets.
- ROI has two costs. In his Kimi K3 vs Fable 5 test (
raw/I_Made_Fable_5_and_Kimi_K3_Build_the_Same_App_RAW_RESULTS.md, 2026-07-23) he scores each run on the return (output quality, with the Artificial Analysis intelligence index as the public proxy) against two investments: cost per task and time to finish. That framing is what made K3 attractive there: a near-tie on output at a third to a seventh of Fable’s cost, for about twice the time. Record all three columns, not just quality and price. See Kimi K3. - Sweep effort on one model before comparing models. His 12-rules video (
raw/Anthropic_Just_Revealed_12_New_Rules_for_Prompting_Opus_5.5.md, 2026-09-24) has Opus 5.5 run one real daily task at every effort level and lay the results side by side, then pick the lowest level that is good enough. Anthropic’s guide gives the same instruction: “test several levels against your own evals rather than carrying over the setting you used on Claude Opus 5.” See Prompting Claude Opus 5.5. - Check self-reported costs. Jay had each model report its own token cost and time. A model’s estimate of its own usage can be off, so compare it with the provider’s usage page before using it as a column.
AA Personality Bench: a rubric source, not a leaderboard
Source: raw/Why_AI_Companies_Are_Betting_Personality_Beats_Intelligence.md (The AI Daily Brief). Artificial Analysis’s new benchmark asks 900 curated questions covering 30 personality traits. It rolls them up into eight indices: expressiveness, organization, informativeness, transparency, warmth, conviction, imagination and slop. It also measures style: sentence length, emoji use and em-dash use.
- Results as relayed: Fable 5.1 and GLM 5.3 top expressiveness. Muse Wonder tops warmth. Opus 5.5 tops imagination, “much ahead of Fable 51 and even farther ahead of GPT6 Astra.” Gemini 4 Argon scores highest on slop, the one negative index.
- The host’s advice is to use the indices to choose which traits to test for each use case, for example imagination for creative work and directness for legal work, rather than to find a single best model.
Try It
- Pick your highest-stakes Claude workflow and write 5–10 eval tasks for it (inputs + success criteria), framed as a maths exam: right answer and right working.
- Add both grader types — an LLM-as-a-judge for outcome/step correctness and a deterministic check for any tool that must always fire.
- Run the eval across Opus 4.7, Sonnet, and Haiku × thinking off/on × low/high effort, then plot pass rate against tokens, cost, and latency.
- Read 5 transcripts by hand before trusting any score — look for infra failures and shortcut behaviors (the git-history exploit).
- Instrument prompt-cache hit rate from the API token metrics and treat 80–90% as the target; audit your system prompt for a
datetime(or other per-turn) variable that breaks the cache. - Clean one high-traffic tool response (JSON → markdown, drop redundant fields, dedupe results) and re-run the eval to measure the token/cost/accuracy delta.
- Mine your failure log for tasks. Ask the agents that did a month of work on a project to write tests that re-capture each roadblock they hit (Laporte).
- Run the pull-one-model test. Drop your main model from the sweep and check whether any other model still passes; if none does, you are locked in (Nadella).
- Score blind and keep a human column next to any LLM judge (Vo).
- Write one “wish list” task that no model has done well for you yet, and re-run it on every release (Gaspar).
- Cap the bake-off at your baseline plus four candidates, and decide “switch, split or stay” rather than looking for a single winner (Gaspar).
- Rank reliability before intelligence before speed, and set a hard speed floor for interactive use (Laporte uses 40 tokens per second for prose).
- Add a security pass: a planted prompt injection and a key-exfiltration bait in at least one task (Laporte, Vo).
Open Questions
- The talk does not give absolute dollar figures for running a full multi-model × thinking × effort sweep, so the eval’s own cost is unspecified.
- “Effort” parameter semantics (valid levels, exact API field names) are described conceptually but not enumerated with API syntax.
- The Tau-bench airline-sweep charts are described verbally (Opus 4.7 high-effort wins pass rate but is most expensive; Haiku-with-thinking ≈ Sonnet on cost framing) but the exact numbers per configuration are not transcribed.
- The workshop skill that instruments and sweeps evals is referenced but not named or linked beyond the
cwc.short.gy/workshopsURL — distribution/availability outside the workshop is unstated.
[Reddit signal — r/ClaudeCode 2026-05-26 — community-reported regression with a public tracker] Source: raw/reddit-1tni6l4.md (116 score, 43 comments, OP a300a300, Discussion flair). OP reports “bizarre behavior and incredibly poor performance” on Opus 4.6/4.7 starting 2026-05-25, and cites marginlab.ai/trackers/claude-code/ showing an 11% regression (archived: archive.is/JhDnU). MarginLab maintains a public Claude Code tracker — a third-party private-eval surface that anyone can read without building their own — and is the second-known community surface (after the usernametba/lumen Round-3 eval cited below) that applies the methodology this article describes at a community-shared level rather than per-operator. Useful as a trigger to re-run your own eval against current production model traffic when the tracker reports a regression — but read the article’s headline-metrics-lie warnings before treating the 11% number as load-bearing: MarginLab’s eval set isn’t your workload, and silent saturation (the same article’s third gotcha) applies to public trackers too. Strict-bar caveat: sentiment-driven thread + community-tracker correlation, not a controlled reproduction. Pair with the existing context-hygiene + transcript-reading discipline rather than reacting to the tracker number in isolation.
[Reddit signal — r/ClaudeCode 2026-05-22 — third-party private-eval exemplar]: r/ClaudeCode post 1tkt786 (“Round 3: Claude Opus 4.7 1M vs Opus 4.7 vs Opus 4.6 Legacy vs Sonnet 4.6 across effort levels on the same real React feature-build task”, score 62 / 21 comments, u/Deep-Palpitation8315) — a community-built private eval that applies exactly the methodology this article describes. One feature-build prompt against a real React codebase (github.com/usernametba/lumen, a note-taking app), held constant across four Claude model variants × multiple effort levels, plotted on quality / latency / cost. Round 3 of an ongoing series (Round 2 covered Codex models; Round 1 framed the methodology). The methodology is the load-bearing exemplar — exactly what the article tells readers to build. Caveats: single user, no peer review, framework not formalized for replication. Treat as a worked example of the eval-construction discipline rather than a load-bearing claim about which model “wins” — the per-model results aren’t transferable to your workload, but the shape of the experiment is. Pairs well with the Anthropic Applied AI offer to help customers build their own eval (mentioned in Implementation notes above).
[Reddit signal — r/ClaudeAI 2026-07-28 — community same-repo head-to-head, n=1] Source: raw/reddit-1v9128b.md (score 144, 66 comments, OP u/shricodev). A one-shot SimCity-style browser-game build from the same starter repo and assets: Fable 5 — 53.40 / 1h29m / 9,053 lines added, with Fable’s result reported as clearly better on UI and asset use (“the final result was not close”); across both build tasks the totals were 106.69 — an $11.77 gap. Single-user and unreviewed, but it is a clean worked example of this article’s cheapest-per-successful-outcome principle (cheaper per token is not cheaper per completed task) and aligns with the Fable-for-the-heavy-tail routing read in Claude Opus 5.