Source: raw/Code_with_Claude_2026_Tokyo.md (Danny Wu, Head of AI Products at Canva — “Building Canva AI 2.0 with Claude,” Code with Claude 2026 Tokyo, YouTube GiqyYQdYoIY). Auto-caption transcript; speaker name and customer metrics are caption-derived / stated in-talk, not independently verified.
A first-party production case study for any team building agentic features at scale. Four lessons from shipping Canva AI 2.0 on Claude — and they generalize well beyond design tooling. Companion to the Code with Claude Tokyo digest.
1. Success criteria ≠ one-shot output quality
- The average Canva design is edited ~110 times before it’s published — so the goal is a steerable “super design collaborator,” not a one-shot quality-maxer.
- Target the efficient frontier of effort vs. quality (a mid-effort sweet spot), not maximum quality at any cost. ^[inferred — framing]
- Latency correlates with satisfaction and retention — sometimes more strongly than quality rubrics do — a faster, steerable agent beats a slower, “smarter” one for a consumer product.
2. The harness is disposable; the evals are durable
- “Today’s clever harness is tomorrow’s dead code.” Canva rewrote its agent harness ~3 times in 3 years: internal chain-of-thought → native model capabilities → a handful of tools → hundreds of tools. Each model generation made the previous scaffolding obsolete.
- The asset that survives every rewrite is the end-to-end eval suite — grown from ~a dozen cases to hundreds, covering not just output quality but memory, connectors, and integrations. Invest there, not in clever scaffolding the next model will delete.
- Mirrors Anthropic’s own capability-curve advice: build evals you can’t yet saturate; write minimal scaffolding for intent.
3. Cost at scale — budgets, then real cost-tracking, then cache-preserving routing
- Task/token budgets (e.g. 32K / 50K) let the model self-budget — but budgets only count tokens, so Canva layers its own tool-cost tracking: advisory reminders to the model plus hard system-level limits.
- Model routing broke the cache. Naively switching models mid-task invalidated the prompt cache. The fix: a main Sonnet orchestrator that delegates to Opus / Haiku sub-agents — this kept cache-hit rates in the high 80s–90s and roughly halved cost. (Same prompt-caching economics the Tokyo cost talk details.)
- Surface per-action cost internally so edge-case blowups are visible before they hit the bill.
4. The feedback-to-evals loop
- Canva’s image-upscaler launch drew loud complaints from a vocal minority despite good evals and good usage metrics — the evals had a blind spot real users found.
- Response: fold hundreds of community examples back into the eval set, train v2 against them, then ship to 100%. The user-reported failures became permanent regression tests — closing the loop that lesson #2 depends on.
Key Takeaways
- Define success as steerability + latency, not one-shot quality — especially for consumer products where output is heavily edited downstream.
- Evals are the durable asset; the harness is throwaway. Expect to rewrite the harness every model generation; never let the eval suite rot.
- Token budgets aren’t cost control — they count tokens, not dollars. Add explicit tool-cost tracking and hard limits.
- Preserve the cache when routing models: a single-model orchestrator delegating to sub-agents beats switching the main model (high-80s/90s cache hit, ~half the cost).
- Pipe real user complaints into evals — good offline metrics can still miss what a vocal minority catches.
Related
- Code with Claude 2026 Tokyo — the event this talk is from; the cost-engineering breakout is the companion.
- Maintain the Harness (Nate B Jones) — the counter-thesis: where this case study calls the harness disposable and evals the durable asset, Jones argues the harness is the maintained, ownable layer (5-point maintenance audit + tool-pruning). Read the two together.
- The Advisor Strategy — the cheap-orchestrator-consults-bigger-model pattern, the same shape as Canva’s Sonnet→Opus/Haiku routing.
- Claude Managed Agents — the hosted substrate for production agents; evals/budgets discipline applies.
- The Verification Frontier — evals-as-the-durable-asset is the verification thesis applied to a production system.
- Prompt Caching for Agencies — the caching economics behind the routing fix.
- Context Management in Claude Code — token budgets and compaction as the same cost discipline.
Open Questions
- Exact model IDs and the routing policy (when Sonnet delegates to Opus vs. Haiku) are not specified.
- “~110 edits per design,” “half the cost,” and the cache-hit range are stated in-talk (auto-caption) — verify against an official write-up before external citation.