Source: raw/How_founders_build_on_Claude_Managed_Agents.md — official Anthropic YouTube channel (@claude), roundtable with three founders: Sahaj (a meeting product), Mihir (agents for sales teams) and Todd (a product-analytics platform), youtube.com/watch?v=hm8NzEd5io0 (fetched 2026-10-30). The Anthropic host is not named, and none of the three companies is named in the transcript.

Three founders each describe one Claude Managed Agents primitive in production. Sahaj grades meeting briefs with outcomes, and his product shows nothing if the grade fails. Mihir’s cross-account Watchtower product uses managed memory for org-wide concepts but keeps per-account memory in-house. Todd runs customer source code in sandboxes to tie analytics to code changes. All three give short time-to-ship figures (one day to two weeks) and agree on a build-vs-buy rule: buy the harness unless the harness is what makes the feature good. The second half covers evals, model migration and cost, including what they want Anthropic to fix (batch mode, cost levers, sandbox pre-warming).


Key Takeaways

Outcomes: grade the work with an independent verifier (Sahaj, meeting product)

  • Wrong output is worse than none. A brief about “the wrong Tom” is “almost destructive.” So when the agent fails, “literally choose not to show the user anything at all”, because that “is way better than giving the user false positive information.”
  • The rubric mixes objective checks and UX. Objective: “did you literally just pull the correct LinkedIn for the right person?” UX: is it scannable, in the right order (who the person is, then why you’re meeting, then what to do), and not a “five-page pre-read” nobody reads. “We’ll look at each of the things that, from a UX perspective, we want the user to experience, turn that into part of the rubric, and then hill climb against that rubric.”
  • Run it ahead of time. Briefs generate about 24 hours before the meeting, which leaves room for grading and retries.
  • Why a separate grader is worth it. “Agents that do the work and agents that evaluate the work tend to just have decoupled assessments.” Outcomes uses “an independent context window, independent verifier agent.” Without that, grading “would just be part of the agent loop itself.”
  • Sahaj’s rule for proactive products: “Help when you can and do not cause harm or annoyance.” Outcomes also tells him what to build into the first-pass agent loop.

Memory: in-house for the core, managed for the rest (Mihir, sales agents)

  • Architecture. Each account gets one agent that “runs forever” over the account’s life cycle. Those per-account agents are not on Managed Agents, and their memory architecture is Mihir’s own. Watchtower, the cross-account product (top five accounts today, quarterly forecast, roll-ups for a manager with 500 accounts), is the managed agent, and it can query the fleet’s memory.
  • A two-by-two of memory. Account vs cross-account, and user vs org. Cross-account concepts (“how forecasting works”, what a company’s 30 overlapping Salesforce fields mean) go into org-wide memory as people use the product, so “we don’t have to… do a bunch of config at the beginning.” When a question is ambiguous, the agent asks the user and saves the answer.
  • Keep core memory, buy commodity memory. Per-account memory “lasts forever”, “can get corrupted over time”, and needs custom indexing to be efficiently queryable, so they manage it themselves. Field meanings, org process and user preferences “need to work… but they don’t need to work any differently for us than they need to work for other people”, so managed memory handles those.
  • Code plus fan-out for big queries. For 500 deals, the agent “will actually write code to be able to answer the question,” uses programmatic tool calling to narrow the candidate set, and then “fan[s] out more work to independent agents that work on every single account” and rolls the results up.
  • Two weeks to ship. Watchtower went from idea to customers in about two weeks, with dogfooding about a week in. It is now “one of our most used features.” Months later they “haven’t rebuilt it or felt the need to go off of Managed Agents.”

Sandboxing: customer code in a safe box (Todd, product analytics)

  • Why a sandbox. The agent snapshots customers’ source code, which can expose secrets, so tools, data sent and actions all need care. The end result is often a pull request pushed back to the customer’s repository. “It’s not core to our value prop. It just needs to work. It needs to be safe and secure.”
  • Session types. Per-PR sessions review each change, including “a lightweight UX review” to keep bad UX from being committed. After merge, the agent opens its own instrumentation PR. Overnight batch jobs check for funnel conversion drops, day over day and week over week across 30 days, trace a drop back to what changed weeks earlier, and suggest a fix. Others review agent-conversation error rates and suggest prompt fixes, and interactive sessions read the code as the source of truth to answer a question. The stated goal is “self-healing software.”
  • Days, then weeks. They replaced a homegrown system that was running into problems: “Within a few days, we had working infrastructure. Within a few weeks, we had the whole product rolled over.” Elsewhere Todd puts it at “under 2 weeks”, with some tuning “to get it within our operational controls.”

Build vs buy

  • The rule (Sahaj). Buy when the harness “is not a core competency of that feature”. Build it yourself when the harness is the capability, such as a “fully voice-controlled Jarvis system.” “It’s a question of basically control versus cost and time.”
  • Speed during the search for product-market fit (Todd). “None of us in this roundtable, we’re not infrastructure folks.” In-house infrastructure may come later, for finer control or cost. Todd adds a fourth case for buying: when you are still testing whether an idea is viable, because “the cost of an experiment is in the time it takes to learn.”
  • What they actually tune. Sahaj built a first version “in a day” and scaled the user base “100 to 1,000X” over a few weeks. The iteration went into prompts, skills, tools and the loop order (search, retrieval, ranking, what to show), not compaction or prompt caching, which the managed harness handles.

Evals and model migration

  • Vibes first, evals once it sticks. “If the thing is not sticky, no amount of evaluation is going to make the thing sticky.” After that, build eval cohorts: regressions, hard scenarios, data partitions and task types, each with rubrics. Keep offline eval rubrics in sync with your outcomes rubrics. Internal dogfooding risks “overfitting to your own users,” and thumbs up/down from external users builds a decent set quickly.
  • Build evals from real customer queries (Mihir). Their sales-team dogfooding was “not representative.” What helped was taking real customer queries and asking the user why they ran it and what they wanted. Abstract evals were “too abstracted.”
  • Unsolved: evals for stateful systems. Nobody has a good offline eval for a system with live memory (trajectory evals only cover the interactions), or for MCP tools whose third-party state keeps changing. “Mocking a customer’s Slack, I’m not getting in that territory.”
  • On model migration, guard against new failure modes rather than tuning for the last 1%. A panelist says “the whole five series of models tends to produce writing in a way that has actually a lot more AI telltale signs… More em dashes”, and that known failure modes of a new model family are what they most want Anthropic to communicate.

Cost and what they want from Anthropic

  • Batch mode would save 50% to 75% (Sahaj). “If we’re running things 24 hours in advance, we could probably save 50%-75% on cost,” but “you can’t run batch mode that easily today.” He calls the infrastructure markup “pretty low.” His policy for now is to “be mindful, and spend well” and to optimize later.
  • Cost attribution (Todd). They want “really fine-grained telemetry of what calls cost what”, especially for expensive overnight batch processing. For now the cheap fixes are to run less often or send less data.
  • Cost levers are all or nothing (Mihir). The Watchtower coordinator needs frontier intelligence, but fanning out across 500 accounts forces cheaper models. Optimizing within Managed Agents “is very constraining because it’s all or nothing.” He asks for flex or batch mode (“a 2X, 50% right there”) and sandbox pre-warming to cut cold-start latency. His assessment: Managed Agents fits best when “you need frontier intelligence, you’re relatively price insensitive, and the task is hard, so it runs for minutes.”

Try It

  1. Add a do-not-show path. For any proactive agent output (briefs, digests, alerts), define an outcomes rubric with at least one objective identity check, and suppress the output when it fails rather than showing a hedged version. See the outcomes cookbook.
  2. Write rubric lines from the UX. List what the user should experience (order, length, scannability) and turn each into a graded rubric line next to the factual checks.
  3. Sort your memory into the two-by-two. Put each memory need in account vs cross-account and user vs org. Manage the core, high-churn store yourself and put the rest in managed memory stores.
  4. Use the build-vs-buy test. Ask whether the harness is what makes the feature good. If not, start on Managed Agents and revisit when cost or control blocks a feature (for example, needing a 90% cost cut).
  5. Seed evals from five real user queries. Contact the users behind them, record what they wanted, and write one eval per query before adding synthetic cases.
  6. Schedule non-urgent work early. If output is needed hours later (like the 24-hour briefs), run it early. Watch for a batch or flex option, which the panel estimates would save 50% or more.


Open Questions

  • The three companies are not named in the transcript. The descriptions (an “agent on every account” sales product, a product-analytics platform) may match known startups, but nothing here confirms which.
  • Several lines in the evals and model-migration section have no clear speaker. “A panelist” is used where the transcript does not show who is talking.
  • “The whole five series of models” writing more AI telltale signs is one panelist’s impression. No model versions or measurements are given.
  • The “50%-75%” batch saving is Sahaj’s estimate, not a measured figure, and the transcript doesn’t say whether Managed Agents has since added batch or flex.
  • Sahaj’s “100 to 1,000X scale-up in the user base” gives no absolute user counts.