Source: raw/Why_PMs_are_suddenly_valuable_again_Tulsee_Doshi_Head_of_Product_Gemini.md

Creator: Lenny Rachitsky, Lenny’s Podcast, with Tulsee Doshi, head of product for Google’s frontier models (Gemini, the generative-media models including Nano Banana and the Lyria music model, the Gemma open models, and the audio models) · URL: https://www.youtube.com/watch?v=qGf2Yvan4Wg · Duration: not recorded · Platform: YouTube (auto-captions, fetched 2026-11-09). Sponsors: WorkOS, DX.

Doshi spent six years leading product for Google’s responsible-AI team before this role. Most of the episode is practical advice for product people working on or with models. Start evals with five prompts. Judge a new model on a task you care about where models are close but not there yet. Build so that 60% works today and 20% is a bet on the next model. Keep AI-written documents to data, with the insight coming from you. Her framing of the job: a model is not predictable, “like Snape’s potions class”, so the PM shapes what goes in and then finds where the result shines.

Key Takeaways

  • Models are grown, so the PM looks at both ends. Set goals, capabilities and eval targets, but expect “a little bit of magic” that surprises you; then “identify where does the model shine… and shape that into something that is amazing for users.”
  • The model has a UX. How transparent it is, how it communicates, what it asks back. Her model-behaviour team writes a spec of “what does good look like for Gemini”, turns it into rubrics, evaluates against them, and feeds that into training data and reward modelling.
  • What she hires for. PMs who “understand the vibes of our models or other models”: they use them daily, have their own tests of whether a model is good, will dig into data and build an eval themselves, and rewrite plans often (“A road map you wrote two months ago, you’re going to be looking at it again”).
  • Evals: start with five prompts. Skip the big benchmark. Even “a mini evaluation where you get to like 20 prompts” that shows a difference between model A and model B is useful. Be the rater yourself at first; on long multi-step trajectories, deciding what good looks like is the hard part and gives “self clarity” before you scale.
  • Evaluate the whole stack. The judge model’s quality matters as much as the model’s; so do your production mitigations and filters, and how the model is served. Her audio example: serving can change naturalness and inflection “even if your model is good”, so test end to end.
  • Check whose scale your eval uses. Her earlier team replaced the six-tone Fitzpatrick scale, built for a medical purpose and skewed to light skin, with Harvard professor Monk’s ten-tone scale (built from years of interviews and research), then re-ran evaluations for Pixel camera and image generation.
  • Vibe-test on something you care about. No single right test: people try 3D animation, writing, financial analysis. Pick a task where models show “signs of life” but are not perfect by your standard, so you can see models differ.
  • The 60/20/20 rule for building ahead. About 60% of what you build works now and the prototype is functional; about 20% works “well enough” and you can see the models getting there soon; the last 20% is where you want models to be, where you see “glimmers of light”. The numbers are “fungible”. Her reason: even she cannot say where a model will be in three months, so use your own product’s signals instead of a fixed “build three months ahead” rule.
  • Don’t build rigidly around today’s model. Customers who built tightly around an older model got stuck when it improved. Lenny relays Boris Cherny’s advice to remove your skills now and then and see if the model does as well without them; Doshi: “consistently retest”, because your structure may be holding a new model back.
  • Harness, defined: “the wrapper around the model”: system instructions, tools, skills and context, “an environment in which to operate”.
  • How she upskills her PM team.
    • PMs who lean in write skills others can use; the team is building a shared skills repository.
    • Workshops run by PMs who are good at reading long traces and data.
    • Time and space. A two-day workshop in November takes every PM off their day job to build with the tools.
    • Her rule: “if you have to do something twice, do you need to do it yourself?” A strategy, yes; repeatable work, no.
    • It takes one example that works to change someone’s habits.
  • AI-written docs: the model brings data, you bring insight. Good use: a usage-and-feedback report two weeks after launch, pulled from dashboards, metrics and social posts. Bad use: the insight itself. The team is investing in skills that fetch the right data so documents are structured well. She sees the “X is the new PRD” posts as one shift: the PM now has to bring “heft” (a prototype, data, an eval) instead of a list of requirements.
  • Burnout comes from two places, she says: churn and chaos, and work you do not enjoy. Long hours on work you care about are tiring but not the same thing. She hopes the tools remove the grunt work.
  • Flash teams. Her Stanford research with Michael Bernstein assembled short-lived crowdsourced teams on Mechanical Turk for a task; she now applies the same principles to setting up subagents.
  • Safety, as she describes Google’s approach: monitors that read the model’s chain of thought to catch actions that do not match its instructions, and defensive security work, including Wiz’s scan-for-good programme finding and patching a hospital system’s vulnerability.
  • On Google’s position. She expected listeners to be using “Gemini 4 Argon” by air date and says Google wants “to win the frontier conversation”. On a Google personal agent using Gmail and Calendar data: “stay tuned… there is more work we need to do.”
  • Favourite non-Google AI product: Wispr Flow, for dictation, because of wrist problems.

Try It

  1. Write five prompts that test the capability you care about most, rate two models’ answers yourself, and grow it to about 20 once you know what “good” means.
  2. Keep one personal vibe test: a task where current models are close but not there, and run it on every new model.
  3. Audit your next roadmap with 60/20/20: label each feature works-now, nearly-there or bet, and check the bets are not more than about a fifth.
  4. Remove your skills or prompt scaffolding for one run after each model upgrade and compare.
  5. Split AI-drafted documents into data sections (AI) and insight sections (you) and say which is which.
  6. Apply “twice means delegate” to your week: list tasks you did twice and hand one to an agent.
  7. When you evaluate a voice or judge-based system, test through the production stack, not the bare model.

Open Questions

  • Doshi says the figure of 14 Google products above a billion users was something she “should check”.
  • The safety and Wiz claims are first-party descriptions, not independently reported here.
  • The internal tool named “Jet Ski” is not described.