Jonathon's AI Wiki

Voice Agents with Claude Code + ElevenLabs (Nate Herk)

11 min read

Source: “Building Realistic Voice Agents Has Never Been Easier” YouTube tutorial by Nate Herk, youtube.com/watch?v=-cdexJWN8YA, fetched 2026-05-04. Author self-attributes through “Hey, I’m Nate’s AI assistant” demo + the 400-YouTube-video knowledge base reference + the GLO team membership disclosure. Sister tutorial to Nate’s AIS-OS masterclass but narrower — single recipe, end-to-end voice-agent build.

A live-build walkthrough showing Claude Code (running inside VS Code) configuring an ElevenLabs voice agent end-to-end via natural language — system prompt, voice selection, knowledge base, tool calls, and embedded widget on a website — without manually clicking through the ElevenLabs dashboard. The headline demo is a voice agent trained on 400 YouTube transcripts that visitors can talk to about Nate’s content; the buildable demo is an AI-consultancy sales agent that books discovery calls into Cal.com directly from the conversation. Sits alongside n8n Vapi voice agents (the workflow-orchestration approach to voice agents) as the Claude-Code-native alternative.

Key Takeaways

  • Voice agent = a four-piece structure. Every ElevenLabs voice agent has the same shape: (1) Persona (the system prompt that defines how the agent talks — warm, professional, rude, joke-after-every-sentence, conversational like the founder); (2) Voice (selected from ElevenLabs’ library or, in Nate’s case, a 4-hour professional voice clone); (3) Knowledge (business info, customer database, transcripts, product docs — whatever the agent needs to answer from); (4) Tools (MCP servers, direct API calls, n8n/Python/Zapier triggers, calendar booking endpoints). Once you understand these four primitives, “build a voice agent” becomes a structured Claude Code prompt instead of an opaque dashboard tour.
  • The agent runs as a loop, not magic. Visitor speaks → microphone captures → speech-to-text → LLM reads + decides → either responds, queries the knowledge base, or calls a tool → response synthesized to speech → loop repeats. Latency comes from the LLM round-trip; tool calls add a hop. The mental model fix matters because it makes failure modes (slow tool calls, knowledge-base misses) predictable.
  • Three deployment modes for a voice agent. (a) Dashboard-only (test/internal) — talk to it inside the ElevenLabs UI; (b) Website widget (covered in this build) — a single <script> snippet pasted onto any HTML page renders a floating “Start Call” bubble; (c) Phone number via Twilio integration — the voice agent picks up inbound calls or makes outbound calls from a paired Twilio number. Same agent definition, three surfaces.
  • The Claude Code unlock — code beats clicks. ElevenLabs’ dashboard requires manual configuration of system prompt, first message, knowledge upload (raw doc vs vector store choice), tool definitions, and endpoint setup. Each step has save-state risk and configuration drift. Claude Code reads the ElevenLabs documentation (or has it in training context), understands every setting, and configures the agent from a high-level natural-language brief: “build me a sales agent for my AI consultancy that books Cal.com discovery calls.” Setup time drops from “hours of clicking” to “15 minutes of talking.”
  • Plan Mode is the entry point. Nate’s recipe: open Claude Code in VS Code, click Plan Mode in the bottom-right, then describe the agent in natural language with the goal stated explicitly (“ultimately what I want to do is push people to book in a discovery call. That’s kind of your job. So you’re kind of like a salesperson”). Plan Mode asks clarifying questions instead of immediately generating code — current ElevenLabs setup state, current Cal.com setup state, agent persona preference, data fields to capture, widget appearance. Answer the questions, then leave Plan Mode and let Claude execute. Same pattern as Six Best Claude Code Skills’ Plan-Mode-first discipline.
  • The starter project is just an HTML landing page. Nate’s pre-existing project: one folder with a vanilla HTML file (a simple “Neural” AI-consultancy landing page). The voice-agent build adds the ElevenLabs <script> widget snippet to that page — that’s it on the frontend side. If you don’t have a website yet, the build still works for the agent; you just won’t have an embed surface at the end.
  • Cal.com → ElevenLabs direct, no n8n bridge. Nate’s deliberate architectural choice: instead of ElevenLabs → n8n → Cal.com booking (the typical workflow-orchestration path), connect ElevenLabs directly to Cal.com via tool definition. “Too many pieces” is the rationale — fewer hops = lower latency + fewer breakage points. Same anti-pattern call-out as in the Higgsfield MCP campaigns tutorial — when a vendor exposes an API, prefer a direct tool call over a workflow detour.
  • Required data fields for booking are the persona’s escape velocity. Cal.com booking needs full name + email at minimum; agents typically also capture company name, the prospect’s stated problem, and a budget signal. These are the fields you tell Claude Code in Plan Mode — they become the conversation graph the agent steers toward. The agent stays “warm, professional B2B sales tone” by default until enough fields are captured to book; then it pivots to closing.
  • Voice clone is the credibility unlock. Nate’s demo agent uses his actual professional voice clone (4 hours of training audio in ElevenLabs). The visitor’s first reaction in the source (“you guys might have noticed that that one sounded like me”) is the proof point — voice fidelity changes the agent from “AI assistant” to “the founder, but available 24/7.” Same surface as HeyGen Avatar V’ identity-faithful video avatars but for audio.
  • VS Code over the standalone Claude Code desktop app. Nate’s stated preference: “I personally don’t love using the desktop app. I actually prefer to use VS Code, which is just an IDE. It’s completely free to download.” Claude Code installs as a VS Code extension; once logged in to a paid Claude account, it opens in the same window as the project. This is a recurring operator-track pattern in the 2026 Claude Code AIOS Pattern — the IDE is the workspace, the desktop app is for one-off tasks.
  • Adjacent tools mentioned in passing (operator stack signals). GLO (faster, private replacement for Whisper for dictation; Nate joined the team) — Nate dictates the entire Plan-Mode brief instead of typing. Superpowers skill for brainstorming flow (“just go to Google and search Claude Code Superpowers”). Firecrawl as the canonical scraping tool when an agent needs web data. These are stack confirmations more than recommendations — the same toolchain shows up in the Six Best Claude Code Skills writeup and Eliot Prince’s Live Artifacts recipes.

Anatomy of the Plan-Mode Prompt

The exact prompt Nate dictates into Claude Code (transcript lines 292-328) is the canonical template — copy and adapt:

“Hey Cloud Code, what I’ve got for you is a bit of a project. So, I have a website which is called Neural. It is inside of this project and it’s basically just our landing page. We run an AI consultancy and this is our landing page. And what I want to do is embed a voice agent widget onto my website. I want this agent to have information about the business. I want it to be able to answer questions from prospective clients. But ultimately what I want to do is try to push people to book in a discovery call. So you’re kind of like a salesperson, a sales agent on my website. So what I want you to do is help me figure out how do I embed you onto the website? I want to use 11 Labs. And then how do I actually prompt you in the right way to be a salesperson. And then how do I actually connect you to cal.com, which is where I’m using kind of like my booking. It’s synced to my calendar. So, it should show people available slots. And how can I give you access to that tool so that you can actually just go ahead and book meetings for people? They would give you their email and full name and then you would just go ahead and use the cal.com tool and book a call for them. So, that’s kind of the high-level goal that I have for you. Help me figure out the best way to do this and ask me any questions that you have if I didn’t explain anything clear enough.”

The structural pieces that make this prompt work:

  • Project context (“Neural” + AI consultancy + landing page exists)
  • Goal stated as outcome (not as features): book discovery calls
  • Role assignment: “you’re kind of like a salesperson, a sales agent”
  • Three sub-tasks named explicitly: embed, prompt, connect-to-Cal.com
  • Permission to ask questions at the end — invokes Plan Mode’s clarification cycle

Implementation

Tool/Service: Claude Code (VS Code extension) + ElevenLabs (paid account; voice agent build) + Cal.com (free or paid; calendar booking) + optional Twilio (paid; phone deployment)

Setup:

  1. Open VS Code, install the Claude Code extension from the Extensions panel, sign in to a paid Claude account.
  2. Open the project folder containing your landing page (or any blank folder if you don’t have a site yet).
  3. Open Claude Code in the panel (top-right button in VS Code).
  4. Click Plan Mode in the bottom-right.
  5. Dictate or type the brief (template above), name the tools (ElevenLabs, Cal.com), and explicitly invite clarifying questions.
  6. Answer Claude’s clarifying questions (current ElevenLabs setup state, current Cal.com setup state, widget appearance preference, voice persona, additional data fields beyond name + email).
  7. Leave Plan Mode and let Claude execute. It will create the agent in ElevenLabs, configure the system prompt, attach the Cal.com tool, and inject the widget snippet into your HTML.

Cost: ElevenLabs voice agent — paid tier required (Conversational AI plan; free tier doesn’t cover production agents). Cal.com — free tier sufficient for individual use. Claude Code subscription required (Pro/Max). Twilio optional for phone deployment (per-minute pricing).

Integration notes:

  • Cal.com integration uses Cal’s API key — Claude Code walks through generating it.
  • ElevenLabs voice clone is a separate one-time training step (4 hours of clean audio) — covered as background context, not in the live build.
  • Knowledge upload: Nate’s demo uses YouTube transcript bulk upload; for support/sales agents, upload product docs, FAQ, ICP notes.
  • Widget snippet is a single <script> tag from the ElevenLabs Widget tab — Claude Code copies it into the right place in your HTML.
  • The integration as shipped in the source is direct ElevenLabs → Cal.com; no n8n / Zapier / Make middleware. Choose this path unless you need workflow logic between booking and your CRM.

Try It

  1. Watch the source video once end-to-end before starting the build — the Plan-Mode dialogue pattern matters more than any single click.
  2. Stand up a minimal HTML landing page (or use the one you already have) — the voice agent needs a surface to embed into.
  3. Sign up for ElevenLabs (Conversational AI plan) and Cal.com. Generate API keys for both.
  4. Open VS Code with the project folder, open Claude Code, enter Plan Mode.
  5. Dictate the brief using the template above, swapping “Neural / AI consultancy” for your actual business name and offer.
  6. Answer the clarifying questions honestly — agent persona and data fields-to-capture are the two that shape the conversation graph the most.
  7. Let Claude execute outside Plan Mode. Test the embedded widget by clicking “Start Call” on your landing page.
  8. Iterate the system prompt by talking to Claude Code, not by clicking around the ElevenLabs dashboard. That’s the whole point.

Open Questions

  • Voice clone training source. Nate’s professional voice clone is “4 hours of my voice” — but the source doesn’t break down how that audio was prepared (clean room? podcast cuts? script reads?). The 4-hour figure is repeatable; the prep matters for fidelity.
  • Knowledge-base mode trade-offs. ElevenLabs supports raw-document upload OR vector store (Supabase / Pinecone / NotebookLM). Nate uses bulk transcript upload; the source mentions vector stores as an alternative without comparing latency or recall. Worth a future article on which choice fits which agent type.
  • Cal.com booking failure modes. What happens if the agent attempts a booking and Cal.com rejects (slot taken, time-zone mismatch, required field missing)? Source’s demo shows the happy path only. The prompt template should probably include error-handling language (“if the booking fails, apologize, ask for an alternative time, retry”).
  • Cost economics at scale. The source pitches “minimal manual configuration” but not “minimal per-conversation cost.” A voice agent on Opus-grade reasoning + ElevenLabs voice synthesis will run noticeably more per call than text chat. The “build a voice agent in 15 minutes” framing under-states ongoing OpEx.

Graph View

Backlinks