Source: raw/How_to_Build_a_Real-Time_AI_Avatar_With_Claude_Voice_AI_Agent_Tutorial.md (auto-caption transcript; “claws set up” normalized to Claude setup, “turn-taken” to turn-taking)

Creator: AI Master (YouTube @iamAImaster) · URL: https://www.youtube.com/watch?v=DqUewrLQQfg · Published: not in the transcript · Sponsor: effectively Synthesia. The video names Synthesia’s interactive avatar API only near the end and points viewers to its beta access page. All capability claims are the creator’s demo plus the vendor’s, so confidence is low.

AI Master builds a spoken AI tutor with a face. Claude is the model, a long system prompt carries the teaching behaviour, and Synthesia’s interactive avatar API adds the face, the voice, lip sync and turn-taking, with LiveKit streaming the audio. Claude Code scaffolds the project from Synthesia’s quick-starts repository. The creator then stress-tests the tutor with an interruption and a language switch. The useful parts are the voice-specific prompting rules and the failure he found: background music triggered the avatar to stop and listen.

Key Takeaways

  • The avatar layer sits on top of your agent. Per the video, the platform “never touches your model or your framework.” It handles real-time rendering, lip sync and the logic for who speaks when. The voice is optional too, so you can keep your own voice pipeline. “The Claude setup stayed as it was, and the face went on top.”
  • Three keys: one for the avatar API, one for Anthropic, one for LiveKit (real-time audio streaming for low latency). They go in an environment file kept out of the repo.
  • Scaffold with a coding agent, not by hand. Download the platform’s quick-starts repo from GitHub (one starter includes a tool-use example), open it in Claude Code and give one short prompt. Claude Code reads the repo and scaffolds a fresh project. “Codex works exactly the same way.”
  • The tutor prompt has three jobs: explain one concept at a time in plain language, check understanding before moving on, and make the learner try writing a prompt out loud. “A tutor that only lectures is just a video with extra steps.” It is also told to keep answers short.
  • The demo tutor coaches instead of grading. When the learner’s practice prompt lacked a goal and details, the tutor said what was missing, asked for another try, and only then confirmed it was right.
  • Interruption test: passed. When cut off mid-answer with a new question, it stopped, answered the new question, and then restated the explanation that had been cut off.
  • Language-switch test: passed. A mid-lesson switch to French (“Parlez-vous français?”) got an answer in French and the lesson continued in French.
  • Failure found: background noise counts as speech. Quiet music in the room made the tutor stop mid-explanation and start listening. Plan for this if users will be somewhere noisy.
  • Voice changes the prompting rules. Rewrite anything that reads fine but “sounds like a manual when spoken.” A spoken wall of text feels much longer than a written one.
  • Latency adds up across the stack. The model, the tools and the avatar each take a turn. “A fast model with short answers feels more alive than a smart model that monologues.” Pick the model with that in mind.
  • Choose a calm avatar for long sessions. For a beginner tutor the creator picked someone “calm and neutral,” since “a hyper-energetic presenter gets tiring over a long lesson.” A custom avatar of a real person, such as the course author, is also possible.

Try It

  1. Get the agent right in text first. “The avatar makes a good agent feel present. It won’t fix a vague one.”
  2. Keep the knowledge small. The creator says the realistic projects are those whose knowledge fits in a prompt or a small knowledge base: tutors and practice coaches, onboarding guides for the setup steps people get stuck on, and website product concierges.
  3. Write the system prompt for the ear. Short turns, one idea per turn, a check-question at the end of each explanation, and no lists or formatting that only work on screen.
  4. Test the two places voice agents usually fail: talk over it mid-sentence, and switch language mid-conversation. Then add a third test with background noise or music.
  5. Pilot on one page. For a customer-facing guide, start on the page where visitors drop off and watch whether they talk to it or scroll past.

Implementation

Tool/Service: Synthesia interactive avatar API (closed beta at the time of the video), LiveKit for real-time audio, Claude via the Anthropic API, Claude Code (or Codex) to scaffold from the quick-starts repo.

Setup: Request beta access on Synthesia’s interactive avatars page, with a concrete use case in mind. Clone or download the quick-starts repo, open it in Claude Code, and prompt it to scaffold a project and swap the default model for Claude. Add the three API keys to .env. Write the system prompt. Pick an avatar in the platform’s avatar builder.

Cost: Not stated in the transcript.

Integration notes: The video mentions a “coding agent skill” alongside the quick-starts. Synthesia says customers have built agents that guide buyers through choosing an insurance plan or booking a car test drive (vendor claim, relayed in the video).

Open Questions

  • Measured latency. The video says to watch the reply gap and the lip sync, but gives no numbers. Compare with the published latency figures in Voice Agent Comparison.
  • Noise handling settings. Whether the platform exposes voice-activity or noise-suppression controls to fix the music problem is not covered.
  • Pricing and general availability of the interactive avatar API.
  • Which Claude model the demo used.