Source: wiki synthesis: moshi-kyutai, voice-agents-elevenlabs-claude-code, openai-realtime-api, cartesia-sonic, whisper-cli. 2026-10-27 addition (new models, host-relayed): raw/AI_Breakthroughs_Everyone_Missed.md, raw/AI_News_-_Dots_GPT-6.1_Sol_Sonnet_5.5_Gemini_4_and_everything_you_need_to_know.md, raw/AI_News_-_6_New_Models_A_Nano_Banana_Killer.md, raw/ai_put_minecraft_in_skyrim_and_open_ai_had_a_dev_day.md, raw/Gemini_4_Argon_Meta_Muse_Wonder_Are_Good._Not_Great..md, raw/8_GitHub_Repos_To_Go_Viral_and_Make_Money.md, raw/Ep._245_-_OpenAI_DevDay_+_Dots_+_Safety_Crisis_Trump_Renames_AI_Anthropic_Pre-Thanksgiving_IPO.md. 2026-10-30 field note (HeyGen official channel, vendor-made): raw/Introducing_HeyGen_Voice_-_The_First_Voice_Model_Built_to_Adapt_to_Video.md, raw/Best_AI_Voice_Generator_-_HeyGen_vs_ElevenLabs_vs_Cartesia_vs_Inworld_vs_Fish.md
Four voice-tech stacks now live in this topic, each occupying a genuinely different point in the design space rather than competing head-on. This article synthesizes them side by side — architecture, latency, cost, and openness — so the choice of stack for a given use case is a lookup, not a fresh research pass each time. whisper-cli sits alongside as the free/offline reference point, though it isn’t a live-conversation competitor to the other three.
Key Takeaways
- Two architectures, not four. Moshi and OpenAI’s Realtime API are native speech-to-speech — one model processes and generates audio directly, no separate STT/LLM/TTS stages. ElevenLabs and Cartesia Sonic are pipeline components: ElevenLabs is a full pipelined stack (STT→LLM→TTS) that also does the orchestration; Cartesia Sonic is TTS-only, one leg of a pipeline you assemble yourself.
- Moshi is the only open, self-hostable option. Code MIT+Apache 2.0, weights CC-BY 4.0. Every other stack here is closed and hosted-only. This is the single sharpest dividing line in the group — it determines whether you can run the model on your own infrastructure at all.
- No single “fastest” stack — it depends which latency number you trust. Moshi publishes ~200ms practical latency. Cartesia markets sub-90ms TTS latency but an independent production benchmark measured P50 188ms with a long tail past 269ms. OpenAI doesn’t publish a headline number at all — independent sources cite 300ms to 2.3 seconds depending on the reasoning-effort setting. ElevenLabs’ pipelined architecture typically runs 500-800ms end-to-end. Vendor-marketed and independently-measured numbers disagree often enough that any single figure should be treated as a starting hypothesis, not a fact, until benchmarked in your own conditions.
- Pricing models aren’t directly comparable. Moshi is free (self-hosted compute cost only). OpenAI is token-based on audio duration (~0.05-0.10/min cached, blended). ElevenLabs and Cartesia are character-or-minute-based subscription tiers (Cartesia: 299/mo Scale; Voice Agents billed separately at $0.06/min+telephony). Converting between these requires knowing your actual usage pattern, not just a quoted rate.
- Independent comparisons converge on a three-way, not four-way, tradeoff: OpenAI wins on conversational intelligence and tool-use reliability; ElevenLabs wins on emotional range and voice-cloning quality; Cartesia wins on raw TTS latency (when the marketed number holds). Moshi’s differentiator isn’t in that same competition — it’s the only one you can self-host, which matters for privacy/cost-at-scale rather than head-to-head conversational quality.
Side-by-Side
| Dimension | Moshi | ElevenLabs | OpenAI Realtime API | Cartesia Sonic |
|---|---|---|---|---|
| Architecture | Native speech-to-speech (single model) | Pipelined (STT→LLM→TTS) | Native speech-to-speech (single model) | TTS-only (one pipeline stage) |
| Open vs. closed | Open-weight, self-hostable | Closed, hosted only | Closed, hosted only | Closed, hosted only |
| Latency (marketed) | ~160ms theoretical / ~200ms practical (L4 GPU) | ~500-800ms typical | Not published | Sonic 3.5: sub-90ms model latency |
| Latency (independently measured) | Not independently benchmarked in sources gathered | Not independently benchmarked in sources gathered | 300ms-2.3s (reasoning-effort dependent) | Coval production P50: 188ms, P75: 269ms |
| Pricing model | Free (self-host compute only) | Character/minute subscription tiers | Token-based (audio in/out) | Character/minute subscription tiers + separate Voice Agents rate |
| Independent-comparison strength | Self-host + latency | Emotional/voice-cloning quality | Conversational intelligence, tool-use | Raw TTS latency (marketed) |
| Voice cloning | Two fixed voices (Moshika/Moshiko) | Yes, full library + cloning | 10 voices (Cedar, Marin + 8 legacy) | Yes, instant clone from 10-second sample |
| Deployment surfaces | PyTorch / MLX (Apple Silicon) / Rust runtimes | Dashboard, website widget, phone (Twilio) | WebRTC, WebSocket, SIP | REST/WebSocket API |
Which One, For What
- Need to self-host, control the model, or avoid per-minute vendor billing entirely → Moshi. The only option that runs on your own infrastructure; MLX runtime makes it genuinely usable on a local Mac.
- Need a full agent (persona + knowledge base + tool calls) with minimal build effort, and voice-cloning/emotional range matters → ElevenLabs. Its pipelined architecture and dashboard tooling (or the Claude-Code-configured path in this vault’s ElevenLabs article) is the fastest path to a working sales/support agent.
- Need strong tool-calling reliability, MCP integration, or telephony (SIP) built in → OpenAI Realtime API.
gpt-realtime-2’s parallel tool calls and remote MCP support are the most agent-native feature set of the four. - Building your own pipeline and want the fastest TTS leg specifically, willing to bring your own STT+LLM → Cartesia Sonic — but benchmark the actual latency in production rather than trusting the marketed figure alone.
- Need free, offline, private transcription (not conversation) → whisper-cli — not a competitor in this table’s conversational-latency race, but the right default for batch/dictation/captioning work where no live back-and-forth is needed.
October 2026 update: new voice models
New models reported by YouTube and podcast hosts in October 2026. They are not yet added to the side-by-side table above, which covers the four stacks with full wiki articles. All claims are the hosts’ own relays.
- Fish Audio Drama 3: TTS directed in plain English (Matt Wolfe,
raw/AI_Breakthroughs_Everyone_Missed.md).- You write stage directions in square brackets in the script (for example “barely holding it together, forcing a smile”). You can scope a direction to specific words, drop in sound effects such as a small laugh or a shaky breath, and use multiple speakers.
- On Artificial Analysis it scores highest on pronunciation, above ElevenLabs. In the blind arena preference vote it does worse: “it reads hard words better than almost anything else, but a lot of people still prefer other voices.”
- Open weights on Hugging Face at 36B parameters. Users report needing about 46 GB of VRAM, or 32 GB with a community 4-bit quant.
- The license is free for commercial use only under 10 per million characters as a launch discount, rising to $50 after November 12.
- Gotcha: set the model to Drama 3 explicitly in the API. It “will quietly fall back to older models and you’ll still get billed”. Takes vary, so generate a few and keep the best.
- ElevenLabs v4. Matt Wolfe calls it ElevenLabs’ “most emotive model”, with better voice cloning; you can tell it how to sound (excited, sleepy). AI For Humans say it was trained on how people used v3’s tags and which takes they kept, so it adds expression automatically. You can still steer it with tags. It handles coughing, whispering, screaming, singing, laughing and applause.
- ElevenLabs instant soundtracks in Eleven Creatives. Upload a video and it generates a synchronized soundtrack and sound effects (Matt Wolfe).
- Microsoft MAI Voice 2.1 with expressive multilingual speech, plus a Microsoft streaming transcription model for near-real-time captions. Wolfe’s side-by-side: “maybe 11 Labs has the edge.”
- Grok 3 voice. AI For Humans say its pitch is low price for call-center use at scale; Gavin Purcell’s verdict was “it wasn’t that special”.
- Voice Studio (Greg Isenberg,
raw/8_GitHub_Repos_To_Go_Viral_and_Make_Money.md). A free desktop app he describes as “like an open-source 11 Labs”: speech generation, cloning from your own recording, and dubbing into other languages, all on local models, so speed depends on your hardware. His use case is translating onboarding and how-to videos, “with a fluent person checking the result”. He says to check the commercial license and to clone only voices you have permission to use. No repo URL or star count is in the transcript. - HeyGen Voice (field note 2026-10-30, HeyGen’s own videos). HeyGen announced its own voice model for avatar video in a 2026-10-14 video. It clones from about 60 seconds of phone audio and adapts the delivery and room acoustics to the avatar’s scene. It is a TTS voice for pre-rendered video, not a real-time agent stack. A HeyGen-made five-way cloning test (same source recording, unseen script, one presenter scoring six categories) ranked HeyGen Voice 28.5/30, ElevenLabs v4 Turbo 27.5, Cartesia Sonic 3.6 26.5, Inworld Realtime TTS-2 25 (20 in its own segment), and Fish 24.5. ElevenLabs v4 Turbo was “the clear winner” on emotion, which matches the emotional-range point above. This is the first voice-quality comparison in the topic and partly answers the open question below, but it is vendor-run and subjective. See HeyGen Voice and AI Voice Generators Compared (HeyGen, 2026).
- Turn-taking benchmark for any live voice or video agent (Paul Roetzer on the AI Show, unsourced). The gap between human speakers is about 200 ms; under 300 ms feels natural, 300 to 500 ms feels awkward, and over 500 ms reads as a dropped call. Compare it with the latency figures above: by this yardstick only sub-300 ms end-to-end stacks will feel natural. ^[inferred] For the real-time video avatar the same episode covers (Tavus Griffin), see HeyGen Avatar V.
Implementation
Tool/Service: N/A — this is a comparison article; see each linked article’s own Implementation section for setup details. Integration notes: None of these four are strictly interchangeable at the API level — switching stacks means re-architecting the STT/LLM/TTS boundary (or removing it entirely for Moshi/OpenAI Realtime). Prototype the conversational flow against one stack before assuming portability to another.
Related
- Moshi — Kyutai Labs’ Full-Duplex Speech Foundation Model
- Voice Agents with Claude Code + ElevenLabs (Nate Herk)
- OpenAI Realtime API — Native Speech-to-Speech Voice Agents
- Cartesia Sonic — Low-Latency Text-to-Speech (State-Space Model)
- whisper-cli (whisper.cpp) — Free Local Speech-to-Text
- Vapi Voice Agents (n8n) — the workflow-orchestration alternative to any of these four for building a voice agent without an ElevenLabs/Claude-Code-direct integration
- OpenClaw on Rabbit R1 — a hardware surface that could pair with Moshi or ElevenLabs at the model layer
- HeyGen Voice — video-adaptive voice cloning for avatar video (TTS, not a live agent)
- AI Voice Generators Compared (HeyGen, 2026) — vendor-made five-way voice-clone quality test, with a reusable method
- The Voice-Agent Lead-Qualification Stack — Model, Orchestration, CRM
Try It
- Match the “Which One, For What” section above against the actual constraint driving the decision (self-host requirement? tool-calling depth? voice-cloning fidelity? raw latency?) before defaulting to whichever vendor is best-known.
- If latency is the deciding factor, don’t trust any single marketed number in the table above — the Cartesia-vs-Coval gap shows why. Run a small production benchmark against the top 2 candidates in your actual network/geographic conditions.
- For a first build with minimal setup, the ElevenLabs + Claude Code recipe has the most complete end-to-end walkthrough already in this wiki.
Open Questions
- No source in this cluster independently benchmarked Moshi’s or ElevenLabs’ production latency the way Coval did for Cartesia — the comparison table’s “not independently benchmarked” cells are a real gap, not just an omission.
- Voice quality / naturalness (as opposed to latency and architecture) was not systematically compared across all four in any single source gathered — this table is latency/cost/architecture-focused, not a blind-listening-test result. (Partial answer 2026-10-30: HeyGen’s five-way cloning test scores ElevenLabs and Cartesia on clone quality, but it is vendor-made, uses a single listener, and leaves out Moshi and OpenAI Realtime.)
- Whether any of these four is actually the right fit for a WEO/OmniPresence dental-client voice-agent use case (e.g. appointment booking, intake) is unexplored — this article is a general-purpose comparison, not a WEO-specific recommendation.