Jonathon's AI Wiki

AI Voice

3 min read

Voice models, voice agents, real-time speech infrastructure, and end-to-end conversational AI. Covers both open-source foundation models (Moshi, Mimi) and commercial voice agent stacks (ElevenLabs Conversational AI). Distinct from AI Video & Content Production (which covers HeyGen avatar models, lipsync, and video composition with audio); this topic is voice-first / audio-first.

Articles

  • Moshi — Kyutai Labs’ Full-Duplex Speech Foundation Model — 7B Temporal Transformer + small Depth Transformer + Mimi neural audio codec. Real-time conversational AI with theoretical 160ms / practical ~200ms latency on L4 GPU — single foundation model handles full-duplex dialogue with no STT→LLM→TTS pipeline. Three runtimes: PyTorch (research, 24GB+ VRAM), MLX (Apple Silicon local), Rust (production with CUDA/Metal). Code MIT + Apache, weights CC-BY 4.0. 10,163 stars.
  • Voice Agents with Claude Code + ElevenLabs (Nate Herk) — Migrated from claude-ai/ 2026-07-03. Live-build walkthrough: Claude Code configures an ElevenLabs voice agent end-to-end via natural language. Four-piece structure (Persona / Voice / Knowledge / Tools), pipelined STT→LLM→TTS architecture with ~500-800ms typical latency, three deployment modes (dashboard, website widget, Twilio phone number).
  • OpenAI Realtime API — Native Speech-to-Speech Voice Agents — OpenAI’s native speech-to-speech voice model (gpt-realtime → gpt-realtime-2, GA Aug 2025 / May 2026). Single model processes and generates audio directly, no STT→LLM→TTS pipeline — same architectural category as Moshi. Token-based pricing (64/1M audio output tokens), adjustable reasoning-effort/latency tradeoff, WebRTC/WebSocket/SIP transports.
  • Cartesia Sonic — Low-Latency Text-to-Speech (State-Space Model) — Cartesia’s TTS-only model built on state-space-model (Mamba lineage) inference rather than Transformers. Sonic 3.5 (May 2026) markets sub-90ms latency; an independent production benchmark measured meaningfully higher and more variable P50 188ms — flagged as an open marketed-vs-measured discrepancy. 42 languages, instant voice cloning from a 10-second sample.
  • whisper-cli (whisper.cpp) — Free Local Speech-to-Text — The free, self-hosted, offline path to OpenAI’s Whisper model weights (not the hosted API). MIT licensed, Metal/CUDA/CPU backends, six model sizes (tiny to large-v3). Not built for live conversational latency (0.5-2s behind live speech) — the offline/batch/privacy tier rather than a real-time competitor to the other three articles here.
  • Fluid Voice and Voice Box — Free Self-Hosted Alternatives to Whisper Flow and ElevenLabs — Two free, self-hosted tools from a 2026-07-06 GitHub-trending roundup: Fluid Voice (local dictation, bundled ~3.5GB voice model, no API keys) and Voice Box (voice cloning + TTS, clones a voice in under 30 seconds from a short sample). The pitch is cost/control, not capability — cancel a paid Whisper Flow ($15/mo) or ElevenLabs subscription in favor of a local, self-hosted equivalent.
  • Voice Agent Comparison — Moshi vs ElevenLabs vs OpenAI Realtime vs Cartesia Sonic — Four-way synthesis across architecture (native speech-to-speech vs. pipelined), latency, pricing, open vs. closed, and use-case fit.

Voice-relevant articles in other topics:

  • yt-dlp — YouTube transcript / audio extraction tool used by bin/yt-transcript in this vault, plus local whisper-cli fallback for captionless videos via ~/.whisper-models/ggml-*.en.bin. Source dependency for last30days and claude-video.
  • HeyGen Hyperframes — HTML video composition with TTS / lipsync layers.
  • HeyGen Studio Automation — Avatar V production pipeline (TTS + lipsync + multi-clip composition).
  • OpenClaw on Rabbit R1 — voice as input to a self-hosted agent fleet. Pocket-hardware voice surface; pairs with Moshi or ElevenLabs at the model layer if you self-host both.
  • Code with Claude 2026 keynote — Anthropic conference frame for voice/agent integration.

7 items under this folder.