Jonathon's AI Wiki

AI Video Tools

14 min read

Full annotated catalog: AI Video & Content Production — Full Catalog — every entry below with its complete description. This index carries the one-line version for navigation.

Practical AI video production: avatar models, automation pipelines, composition frameworks, and motion graphics. Each article is compiled from primary sources (product pages, technical reports, tutorials, and open-source repos) rather than press summaries.

WEO project status (2026-06-27). The first post-production pilot — WEO Marketly brand promotion (the agency itself, not client/OmniPresence work) — is in post-production / delivery. Shoot completed April 5-6, 2026; the WEO Media + Marketly Digital merger is public as of 2026-06-18 (CTA domain weomarketly.com). The built pipeline is HeyGen Avatar V twins (v3 API + the weo_video library) + HyperFrames motion-graphics + real-footage b-roll, finished with PIL/PNG captions, two-pass −14 LUFS audio, mandatory deband, and Drive delivery. Higgsfield generative b-roll was tried and dropped (502s + charged failures); VEO/Runway are not in the path; Remotion is a reference, not the chosen tool. Details: the “WEO Marketly Promo” and “Content Production Workflow” articles (both internal, unpublished — see the internal-articles list at the end of this index).

AI Avatars & Generation Models

  • HeyGen Avatar V — Production-scale video-reference avatar model. 15s webcam clip → unlimited-duration 1080p twins with preserved identity, talking rhythm, and gestures. 175+ language lip-sync.
  • HeyGen Studio Automation with Claude Code — Three-tool production pipeline (ElevenLabs + HeyGen + Remotion) orchestrated by Claude Code.
  • HeyGen Studio CLAUDE.md Template (Bootstrap Pattern) — HeyGen-shipped CLAUDE.md template for bootstrapping a HeyGen Studio project under Claude Code.
  • skills (Official HeyGen Skills Bundle) — Vendor-published 3-skill bundle (MIT, 232 stars, v3.1.0 2026-04-27, Shell): heygen-avatar (persistent digital twins from photos + voice synthesis), heygen-video (idea-to-scripted-video with avatar delivery + style recommendations), heygen-translate (175+ language localization with voice cloning + lip sync).
  • FLUX 3 (Black Forest Labs) — Multimodal Video, Audio, Image, and Action Model — BFL’s multimodal foundation model: one architecture jointly trained on images, video, and audio (Self-Flow scaled up), generating up to 20s of video with native audio in a single pass, chained multi-minute character-consistent clips, and multilingual dialogue.

Image Models

  • Nano Banana — Google’s Gemini Image Model (and Nano Banana Pro) — Google’s Gemini-native image generation/editing family: the original Nano Banana (Gemini 2.5 Flash Image, Aug 2025), the studio-quality reasoning tier Nano Banana Pro (Gemini 3 Pro Image, Nov 2025), and the current Flash-tier Nano Banana 2 / 2 Lite (Gemini 3.1 Flash Image / Flash Lite Image).

Search & Discovery

  • SentrySearch — Semantic Search Over Videos (Gemini + Qwen3-VL) — Local-first CLI that indexes video footage (default 30s chunks with 5s overlap, 480p / 5 FPS) and searches by natural language using Gemini Embedding 2 (cloud) or Qwen3-VL (local or DashScope).
  • watch) — Give Claude the Ability to Watch Any Video — MIT Claude skill (taoufik123-collab, 58★) that lets Claude watch a video: scene-change frame extraction (one frame per cut — token cost bounded by shots, not duration), a 0–10s “hook microscope” (2fps + word-level Whisper on the opening, for studying why a viral hook works), a fixed-schema report.md, and optional Obsidian auto-save that mirrors this wiki’s own clip→compile pattern. yt-dlp + ffmpeg + free captions (Whisper only as fallback).

Editing & Assembly

Creator Workflows

  • Claude + VidIQ YouTube Video Workflow — a zero-to-recorded talking-head workflow chaining four Claude features: the new VidIQ Claude MCP connector (titles grounded in real YouTube search volume / competition / opportunity score), an interview→outline skill (scoped to one video), a slide-deck-plan skill, and a Claude Design deck whose presenter-notes pane doubles as a teleprompter (record screen+face, overlay the talking head on the notes half).

Performance Measurement & A/B Testing

  • B Testing — Closes the post-publish gap the production pipeline stops short of: reading YouTube retention curves (sharp early drop / gradual decline / mid-video cliff / flat, mapped to script fixes), Meta’s and TikTok’s metric hierarchies, and native A/B testing tools on both platforms.

Composition & Motion Graphics

  • HeyGen Hyperframes — Open-source HTML-based video composition framework (Apache 2.0, v0.7.x; “Write HTML.
    • HyperFrames in Claude Design — draft a valid first-draft video in claude.ai/design → download ZIP → refine in any coding agent; includes the Open Design BYOK twin of the same loop.
    • HyperFrames Quickstart & CLI — the under-2-minute path plus the complete non-interactive hyperframes command surface (incl. cloud / lambda / cloudrun render backends).
    • HyperFrames HTML Schema & Compositions — the authoring contract: composition document structure, the full data-* timing-attribute set, and the window.__timelines GSAP handshake.
    • HyperFrames Core Concepts — deterministic seek-driven rendering (frame clock → seek → BeginFrame capture → encode) + the Frame Adapter API + a fair HyperFrames-vs-Remotion teardown.
    • HyperFrames GSAP Animation — the paused-timeline contract, supported methods/properties, and the timeline-duration-equals-composition-duration rule.
    • Prompting HyperFrames — cold-start vs warm-start prompt shapes plus the easing/caption/transition vocabulary cribs and per-agent notes.
    • HyperFrames Rendering — output formats (MP4/MOV/WebM/GIF/PNG), the full render flag surface, local-vs-Docker determinism, GPU/workers, batch + transparent overlays.
    • HyperFrames Website to Video — capture any URL and turn it into a production video in one prompt; the 7-step capture → design → script → storyboard → VO → build → validate pipeline.
    • HyperFrames Common Mistakes & Troubleshooting — the 8 authoring pitfalls (symptom → cause → fix), environment/render troubleshooting, and the lint/doctor pre-render gates.
    • HyperFrames Block Catalog — the ~109-block registry grouped by category (shader transitions, social overlays, lower thirds, code blocks, data viz, VFX/device) plus per-block anatomy.
    • HyperFrames Packages — the ~13-package monorepo (core/engine/producer/studio/player/shader-transitions + CLI, plus sdk/aws-lambda/gcp-cloud-run), with @hyperframes/player embedding depth.
    • HyperFrames Launch-Video Playbook (builder interview) — Bin Liu (VP Eng, HeyGen) & Jake Moran (PMM) on the operator workflow: the three setup paths, the website-to-video one-shot (Spotify / Fable 5), Jake’s net-new launch playbook (frame.md from hyperframes.dev/design → key-events table → component reuse from the open-sourced launch-video repo → storyboard.html → Studio/Inspector last-mile editing), model guidance (Fable 5 / GPT-5.5 top tier, Gemini for cost), and the spatial-vs-temporal-aesthetics thesis behind why launch videos are hard for agents.
    • HyperFrames Pipeline & Getting Started — the canonical 7-step capture → design → script → storyboard → VO+timing → build → validate flow, the on-disk project layout, the minimal write-HTML/preview/render loop, and the open-source launch-video projects.
    • HyperFrames Skills Catalog & Agent Setup — the full ~18-skill catalog (router + 10 creation workflows + 7 domain skills) plus per-agent wiring for Google Antigravity and GitHub Copilot CLI.
    • HyperFrames Deployment & Cloud Rendering — HeyGen-hosted cloud rendering + self-hosted AWS Lambda / GCP Cloud Run, one-click preview+render-API templates, templates-on-Lambda batches, and the Remotion-Lambda migration path.
    • HyperFrames Advanced Rendering — native/supersampled 4K (--resolution), HDR10 (--hdr), preview/render performance tuning, and local transparent-video matting (remove-background).
    • HyperFrames Timeline Editing, Keyframes & Studio DOM Editing — Studio’s three editing surfaces (timeline, visual keyframe/Arc-Motion tools, capability-gated manual DOM inspector), all round-tripping deterministically to authored HTML.
    • HyperFrames Video Components, HTML-in-Canvas & Variables — embedding/trimming video, capturing live DOM as a WebGL texture for shaders/3D, and data-composition-variables for parameterizing one composition into many (templating).
    • HyperFrames SDK, Authentication & Feedback — the headless @hyperframes/sdk editing engine, HeyGen auth (OAuth or API key, env-var-first, per-capability provider hierarchy with offline fallback), and feedback collection.
    • HyperFrames Video Editor Cheatsheet — fast operator quick-reference: common edit prompts, CLI commands, Studio shortcuts, timing attributes, render presets, and quick fixes.
    • media-use) — 2026-07-17 launch — HeyGen gave HyperFrames a media library: 10k+ music tracks, 75k+ images, SFX, and logos, plus access to HeyGen’s generative models (voice/image + new avatar-video), reached through the /media-use skill + the heygen CLI (≥ v0.3.0) and cached to .media/ + ~/.media/.
    • Catalog deep-dives (133 entries, 2026-06-27): Code Blocks & Snippet Themes, Transitions & Shader Effects, Lower Thirds & Tickers, Maps & Data Viz, Social Cards & Platform UI, VFX, Liquid Glass & Device Frames, Caption Styles & Overlay Components — per-block anatomy for every block + component, indexed from Block Catalog.
  • WEO Motion Library — One Vocabulary, Two Runtimes (CSS + GSAP) — first-party build documenting a reusable craft pattern: a 38-effect brand motion vocabulary implemented twice because CSS animation is wall-clock and does not frame-seek, while a headless video renderer captures frames by seeking a paused GSAP timeline on window.__timelines.
  • Remotion Motion Graphics — AI motion graphics generator converting natural language prompts into React-based Remotion animations.
  • Claude Code Video Toolkit (Digital Samba) — Open-source AI-native video production workspace for Claude Code (MIT, 890 stars). 10 skills, 13 slash commands, templates, brand profiles, transitions library.

Higgsfield (API-first generative video)

Multi-Model Claude Skills (filmmaking)

  • FREE Seedance 2.0 Claude Skill — Multi-Model AI Filmmaking Workflow (LTX Studio) — Single free Claude Skill that prompts Nano Banana Pro, GPT Image 2, and Seedance 2 in their own natural prompt languages for three distinct AI-filmmaking jobs: character sheet, 3×3 cinematic storyboard grid, and Seedance 2 shot (continuous + timestamp-with-dialogue paths).
  • AI Animated Short Film Pipeline — Seedance 2.0 + Codex + ElevenLabs (MattVidPro) — One-person ~3-min animated comedy short built almost entirely in Seedance 2.0. Codex runs pre-production (story + all generation prompts + GPT Image 2 art + an HTML reference site); ~30-50 image-to-video clips on Polo AI; ElevenLabs re-voices every line (native Seedance dialogue clones famous voice actors); royalty-free ambiance + manual SFX in post.
  • Fable 5 + Seedance 4K Short-Film Workflow (Higgsfield Cinema Studio, Adil) — a distinct end-to-end hyperreal-live-action pipeline built in Higgsfield Cinema Studio (not LTX or Polo AI) at Seedance 2.0 4K: the Cinema Studio “elements” asset system, gray-background sheet tuning, per-asset image-model switching (GPT Image 2 → Nano Banana Pro → Soul Cinema), a VFX-free video-in-video composite (6s reference + exact-duration match), multi-variant crowd sheets, and red-arrow action annotation.
  • Video-to-Video Trigger Editing — Editing Real Footage with Time-Gated Prompts (Maker Zero) — A real-footage editing playbook using video-to-video models (not text-to-video): a source clip (≤~10s) + a hyper-specific prompt structured as TRIGGER (a time-gated moment — a finger-snap, a spoken word, an exact timestamp) + CHANGE (e.g. “right after the man says this at exactly 2.9 seconds, change his outfit to a hoodie with a chain”), run through Omni/Gemini-Omni or Kling via the Higgsfield aggregator. ~20% per-attempt hit rate → generate 5 in parallel and pick the best ($0.50–1 each); 720p seam-hiding discipline (never cut degraded 720p AI output straight back to real talking-head footage — cut to a different scene type); proceduralize via the Higgsfield MCP from Claude Code/Codex.
  • Higgsfield as a Creative Agency in Claude (Nate Herk) — Fourth Higgsfield+Claude tutorial, three new dimensions vs the prior trio: (1) CLI-over-MCP architectural call for agentic work on token-cost grounds (“the MCP has all those tools, so from a token perspective it’s actually more expensive — the CLI is just better for agents”); (2) skill reverse-engineering workflow turning a single winning prompt into a reusable .claude/skills/hypermotion-video/SKILL.md recipe that compounds across runs; (3) two-routine scaling pattern (Sunday-plan + Monday-generate) that grows asset bank from 50 → 100 → 200 ads per week while operator sleeps, with a Google Workspace CLI–created Sheet acting as the cross-routine asset database.
  • Higgsfield Image-to-Video — Three featured models (higgsfield-ai/dop/preview, Bytedance Seedance Pro, Kling v2.1 Pro).
  • Higgsfield SDK (Python) — pip install higgsfield-client. Auth via HF_KEY or HF_API_KEY+HF_API_SECRET env vars.
  • Higgsfield Webhooks — Add hf_webhook query param to submit URL. Delivers completed/failed/NSFW final states. 2-hour retry window.
  • Higgsfield Training Framework (OSS Origin) — Historical context. Apache-2.0 distributed-training framework at higgsfield-ai/higgsfield (3.6k stars, last push 2024-05-25).

Model releases

HeyGen tutorials (2026-05-17 cohort)

  • HeyGen Instant Highlights V2 — Auto-Clip Long-Form Video to Short-Form Clips — Drop a long-form video (up to 10 GB) or URL, the tool analyzes for highlight moments (speech energy, importance signals, viewer-save likelihood), and auto-cuts to short-form clips with optional captions in 9:16 / 16:9 / 1:1 formats.
  • Style) — Prompt-engineering framework for HeyGen’s Avatar Shots feature (Avatar 5 + Seedance 2.0). 5-element prompt structure (subject / action / environment / camera / style) with one camera move per shot, layered into multi-shot beats (timestamps + per-beat camera moves), multi-avatar choreography (up to 3 avatars/scene with explicit blocking + relative-motion), and elements references (pre-uploaded locations / outfits / products for continuity).

Internal production articles (migrated to weomarketly-vault, 2026-07-09)

The six WEO production-stack articles formerly hosted here unpublished (OmniPresence System; Content Production Workflow; WEO Marketly Promo; Voice Profile Extraction; Banned AI Patterns; Mel’s Feedback Rules) were migrated to the internal weomarketly-vault on 2026-07-09 per the vault-separation rule, following the weo-ai-governance precedent. Each file remains in this folder as an archived migration stub so inbound links resolve; the full content lives at weomarketly:projects/omnipresence-system, weomarketly:projects/content-production-workflow, weomarketly:marketing/weo-marketly-promo, weomarketly:playbooks/voice-profile-extraction, weomarketly:playbooks/banned-ai-patterns, and weomarketly:playbooks/mels-feedback-rules (vault root: ~/Auto1111/Claude/weomarketly/weomarketly-wiki/wiki/).

72 items under this folder.