Source: raw/Claude_Rebuilt_This_AI_Video_in_Unreal_Engine_Zero_Imported_Assets.md (MattVidPro) + raw/AI_Breakthroughs_Everyone_Missed.md (Matt Wolfe, world-model segment)
Creator: MattVidPro AI (main walkthrough); Matt Wolfe (world-model segment) URL: https://www.youtube.com/watch?v=B0HN9kLxf7A ; https://www.youtube.com/watch?v=umGZrkyTRtw Duration: not captured in the transcripts Platform: YouTube
“Video to code” takes a clip from a video model and has a coding agent rebuild it as real, navigable code: a GLSL shader that runs in the browser, or a procedural Unreal Engine scene. MattVidPro ran both versions with Claude Code on Opus 5.5 from Seedance 2.5 clips. The shader route took about 45 minutes; the Unreal route took about 2.5 hours and roughly $35 at API prices. Both produced interactive scenes. The method compares the build against the source clip and fixes the biggest differences, which is the same loop as other verify-against-a-reference agent work. The MattVidPro video is sponsored by Crusoe, a GPU cloud that is not part of the workflow.
Key Takeaways
- The idea. A video model “hallucinates a non-physically grounded believable world”; an LLM then rebuilds it “in a reality-constrained coded environment”, which per MattVidPro keeps “80% of the excitement” of the clip “in about a thousand times the practicality”. The demo shader he opens with is “just 282 lines of code” and runs in real time in the browser.
- The skill. “Shader Skills by GoKay” on GitHub. Paste the repo link into Claude Code and ask it to install the skills; Claude offered to install them for Codex too.
- Shader result (about 45 minutes). A Seedance rover clip became a browser demo you can drive, with sounds, ambient occlusion, a chip you can pick up and dock, plus a separate “cinematic exact” version matched to the reference. MattVidPro judged it less faithful than GoKay’s own showcase, but “very much serviceable”.
- Unreal result (about 2.5 hours, about $35 at standard API pricing). He had Claude write a “video to Unreal” skill that keeps GoKay’s method: break down the clip, build a rough version, compare “at matching moments”, fix the biggest misses. It drives Unreal 5.8 through the official Unreal MCP, and every model and material is generated inside Unreal: “No Fab, Mega Scan, starter content, Blender imports, nothing.”
- What the Unreal build got right. Interactive props (atlas, captain’s log, a candle you can blow out), storm and day modes, close lighting, and, unprompted, a side-by-side comparison with the source and a 3D recreation of the Seedance camera move.
- What it got wrong. Curved wood and window joins left gaps, the spectacles were rotated the wrong way, and it built only the room (“basically just a floating room”). His estimate for a full, equally detailed ship: “days of agent work and hundreds of dollars”.
- His decision rule. Use the GLSL shader for fast in-browser ideation and sharing; move to Unreal “when your passions outgrow the web browser”.
- Spin-off (secondhand). MattVidPro relays that Epic Games’ Daniel Vogel pointed Opus 5.5 at the same idea for texture compression and cut a game’s size on disk from 44 MB to 1.5 MB with “zero noticeable loss in quality”.
- The mirror-image pattern: world models where code runs the world and a model paints it. Matt Wolfe covers two:
- Skywork Matrix Game 4.0 runs an ordinary game engine that draws a blocky low-poly frame, and a per-game video model (about 5B parameters each) repaints it. When you ask for something the rules don’t cover, a coding agent writes new rules into the engine. Output is 480 by 864; weights and inference code are Apache 2.0, training code is not included.
- NVIDIA StreamRender H3 sends the model only a flat colour-coded map. Per Wolfe, the team “handed Claude Code the job of building a faster video decoder” with a speed target and had it working in the pipeline in about 3 hours. It needs eight data-centre GPUs and has only been trained on racing.
- Video-to-code and these world models split the work the same way, approached from opposite ends: code holds the geometry and rules, and a model supplies the look.
Try It
- Generate a short, simple clip first (MattVidPro’s first test was a rover in a sparse environment). He had the LLM research Seedance 2.5 prompting before writing the clip prompt.
- Install GoKay’s shader skills from the GitHub link in Claude Code and ask for both a playable version and a “cinematic exact” version, so you can compare the two against the source.
- Only move to Unreal for a scene you will keep building. Expect about 4x the time of the shader route, per his estimate. Require procedural assets if you want to test the model rather than an asset library.
- Keep the compare step in the skill: render at matching moments of the source clip and fix the largest misses first. Check the details a model tends to get wrong, such as object orientation, curved geometry and joins.
Implementation
Tool/Service: Claude Code (Opus 5.5), GoKay’s Shader Skills (GitHub), Seedance 2.5 for source clips, Unreal Engine 5.8 with the official Unreal MCP.
Setup: Install the skills by pasting the repo URL into Claude Code. For Unreal, have Claude write a “video to Unreal” skill that keeps the same breakdown, rough build, compare and fix loop.
Cost: Shader route: about 45 minutes (no cost figure given). Unreal route: about 2.5 hours and about $35 at standard API pricing, per MattVidPro. The Unreal method itself is “still free” to try.
Integration notes: The shader output runs locally in the browser (he launched it with a play.bat). Unreal output is a real level you can extend.
Open Questions
- The GoKay repository URL and star count are not in the transcript. Check the link in the video description before citing it.
- The $35 figure is MattVidPro’s API-price estimate; he ran it on a subscription. No token counts were given.
- Is the Unreal MCP he used Epic’s own release? He calls it “official” but does not name it.
- The Daniel Vogel texture-compression result comes from an X post relayed in the video; the post itself was not captured.
Related
- Open Video Model Wave (Seedance 2.5): the clip generator used here.
- Claude Motion Graphics: Video Model vs Code vs After Effects: the same video-model-versus-code split for motion graphics.
- Verifier-First Loops: the compare-against-a-reference loop this method depends on.
- The Gauntlet Loop: rebuilding a game by comparing against the source, at larger scale.
- Claude Opus 5.5: the model used for both builds.
- HyperFrames Pipeline: the other code-first video route in this wiki, for 2D compositions.