--- name: animated-video-studio description: Make cinematic promo, launch, explainer and animated character videos entirely in code. Every frame is rendered from three.js / HTML in headless Chromium, with a synthesized music score, an AI voiceover (free local Kokoro or ElevenLabs), burned-in subtitles, and native 9:16 (TikTok/Reels/Shorts) plus 16:9 deliverables, then checked with numeric QA. Two tracks - a low-poly 3D character short (characters at a desk, a mascot, a laptop UI, night-city bokeh) and a 2.5D motion-graphics product spot (UI at macro scale, cursor clicks, bloom). Use whenever the user wants a video, hype video, ad, promo film, launch video, explainer, animated skit, TikTok/Reel/Short, or wants a reference video recreated "in this style", including asks like "make a video like this", "add a voiceover", "make it faster paced", or "change the line to ...". --- # Animated Video Studio Make short promo, launch, explainer and character videos entirely in code: three.js or HTML scenes rendered frame by frame in headless Chromium, a local or ElevenLabs voiceover, a synthesized score, burned-in subtitles, and 9:16 plus 16:9 MP4s checked with numeric QA. Use it when someone asks for a video, ad, Reel/TikTok/Short, or wants a reference video remade in this style. You need Node 20+, Python 3 and the ffmpeg CLI. Films rendered from code look designed rather than templated: you control every frame, every cut lands on the audio, and any change ("change this line", "less zoom") is a re-render, not a re-shoot. This pipeline was used on two finished films; the gotchas at the end are what broke along the way. - **Track A, 3D character short** (three.js, low-poly, flat-shaded): two characters talk at a desk at night, a laptop screen carries the plot (a prompt, a reply, a catalog, a before/after), shallow depth of field, warm lamp key, city bokeh. - **Track B, 2.5D motion-graphics spot** (HTML/CSS 3D transforms): macro UI shots on black, bloom, grain, cursor clicks, end cards. Same renderer, timeline, audio and QA. ## Taste rules (learned from feedback on finished films; apply by default) 1. **State the premise on screen in the first 2-3 s**, before any dialogue: e.g. a point-of-view shot of the request being typed in big readable type. If the premise is not clear in the first few seconds, assume viewers scroll past. 2. **Show the payoff as a literal before/after** (e.g. a divider sliding across a screen: "Before: a script" | "After: a video"), held ~1.3 s on the split, then the result full screen for ~2 s. 3. **Pace:** voice at ~3.5-4 words/s, 0.15-0.25 s between lines, 1-2.5 s shots, camera moves that land within ~1.3 s and then only creep. Emphasis beats (the hook, the signature moment, the payoff) may breathe; the rest of the flow must not, or it reads as slow motion. 4. **Don't over-zoom.** Payoff screens and UI stay fully in frame with breathing room; keep true macro for 2-3 deliberate beats. 5. **Copy:** plain, short, declarative lines. No em dashes, no invented stats, no copywriter flourishes. Never clone a real company's UI or logo (the client's own brand is fine). End card = tagline, logo, URL. Nothing else unless asked. 6. **Characters are actors:** reactions start on the frame of their cause; no idle wobble; every gesture means something (hands up, head grab, hand on chin, a smile growing, a fist bump that visibly connects). ## Workflow 1. **Study the reference.** Extract frames: `ffmpeg -i ref.mov -vf fps=2,scale=228:-1,tile=9x4 sheet_%02d.jpg` (one sheet per 18 s). Note format, shot types, subtitle style/position, palette, pacing. Screen recordings usually have **no audio track** (check with `ffmpeg -i`): you cannot hear the reference voice, so say so and ask for the original post link or voice names. 2. **Write the script as a beat table** (who / line / picture) and show it. Lines the user approves are locked. 3. **Record the voice first. Audio is the clock.** Generate every line, measure it, and build the timeline from the durations. 4. **Build:** for Track A write a `CONTRACT.md` first (below), then build modules (in parallel with subagents if available). 5. **Look at stills early and often** (contact sheets every 0.5-1 s), both aspect ratios. 6. **Score** from the timeline's events, mix the voice, master to -14 LUFS. 7. **Render** both aspect ratios (they can run in parallel): `AR=9x16 node render.mjs video out/silent-9x16.mp4 & NO_EVENTS=1 AR=16x9 node render.mjs video out/silent-16x9.mp4 & wait`. Then mux, **verify** (QA section), deliver. 8. **Version everything** (`versions/vN/`: sources + mp4s) before the next round of changes. ## Setup These commands download packages from npm and PyPI and a Chromium build from Playwright; Kokoro and faster-whisper also download model weights on first use. Tell the user before running them. ```bash mkdir -p my-film/js my-film/vo/raw my-film/out && cd my-film && npm init -y && npm pkg set type=module npm i three@0.186 playwright@1.64 ffmpeg-static && npx playwright install chromium brew install ffmpeg # the ffmpeg CLI used below (Linux: sudo apt install ffmpeg) python3 -m venv .venv && .venv/bin/pip install numpy scipy faster-whisper # score, mixing, QA; run with .venv/bin/python ``` No system ffmpeg? Prefix commands with `PATH="$(dirname "$(node -p "require('ffmpeg-static')")"):$PATH"` (it has no ffprobe; read stream info from `ffmpeg -i`). Shell state does not persist between Claude Code Bash calls, so repeat the prefix. Headless Chromium often fails to launch inside Claude Code's command sandbox. If a render fails for that reason, stop and tell the user: they can approve that one render command through Claude Code's normal permission prompt, or run `node render.mjs ...` in their own terminal. Do not change sandbox or permission settings, and keep every other command sandboxed. On a Mac the launch flags below let WebGL use the GPU (Metal) instead of the slow software renderer. Budget about 0.3 s per frame with full post-processing: a 35 s film (1043 frames) took about 5.5 min per aspect ratio, with both ratios rendering in parallel. ### render.mjs (local server + frame-accurate capture) ES modules do not load from `file://`, so the renderer serves the project over a tiny local HTTP server. ```js // node render.mjs stills 1.5 6 ... | node render.mjs video [out/silent-.mp4] // env: AR=9x16|16x9 PAGE=film.html STILLS_DIR=stills/ NO_EVENTS=1 FROM= TO= import { chromium } from 'playwright'; import { spawn } from 'node:child_process'; import { writeFileSync, mkdirSync, readFile } from 'node:fs'; import http from 'node:http'; import path from 'node:path'; import { fileURLToPath } from 'node:url'; import { createRequire } from 'node:module'; const here = path.dirname(fileURLToPath(import.meta.url)), ffmpeg = createRequire(here + '/')('ffmpeg-static'); const [mode, ...args] = process.argv.slice(2), AR = process.env.AR || '9x16', [W, H] = AR === '16x9' ? [1920, 1080] : [1080, 1920]; const MIME = { '.html': 'text/html', '.js': 'text/javascript', '.mjs': 'text/javascript', '.css': 'text/css', '.json': 'application/json', '.png': 'image/png', '.jpg': 'image/jpeg', '.webp': 'image/webp', '.svg': 'image/svg+xml', '.woff2': 'font/woff2', '.wav': 'audio/wav', '.mp3': 'audio/mpeg', '.mp4': 'video/mp4' }; const srv = http.createServer((q, res) => { const f = path.join(here, decodeURIComponent(q.url.split('?')[0])); if (!f.startsWith(here + path.sep)) { res.writeHead(403); return res.end(); } readFile(f, (e, d) => { if (e) { res.writeHead(404); return res.end(); } res.writeHead(200, { 'content-type': MIME[path.extname(f)] || 'application/octet-stream' }); res.end(d); }); }).listen(0, '127.0.0.1'); await new Promise(r => srv.on('listening', r)); const url = `http://127.0.0.1:${srv.address().port}/${process.env.PAGE || 'film.html'}?ar=${AR}`; // GPU flags so WebGL renders on the Mac GPU; this browser only opens the local film page. Drop them on Linux/Windows. const browser = await chromium.launch({ args: ['--use-angle=metal', '--enable-gpu', '--ignore-gpu-blocklist'] }); const page = await browser.newPage({ viewport: { width: W, height: H }, deviceScaleFactor: 1 }); page.on('pageerror', e => console.log('[page error]', e.message)); page.on('console', m => ['error', 'warning'].includes(m.type()) && console.log(`[page ${m.type()}]`, m.text())); page.on('response', r => r.status() >= 400 && console.log(`[http ${r.status()}]`, r.url())); const load = async () => { await page.goto(url); await page.waitForFunction(() => window.ready !== undefined); await page.evaluate(() => window.ready); }; await load(); const film = (await page.evaluate(() => window.FILM)) || { FPS: 30, DUR: 10 }; if (!process.env.NO_EVENTS && film.VO) writeFileSync(path.join(here, 'events.json'), JSON.stringify(film, null, 1)); if (mode === 'stills') { const dir = path.resolve(here, process.env.STILLS_DIR || `stills/${AR}`); mkdirSync(dir, { recursive: true }); for (const t of args.map(Number)) { await page.evaluate(t => window.render(t), t); await page.screenshot({ path: `${dir}/t_${t.toFixed(2)}.jpg`, type: 'jpeg', quality: 88 }); } } else if (mode === 'video') { const out = path.resolve(here, args[0] || `out/silent-${AR}.mp4`); mkdirSync(path.dirname(out), { recursive: true }); const f0 = Math.round((+process.env.FROM || 0) * film.FPS), f1 = Math.round((process.env.TO ? +process.env.TO : film.DUR) * film.FPS); const ff = spawn(ffmpeg, ['-y', '-v', 'error', '-f', 'image2pipe', '-framerate', String(film.FPS), '-i', '-', '-c:v', 'libx264', '-preset', 'slow', '-crf', '14', '-pix_fmt', 'yuv420p', '-vf', 'scale=out_color_matrix=bt709:out_range=tv', '-colorspace', 'bt709', '-color_primaries', 'bt709', '-color_trc', 'bt709', out], { stdio: ['pipe', 'inherit', 'inherit'] }); for (let f = f0; f < f1; f++) { if (f > f0 && (f - f0) % 300 === 0) await load(); // long runs slow down: start fresh every 300 frames await page.evaluate(t => window.render(t), f / film.FPS); const buf = await page.screenshot({ type: 'png', timeout: 120000 }); if (!ff.stdin.write(buf)) await new Promise(r => ff.stdin.once('drain', r)); } ff.stdin.end(); await new Promise(r => ff.on('close', r)); } await browser.close(); srv.close(); ``` Pass still times as separate arguments. In zsh a variable list must be split with `${=T}`; an unsplit string becomes `NaN` and fails deep inside canvas code with "non-finite". ### film.html + js/main.js (the page contract) The page exposes `window.ready` (a Promise), `window.render(t)` (draw the frame at time t, **pure**: no `Math.random`, `Date`, CSS animations or state carried between frames, because frames render in any order) and `window.FILM` (`{FPS, DUR, SC, M, VO, SUBS, EVENTS}`, written to `events.json` for the score). ```html
Your tagline.
``` ```js // js/main.js: timeline, shots, render(t) import * as THREE from 'three'; const V = (new URLSearchParams(location.search).get('ar') || '9x16') !== '16x9', W = V ? 1080 : 1920, H = V ? 1920 : 1080, FPS = 30; const clamp = (x, a = 0, b = 1) => Math.min(b, Math.max(a, x)), lerp = (a, b, k) => a + (b - a) * k; const prog = (t, a, b) => (b <= a ? +(t >= b) : clamp((t - a) / (b - a))); const E = { o3: x => 1 - (1 - x) ** 3, io2: x => (x < .5 ? 2 * x * x : 1 - (-2 * x + 2) ** 2 / 2), oback: x => 1 + 2.70158 * (x - 1) ** 3 + 1.70158 * (x - 1) ** 2 }; const mix = (a, b, k) => a.map((x, i) => lerp(x, b[i], k)); // camera moves land fast, then only creep: a long shot never reads as slow motion const land = (lt, d, T = 1.3) => 0.93 * E.o3(prog(lt, 0, Math.min(d, T))) + 0.07 * prog(lt, 0, d); const fov16 = f => 2 * Math.atan(Math.tan(f * Math.PI / 360) * 0.6) * 180 / Math.PI; // fov is the 9:16 vertical fov // ---- the timeline is built from measured voice lengths (vo/lines_measured.json: [{i, dur}]) const LINES = [{ who: 'G', text: 'I asked for a launch video.', est: 1.6 }, { who: 'C', text: 'I can.', est: 0.6 }]; const SC = {}, M = {}, VO = [], EVENTS = []; let DUR = 0; const ev = (t, type, x = {}) => EVENTS.push({ t: +t.toFixed(4), type, ...x }); function buildTimeline(D) { const say = (i, t) => { VO.push({ i, ...LINES[i - 1], start: t, dur: D[i] }); return t + D[i]; }; let t; SC.a0 = [0, 2.2]; // the hook: premise on screen, no dialogue M.l1 = SC.a0[1] + 0.15; t = say(1, M.l1); // lines start ~0.15 s after their cut SC.a1 = [SC.a0[1], t + 0.2]; SC.b1 = [SC.a1[1], 0]; M.l2 = SC.b1[0] + 0.3; t = say(2, M.l2); SC.b1[1] = t + 0.4; SC.end = [SC.b1[1], SC.b1[1] + 3.5]; DUR = Math.ceil(SC.end[1] * FPS) / FPS; // subtitle end = min(voice end + 0.3, next line start - 0.04 - 2 frames, end card, first cut after the voice ends) const cuts = Object.values(SC).map(s => s[0]).sort((a, b) => a - b); VO.sort((a, b) => a.start - b.start).forEach((L, j) => { const end = L.start + L.dur, nx = VO[j + 1]; L.subEnd = Math.min(end + 0.3, nx ? nx.start - 0.04 - 2 / FPS : Infinity, SC.end[0], cuts.find(c => c >= end) ?? Infinity); }); for (const k in SC) ev(SC[k][0], 'cut', { scene: k }); } // ---- shots: (local t, duration) -> { pos, tgt, fov, fov16?, pos16?, tgt16?, roll, focus } const SHOTS = { a0: (lt, d) => { const k = land(lt, d); return { pos: mix([0, 1.3, 1.4], [0, 1.25, 1.2], k), tgt: [0, 1.0, 0], fov: 36 }; }, a1: (lt, d) => ({ pos: [0.6, 1.0, 1.0], tgt: [0, 1.0, 0], fov: 38 }), b1: (lt, d) => ({ pos: [-0.6, 1.0, 1.0], tgt: [0, 1.0, 0], fov: 38 }), end: (lt, d) => ({ pos: [0, 1.4, 1.8], tgt: [0, 1.0, 0], fov: 40 }), }; let renderer, scene, camera, hero; const $ = id => document.getElementById(id); window.render = t => { t = clamp(t, 0, DUR - 1e-6); const sc = Object.keys(SC).find(k => t >= SC[k][0] && t < SC[k][1]) || 'end', c = SHOTS[sc](t - SC[sc][0], SC[sc][1] - SC[sc][0]); const pos = !V && c.pos16 ? c.pos16 : c.pos, tgt = !V && c.tgt16 ? c.tgt16 : c.tgt; camera.position.set(...pos); camera.lookAt(...tgt); camera.rotateZ(c.roll || 0); camera.fov = V ? c.fov : (c.fov16 ?? fov16(c.fov)); camera.updateProjectionMatrix(); hero.rotation.y = t * 0.5; // everything is a function of t renderer.render(scene, camera); // swap for post.render(...) once post-processing exists const line = VO.find(L => t >= L.start - 0.04 && t < L.subEnd); $('sub').textContent = line ? line.text : ''; $('card').style.opacity = clamp((t - SC.end[0] - 0.3) / 0.4); $('fade').style.opacity = prog(t, DUR - 0.45, DUR - 1 / FPS); // last frame black, so the loop is clean }; window.ready = (async () => { document.body.classList.add(V ? 'v' : 'h'); Object.assign($('stage').style, { width: W + 'px', height: H + 'px' }); // fonts load lazily: canvas text and not-yet-shown subtitles would draw in a fallback font, so load every face up front await Promise.all(['500 16px Inter', '600 16px Inter', '700 16px Inter', '800 16px Inter', '500 16px "JetBrains Mono"', '700 16px "JetBrains Mono"'].map(f => document.fonts.load(f))); await document.fonts.ready; const D = {}; LINES.forEach((l, i) => (D[i + 1] = l.est)); const r = await fetch('/vo/lines_measured.json'); if (r.ok) for (const m of await r.json()) D[m.i] = m.dur; else console.warn('vo/lines_measured.json missing: timing from est'); buildTimeline(D); renderer = new THREE.WebGLRenderer({ antialias: true, preserveDrawingBuffer: true }); renderer.setSize(W, H); renderer.shadowMap.enabled = true; renderer.shadowMap.type = THREE.PCFShadowMap; $('stage').prepend(renderer.domElement); scene = new THREE.Scene(); scene.background = new THREE.Color('#141a4a'); camera = new THREE.PerspectiveCamera(40, W / H, 0.01, 80); hero = new THREE.Mesh(new THREE.BoxGeometry(0.26, 0.19, 0.19), new THREE.MeshStandardMaterial({ color: '#e06a40', flatShading: true, roughness: 0.85 })); hero.position.set(0, 0.9, 0); scene.add(hero); const key = new THREE.SpotLight('#ffb36b', 30, 4, 0.6, 0.5, 1); key.position.set(0.8, 1.6, 0.4); scene.add(key); scene.add(new THREE.HemisphereLight('#5a66ff', '#120c20', 1.2)); window.FILM = { FPS, DUR, SC, M, VO: VO.map(L => ({ ...L, end: L.start + L.dur })), SUBS: VO.map(L => ({ i: L.i, text: L.text, start: L.start - 0.04, end: L.subEnd })), EVENTS: EVENTS.sort((a, b) => a.t - b.t) }; return true; })(); ``` `node render.mjs stills 1 3 5` must produce frames before anything else is built. Derive every time from marks (`M.click = t + 0.15`) rather than literals, so a changed line retimes the whole film. ## Track A: low-poly 3D character short ### Write the contract first Parallel builders (characters / set + post / screen UI / voice) only plug together if a `CONTRACT.md` fixes: - **World layout in metres** (Y up, the human faces -Z, his right is +X): floor y 0; desk top y 0.74; laptop centred on the desk, screen 16:10 facing the human; mascot anchor on the desk beside the laptop; human anchor (floor under the pelvis, z ~0.8); lamp; windows behind the desk **and** behind the human (reverse shots also need bokeh). - **Every pose field with its sign convention** (`headYaw` + = toward +X, `yaw` for the mascot = `rotation.y = -yaw`, ...). Ambiguous signs were the main integration risk. - **Module APIs:** `buildGuy()/buildMascot() -> {root, set(p)}` (absolute pose; omitted fields = neutral), `buildSet(scene) -> {screen:{canvas, ctx, texture, mesh, center, normal}, setScreenLight(color, k), update(t), anchors}`, `buildPost(renderer, scene, camera, W, H) -> {render({focus, aperture, t})}`, `drawScreen(ctx, W, H, state)`. ### Characters (flat-shaded, low segment counts) - **Human:** faceted head (`IcosahedronGeometry(r, 1)` scaled, ~0.30 m tall, big for the body), hair from 6-10 low-poly cones, eyes as near-flat dark discs laid on the facets (decals with `polygonOffset`, never thick blocks that poke out in 3/4 view), pyramid nose, a dark mouth slab whose height follows the voice envelope, hoodie torso + hood ring + drawstrings. Arms are two-bone IK to a palm target. Named poses blended by weight: `rest, type, handsUp, headGrab, shrug, chin, point, fistOut`. Contact poses (head grab, hand on chin, fist bump) need measured targets: test hand-to-surface distance numerically. - **Mascot:** a box wider than tall, two vertical-rectangle eyes, side nubs, four short legs. It has no mouth, so speech is a body bob + tiny scale pulse + eye squint driven by the envelope. Also `hop`, volume-preserving `squash`, `happy` (^ ^ eyes), `glow`. Keep the feet planted: lean/tilt pivot the body about its bottom, not the legs. Cap emissive glow (~+0.2) or it flattens into a cut-out; let bloom make the halo. - **Lip flap:** mouth = `smoothstep(0.18, 0.85, env) * 0.9` so it closes between syllables, where `env` comes from `vo/env.json` (per-frame RMS over 1/30 s, divided by that line's 95th-percentile RMS, clipped 0..1; interpolate between frames). - **Blinks:** a hand-placed list of times per character, moved off key frames; let a few follow line ends. ### Set, light, post - Night interior. Navy/indigo ambient (shadow sides go blue-violet), **one warm amber key** from a desk lamp, a cool fill from the window, and the laptop screen as a soft light whose colour follows what is on screen (a `RectAreaLight` the size of the display, a few mm in front of it; call `RectAreaLightUniformsLib.init()` from three/addons/lights/ once). A cookie or tight cone keeps the lamp off faces; a separate warm cheat light gives the hot cheek. - **City bokeh** is the signature: large soft round discs (warm yellow, a few white, a few small red) on shells 10-26 m out, visible through both windows. Pre-shaped disc sprites look better than relying on DOF alone. - Laptop display: `MeshBasicMaterial({ map: CanvasTexture, toneMapped: false })`; redraw the 1600x1000 canvas only when its state changes (key the state with `JSON.stringify`). Canvas text never triggers a font load: `document.fonts.load` every family and weight `drawScreen` uses before the first frame. - **Post:** three's addons (`RenderPass` -> `BokehPass` -> `UnrealBloomPass` -> grade `ShaderPass` -> `OutputPass`) are a quick start. The finished film used its own chain instead: the opaque room renders into an MSAA half-float target with a depth texture and a transparent clear; a per-pixel circle of confusion comes from depth (focus = camera-to-subject distance); a gather DOF never lets background spread over sharper foreground; the city is a separate layer (blurred skyline + bokeh discs sized from the same CoC) composited where the room is transparent; transparent objects (cards, sparks) are drawn sharp on top with no DOF, so keep them near the focus distance; then a multi-level bloom and a grade (exposure, ACES, sRGB, cool-shadow/warm-highlight split, vignette, grain seeded by frame number). ### Screen states carry the plot Write the laptop UI as a pure `drawScreen(ctx, W, H, state)` with modes: `compose` (one big prompt box: the hook must be readable with the whole laptop in frame, ~56 px mono on a 1600 px canvas), `chat` (prompt, streaming reply, a green "skill installed" line, a progress bar), `catalog` (product page, cursor, Install press -> Installed), `compare` (before/after divider over two states), `video` (the result playing, kinetic type in the brand palette). Return a matching `screenGlow(state)` for the light. Generic UI only: no real company's interface or logo. ### Camera grammar - Each scene has a shot function. Vary scale: point-of-view on the hook, over-the-shoulder on the screen, low dutch on the frustrated line, three-quarter on the mascot from the human's side, a signature orbit, whole-laptop medium for the payoff, over-the-shoulder two-shot for the finale. - **Real-world scale:** a 0.26 m mascot needs the camera ~1.1-1.25 m away at a 38 deg vertical fov in 9:16 (the orbit ran 1.0-1.1 m at 46 deg). Width covered = `2 d tan(fov/2) * W/H`. First attempts at 0.5 m filled the frame with one face. - If a shot's camera would sit inside a character (frontal screen shots, a POV), hide that character or its head for that shot. - Screen shots: compute camera positions from canvas pixels (map a pixel through the screen mesh's bounding box and `matrixWorld`, then place the camera `d` metres along the screen normal). - Check every shot in both aspect ratios. Subtitles sit at ~36% height in 9:16 and near the bottom in 16:9; frame faces below them. ## Track B: 2.5D motion-graphics spot Same renderer and timeline; scenes are DOM layers. A `cam` div per scene gets `translate(960px,540px) perspective(1700px) rotateX() rotateY() rotateZ() scale(s) translate(-fx px,-fy px)` (rotations pivot on the focus point) from a shot table: a hard cut, a punchy move over 0.3-0.6 s with o3/o4, then a 3% creep. Rebuild UI natively at macro scale (search field, results, chat, toggles, button), monochrome with bloom (SVG filter: highlight extract -> two blur radii screened back -> tone curve crushing near-black), seeded grain, vignette. Depth of field via blurred clones masked by distance; on steep planes use per-element blur instead (layered DOF drops whole layers). Use background/border brightness instead of `box-shadow` on many elements, or frames time out. An oversized cursor walks to the target; the click compresses the button (0.93) and every reaction starts on the click frame. End cards: tagline, then logo + URL; the logo may be the only colour. ## Voice **Kokoro (free, local):** `npm i kokoro-js` (downloads a ~330 MB model on first use; ask first). Output is 24 kHz mono. ```js import { mkdirSync } from 'node:fs'; import { KokoroTTS } from 'kokoro-js'; mkdirSync('vo/raw', { recursive: true }); const tts = await KokoroTTS.from_pretrained('onnx-community/Kokoro-82M-v1.0-ONNX', { dtype: 'fp32', device: 'cpu' }); await (await tts.generate("Why can't you make a video?!", { voice: 'am_michael', speed: 1.05 })).save('vo/raw/line_03.wav'); ``` Two characters = two voices (e.g. `am_michael`, and `am_puck` pitched +1 semitone with `asetrate=24000*1.05946,aresample=48000` before the chain). asetrate also speeds the line up by the same ratio, so generate that voice at `speed / 1.05946`. **ElevenLabs (optional, paid):** ask the user to set `ELEVENLABS_API_KEY` themselves (shell profile or a git-ignored `.env`). Use the key only in requests to `api.elevenlabs.io`. Never print, log or commit it, and never write it into project files. If the user pastes a key into the chat, do not repeat it; with their go-ahead, save it as one header line (`xi-api-key: ...`) in a file outside the project that only they can read (`umask 077`), pass it with `curl -H @that-file` so it never appears on a command line, delete the file when the voice lines are done, and recommend rotating the key since it is now in the chat history. A key limited to text-to-speech can get a 401 from `GET /v1/voices`; that is expected, so use premade voice IDs instead of listing voices. ```bash mkdir -p vo/raw; VOICE_ID=iP95p4xoKVk53GoZ742B # Chris; see premade IDs below curl -sS --fail-with-body -X POST "https://api.elevenlabs.io/v1/text-to-speech/$VOICE_ID?output_format=mp3_44100_128" \ -H "xi-api-key: $ELEVENLABS_API_KEY" -H "Content-Type: application/json" \ -d '{"text":"[frustrated] Why can'\''t you make a video?!","model_id":"eleven_v3","voice_settings":{"stability":0.5,"similarity_boost":0.75},"seed":1003}' \ -o vo/raw/line_03.mp3 || cat vo/raw/line_03.mp3 ``` `eleven_v3` reads inline acting tags ([frustrated], [warmly], [curious], [impressed], [playful]). Render up to 3 seeded takes per line, run each through the chain below, and keep the one Whisper hears word for word, with no tag spoken, that fits the line's time budget. Premade IDs that worked: Chris `iP95p4xoKVk53GoZ742B` (natural young male), River `SAz9YHcvj6GT2YYXdXww` (light, warm, gender-neutral: a good mascot). Audition 3-5 voices on the hardest line and pick by transcript accuracy, pitch separation between the two characters, and length. Never clone a real person's voice without consent. **Processing chain** per line (48 kHz mono): ```bash ffmpeg -y -i vo/raw/line_03.wav -af "aresample=48000,highpass=f=75,equalizer=f=260:t=q:w=1.1:g=-2,equalizer=f=3400:t=q:w=1:g=2.5,highshelf=f=8000:g=1.5,acompressor=threshold=-20dB:ratio=3:attack=5:release=90:knee=4:makeup=2,silenceremove=start_periods=1:start_threshold=-50dB:start_silence=0.012:detection=peak,areverse,silenceremove=start_periods=1:start_threshold=-54dB:start_silence=0.025:detection=peak,areverse,loudnorm=I=-19:TP=-1.5" -ar 48000 -ac 1 vo/line_03.wav ``` (`loudnorm` outputs 192 kHz unless `-ar` is set.) A tiny synthetic room (convolve with a short noise decay, RT60 ~0.16 s, wet -21 dB, fixed seed) before the trim places the voices in the room. Then write `vo/lines_measured.json` `[{i, who, text, dur}]` and `vo/env.json` `{fps:30, lines:[{i, env:[...]}]}`. **Pace** (on the raw takes, before the chain; then re-run the chain so the durations and envelopes match): cap every internal pause at ~80 ms (silence = 10 ms windows under -42 dBFS; keep a deliberate beat where the acting needs one) and `atempo` 1.06-1.18 so lines of 5+ words land at ~3.6-3.9 words/s (atempo keeps pitch). Spell acronyms out for TTS only (`LLMs` -> `L L Ms`); keep the subtitles unchanged. ## Score (numpy, no samples, nothing to license) Write `audio.py` (numpy + scipy) that reads `events.json` and synthesizes everything: e-piano chords (FM sine), swung lo-fi drums, pads (detuned saws, low-passed), plucks, bells, risers, whooshes, key clicks, UI clicks, hops, thuds. Make the music follow the story: - frustration: lo-fi keys, then a tape-stop (time-warp the music bus to a halt over ~0.38 s, landing on the frame of the breaking point); - a beat of near-silence for the turn; - a soft magic pad and a sparkle arpeggio for the idea / signature moment; - a success chime on the click; a riser that stops 60 ms before the payoff; - the result's own music, band-limited "from laptop speakers" (that loses ~15 dB: compensate), then opening to full range on the reaction shot; - a resolution under the tagline, and a logo hit. Foley lands on the first frame that shows the action: `fq = t => Math.ceil(t * FPS - 0.005) / FPS`. **Mix:** place each `vo/line_NN.wav` at its `start` (70 Hz high-pass on the voice bus). Dialogue bus: a 4:1 compressor with its threshold at active-speech RMS + 4 dB (2 ms attack, 60 ms release), then a 3 ms look-ahead voice-only limiter at speech RMS + 10 dB (speech crest is ~20 dB and otherwise pins the master limiter). Duck music (not foley) 8 dB while speech is active: gate = voice envelope above -38 dB re its peak, extended 120 ms early, smoothed with 60 ms attack and 350 ms release. Then scale the voice so it sits 10 dB over the ducked music's RMS during speech. **Master:** 30 Hz high-pass, mono below 140 Hz, a true-peak (4x oversampled) look-ahead limiter at **-3.4 dBFS** (AAC added only 0.2-0.3 dB here, so this clears -1 dBTP with margin), loudness to **-14 LUFS** (BS.1770 K-weighting, gated). Until `audio.py` exists, an ffmpeg-only mix works (voices delayed to their starts in ms, music ducked by sidechain, mastered, trimmed to DUR): ```bash ffmpeg -y -i music.wav -i vo/line_01.wav -i vo/line_02.wav -filter_complex "[1]adelay=2350:all=1[a1];[2]adelay=4450:all=1[a2];[a1][a2]amix=inputs=2:normalize=0,apad,asplit[v][k];[0][k]sidechaincompress=threshold=0.02:ratio=6:attack=60:release=350[m];[m][v]amix=inputs=2:normalize=0:duration=first,atrim=end=8.966667,loudnorm=I=-14:TP=-1.5:LRA=11" -ar 48000 -ac 2 score.wav ``` **Mux** (`apad` lets the video set the length, so no frames are dropped; use `-c:v copy` instead of the libx264 flags to skip a second encode): ```bash ffmpeg -y -i out/silent-9x16.mp4 -i score.wav -map 0:v -map 1:a -c:v libx264 -crf 18 -preset slow -pix_fmt yuv420p -af apad -c:a aac -b:a 256k -shortest -movflags +faststart out/film-9x16.mp4 ``` ## QA (never hand over without it) - Contact sheets of the mp4: `ffmpeg -i out/film-9x16.mp4 -vf "fps=2,scale=150:-2,tile=17x4" sheet_%02d.jpg` (one sheet per 34 s, works for both aspect ratios). Look at every sheet. - Frames: the count must equal `round(DUR * FPS)` (`ffmpeg -i f.mp4 -map 0:v -f null -`). Loudness on the **mp4**: `ffmpeg -nostats -i out/film-9x16.mp4 -af ebur128=peak=true -f null - 2>&1 | tail -12` (-14 +/- 1 LUFS, true peak <= -1 dBTP). - Dialogue: transcribe with faster-whisper `small` (CPU, int8) and match each locked line **in order** through the transcript (a line passes when >= 80% of its words are found in order, each within 8 tokens of the last). Do not use word timestamps for timing: Whisper can stamp a word up to 0.7 s early, stretched back across a pause. - Exact sync: write a short numpy check that cross-correlates each `vo/line_NN.wav` with the mix around its `start`; every line must land within 40 ms (expect about 1 ms). - Pass bar: every subtitle matches its voice, foley on the action frame, no cropped payoff UI, no black frames except the designed end fade (last frame black for a clean loop), both aspect ratios checked. - With subagents: run critics in parallel (story/acting, camera vs reference, characters, on-screen copy, audio/sync) -> one planner merges and ranks -> one editor applies -> adversarial verifiers re-check stills. Tell critics which lines are locked; they otherwise rewrite approved copy. Show the user a draft early: a full critique loop can take an hour. ## Gotchas that actually happened 1. Chromium silently uses the software renderer unless launched with GPU flags; check `WEBGL_debug_renderer_info`. 2. three r186 removed `PCFSoftShadowMap` (warns on every load): use `PCFShadowMap` + `shadow.radius`. 3. Long renders slow down until a screenshot times out: reload the page every 300 frames. 4. Events must use absolute film time; scene code works in local time. 5. Duplicate element ids (a scene id equal to a span id) silently blank a scene. 6. `node -e "import('render.mjs')"` runs a whole render; syntax-check with `node --check`. 7. Shortening a scene can make a range end before it starts; `prog(t, a, b)` with a >= b becomes a hard step (a pop). Guard ranges after retiming. 8. Subtitles must not outlive their scene or crowd the next line: end = min(voice end + 0.3, next start - 0.04 - 2/FPS, end-card start, first cut after the voice ends). 9. Fade to black must finish one frame early (`DUR - 1/FPS`) or the loop pops. 10. TTS leading/trailing silence breaks timing: trim, then measure. A stray click before a line defeats `silenceremove`: trim the raw take. 11. Coloured screen light on faces overshoots fast (a magenta face); scale video glow down (~0.5). 12. A 3D camera placed by numbers alone ends up inside heads, hoodies and lamps: render stills of every shot before any video. 13. Critic subagents "improve" approved lines and claims: lock them in their brief. 14. Whisper writes "Claude" as "Clawd"/"Clawed", "Whoa" as "Woah", and can hear a drawn-out "Whoa" as "Well": map known homophones before matching. 15. Screen-capture temp files on macOS vanish within minutes: copy a reference out of temp folders immediately. ## Credits Built on principles from these community skills (listed on augmentclaude.com, each by its own author): motion-doctrine by heygen-com (continuous motion, causal reactions, no idle wobble), launch-video by ucsandman (audio is part of done, shot plan before production, independent review), hyperframes-motion-director by geekjourneyx (anti-slide rule, a signature motion moment, rendered-frame integrity) and video-shotcraft by vincentwei1021 (one motion idea per shot, hold after key info, deterministic seeded rendering).