> Canonical URL: https://codingcat.dev/course/local-gpu-audio-diffusion-chatterbox-faster-whisper-comfyui/lesson/local-gpu-audio-diffusion-chatterbox-faster-whisper-comfyui-lesson-4-aj-visemes # Lesson 4: Driving 2D & 3D AJ Lip-Sync Visemes from Word & Beat Envelopes | Flagship 4: Local GPU Audio & Diffusion Pipelines with Chatterbox, faster-whisper & ComfyUI | CodingCat.dev Course [🧶 Flagship 4: Local GPU Audio & Diffusion Pipelines with Chatterbox, faster-whisper & ComfyUI](/course/local-gpu-audio-diffusion-chatterbox-faster-whisper-comfyui)Section 1.4 · Lesson 4 of 5[![](/logos/tags/video-symbol-light.svg)![](/logos/tags/video-symbol-dark.svg)Video](/topic/video)[← Prev Lesson](/course/local-gpu-audio-diffusion-chatterbox-faster-whisper-comfyui/lesson/local-gpu-audio-diffusion-chatterbox-faster-whisper-comfyui-lesson-3-faster-whisper)[Next Lesson →](/course/local-gpu-audio-diffusion-chatterbox-faster-whisper-comfyui/lesson/local-gpu-audio-diffusion-chatterbox-faster-whisper-comfyui-lesson-5-comfyui-api) # Lesson 4: Driving 2D & 3D AJ Lip-Sync Visemes from Word & Beat Envelopes ![Lesson 4: Driving 2D & 3D AJ Lip-Sync Visemes from Word & Beat Envelopes Architecture Diagram](https://cdn.sanity.io/images/hfh83o0w/production/7a43567757f12f6fbe23f101ee5f4398ec9f640e-1376x768.png?w=1376&q=82&fit=max&auto=format) ## 1. The 15-Second Hook Welcome to Lesson 4 of Flagship 4: Local GPU Audio & Diffusion Pipelines with Chatterbox, faster-whisper & ComfyUI on CodingCat.dev. Today we break down Driving 2D & 3D AJ Lip-Sync Visemes from Word & Beat Envelopes and see how it powers our deterministic 3D AJ and code-rendered video pipeline. ## 2. Architecture Mental Model (Visual Diagram) Look at the architecture diagram: on the left, vibe-prompted animations suffer from timer drift and unreviewed layout bugs. On the right, AJ Visemes and 120 BPM Sync enforce a deterministic frame contract. ## 3. Show Me the Code (tools/video-studio/align-audio-visemes.py) Copied! 🐾📄 tools/video-studio/align-audio-visemes.pypython ``` from faster_whisper import WhisperModel model = WhisperModel("large-v3", device="cuda", compute_type="float16") segments, _ = model.transcribe("lesson-voiceover.wav", word_timestamps=True) viseme_events = [ {"word": w.word.strip(), "start": round(w.start, 3), "end": round(w.end, 3)} for seg in segments for w in (seg.words or []) ] ``` In tools/video-studio/align-audio-visemes.py, we wire up the production implementation so both our browser runtime and headless renderer stay 100 percent synchronized. ## 4. Production Trade-Off Matrix DimensionDeterministic 2026 PatternLegacy Anti-PatternWhy It Matters in ProductionDeterminismPure function of time t via window.seek(t)Non-deterministic timers & Math.random()Any frame renders identically on demand.Visual PolishContact-sheet critique loop + W3C tokensFirst unreviewed render ships as finalEliminates overlapping text and dead beats.3D & Mascot RigUnified 57-path 2D SVG + Three.js 3D GLB AJStatic flat raster imagesInteractive orbital 3D + 120 BPM lip-sync.Production Cost$0 open-source local RTX 5090 pipeline$1,200+ agency fee or per-minute SaaS cliffScales across 95+ lessons with zero marginal cost. When we compare this deterministic 2026 pattern against legacy workflows in our trade-off matrix, we get exact frame reproducibility, automated critique gates, and zero licensing cost. ## 5. Verdict & AJ Pro-Tip > AJ Pro-Tip: Always verify frame 300 twice during your contact-sheet critique pass to confirm your AJ Visemes setup has zero non-deterministic state leaks. Adopt AJ Visemes inside your studio folder today so every future launch film and 3D mascot animation builds on a reusable harness. Hit Subscribe and let's continue! ✏️ Visual Architecture Diagram ## how this course connects together. Follow AJ from first principles to production-ready architecture. Let’s build it!✏️ Visual Diagram: Flagship 4: Local GPU Audio & Diffusion Pipeline…✓ Good PatternVISUAL DIAGRAM[Diagram Walkthrough]Step-by-step diagram breakdown for Flagship 4: Local GPU Audio & Diffusion Pipelines with Chatterbox, faster-whisper & ComfyUI: Lesson 4: Driving 2D & 3D AJ Lip-Sync Visemes from Word & Beat Envelopes: start with Core Architecture & Deterministic Implementation, implement Flagship 4: Local GPU Audio & Diffusion Pipelines with Chatterbox, faster-whisper & ComfyUI, and ship to production.AAA Contrast Verified ## related deep dives. [View All→](/blog)[![Chrome DevTools MCP: Self-Healing Agent Visual & Video Inspection](https://cdn.sanity.io/images/hfh83o0w/production/b144a4ce077ac03d081ed66396a7cffebbcdb24f-1376x768.png?w=1376&q=82&fit=max&auto=format)](/course/chrome-devtools-mcp-visual-agent-loops)November 4, 2026[![](/logos/tags/react-native-symbol-light.svg)![](/logos/tags/react-native-symbol-dark.svg)React Native](/topic/react-native)[![](/logos/tags/threejs-symbol-light.svg)![](/logos/tags/threejs-symbol-dark.svg)Three.js](/topic/threejs) ### [Chrome DevTools MCP: Self-Healing Agent Visual & Video Inspection](/course/chrome-devtools-mcp-visual-agent-loops) Give your AI agent eyes inside the browser. Use Chrome DevTools MCP so your agent inspects its own rendered video frames, WebGL canvases, and DOM layouts before exporting. [![Alex Patterson](https://cdn.sanity.io/media-libraries/mlttp0BZbCpV/images/a606dfaa381aab08ddbe16a33fe1d3e1fb37c927-400x400.jpg?w=96&h=96&q=82&fit=max&auto=format)Alex Patterson](/author/alex-patterson)[![FFmpeg for AI Video Pipelines: Contact-Sheet Critics & Audio Mastering](https://cdn.sanity.io/images/hfh83o0w/production/9e096b31bcd2d92f9b5a57cbc93875ec8c6ce16c-1376x768.png?w=1376&q=82&fit=max&auto=format)](/course/ffmpeg-ai-video-contact-sheet-critics)November 4, 2026[![](/logos/tags/ai-symbol-light.svg)![](/logos/tags/ai-symbol-dark.svg)AI & Agents](/topic/ai)[![](/logos/tags/ffmpeg-symbol-light.svg)![](/logos/tags/ffmpeg-symbol-dark.svg)FFmpeg](/topic/ffmpeg) ### [FFmpeg for AI Video Pipelines: Contact-Sheet Critics & Audio Mastering](/course/ffmpeg-ai-video-contact-sheet-critics) Master the exact FFmpeg filtergraphs that turn raw agent frames and synthesized audio into broadcast-ready MP4s, including automated contact-sheet defect detection. [![Alex Patterson](https://cdn.sanity.io/media-libraries/mlttp0BZbCpV/images/a606dfaa381aab08ddbe16a33fe1d3e1fb37c927-400x400.jpg?w=96&h=96&q=82&fit=max&auto=format)Alex Patterson](/author/alex-patterson)[Course Outline](/course/local-gpu-audio-diffusion-chatterbox-faster-whisper-comfyui) ## Flagship 4: Local GPU Audio & Diffusion Pipelines with Chatterbox, faster-whisper & ComfyUI 4/51 ### Core Architecture & Deterministic Implementation - [1.1Lesson 1: Zero-Shot Local Voice Cloning & Emotion Control with Chatterbox](/course/local-gpu-audio-diffusion-chatterbox-faster-whisper-comfyui/lesson/local-gpu-audio-diffusion-chatterbox-faster-whisper-comfyui-lesson-1-chatterbox) - [1.2Lesson 2: Phonetic Code Identifier Pre-Processing for Technical Narration](/course/local-gpu-audio-diffusion-chatterbox-faster-whisper-comfyui/lesson/local-gpu-audio-diffusion-chatterbox-faster-whisper-comfyui-lesson-2-phonetics) - [1.3Lesson 3: Sub-20ms Word Timestamp Extraction with faster-whisper on CUDA](/course/local-gpu-audio-diffusion-chatterbox-faster-whisper-comfyui/lesson/local-gpu-audio-diffusion-chatterbox-faster-whisper-comfyui-lesson-3-faster-whisper) - [1.4Lesson 4: Driving 2D & 3D AJ Lip-Sync Visemes from Word & Beat Envelopes▶](/course/local-gpu-audio-diffusion-chatterbox-faster-whisper-comfyui/lesson/local-gpu-audio-diffusion-chatterbox-faster-whisper-comfyui-lesson-4-aj-visemes) - [1.5Lesson 5: ComfyUI API Graphs for Character Turnarounds & Generate-Then-Trace](/course/local-gpu-audio-diffusion-chatterbox-faster-whisper-comfyui/lesson/local-gpu-audio-diffusion-chatterbox-faster-whisper-comfyui-lesson-5-comfyui-api) @CodingCatDev on YouTube62.5K / 100K Subscribe for new course lessons and architecture deep dives every week. [Subscribe for New Lessons →](https://www.youtube.com/@codingcatdev?sub_confirmation=1)