> Canonical URL: https://codingcat.dev/course/faster-whisper-word-level-captions/lesson/faster-whisper-word-level-captions-lesson-4-4-visemes

# Lesson 4: Mapping Spoken Syllables to AJ's 4 Mouth Visemes (`smile-open`, `talking`, `surprised-o`, `grin`) | faster-whisper on Local GPU: Word-Level Captions & Mascot Lip-Sync | CodingCat.dev Course

[🧶 faster-whisper on Local GPU: Word-Level Captions & Mascot Lip-Sync](/course/faster-whisper-word-level-captions)Section 1.4 · Lesson 4 of 5[![](/logos/tags/video-symbol-light.svg)![](/logos/tags/video-symbol-dark.svg)Video](/topic/video)[← Prev Lesson](/course/faster-whisper-word-level-captions/lesson/faster-whisper-word-level-captions-lesson-3-srt-json)[Next Lesson →](/course/faster-whisper-word-level-captions/lesson/faster-whisper-word-level-captions-lesson-5-keyword-cut)

# Lesson 4: Mapping Spoken Syllables to AJ's 4 Mouth Visemes (`smile-open`, `talking`, `surprised-o`, `grin`)

![Lesson 4: Mapping Spoken Syllables to AJ](https://cdn.sanity.io/images/hfh83o0w/production/7182e6cb695a6dad9147f7a14ff87816c599c0aa-1376x768.png?w=1376&q=82&fit=max&auto=format)

## 1. The 15-Second Hook

Welcome to Lesson 4 of faster-whisper on Local GPU: Word-Level Captions & Mascot Lip-Sync on CodingCat.dev. Today we break down Mapping Spoken Syllables to AJ's 4 Mouth Visemes (`smile-open`, `talking`, `surprised-o`, `grin`) and see how it powers our deterministic 3D AJ and code-rendered video pipeline.

## 2. Architecture Mental Model (Visual Diagram)

Look at the architecture diagram: on the left, vibe-prompted animations suffer from timer drift and unreviewed layout bugs. On the right, 4 Visemes and Lip-Sync enforce a deterministic frame contract.

## 3. Show Me the Code (tools/video-studio/synthesize-voiceover.py)

Copied! 🐾📄 tools/video-studio/synthesize-voiceover.pypython

```
from faster_whisper import WhisperModel

whisper = WhisperModel("base.en", device="cuda", compute_type="float16")
segments, info = whisper.transcribe(wav_path, word_timestamps=True)
for seg in segments:
    for w in (seg.words or []):
        print(f"{w.start:.3f}s -> {w.end:.3f}s: {w.word}")
```

In tools/video-studio/synthesize-voiceover.py, we wire up the production implementation so both our browser runtime and headless renderer stay 100 percent synchronized.

## 4. Production Trade-Off Matrix

DimensionDeterministic 2026 PatternLegacy Anti-PatternWhy It Matters in ProductionDeterminismPure function of time t via window.seek(t)Non-deterministic timers & Math.random()Any frame renders identically on demand.Visual PolishContact-sheet critique loop + W3C tokensFirst unreviewed render ships as finalEliminates overlapping text and dead beats.3D & Mascot RigUnified 57-path 2D SVG + Three.js 3D GLB AJStatic flat raster imagesInteractive orbital 3D + 120 BPM lip-sync.Production Cost$0 open-source local RTX 5090 pipeline$1,200+ agency fee or per-minute SaaS cliffScales across 95+ lessons with zero marginal cost.

When we compare this deterministic 2026 pattern against legacy workflows in our trade-off matrix, we get exact frame reproducibility, automated critique gates, and zero licensing cost.

## 5. Verdict & AJ Pro-Tip

> AJ Pro-Tip: Always verify frame 300 twice during your contact-sheet critique pass to confirm your 4 Visemes setup has zero non-deterministic state leaks.

Adopt 4 Visemes inside your studio folder today so every future launch film and 3D mascot animation builds on a reusable harness. Hit Subscribe and let's continue!

✏️ Visual Architecture Diagram

## how this course connects together.

Follow AJ from first principles to production-ready architecture.

Let’s build it!✏️ Visual Diagram: faster-whisper on Local GPU: Word-Level Captions…✓ Good PatternVISUAL DIAGRAM[Diagram Walkthrough]Step-by-step diagram breakdown for faster-whisper on Local GPU: Word-Level Captions & Mascot Lip-Sync: Lesson 4: Mapping Spoken Syllables to AJ's 4 Mouth Visemes (`smile-open`, `talking`, `surprised-o`, `grin`): start with Core Architecture & Deterministic Implementation, implement faster-whisper on Local GPU: Word-Level Captions & Mascot Lip-Sync, and ship to production.AAA Contrast Verified

## related deep dives.

[View All→](/blog)[![HyperFrames: Turning HTML & Canvas into 60fps Video for AI Agents](https://cdn.sanity.io/images/hfh83o0w/production/40ad87fae480aff0c65b6d22225ccf69217ce6e5-1376x768.png?w=1376&q=82&fit=max&auto=format)](/course/hyperframes-html-to-video-agents)November 4, 2026[![](/logos/tags/html-symbol-light.svg)![](/logos/tags/html-symbol-dark.svg)HTML](/topic/html)[![](/logos/tags/css-symbol-light.svg)![](/logos/tags/css-symbol-dark.svg)CSS](/topic/css)

### [HyperFrames: Turning HTML & Canvas into 60fps Video for AI Agents](/course/hyperframes-html-to-video-agents)

Learn how HeyGen's open-source HyperFrames compiler turns self-contained HTML, CSS, and Canvas compositions into deterministic 1080p60 videos built by AI agents.

[![Alex Patterson](https://cdn.sanity.io/media-libraries/mlttp0BZbCpV/images/a606dfaa381aab08ddbe16a33fe1d3e1fb37c927-400x400.jpg?w=96&h=96&q=82&fit=max&auto=format)Alex Patterson](/author/alex-patterson)[![Alex Patterson next to a dark-mode architecture diagram of 3D AJ and the 14 open-source video repositories](https://cdn.sanity.io/images/hfh83o0w/production/1a6c5bade322565ed55c3d0999dfe3c0332055c4-1376x768.png?w=1376&q=82&fit=max&auto=format)](/post/14-open-source-repos-code-driven-video-how-we-built-3d-aj)November 4, 2026[![](/logos/tags/ai-symbol-light.svg)![](/logos/tags/ai-symbol-dark.svg)AI & Agents](/topic/ai)[![](/logos/tags/video-symbol-light.svg)![](/logos/tags/video-symbol-dark.svg)Video](/topic/video)

### [The 14 Open-Source Repos Powering Code-Driven Video in 2026: How We Built 3D AJ the Cat](/post/14-open-source-repos-code-driven-video-how-we-built-3d-aj)

Why viral 'one-prompt' AI videos are actually 9,500-character production harnesses backed by deterministic frame functions, closed-form springs, and contact-sheet critique loops. Here is the full architectural breakdown of all 14 open-source repos and how we used them to build our interactive 3D AJ the Cat mascot and Course Factory.

[![Alex Patterson](https://cdn.sanity.io/media-libraries/mlttp0BZbCpV/images/a606dfaa381aab08ddbe16a33fe1d3e1fb37c927-400x400.jpg?w=96&h=96&q=82&fit=max&auto=format)Alex Patterson](/author/alex-patterson)[Course Outline](/course/faster-whisper-word-level-captions)

## faster-whisper on Local GPU: Word-Level Captions & Mascot Lip-Sync

4/51

### Core Architecture & Deterministic Implementation

- [1.1Lesson 1: Why CTranslate2 (`faster-whisper`) Is 4x Faster with Half the VRAM](/course/faster-whisper-word-level-captions/lesson/faster-whisper-word-level-captions-lesson-1-ctranslate2)
- [1.2Lesson 2: Extracting Word-Level Timestamps & Filtering Hallucinated Silence](/course/faster-whisper-word-level-captions/lesson/faster-whisper-word-level-captions-lesson-2-word-clock)
- [1.3Lesson 3: Generating Dynamic SRT/VTT Captions & Active-Word Highlight JSON](/course/faster-whisper-word-level-captions/lesson/faster-whisper-word-level-captions-lesson-3-srt-json)
- [1.4Lesson 4: Mapping Spoken Syllables to AJ's 4 Mouth Visemes (`smile-open`, `talking`, `surprised-o`, `grin`)▶](/course/faster-whisper-word-level-captions/lesson/faster-whisper-word-level-captions-lesson-4-4-visemes)
- [1.5Lesson 5: Locking Scene Cuts & Diagram Arrows to Spoken Keyword Timestamps](/course/faster-whisper-word-level-captions/lesson/faster-whisper-word-level-captions-lesson-5-keyword-cut)

@CodingCatDev on YouTube62.5K / 100K

Subscribe for new course lessons and architecture deep dives every week.

[Subscribe for New Lessons →](https://www.youtube.com/@codingcatdev?sub_confirmation=1)