AI 短视频工厂｜HTML-to-Video Studio

概述

AI 短视频工厂是一个基于 HyperFrames 的 AI 视频生成 Skill，可将文字创意、结构化脚本或内容大纲自动转化为 HTML + CSS + GSAP 动画视频，并通过 Headless Chrome + FFmpeg 确定性渲染为 MP4。无需 React，无按次收费，专为 AI Agent 设计。

核心能力：

• 🎬 从文字描述一键生成完整视频（15s ~ 3min）
• 🎨 完全自定义：布局、配色、动画、字体均由 HTML/CSS 控制
• 🔄 AI 全流程自动化：初始化 → 编写 → 动画 → 渲染 → 交付
• 🎵 支持字幕同步、原片音频保留、按需 BGM 合成与音量调校
• ✅ 自检管线确保视觉完整性和技术正确性

快速开始

一句话生成视频：

"帮我做一个 30 秒的科技资讯短视频，主题是 AI 编程工具的发展趋势"

AI 会自动完成全流程：

1. 初始化项目 → npx hyperframes init
2. 编写 HTML 视频内容（场景规划 + 动画编排）
3. 渲染为 MP4 → npx hyperframes render
4. 按需处理音频 → 保留原片音频，或在用户要求时 FFmpeg 后置混音
5. 执行自检管线 → 确保质量达标后交付

最小示例：

# 创建项目
npx hyperframes init my-video --non-interactive
# 编辑 index.html（AI 自动生成内容）
# 渲染
npx hyperframes render --non-interactive

适用场景

Overview (English)

HyperFrames is an open-source (Apache 2.0) HTML-native video rendering framework by HeyGen. Write video as HTML + CSS + seekable animations, then render deterministically to MP4 via headless Chrome + FFmpeg. No React required, no per-render fees, designed for AI agents.

Prerequisites Check & Installation

Before any operation, run the environment diagnostic:

npx hyperframes doctor

This reports all dependencies and their status. If any check fails, STOP and resolve before proceeding.

Required Tools (ALL mandatory)

Optional Tools (for advanced workflows)

First-Time Setup Sequence

For a brand new environment, execute these steps in order:

# 1. Verify Node.js (should be pre-installed by WorkBuddy)
node --version  # must be >= 22.0.0

# 2. Install FFmpeg if missing
which ffmpeg || brew install ffmpeg
ffmpeg -version  # confirm installation

# 3. Download Chrome Headless Shell (auto-cached at ~/.cache/hyperframes/chrome/)
npx hyperframes browser ensure

# 4. Run full diagnostic — all checks must pass
npx hyperframes doctor

Troubleshooting Common Setup Issues

Pre-Flight Check (AI Must Execute Before Every Project)

# Quick 3-command pre-flight (run at start of every video task)
npx hyperframes doctor          # full diagnostic
which ffmpeg && ffmpeg -version  # confirm FFmpeg accessible
echo "Pre-flight OK"

If doctor reports any failure, resolve it BEFORE writing any HTML. Do NOT proceed with a broken environment — it will waste render time and produce invalid output.

Workflow Decision Tree

1. 从零生成视频 → Step 1 (Init) → Step 2 (Compose) → Step 3 (Animate) → Step 4 (Lint) → Step 5 (Render)
2. 口播后期加工（加字幕/特效/换背景） → 详见「口播后期处理管线」章节完整流程
3. 已有口播 + 去背景 + 换场景 → Remove background → Write composition with layers → Add captions → Add music → Render
4. 修改已有 composition → Read files → Modify → Lint → Render
5. 数据可视化视频 → Plan data scenes → Compose with animated charts → Render
6. 多段素材拼接 → FFmpeg concat → HyperFrames 加统一字幕/特效/转场 → Render

Step 1: Initialize Project

npx hyperframes init  --non-interactive
npx hyperframes init my-video --example blank
npx hyperframes init my-video --video clip.mp4        # with existing video
npx hyperframes init my-video --audio track.mp3       # with audio
npx hyperframes init my-video --tailwind              # Tailwind v4 support

Available templates: blank, warm-grain, play-mode, swiss-grid, vignelli, decision-tree, kinetic-type, product-promo, nyt-graph

CRITICAL: Rendering Requirements

These rules are MANDATORY — violating them causes the compile phase to hang indefinitely:

1. GSAP: Keep CDN URL — Always use . The Compiler automatically inlines CDN scripts. NEVER use a local lib/gsap.min.js — file serving breaks during render.
2. Fonts: DO NOT use @import — Never write @import url('https://fonts.googleapis.com/...'). Instead, just declare font-family: "Inter", sans-serif in CSS. The Compiler auto-resolves common Google Fonts and caches them to ~/.cache/hyperframes/fonts/. Supported fonts include: Inter, JetBrains Mono, Roboto, etc. Noto Sans SC is NOT auto-resolved — avoid it.
3. WorkBuddy sandbox — npx hyperframes render requires dangerouslyDisableSandbox: true because it launches Headless Chrome (Puppeteer).
4. Always use --non-interactive — Required for WorkBuddy automation.

CRITICAL: Prompt Priority & Creative Freedom Protocol

Priority Rule (MOST IMPORTANT — Governs Everything Below)

用户提示词的明确意图 > 硬约束（技术安全）> 软默认（参考建议）

Two modes of operation:

Creative License（创意许可）： 当用户的提示词包含具体的视觉描述、风格要求、布局指示、动效要求时，视为创意许可——AI 应优先实现用户的视觉意图，所有软默认规则自动退让。AI 不应将每个视频拉回同一个标准模板。

无论哪种模式，AI 必须在最终输出前执行 §9 AI自检管线，确保技术安全。

Rule Classification: Hard Constraints vs Soft Defaults

🔴 硬约束（HARD CONSTRAINTS — 永远生效，即使用户提示词冲突也不可违反）

这些是技术限制，违反会导致渲染崩溃、输出错误或不可预期行为：

🟢 软默认（SOFT DEFAULTS — 仅当用户提示词未指定时生效，一旦用户有明确意图即让位）

这些是经过验证的良好实践，但不是唯一正确答案：

1. Prompt Compliance Checklist

Before writing HTML, create a compact production checklist with:

• Topic/title and target audience
• Required scenes or content points
• Required visual style and forbidden styles
• Required color palette — ALWAYS follow the user's prompt specification; never substitute or override
• Required BGM style, whether it must cover the whole video, and target volume
• Duration target and final scene end time
• Any requested font/card/icon scale preferences
• Creative direction signal: 用户是否给了详细视觉指示？(YES → 严格执行，软默认让位; NO → AI自由发挥，参考软默认)

After rendering, check the output against this list. If the user asked for a specific BGM style, do not replace it with a different style unless explicitly approved. If the user specified a color palette, do not swap it for a "safer" or "more generic" scheme.

2. Color Palette Rule

User's prompt is the ONLY source of truth for color decisions. The Skill provides NO default theme colors. When the user provides a color scheme (e.g., "霓虹青绿玫红橙紫"), implement it exactly. When the prompt omits color requirements, AI may freely choose a palette that best serves the content and mood — no need to ask unless the user is unsatisfied.

3. Safe Typography Scale for 1920x1080 (Soft Default — 参考范围)

These are reference ranges for when the user hasn't specified typography preferences. If the user describes a specific visual style (e.g., "极简大字报风格", "数据密集仪表盘"), AI should adapt freely while ensuring readability.

Note: Hard max 仅为防溢出，不是风格限制。如果用户明确要超大标题效果且布局容得下，AI 应实现。

4. Layout Safe Zones (Soft Default — 标准布局参考)

以下是 1920x1080 的标准布局参考。当用户提示词未指定布局时使用；当用户描述了不同的布局需求（如全屏沉浸式、非规则排列、斜切设计、居中对称等），AI 应自由设计，仅确保不违反硬约束 H6（内容不超出画布）和 H7（禁止 inline top%）。

┌──────────────────────────────────────────────┐
│ Title Safe Zone: top 50px, height ≤ 150px    │ ← 标准参考
├──────────────────────────────────────────────┤
│ Content Safe Zone: top 240px ~ bottom 980px  │ ← 标准参考
├──────────────────────────────────────────────┤
│ Bottom Safe Margin: bottom 100px             │ ← 建议保留
└──────────────────────────────────────────────┘

硬约束（始终生效）：

• H6: 所有可见内容必须在 01920px (水平) 和 01080px (垂直) 范围内
• H7: 禁止 inline style="top:XX%" — 必须使用 CSS class 或固定 px 值

软默认（未指定时使用）：

• 标题区: top: 50px, 居中, height ≤ 150px
• 内容区: top: 240px, 底部不超过 980px
• 左右边距: ≥ 120px
• 标题与内容间距: ≥ 40px

Content height budget (标准布局下的参考计算):

available_height = 980px - 240px = 740px
max_card_height = (available_height - (rows-1) × gap) / rows

Example: 3 rows, gap=24px → max per row = (740 - 48) / 3 = 230px ✓
Example: 6 cards 2×3, gap=28px → max per row = (740 - 28) / 2 = 356px ✓
Example: 4 vertical items, gap=18px → max per item = (740 - 54) / 4 = 171px ✓

If content_height > available_height (overflow risk), apply fixes:

1. Reduce card padding (28px → 20px)
2. Reduce gap (36px → 24px → 18px)
3. Reduce icon/text size within cards
4. Split into two consecutive scenes

Card layout dimension reference (soft defaults):

5. Font White List — HARD CONSTRAINT (H2)

Only use fonts that HyperFrames Compiler can auto-resolve:

/* ALLOWED */
font-family: "Inter", sans-serif;
font-family: "JetBrains Mono", monospace;  /* for code blocks */
font-family: "Roboto", sans-serif;

/* PROHIBITED — cause lint errors or render failures */
font-family: "PingFang SC";        /* macOS only, not bundled */
font-family: "Microsoft YaHei";    /* Windows only */
font-family: "Noto Sans SC";       /* not auto-resolved */
font-family: "Source Han Sans";    /* not auto-resolved */

Chinese text rendering: rely on sans-serif fallback (renders correctly in Chrome). The visual difference is negligible for video output.

6. Deterministic Rendering — HARD CONSTRAINT (H1, H8)

HyperFrames requires frame-level determinism for multi-pass rendering:

// ❌ PROHIBITED — non-deterministic (H1)
Math.random()
Date.now()
new Date()
performance.now()  // for positioning

// ✅ REQUIRED — use seeded PRNG
function mulberry32(seed) {
  return function() {
    seed |= 0; seed = seed + 0x6D2B79F5 | 0;
    let t = Math.imul(seed ^ seed >>> 15, 1 | seed);
    t = t + Math.imul(t ^ t >>> 7, 61 | t) ^ t;
    return ((t ^ t >>> 14) >>> 0) / 4294967296;
  }
}
const rand = mulberry32(42);  // fixed seed

Also for GSAP repeat calculations (H8):

// ❌ May overshoot composition duration
repeat: Math.ceil(duration / cycle) - 1

// ✅ Guaranteed to stay within bounds
repeat: Math.floor(duration / cycle) - 1

7. Scene Duration and Root Duration

The root composition duration must equal the last scene end time:

root_duration = max(scene.data_start + scene.data_duration)

Never leave data-duration shorter than the final scene. This causes unreliable render behavior and confusing QA results.

8. BGM Must Match Prompt and Cover Full Video

For videos longer than 30s, do not rely on short audio loops inside HyperFrames. Use this reliable workflow:

1. Generate or prepare a full-length WAV whose duration is at least root_duration + 3s.
2. Render the visual video with HyperFrames (audio data-duration can match video, but DO NOT trust the rendered audio).
3. Replace/mux final audio with FFmpeg from the WAV source:

ffmpeg -y -i rendered.mp4 -i bgm_full.wav -map 0:v:0 -map 1:a:0 
  -c:v copy -c:a aac -b:a 192k -t  
  -movflags +faststart final.mp4

4. Verify the final MP4 by extracting the audio stream and checking real decoded duration and per-second RMS:

ffmpeg -y -i final.mp4 -vn -ac 1 -ar 44100 extracted_audio.wav
ffprobe -v quiet -show_entries format=duration -show_entries stream=codec_type,duration -of default=noprint_wrappers=1 final.mp4
ffmpeg -ss  -t 12 -i final.mp4 -af volumedetect -f null /dev/null 2>&1 | grep -E "mean_volume|max_volume"

Bundled helper option:

python scripts/verify_audio.py final.mp4 --min-duration  --tail-seconds 12

A successful BGM check must prove:

• Extracted audio duration >= video duration - 0.1s
• No silent seconds in the final 12 seconds (RMS > -30dB per second)
• The BGM style matches the user's requested or previously approved style
• Volume is audible but not overpowering (RMS mean between -15dB ~ -20dB)

BGM generation fallback: Only generate BGM when the user explicitly requests new music or when the source material has no audio to preserve. If numpy is unavailable, use Python standard library struct + wave + math to synthesize. A 130BPM electronic track can include: kick (4-on-floor), hi-hat (8th notes), bass (sub oscillator), pad (chord progression), lead melody, and arpeggio layers. For existing edited videos with original audio, preserve and post-mux the original audio instead of synthesizing replacement music.

Bundled helper option for generated BGM projects:

python scripts/gen_bgm.py bgm.wav --duration  --bpm 110 --volume 0.25

9. AI Self-Check Pipeline (Mandatory Before Delivery — 两种模式通用)

无论用户提示词是详细还是简单，最终交付前都必须执行此管线。 此管线验证的是技术安全和视觉完整性，不是风格是否符合某个模板。

Execute ALL checks in this exact order. Do NOT skip any step.

Phase A: Pre-Write Validation (before writing HTML)

Phase B: Post-Write / Pre-Render Validation

# B1: Lint must pass with 0 errors (HARD — always)
npx hyperframes lint

# B2: Check for prohibited patterns in HTML (HARD — H1, H2)
grep -n "Math.random|Date.now|PingFang|Microsoft YaHei|Noto Sans" index.html
# → must return empty

# B3: Check for inline top% overrides (HARD — H7)
grep -n 'style=.*top:.*%' index.html
# → If found: REMOVE immediately (use CSS class or fixed px)

# B4: Check root data-duration matches expected
grep 'data-composition-id.*data-duration|data-start.*data-duration' index.html
# → Verify last scene end = root duration

Phase C: Post-Render Validation

Phase D: Layout & Visual Integrity Check (关键步骤)

For every video, extract key frames and verify visual integrity:

# Extract key frames at scene midpoints
ffmpeg -ss  -i final.mp4 -frames:v 1 -q:v 2 check_scene_N.jpg

通用视觉检查（两种模式都执行）：

• ✅ 所有文字和内容在画布范围内（H6）— 无裁切
• ✅ 无文字互相重叠导致不可读
• ✅ 卡片/元素间距均匀，无挤压变形
• ✅ 关键信息可读（字号足够，对比度足够）
• ✅ 动画过渡流畅，无跳切（除非用户要求跳切风格）

额外检查（仅标准布局模式——使用软默认时）：

• 标题在顶部可见，不与内容重叠
• 所有卡片在安全区范围内

If ANY check in Phases B-D fails → fix → re-render → re-verify. Maximum 2 retry cycles; if still failing after 2 retries, report the specific issue to the user.

10. Auto-Completion Defaults (Soft — 仅简单提示词模式)

当用户的提示词未指定以下细节时，AI 可参考这些经过验证的默认值作为起点，但鼓励在此基础上发挥创意：

Key principle: 默认值是灵感参考，不是束缚。AI 应根据视频主题和内容自然选择最佳表达方式。

11. Common Pitfalls Registry (Quick Reference)

Step 2: Write HTML Composition

A composition is an HTML file. The root container defines the video canvas.

Simple Single-Scene Example

  


  


    
    

    
    

      Product Launch
    

    
    
  

  

  

Multi-Scene Standard Skeleton (Soft Default — Reference Starting Point)

For videos with 3+ scenes when the user hasn't specified a custom layout, this skeleton provides a proven starting point. If the user's prompt describes a different visual structure (e.g., full-screen transitions, split-screen, non-linear navigation, cinematic parallax), AI should design freely — only the Hard Constraints (H1-H9) remain in effect.

  




  
  

    
    

    

  

  
  

    

      
    
    

      
    
  

  
  

    

      
场景标题
      
副标题说明
    
    

      
    
  

  
  

  
  

Skeleton naming conventions (recommended for consistency, not mandatory):

• Scene wrappers: #scene-1, #scene-2, ... #scene-N
• Scene titles: #s1-title, #s2-title, ... #sN-title
• Content containers: #s1-content, #s2-cards, #s3-items, ... (descriptive suffix)
• Card elements: .glass-card, .data-card, .app-card (semantic naming)
• Background: #bg-layer, #grid-bg, #particles-container

Scene wrapper pattern (recommended, AI may use alternative transition approaches if user requests):

// Scene entry (instant opacity on)
tl.to("#scene-N", { opacity: 1, duration: 0.01 }, SCENE_START);
// Scene content animations
tl.from("#sN-title", { /* entrance */ }, SCENE_START + 0.1);
tl.from("#sN-content ...", { /* entrance */ }, SCENE_START + 0.3);
// Scene exit (0.6s before end) — EXCEPT final scene
tl.to("#scene-N", { opacity: 0, duration: 0.5 }, SCENE_END - 0.6);

Data Attributes Reference

Video and Audio Rules

• Video MUST be muted playsinline — audio is always a separate element
• Never call video.play()/audio.play() — the framework owns playback
• Never nest video inside a timed div — use a non-timed wrapper
• Long BGM warning: HyperFrames' built-in audio processing may truncate long background music around ~32s even when the source audio and data-duration are longer. Always verify the rendered MP4 by extracting its audio stream and checking the decoded audio duration/RMS; volumedetect -ss 30 -t 12 can be misleading because it only analyzes available samples.
• Reliable long-BGM workflow: render the visual video first, then replace/mux the audio with FFmpeg from a full-length WAV source: ffmpeg -i rendered.mp4 -i bgm_full.wav -map 0:v:0 -map 1:a:0 -c:v copy -c:a aac -b:a 192k -t -movflags +faststart final.mp4. Verify by extracting final.mp4 audio to WAV and checking per-second RMS through the end.

Step 3: Animate with GSAP

Non-Negotiable Rules

1. All timelines MUST start { paused: true } — player controls playback
2. Register every timeline: window.__timelines[""] = tl
3. Duration comes from data-duration, NOT from GSAP timeline length
4. No repeat: -1 — calculate exact repeats: repeat: Math.ceil(duration / cycleDuration) - 1
5. No Math.random(), Date.now() — use seeded PRNG if needed
6. No async timeline construction — no setTimeout, await, Promises
7. Only animate visual properties: opacity, x, y, scale, rotation, color, backgroundColor
8. Never animate visibility, display
9. Offset first animation 0.1–0.3s from t=0

Scene Transitions (multi-scene compositions)

1. ALWAYS use transitions between scenes — no jump cuts
2. ALWAYS use entrance animations (gsap.from()) on every element
3. NEVER use exit animations except on the final scene — the transition IS the exit
4. Final scene only may fade elements out

Layout Before Animation

Build the end-state first as static CSS, then add motion:

1. Position elements at their most visible moment
2. Add entrances with gsap.from() — animate FROM offscreen TO CSS position
3. Add exits with gsap.to() — only on final scene

Step 4: Lint & Inspect

npx hyperframes lint              # Structure/code check (fast)
npx hyperframes lint --json       # Machine-readable
npx hyperframes inspect           # Visual layout check (launches Chrome)
npx hyperframes inspect --json    # Agent-readable findings

Fix all errors before rendering. Warnings should be addressed.

Step 5: Render

npx hyperframes render                          # Standard MP4
npx hyperframes render --quality draft          # Fast iteration (~3x faster)
npx hyperframes render --quality high --fps 60  # Final delivery
npx hyperframes render --output final.mp4       # Custom filename
npx hyperframes render --format webm            # Transparent WebM
npx hyperframes render --docker                 # Byte-level consistency

Deliver the rendered MP4 to user via deliver_attachments.

Media Processing

Text-to-Speech (TTS)

Local inference with Kokoro-82M. No API key needed.

npx hyperframes tts "Your script here" --voice af_heart --output narration.wav
npx hyperframes tts script.txt --voice bf_emma --output narration.wav
npx hyperframes tts --list                      # List all 54 voices

Voice selection:

Chinese voices use z prefix: e.g., zf_xiaobei, zm_yunjian.

Requirements: Python 3.8+, pip install kokoro-onnx soundfile. Non-English needs espeak-ng.

Transcription (for captions)

Local Whisper inference. No API key needed.

npx hyperframes transcribe audio.mp3                      # → transcript.json
npx hyperframes transcribe video.mp4 --model small        # Default model
npx hyperframes transcribe video.mp4 --model medium       # Better accuracy
npx hyperframes transcribe subtitles.srt                  # Import existing

Output: transcript.json with word-level timestamps:

[{"id": "w0", "text": "Hello", "start": 0.0, "end": 0.5}, ...]

CRITICAL: Never use .en models unless audio is confirmed English. .en models TRANSLATE instead of transcribe.

Background Removal

Local u2net_human_seg model. No API key.

npx hyperframes remove-background talking-head.mp4 -o transparent.webm
npx hyperframes remove-background subject.mp4 -o subject.webm --background-output plate.webm
npx hyperframes remove-background portrait.jpg -o cutout.png

Output formats:

• .webm (VP9 + alpha) — for direct use in composition
• .mov (ProRes 4444) — for editing in external tools
• .png — single image cutout

Common Video Recipes

Recipe 1: Talking-Head + Captions + Background Music

# 1. Remove background from talking-head video
npx hyperframes remove-background talking-head.mp4 -o transparent.webm

# 2. Transcribe for captions
npx hyperframes transcribe talking-head.mp4 --model small

# 3. Init project and compose
npx hyperframes init captioned-video --non-interactive

Composition structure:

• Track 0: Background (gradient, image, or video)
• Track 1: Transparent talking-head (transparent.webm)
• Track 2: Animated captions (driven by transcript.json)
• Track 3: Background music ( with data-volume="0.3")

Recipe 2: Product Launch Video

npx hyperframes init product-launch --example product-promo --non-interactive