Generating One-Minute Videos with Test-Time Training

This groundbreaking research tackles a major hurdle in AI video generation: creating longer, multi-scene videos from text. While recent models excel at short, visually stunning clips, generating minute-long narratives presents a significant challenge due to the sheer volume of information required. This new approach, developed by NVIDIA, Stanford, UC Berkeley, and others, leverages Test-Time Training (TTT) to overcome these limitations.

Table of Contents

• The Long Video Challenge
• TTT: A Dynamic Solution
• One-Minute Video Examples with TTT
• How TTT Works
• The Tom & Jerry Dataset
• Performance Evaluation
• Artifacts and Limitations
• TTT's Unique Advantages
• Future Research Directions
• TTT vs. Other Leading Models
• Conclusion

The Long Video Challenge

Current video generation models, often based on Transformers, struggle with longer videos due to the quadratic computational cost of self-attention mechanisms. Generating a minute of high-resolution video requires processing hundreds of thousands of tokens, leading to inefficiency and narrative inconsistencies. While RNN-based approaches like Mamba or DeltaNet offer linear-time context handling, their fixed-size hidden states limit expressiveness.

TTT: A Dynamic Solution

This research introduces TTT layers—small, trainable neural networks (MLPs) integrated into RNNs. These layers adapt dynamically during inference, learning from the evolving video context using a self-supervised loss. This allows the model to adjust its internal "memory" as the video progresses, improving narrative coherence and motion smoothness.

One-Minute Video Examples with TTT

The researchers demonstrate TTT's capabilities by generating one-minute Tom & Jerry videos from detailed text prompts. These examples showcase improved temporal consistency and motion smoothness compared to baseline models.

Video 1: Jerry stealing cheese

Video 2: Tom and Jerry kitchen chase

Video 3: Example of limitations

How TTT Works

The system incorporates TTT layers into a pre-trained Diffusion Transformer model (CogVideo-X 5B). Self-attention is limited to short segments, while TTT layers manage global narrative understanding. Gating mechanisms prevent performance degradation during early training. Bidirectional sequence processing and scene-segmented annotations (storyboard format) further enhance training.

The Tom & Jerry Dataset

The research utilizes a dataset derived from classic Tom & Jerry cartoons, annotated into 3-second segments with detailed descriptions. This controlled environment simplifies the task, focusing on narrative coherence and motion dynamics.

Performance Evaluation

TTT-MLP significantly outperforms baselines (Mamba 2, Gated DeltaNet) in human evaluation, achieving a 34-point Elo score improvement. It excels in motion naturalness, temporal consistency, and overall aesthetic quality.

Artifacts and Limitations

Despite the progress, artifacts like inconsistent lighting and unnatural motion remain. These are likely due to limitations of the base model and the computational cost. While faster than full self-attention, TTT-MLP is slower than some RNN approaches. However, only fine-tuning is needed, making it more practical.

TTT's Unique Advantages

• Expressive memory through trainable hidden states
• Adaptability during inference
• Scalability to longer, more complex videos
• Efficient fine-tuning

Future Research Directions

Future work includes optimizing TTT kernels, experimenting with different backbone models, exploring more complex storylines, and using Transformer-based hidden states.

TTT vs. Other Leading Models

Model	Core Focus	Input Type	Key Features	How It Differs from TTT
TTT (Test-Time Training)	Long-form video generation with dynamic adaptation	Text storyboard	Adapts during inference, handles 60 sec videos, coherent multi-scene stories	Designed for long videos; updates internal state during generation for narrative consistency
MoCha	Talking character generation	Text Speech	Speech-driven full-body animation	Focuses on character dialogue & expressions, not full-scene narrative videos
Goku	High-quality video & image generation	Text, Image	Rectified Flow Transformers, multi-modal input support	Optimized for quality & training speed; not designed for long-form storytelling
OmniHuman1	Realistic human animation	Image Audio Text	Multiple conditioning signals, high-res avatars	Creates lifelike humans; doesn’t model long sequences or dynamic scene transitions
DreamActor-M1	Image-to-animation (face/body)	Image Driving Video	Holistic motion imitation, high frame consistency	Animates static images; doesn’t use text or handle scene-by-scene story generation

(Links to related articles on MoCha, DreamActor-M1, Goku, and OmniHuman1 would be inserted here.)

Conclusion

TTT represents a significant advancement in long-form video generation. Its ability to adapt during inference enables more coherent and engaging storytelling, paving the way for more sophisticated AI-generated media.

The above is the detailed content of Generating One-Minute Videos with Test-Time Training. For more information, please follow other related articles on the PHP Chinese website!