VideoPoet:

A Large Language Model for Zero-Shot Video Generation

ICML 2024 Best Paper�Presenter: Lijun Yu 于力军

https://sites.research.google/videopoet/

​

Video Poets

Dan Kondratyuk^

Lijun Yu*

Xiuye Gu

José Lezama

Jonathan Huang*^

Grant Schindler

Rachel Hornung

Vighnesh Birodkar^

Jimmy Yan

Ming-Chang Chiu^

Krishna �Somandepalli

Hassan Akbari^

Yair Alon

Yong Cheng

Josh Dillon^

Agrim Gupta^

Meera Hahn

Anja Hauth

David Hendon

Alonso Martinez^

David Minnen

Mikhail Sirotenko

Kihyuk Sohn^

Xuan Yang

Hartwig Adam

Ming-Hsuan Yang

Irfan Essa

Huisheng Wang

David A. Ross*

Bryan Seybold*

Lu Jiang*^

Equal contribution

* Corresponding authors

^ Work done while at Google

~10 Years of Video Generation

VGAN (16’)�http://www.cs.columbia.edu/~vondrick/tinyvideo/�Golf Baby

VideoPoet (24’)

~10 Years of Video Generation

VGAN (16’)�http://www.cs.columbia.edu/~vondrick/tinyvideo/�Golf Baby

VideoPoet (24’)

~10 Years of Video Generation

VGAN (16’)�http://www.cs.columbia.edu/~vondrick/tinyvideo/�Golf Baby

~10 Years of Video Generation

VGAN (16’)�http://www.cs.columbia.edu/~vondrick/tinyvideo/�Golf Baby

Latent diffusion has been dominating text to video

~10 Years of Video Generation

VGAN (16’)�http://www.cs.columbia.edu/~vondrick/tinyvideo/�Golf Baby

A skeleton drinking a glass of soda.

Latent diffusion has been dominating text to video

Is latent diffusion the only way to generate nice videos?

~10 Years of Video Generation

VGAN (16’)�http://www.cs.columbia.edu/~vondrick/tinyvideo/�Golf Baby

A skeleton drinking a glass of soda.

Is latent diffusion the only way to generate nice videos? Absolutely NOT!

VideoPoet is an autoregressive LLM that �synthesizes videos with high-fidelity motion and matching audio, �from a large variety of condition signals

VideoPoet Framework

VideoPoet Framework

Modality-specific tokenizers

Raw signal

Token

➡️ Encode & Compress ➡️

⬅️ Decode & Decompress ⬅️

MAGVIT-v2 Video Tokenizer

Yu et al. Language Model Beats Diffusion – Tokenizer is Key to Visual Generation. In ICLR 2024.

Defining the visual “language”

• Quantized VAE w/ temporally causal 3D CNN
• Image as a prefix of video for joint training
• Seamless support for long videos

MAGVIT-v2 Video Tokenizer

• Scalable quantizer w/ 2¹⁸ large vocabulary for higher prediction bandwidth
• Better compression than VVC with reconstructive and adversarial training

Yu et al. Language Model Beats Diffusion – Tokenizer is Key to Visual Generation. In ICLR 2024.

Defining the visual “language”

• Quantized VAE w/ temporally causal 3D CNN
• Image as a prefix of video for joint training
• Seamless support for long videos

SoundStream Audio Tokenizer

• Residual vector quantizer
• Better compression than Opus

Zeghidour et al. SoundStream: An End-to-End Neural Audio Codec. TASLP 2021.

Defining the audio “language”

• Quantized VAE w/ causal 1D CNN

VideoPoet Framework

Out-of-the-box LLM transformer on discrete token sequences

​

VideoPoet Framework

Out-of-the-box LLM transformer on discrete token sequences

• Flexibility: any to any task setup in a single model
• Training efficiency: learning entire sequence in a single step with causal attention
• Inference efficiency: various acceleration techniques such as caching,� where full decoding FLOPs equal to one full forward pass

Training Data

Mixture of pre-existing sources and formats, in two training phases

• 🔥 Pretraining uses everything, including unlabeled and noisy data

​

Training Data

Mixture of pre-existing sources and formats, in two training phases

• 🔥 Pretraining uses everything, including unlabeled and noisy data
• 🔧 Task adaptation uses task-specific high quality data

Training Tasks

Self-supervised

Training Tasks

Self-supervised

Training Tasks

Self-supervised + Supervised

MAGVIT Superresolution

Masked transformer with non-autoregressive decoding

• Faster at small scale
• Multi-axis windowed attention for long sequence

​

Yu et al. MAGVIT: Masked Generative Video Transformer. In CVPR 2023.

Automatic Benchmarks

Zero-shot text-to-video generation comparison with state-of-the-art

Human Evaluations

Zero-shot text-to-video generation comparison with prior and concurrent works

🔠 → 🎥�Text to Video

More at https://sites.research.google/videopoet/

🌁 → 🎥 Image to Video

(and 3D)

More at https://sites.research.google/videopoet/

🌁 → 🎥 Image to Video

🎥 → 🎨 Video Stylization

(and 3D)

Video Editing

More at https://sites.research.google/videopoet/

A kite surfer riding a shark.

On text-to-video generations, without text prompt

🎥 → 🔊

Video to Audio

More at https://sites.research.google/videopoet/

Future Research Directions

Context:

This is the logo of ICML

This is Lijun

Query:

Generate a video of Lijun dancing like this wearing a hoodie with the ICML logo.

This is a new dance

Future Research Directions

• Real-time streaming video generation�interactive neural gaming, neural user interface for OS / APPs
• Universal multimodal generative model�SOTA generation of text & video (& audio & …) and reasoning�c.f., human-level machine translation (~18’) -> ChatGPT (23’)

Query: Can you show me how to tie this shoe with a single hand? 🥾

Summary

VideoPoet represents a distinct approach to video generation

• State-of-the-art quality, �challenging the diffusion monopoly
• Multi-task flexibility, �going beyond the text to video translation paradigm
• Video-first foundation model, �building upon LLM infrastructure for native integration

​

Thank

you.