Language Model Beats Diffusion�– Tokenizer is Key to Visual Generation

Lijun Yu

04/2024

lijun@lj-y.com

Motivation

• LMs (e.g., GPT-4) have dominated generative tasks in language
• LMs can also generate images and videos, e.g., DALL·E, MaskGIT
• But they do not perform as well as diffusion models, e.g., LDM
• A significant gap exists on the gold standard ImageNet benchmark �(FID 3.4 vs. 1.8)

​

• Why do language models lag behind diffusion models in visual generation?

​

Here LMs refer to transformer models that learn discrete token sequences

​

Background: LMs in Visual Generation

• Pixels are mapped into a sequence of discrete tokens by a visual tokenizer, �then processed by an LM transformer as if they are lexical tokens.
• Tokenizer remains the key bottleneck that controls sequence length and generation quality.

Preliminary: Image Tokenization

• Usually designed around the VQ-VAE framework
• Autoencoder with a discrete bottleneck
• Vector quantization with a learned codebook
• Spatial down sampling with CNN or ViT encoders
• Variants with different setups
• DALL·E dVAE uses ELB with gumble-softmax
• VQGAN adds perceptual and GAN losses
• ViT-VQGAN uses StyleGAN discriminator

​

Video Tokenization

• Naively: frame-by-frame tokenization
• Suffer from consistency issues, esp. for VQGAN
• Long redundant token sequence is a burden
• MAGVIT 3D VQGAN �– prior best video-native tokenizer
• Inflated 3D CNN architectures for better motion and consistency
• Spatial-temporal down sampling to reduce redundancy
• Losses: L2, perceptual, GAN, commitment, codebook, entropy, LeCAM

Issues with MAGVIT Tokenizer

• AR-LM and MLM do not scale well beyond 2B parameters.
• Performance mainly bounded by the tokenizer
• Vocabulary is limited around 1-8k, compared to ~200k used in LLMs
• Larger vocabulary hurts generation performance
• Only supports 16-frame clips, not images or longer videos
• Convolution padding results in strong implicit temporal encoding
• Hinders joint training with large-scale image data and long video generation

Introducing MAGVIT-v2 Tokenizer

• Lookup-free quantizer enables scalable vocabulary that helps generation
• Temporally causal 3D CNN jointly supports images and videos of variable length
• A collection of enhancements for visual quality

​

• State-of-the-art image and video generation on standard benchmarks.
• Better video compression than HEVC and VVC
• Stronger video understanding than MAGVIT

​

• Enabling LMs to scale! E.g., VideoPoet

​

---- Previously ----�SPAE: Semantic Pyramid AutoEncoder

• Anchored frozen language codebook (>65k)
• Hierarchical vector quantization
• Semantic loss

SPAE Tokenization Example

Text to Image �with frozen LLMs

The first time a frozen LLM generates images�without relying on external models, e.g., stable diffusion

With SPAE, we can transform any LLM into a Gemini-style multimodal model even without tuning.

Lookup-Free Quantization

Lookup-Free Quantization

Joint Image-Video Tokenization

• Utilizing large-scale labeled image data has been shown beneficial for video models
• E.g., make-a-video, phenaki, etc.
• Native 3D CNNs in MAGVIT face challenges to tokenize single images due to temporal receptive field
• Existing solution: C-ViViT from phenaki ➡️
• Hard to generalize to different spatial resolutions
• Worse visual quality
• Worse spatial causality of tokens

Joint Image-Video Tokenization

Exploring two new designs:

​

• Combining C-ViViT and MAGVIT
• 3D CNNs replace the spatial transformer and process 4-frame blocks.

​

• Temporally causal 3D CNN,�via custom convolution padding and upsampling
• The first frame remains independent.
• Allowing for videos of variable length.

Joint Image-Video Tokenization

Comparing joint/causal tokenization architectures on UCF-101.

FID is calculated on the first frame.

Architecture Ablations

Image Reconstruction

• VQGAN fails to reconstruct facial details
• MAGVIT-v2 does a much better job when trained on the same dataset
• Much larger vocabulary
• More powerful decoder
• And it further scales to larger data

Original

VQGAN

(ImageNet)

MAGVIT-v2

(ImageNet)

MAGVIT-v2

(Web Images)

Video Compression

MAGVIT-v2 is preferred over MAGVIT, HEVC (H.265), and VVC (H.266) in subjective rater study.

MAGVIT: Masked Generative Video Transformer

Squeezing something

Frame

Prediction

Frame

Interpolation

Out painting

Inpainting

Class-conditional �generation

The first multi-task masked transformer for video generation,�with state-of-the-art generation quality and efficiency.

Masked Video Synthesis

Masked tokens at each step: �Mask Condition Prediction

​

Input video

Generated videos

Sampling progress

Here we decode intermediate states for visualization, which does not happen in standard sampling.

• Initial state includes the condition
• Keep some generated tokens at each step
• All output tokens are predicted by the model, including condition

Token Factorization

Image Generation

Image Generation

Video Generation

• MAGVIT-v2 surpasses all prior arts
• MAGVIT-v2 significantly outperforms MAGVIT
• Using the same MLM backbone and decoding procedure
• Highlighting the importance of a good tokenizer

Video Generation: Frame prediction on Kinetics-600 and class-conditional generation on UCF-101

Video Generation

MAGVIT-v2 enables remarkable video generation quality with transformers using various objectives

• MLM – shown so far
• AR-LM – VideoPoet
• LDM – W.A.L.T

VideoPoet: �A Large Language Model for Zero-Shot Video Generation

• Synthesis of high-quality video with matching audio, from a large variety of condition signals
• Highlight: high fidelity motion
• A purely token-based approach, without diffusion

VideoPoet

• Multi-modal multi-task in a unified sequence model
• MAGVIT-v2 tokenizer for image/video tokenization
• SoundStream tokenizer for audio tokenization
• T5 for text embedding

Training

• Prefix LM with UL2-style objective
• A large mixture of tasks with different datasets in a single model
• Unconditional generation / text-to-image/video
• Video continuation / image-to-video
• Video inpainting / outpainting / stylization
• Video-to-audio / audio-to-video / audio-visual continuation
• Notably, we design sequence formats for easier transfer of capabilities, �e.g., text-to-image becomes a prefix of text-to-video

​

Preliminary Scaling

• Model: 300M, 1B, 8B parameters
• Data: 10B, 37B, 58B tokens

​

Video Generation

Audio Generation

Text-to-Video

Zero-shot text-to-video evaluation on MSR-VTT

See more: https://sites.research.google/videopoet

Text-to-Video

Human Evaluation

Image-to-Video

Source: https://twitter.com/AIsonesone

Video-to-Audio

On generated videos

W.A.L.T

• Windowed Attention Latent Transformer
• Joint image-video latent from MAGVIT-v2
• Joint diffusion training on image and video

W.A.L.T

Related: NaViT

• Variable aspect ratio and resolution training for ViTs �via packing & padding
• Originally used in LMs for sequences of naturally variable lengths
• We have been assuming fixed shape/length for vision models incl. ViTs� – another inductive bias to get rid of
• Can be efficiently implemented via attention masking� – non-trivial for CNNs, though data augmentation tricks have been helpful
• Note: each token still represents a fixed patch size – is it necessary?

Dehghani et al. Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution. In NeurIPS 2023.

Related: Sora

Our best guesses – it’s (almost) all about execution

• Image-video encoder & decoder �~> MAGVIT-v2 in W.A.L.T, SD VQGAN w/o quantization
• Latent diffusion transformer with variable output shapes�~> DiT, NaViT, W.A.L.T
• Data recaptioning ~> DALL·E 3
• No cascaded modules
• Scaling model, data, and compute…

Brooks et al. Video generation models as world simulators. 2024.

Related: Genie

Bruce et al. Genie: Generative Interactive Environments. 2024.

Takeaways

We have covered a lot:

MAGVIT, SPAE, MAGVIT-v2, VideoPoet, W.A.L.T, …�

One point that may be noteworthy:

Language models are at least as good as diffusion models on visual synthesis, if a good tokenizer/representation is available.

And language models are much more general (and scalable).

​

Just generating good-looking videos will never be our end goal.

Multi-modal native, interactive world models, learn from raw signals

Super intelligence?

​

Thank you!