Autoregressive Conditional Generation using Transformers

Paritosh Mittal, Yen-Chi Cheng,

Maneesh K. Singh, Shubham Tulsiani

Introduction

2

3

4

*https://unsplash.com/photos/EPy0gBJzzZU

5

an armchair in the shape of an avocado

[1] https://arxiv.org/pdf/1705.00389v2.pdf

[2] https://openai.com/blog/dall-e/

[3] ShapeNet 3D

Image taken from the original paper [1]

Example taken from original Dall-E work [2]

Image is a visualization from ShapeNet [3]

Autoregressive models

6

Autoregressive models

Autoregressive modeling in time-series domain means that observations from past time-steps are used to predict the value at current time

Benefits of likelihood based methods over Generative Adversarial Networks

• Training regime is stable
• Return explicit probability densities
• Can generate diverse outputs
• Easier evaluation and measure of generalization

7

s_i-4

s_i-3

s_i-2

s_i-1

s_i

Conditional Image Generation with PixelCNN Decoders

• This is an autoregressive form of image generation�
• The value of a pixel x_i in image x is conditioned on previously generated pixels��

8

Pixel CNN is used as decoders to generate realistic images given latent embeddings as input

• CNN features are masked to restrict the receptive field

*https://arxiv.org/pdf/1606.05328.pdf

Image taken from the original paper*

Image taken from the original paper*

Pixel CNN (Pros v. Cons)

• Models return explicit probability densities�
• Fast(er) training

• Results are poor compared to PixelRNN�
• Blind spots in receptive fields

Further modifications, example Gated PixelCNN, PixelCNN++, etc. try to alleviate some of these concerns.

9

Pros

Cons

*https://arxiv.org/pdf/1606.05328.pdf

Discrete - Compact Latent Representations

10

Vector Quantized - Variational AutoEncoders

VAEs regularize the latent space during training. VQ-VAEs propose to use discrete latent variables

• Discrete representations are more common across different modalities�
• Discretization reduces the amount of info. stored in latent space, however most important info is stored�
• Models like Transformers are designed to work with discrete inputs

We need a codebook or embedding table for Vector Quantization step

11

*https://arxiv.org/pdf/1711.00937.pdf

VQ-VAE / VQ-VAE - 2

• VQ-VAE 2 could generate high fidelity images�
• Used a hierarchical method for learning the embeddings�
• Used PixelCNN as an autoregressive decoder

12

*https://arxiv.org/pdf/1906.00446.pdf

Image taken from the original paper*

VQ-VAE 2

VAEs regularize the latent space. VQ-VAEs propose to use discrete latent variables

• Discrete representations are more common across different modalities�
• Discretization reduces the amount of info. stored in latent space, however most important info is stored

13

*https://arxiv.org/pdf/1906.00446.pdf

Image taken from the original paper*

VQ-VAE - VQ step

• Encoder gives an output of dim (n,h,w,d). Resize this to (n*h*w,d)
• Find the closest among k vectors in codebook (Argmin).
• Reshape to (n,h,w,1)
• Replace the latent variables with entries from codebook (n,h,w,d)

14

*https://arxiv.org/pdf/1711.00937.pdf

Image taken from the original paper*

Using Transformers

15

Transformers

Transformers are a form of autoregressive models which can model long-range relationships from its inputs

Why use Transformers?

• Can better capture long range relations�
• No inductive bias towards local interactions

16

Taming Transformers for High-Resolution Image Synthesis

This method builds on top of VQ-VAE

• Use GANs in place of VAE to learn effective codebook
• Use Transformers to learn the global relations in latent space

17

*https://arxiv.org/pdf/2012.09841.pdf

Image taken from the original paper*

Taming Transformers for High-Resolution Image Synthesis

High Resolution image generation:

• Patchwise generation is adopted to limit sequence (16x16)
• Transformer is used in sliding window manner

Further works like Dall-E can generate interesting results with 1.2 B params

18

*https://arxiv.org/pdf/2012.09841.pdf

Image taken from the original paper*

Our Goal

Use the autoregressive ability of Transformers to generate complete high quality representations of visual information from partial inputs

19

Social Motivation

20

Social Motivation

Developing generative models for high quality visual content can:

• Enable users to auto-generate content from partial inputs

• Enable researchers to better understand visual world

21

Partial sketch -> complete image

Text -> images/Videos

Frame -> video

Parts -> 3D Shape

Ways to represent content

Work across modalities

Better understanding of image formulation

Realistic Data Augmentation

This project aims to leverage from the learnings of

• Autoregressive generation�
• Discrete latent representations�
• Transformers

22

Q/A

23

Problem Formulation

• Given a sparse input signal, we aim to generate the full signal with high quality via an autoregressive model

24

Sparse input signal

Full signal

Problem Formulation: Examples

25

Image taken from: alamy¹, conceptdraw², flickr³, this paper⁴

• Challenge: sparse input, multi-modality

Model - Learning Vector Quantized Codebook

26

Codebook

Lookup�Codebook

Quantization

Input�(HxW)

Recon.

(Hz x Wz)

Model - Generative Models on Quantized Vectors

27

Quantization

Encoder

Decoder

Transformer

33

33

?

Input

Output

Model - Conditional Generation on Sparse Input

28

Quantization

Encoder

Decoder

Transformer

?

?

?

2

33

33

8

?

15

Complete Quantized Vector

Partial Quantized Vector

Input

Recon.

Learning Vector Quantized Codebook Across Domains

• We first verify that the VQVAE model could learn good representation across different domains
• Images
• 3D Object

29

Ref: MNIST, CIFAR10, Cats, ShapeNet

MNIST

CIFAR10

Cats

ShapeNet

VQVAE on Images - MNIST

30

Input

Recon.

Resolution: 32x32

k: 10 d: 64 z dimension: 64x8x8

VQVAE on Images - CIFAR10

31

Input

Recon.

Resolution: 64x64

k: 128 d: 256 z dimension: 256x16x16

VQVAE on Images - Cats

32

Input

Recon.

Resolution: 256x256

k: 1024 d: 256 z dimension: 256x16x16

Learning Vector Quantized Codebook for 3D Object with SDF

• How to represent 3D object? Signed Distance Function (SDF).�
• Benefits of using SDF values�
• Truncated SDF or SDF

33

*image taken from this paper

*SDF to the Stanford bunny

Visualization of SDF

• Visualize slice by slice along x, y, z dimension�����
• Visualization in Mesh: extract mesh using Marching Cubes and rendered with pytorch3d

34

Model - Learning Vector Quantized Codebook

35

Codebook

Lookup�Codebook

Quantization

Encoder

Decoder

Input: SDF (DxHxW)

Recon: SDF�(DxHxW)

Dataset: ShapeNet Chair

(Dz x Hz x Wz)

VQVAE on SDF - ShapeNet

36

Input

Recon.

Resolution: 64x64x64

k: 512 d: 256 z dimension: 256x8x8x8

VQVAE on TSDF - ShapeNet

37

Input

Recon.

Resolution: 64x64x64

k: 512 d: 256 z dimension: 256x8x8x8

t: 0.2

Metrics: IoU

• Setting
• Consider SDF > 0 as free space (set to zero)
• Consider SDF <= 0 as occupied space (set to one)
• Compute the 3D IoU accordingly�
• Example

• Comparison (mean IoU across testset)
• TSDF v.s. SDF: 0.5104 v.s. 0.4084

38

0.7695

0.3591

0.1186

Future Steps

39

Quantization

Encoder

Decoder

Transformer

33

33

?

1. Learning Vector Quantized Codebook

2. Generative Models on Lower Dimension

1. Learning Vector Quantized Codebook
2. Generative Models on Lower Dimension
3. Extend to Conditional Generation on Sparse Input

Future Steps

40

Quantization

Encoder

Decoder

Transformer

?

?

?

2

33

33

8

?

15

Complete Quantized Vector

Partial Quantized Vector

3. Extend to Conditional Generation on Sparse Input

• Learning Vector Quantized Codebook
• Generative Models on Lower Dimension
• Extend to Conditional Generation on Sparse Input

Q/A

41

42

Thanks

43

Appendix