1 of 19

Imagen

Text-to-Image Diffusion Models

Vinod

2 of 19

Given a brief description of a scene, Imagen can generate photorealistic, high-resolution images of the scene.

Able to combine unrelated concepts and objects in semantically plausible ways.

capturing spatial relationships, understanding cardinality, and properly interpreting how words in the description relate to one another

No database and no stitching of pre-existing sub-images

Introduction

3 of 19

Text encoder

encoder converts the textual caption to a numerical representation that encapsulates the semantic information

Image generator

Create image with noise
Guide the process with the encoded text – telling the model what is in the caption
Output small image

Super resolution model 1

Grows the image to a higher resolution
Takes the text encoding as input to helps the model on how to fill in the gaps of missing information
Output medium sized image

Super resolution model 2

Operates similar to super resolution model 1 with medium sized image as input
Output 1024 x 1024 sized image

How Imagen Works: A Bird’s-Eye View

4 of 19

Transformer encoder

Self-attention: ensures that the text encoding understands how the words within the caption relate to one another

Text-encoder is frozen: only generate encodings

Text Encoder

5 of 19

Diffusion model

iteratively corrupt the image with Gaussian noise in a series of timesteps, ultimately leaving pure Gaussian noise
learning how to isolate and remove the noise at each timestep, undoing the destruction process that just occurred
Once trained, the model can then be “split in half” – random noise to denoise
No control over what image is output

Caption conditioning

Create images that encapsulate the semantic information of the caption
Condition diffusion model on sequence of vectors

Image Generator

6 of 19

Diffusion model

Upsample model to 1024 x 1024 image

Diffusion model conditioned on caption encoding and the smaller image – medium size image

To upscale this image to the final 1024x1024 resolution, yet another super-resolution model is used

Image Super Resolution

7 of 19

T5 (Text-to-Text Transfer Transformer)

Released by Google in 2019
General framework for many NLP tasks -translation, sentence-acceptability determination (cola), sentence-similarity estimation (stsb), and summarization
T5 sees performance comparable to BERT even with only 25% of the training time thanks to its dataset

Why T5 used in Imagen

DALL-E 2 use text-encodes which are trained on image-caption pairs
Extremely large language models, by virtue of their sheer size alone, may still learn useful representations

Text Encoder

8 of 19

Input and output dimensionalities must be same

U-Net architecture for diffusion process

Each residual block composed of two sub-blocks and each of these composed of

Batch Normalization
ReLU
3x3 Convolution in sequence

Image Generator – Network Architecture

9 of 19

Inject timestep information into the model

Positional encoding

A unique timestep encoding vector is generated for each timestep

The vector is projected to having #channels component
After projection, each component of vector is added to corresponding channel in the image

Image Generator – Timestep Conditioning

10 of 19

The output vectors from T5 text encoder are pooled and added into the timestep embedding

U-Net model conditioned on entire encoding sequence by adding cross attention over the text embeddings at several resolutions

Image Generator – Caption Conditioning

11 of 19

Classifier-free guidance is a method of increasing the image fidelity of a Diffusion model at the cost of image diversity

Achieved by training a Diffusion model to be both conditional and unconditional at same time

Cast a conditional model
Randomly drop out a small fraction by assigning NULL value

Image Generator – Classifier-Free Guidance

12 of 19

Classifier-Free Guidance is a very powerful way to improve the caption alignment of generated images

extremely high guidance weights damage fidelity

Why fidelity occurs?

train-test data scale mismatch

How is it addressed?

Static thresholding
Dynamic thresholding

Dynamic thresholding significantly improves performance, yielding reasonable and unsaturated images even to high guidance weights

Image Generator – Large Guidance Weight Samplers

13 of 19

Trick: Add noise conditioning augmentation to base image
Small-to-Medium (STM)

64 x 64 image -> 256 x 256 image
Caption encoding + low resolution image
Cross attention
Efficient U-Net

Shifting model parameters
Scaling Skip connections
Changing order of operations

Medium-to-Large (MTL)

256 x 256 image -> 1024 x 1024 image
Same as STM

Super-Resolution Models

14 of 19

Quantitative

COCO dataset to evaluate text-to-image models
Metric: FID and CLIP
FID: 7.27
State of the art

Qualitative

Human evaluation to assess quality and caption similarity
200 test set
Subjects were shown batches of 50 of these images

Results and Analysis

Quality

Human: Which image is more photorealistic?
Preference rate (Imagen over ref image)
Preference rate : 39.2%

Caption Similarity

Human: Does the caption accurately describe the above image?
“yes” (100), “somewhat”(50) or “no (0)
Imagen on par with ref image

DrawBench

15 of 19

DALL-E 2 uses a contrastive objective to determine how related a text encoding is to an image (essentially CLIP).
Sheer Size

Imagen’s text encoder is much larger than DALL-E 2 and trained on more data
The effect of scaling up the text encoder is shockingly high, and that of scaling up the U-Net is shockingly low

The Image Encoder Crutch

CLIP: Trade off between text encoder and image encoder

Similar Concepts in Different Data Points

the blanket objective of maximizing the cosine similarity of corresponding caption-image pairs while minimizing that of differing ones does not account for similar concepts in distinct data points

Why is Imagen Better than DALL-E 2?

16 of 19

Scaling the text encoder is very effective
Scaling the text encoder is more important than U-Net size
Dynamic thresholding is critical
Noise conditioning augmentation in the super-resolution models is critical
Text conditioning via cross attention is critical
Efficient U-Net is critical

Key Takeaways

17 of 19

T5-XXL - ~11B parameter model (frozen)
Image generator– 2B parameter model
STM – 600M parameter model
MTL – 1024 parameter model
Epochs: 2.5M training steps for all models
Batch size: 2048
Hardware

256 TPU-v4 chips for Image generator
128 TPU-v4 chips for STM and MTL

Training details