CSE 5539: �Generative Models

Generative models

2

Generative models

Easily samplable distribution

[Credits: Tutorial on Diffusion Models]

Generative models

• How to generate data (e.g., images)?
• By design
• By learning from “real” data

​

• Scenarios:
• From unlabeled data: p(x)
• From labeled data: p(y)p(x | y)

​

​

Cat

​

​

Dog

​

​

Bird

​

​

Lion

​

​

Tiger

​

​

Penguin

What and how to learn?

5

Objective 1: maximum likelihood estimation (MLE)

similar

Maximum likelihood (ML)

Maximum a posterior (MAP)

Variational, EM, …

Similar models OR data

How likely the model can generate the true data?

Objective 1: maximum likelihood estimation (MLE)

• How to build:
• Kullback–Leibler (KL) divergence

Density estimation

MLE =

min KL with empirical distribution

Generative models

• What to build:
• Explicit density estimation: and sample from it

Learn with maximum likelihood or its variants

​

• Implicit density estimation: learn a model to sample from

Learn with other objectives

​

[Ian Goodfellow, Tutorial on generative adversarial models, 2017]

[Stanford CS231n]

Generative adversarial net (GAN)

10

Generator

Discriminator

[Credits: Mengdi Fan and Xinyu Zhou, CSE 5539 course presentation]

Example results (by Style-GAN)

11

[A Style-Based Generator Architecture for Generative Adversarial Networks, CVPR 2019]

Other generative models

• Denoising Diffusion Probabilistic Models (PPDM)
• Learn to inverse the diffusion process
• Can generate very high-quality images

12

[Denoising Diffusion Probabilistic Models, NeurIPS 2020]

Other generative models

• Denoising Diffusion Probabilistic Models (PPDM)

​

13

[Denoising Diffusion Probabilistic Models, NeurIPS 2020]

Other generative models

14

Big-GAN

Diffusion models

Real

[Diffusion Models Beat GANs on Image Synthesis, NeurIPS 2021]

Conditional image generation

15

[Zhu et al., 2017]

[Wang et al., 2018]

Conditional image generation

16

[Hierarchical Text-Conditional Image Generation with CLIP Latents, arXiv 2022]

Diffusion models

Reference

• What are Diffusion Models? https://lilianweng.github.io/posts/2021-07-11-diffusion-models/
• Denoising Diffusion Probabilistic Models, NeurIPS 2020
• Denoising Diffusion Implicit Models, ICLR 2020
• Improved Denoising Diffusion Probabilistic Models, ICML 2021
• Diffusion Models Beat GANs on Image Synthesis, NeurIPS 2021
• Classifier-Free Diffusion Guidance, NeurIPS-W 2021
• Cascaded Diffusion Models for High Fidelity Image Generation, arXiv 2021

​

Reference

• GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models, arXiv 2021
• Hierarchical Text-Conditional Image Generation with CLIP Latents, arXiv 2022 (DALLE-v2)
• Junan Chen, Xiangyu Chen, Christian Belardi, Khiem Pham, Tutorial on Diffusion Models, 2022

​

(Probabilistic) generative models

Easily samplable distribution

[Credits: Tutorial on Diffusion Models]

Existing (probabilistic) generative models

[Credits: What are Diffusion Models?]

Problems:

​

​

Training stability and sample diversity

​

​

Surrogate loss and encoder model

​

​

Specialized architecture for reversible transform

Diffusion models

• Forward: a Markov chain of diffusion steps to slowly add random noise to data
• Reverse: learn to reverse the diffusion process to construct desired data samples from the noise.
• Latent variables: same size as the input

[Credits: What are Diffusion Models?]

Diffusion in (non-equilibrium) thermodynamics

Diffusion Process

Denoising Process

(Reverse of Diffusion)

[Credits: Tutorial on Diffusion Models]

Steps to understand diffusion models

• Modeling, forward

​

• Modeling, reverse

​

• Training

​

• Extension

Let’s look again

diffuse

denoise

“approximate” denoise

[Credits: Denoising Diffusion Probabilistic Models]

Diffusion/forward process

• Markovian steps (add noise gradually)

​

​

• Conditional joint probability

​

​

​

Scale down + add noise

Hyper-parameters

Diffusion/forward process

Let’s look again

diffuse

denoise

“approximate” denoise

Steps to understand diffusion models

• Modeling, forward

​

• Modeling, reverse

​

• Training

​

• Extension

​

Denoising/reverse process

Denoising/reverse process

Denoising/reverse process

[Credits: Sohl-Dickstein et al., 2015]

Steps to understand diffusion models

• Modeling, forward

​

• Modeling, reverse

​

• Training

​

• Extension

​

How to minimize (1)

How to minimize (2)

Ignore!

Auto-encoder:

Separately learned

How to minimize (3)

Short recap: diffusion models

• To learn:

​

​

​

• To generate by:

DDPM

DDPM

diffuse

denoise

“approximate” denoise

Generation

Summary

• Pros:
• DDPM is tractable
• DDPM is flexible: can fit arbitrary structures in data (multiple steps)

​

• Cons:
• Relies on long Markov Chain diffusion and denoising: very slow to generate
• Usually, we need to train T neural networks and apply all of them for generating one image

Why is it the right way to go? (by Harry)

• The whole diffusion process essentially goes through T times L layers of neural networks (U-Net).
• Every step only does a little, so we do not need to specifically design a generator architecture that directly goes from noise to images.
• We have deep supervised signal in the middle, and those supervised signal can be derived easily without additional learning.

Fun facts

Fun facts

Steps to understand diffusion models

• Modeling, forward

​

• Modeling, reverse

​

• Training

​

• Extension

​

(Class) conditioned generation

(Class) conditioned generation

(Class) conditioned generation

[Diffusion Models Beat GANs on Image Synthesis]

Class-conditioned DDPM (sampling)

(Class) conditioned generation

Left to right

• Big GAN
• Diffusion Model
• Test set

(Class) conditioned generation

[GLIDE]

(Class) conditioned generation

[Classifier-Free Diffusion Guidance]

More traditional methods

[Ian Goodfellow, Tutorial on generative adversarial models, 2017]

[Stanford CS231n]

Type of generative models

Tractable density

​

Approximate density: MC

Approximate density

Implicit density:

Direct

[credits: Shakir Mohamed DL Summer School 2016]

Fully-observed models

Directed decomposition:

x[1]

x[2]

x[3]

x[d]

Chain rules

Fully-observed models

Directed decomposition:

• Pixel-RNN [van der Oord et al.]

​

• Generate pixel by pixel
• Model p(.|.) by RNN (LSTM)

​

​

​

[credits: Stanford CS231n]

Fully-observed models

Directed decomposition:

• Pixel-CNN [van der Oord et al.]

​

• Generate pixel by pixel
• Model p(.|.) by CNN
• Prediction given the context
• Can learn in parallel

​

​

​

[credits: Stanford CS231n]

Fully-observed models

Undirect decomposition:

• Markov random field:

​

• clique: a complete sub graph
• : potential function
• Example:

​

​

​

​

Fully-observed models

• Directed decomposition:

​

• Easy to learn: tractable density (likelihood is directly computable)
• Order sensitive, need sequential generation

​

• Undirect decomposition:

​

• Hard to learn: the normalization constant can be intractable
• Need (iterative) sequential generation

Examples

……

p(x[1])

p(x[2] | history)

p(x[3] | history)

p(x[d] | history)

x[1]

x[2]

x[d-1]

Latent variable models

• Easy to sample and generate data
• Easy to include hierarchy, encode structure, avoid order dependency
• Hard to learn with maximum likelihood: intractable density

[credits: Shakir Mohamed DL Summer School 2016]

Transformation models

• Easy to sample and generate data
• Hard to learn for general models using “maximum likelihood”

[credits: Shakir Mohamed DL Summer School 2016]

Normalizing

Flow!

Two-sample tests

similar

If two populations --- real and generated data --- are similar

Two-sample tests

real

generated

learning objective

Backup

How to minimize (1)

How to minimize (2)

How to minimize (2)

How to minimize (2)

Ignore!

Auto-encoder:

Separately learned

How to minimize (3)

How to minimize (3)

How to minimize (3)