Diffusion For Babies

A Tutorial

​

Christian Simon

4

Meta -Emu

Runway

Imagen

Stable Image

Diffusion in Physics

Can we borrow this concept for machine learning problems?

How to get to this point?

Neural Network Training

​

​

1. What is exactly neural network trying to do?
2. How is neural network trained?

Neural Network Training

​

​

• What is exactly neural network trying to do?
• How is neural network trained?

x

y

f(.;θ)

Gaussian Distribution

Love Distribution

Decoder θ

p_θ(x)

Training a function to map from Gaussian dist. to ♡ dist.

Gaussian Distribution

Love Distribution

Decoder θ

p_θ(x)

Dataset: {x1, x2, x3, …, xN} ~ p_data(x)

How to estimate θ?

Decoder θ

Gaussian Distribution

Love Distribution

Diffusion

Decoder θ

p_θ(x)

Dataset: {x1, x2, x3, …, xN} ~ p_data(x)

Maximum Likelihood Estimation (MLE)

• What is MLE? - a method of estimating the parameters of an assumed probability distribution, given some observed data.
• Take for example, when fitting a Gaussian to our dataset, we take the sample mean and sample variance, and use it as the parameter of our Gaussian.

Gaussian Distribution

Love Distribution

Diffusion

Decoder θ

p_θ(x)

Dataset: {x1, x2, x3, …, xN} ~ p_data(x)

Let’s make it easier to compute!

Let’s make it easier to compute!

Remember expectation to integral

Remember expectation to integral

Similarity Between 2 Distributions

Matching Two Distributions

Gaussian Distribution

Love Distribution

Diffusion

Estimation

True/Data

Decoder θ

p_θ(x)

Dataset: {x1, x2, x3, …, xN} ~ p_data(x)

p_data(x)

KL

Div.

p(z)

Matching Two Distributions

Gaussian Distribution

Love Distribution

Decoder θ

p_θ(x)

p(z)

Remember: p(x,z) = p(z|x)p(x)

Simplifying the Formulation

Gaussian Distribution

Love Distribution

Decoder θ

p(z)

z

q(z | x)

Dataset: {x1, x2, x3, …, xN} ~ p_data(x)

Encoder

p_θ(x|z)

We can use this integral to 1, depending on preference.

Remember integral to expectation

Derive

Remember:

Derive

Derive

Derive

REMEMBER THIS?????

Variational AutoEncoder

How do we connect these two algorithms?

Encoding is a function to add noise

Decoding is (trainable) parameterized function to denoise

Layer is represented as denoising iterations, while VAE is not

q

p

Remember:

q

q

Back to Distribution Matching using KL Divergence

Recursive

Simplify

​

Inference DDIM

DDIM >>>> DDPM (Sampling in standard diffusion models)

Denoising Diffusion Implicit Models - DDIM

2010.02502 (arxiv.org)

200 steps >>>> 1000 steps

Stable Diffusion

Diffusion for Classification

Diffusion Classifier (diffusion-classifier.github.io)

Diffusion - Interrupted Optimization

Drawbacks

• Diffusion is hard to control. The randomness in diffusion process makes the generation hard to predict and controlled.

​

• Slow generation. DDPM might take 20 minutes on a low end GPU.

​

​

• Generation is hard to quantify with existing metrics. Most of the time, evaluation requires human in the loop to check the fidelity and quality of generated samples

Some Open Topics

• ​
• Fast diffusion model inference -1 step?
• Diffusion for classification purposes.
• Diffusion that can be combined with existing large models (e.g., LLMs).
• Diffusion models that can work on multimodalities