1 of 31

Diffusion Models

Prof. Seungchul Lee

Industrial AI Lab.

2 of 31

State-of-the-art Generation with Diffusion Models

2

https://www.youtube.com/watch?v=HK6y8DAPN_0&t=105s&ab_channel=OpenAI

Open AI’s SORA

3 of 31

Generative Models Timeline

3

VAE

2013

2024

2014

2015

2020

2022

GAN

Diffusion

DDPM

(Denoising Diffusion Probabilistic Models)

Stable diffusion

DALL-E

4 of 31

Diffusion Models Beat GANs

4

Dhariwal & Nichol, NeurIPS, 2021.

Diffusion models beat GANs

on image synthesis

Diffusion models beat GANs

on topology optimization

Diffusion models beat GANs

on talking-face generation

Maze & Ahmed, AAAI, 2023.

Stypułkowski et al., WACV, 2024.

5 of 31

Commercially Available Generative Models

5

DALL-E

Stability AI

Midjourney

6 of 31

Generative Models (Revisited)

  • Discriminative/predictive models
    • Learn the boundary between data classes (e.g., predicting material properties from composition)
    • Focus on mapping input to output (e.g., regression or classification tasks)
  • Generative models
    • Learn the underlying distribution of data, capturing the patterns and relationships within
    • Aim to generate new samples similar to the training data

6

Learn the distribution of data!

 

Discriminative/predictive models

Generative models

7 of 31

Comparison of Generative Models

  • Variational Autoencoders (VAEs)
    • Excel in mode coverage (diversity) but often produce lower-quality samples
  • Generative Adversarial Networks (GANs)
    • Generate high-quality samples but struggle with mode collapse and diversity
  • Diffusion Models
    • Achieve a balance between high-quality samples and diversity but can be computationally expensive

7

https://pub.towardsai.net/diffusion-models-vs-gans-vs-vaes-comparison-of-deep-generative-models-67ab93e0d9ae

8 of 31

Introduction to Diffusion Models

8

9 of 31

Principles of Data Generation in Diffusion Models

  • Generate new data gradually from latent input
    • Improved stability
    • Better sampling

​

9

Data generation with GAN/VAE

Data generation with diffusion models

Latent input

(noise)

 

Generated data

Latent input

(noise)

Generated data

Denoise

[1] Ho et al., NeurIPS 2020

[2] Song and Ermon, NeurIPS 2020

[1], [2]

[1]

 

10 of 31

Physical Intuition of Diffusion Models

  • Inspired by non-equilibrium thermodynamics
  • Formulated as Markov chains
    • Gaussian transition for a sufficiently small timestep
  • Analogous to adding noise in an image

​

10

https://www.mrdubuque.com/home/biodub-my-gifs-to-you-diffusion-osmosis

  1. Markovian property
  2. Gaussian transition

https://dzdata.medium.com/intro-to-diffusion-model-part-1-29fe7724c043

Physical diffusion process

Diffusion of noise in images

11 of 31

Diffusion Models Overview

  • Learn to generate by denoising
  • Consists of two processes
    • Forward process: gradually adds noise to input to destroy information
    • Reverse process: iteratively removes noise to generate sample

​

11

Data

Noise

 

 

 

 

 

 

 

Forward process

Reverse process

12 of 31

Diffusion Models Overview

  • Learn to generate by denoising
  • Consists of two processes
    • Forward process: gradually adds noise to input to destroy information
    • Reverse process: iteratively removes noise to generate sample

​

12

Data

Noise

Forward process

Reverse process

 

 

 

 

 

 

13 of 31

Forward Diffusion Process

  •  

13

 

+

 

Noise

 

 

 

 

 

14 of 31

Forward Diffusion Process

  •  

14

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Reparameterization trick

 

 

 

15 of 31

Forward Diffusion Process

  •  

15

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

16 of 31

Diffusion Models Overview

  • Learn to generate by denoising
  • Consists of two processes
    • Forward process: gradually adds noise to input to destroy information
    • Reverse process: iteratively removes noise to generate sample

​

16

Data

Noise

Forward process

Reverse process

 

 

 

 

 

 

17 of 31

Reverse Diffusion Process

  •  

17

 

 

 

 

 

 

Loss (simplified)

 

18 of 31

Reverse Diffusion Process

  •  

18

 

 

 

 

 

 

 

 

 

 

 

19 of 31

Reverse Diffusion Process

  •  

19

 

 

 

 

 

 

 

Ho et al. NeurIPS 2020

 

 

 

 

20 of 31

Reverse Diffusion Process

  •  

20

 

 

 

 

 

 

 

Ho et al. NeurIPS 2020

 

Predict noise to be removed

 

 

 

 

21 of 31

Reverse Diffusion Process

21

 

 

 

Ho et al. NeurIPS 2020

 

 

 

22 of 31

Reverse Diffusion Process

  • Neural network is trained to predict noise used during forward diffusion

22

 

 

Noise from forward process

 

 

 

23 of 31

Notes on Noise Subtraction

  • What happens if we simply subtract noise?

23

 

 

 

 

 

 

https://medium.com/@kemalpiro/step-by-step-visual-introduction-to-diffusion-models-235942d2f15c

 

 

NOT

 

OR

24 of 31

Data Generation

  •  

24

 

Data

Noise

(Latent input)

 

 

 

Reparameterization trick

 

25 of 31

Data Generation (Intuitively)

  •  

25

 

Data

Noise

(Latent input)

 

 

 

 

 

26 of 31

Diffusion Implementation in TensorFlow

26

27 of 31

Forward Process Implementation

27

 

28 of 31

Neural Network Architecture

28

 

+

+

+

 

 

 

29 of 31

Training

29

 

30 of 31

Reverse Process Implementation (After training)

30

Retake screenshot

 

31 of 31

Generated Images

31

  • Trained on Fashion MNIST

Training data

Generated