Lecture 16:

Diffusion Models

Spring 2026

Taxonomy of Generative Models

Diffusion

Generative models

Explicit density

Implicit density

Tractable density

Approximate density

Variational

Stochastic

Direct

Stochastic

Generative Adversarial Networks (GAN)

Variational Autoencoders (VAE)

Boltzmann Machine

Fully Visible Belief Nets

PixelRNN/CNN

Today

Spring 2025

2

Score-based Generative Modeling

Spring 2025

3

Score-based Generative Modeling

• Dataset consists of i.i.d. samples from unknown data distribution:

​

• Define score of probability density as:
• “the gradient of the log probability density of the data”

​

• A score network s_θ is trained to approximate the score:

Spring 2025

4

Score-based Generative Modeling

• Score-based generative modeling in a nutshell:

https://arxiv.org/pdf/1907.05600

Spring 2025

5

Score-based Generative Modeling

• Score estimation
• Directly train a score network s_θ(x) to estimate score, eventually the data distribution.
• The objective is to minimize

This part is constant�(independent of the score function s_θ).

x ∈ R^D

s_θ(x) : R^D → R^D

▽_xs_θ(x) ∈ R^D×D

Spring 2025

6

Challenge 1: Limited Support

In the low data density regions, the score estimation becomes inaccurate.�→ Due to the lack of data samples, there is no sufficient evidence to estimate score functions accurately in these regions.

Spring 2025

7

Proposed Solution: Denoising Score Matching

• Employs score matching to estimate the score of the perturbed data distribution q_σ(x̃ | x).
• The objective function is changed:�����
• If the noise is small enough, .

A similar idea to the denoising autoencoders.

Spring 2025

8

Proposed Solution: Denoising Score Matching

• q_σ(x̃ | x) is a perturbed distribution of data point x with a pre-specified noise.�����
• The score direction, moving x̃ back to x, is given by

Because q_σ(x̃ | x) = 𝓝(x̃; x, σ²)!

A density function that is added with a pre-specified Gaussian noise (𝝈)

q_σ(x̃ | x) = 𝓝(x̃; x, σ²).�That is, x̃ = x + σε, where ε~𝓝(0, σ²).

Spring 2025

9

Sampling with Langevin Dynamics

• Langevin dynamics:
• Def. A mathematical modeling of the dynamics of molecular systems, characterized by the use of simplified models while accounting for omitted degrees of freedom by the use of stochastic differential equations. -- Wikipedia
• ���
• Sampling with Langevin dynamics
• Produce samples from a probability density using only the score function.�→ We train a score network, then obtain samples using Langevin dynamics!
• Langevin method recursively computes the following:

Gradient of the force calculated from the particle interaction potentials

A zero-mean stationary Gaussian process

Score function s_θ(x)

z_t ~ 𝓝(0, I): to avoid local minimum

Spring 2025

10

Challenge 2: Incorrect Relative Density

When two modes of the data distribution are separated by low density regions, it is not able to correctly recover the relative weights of these two modes in reasonable time.�→ Langevin dynamics have incorrect relative density between the two modes.

Images from https://yang-song.net/blog/2021/score/

Spring 2025

11

Proposed Solution: Multiple Noise Levels

• Recall that perturbing data with random Gaussian noise makes the data distribution more amenable to score-based generative modeling.
• The perturbed data will not be confined to a low dimensional manifold any more.
• Being free from the low-dimensional manifold issue, the score estimation is well-defined anywhere now. 😊�
• To resolve the Challenge 2, a sequence of perturbed distributions with multiple noise levels are used to converge to the true data distribution.
• A single conditional score network is trained to estimate scores corresponding to all noise levels simultaneously.
• A large Gaussian noise has the effect of filling low density regions in the original unperturbed data distribution.

Spring 2025

12

Annealed Langevin Dynamics

• Distributions perturbed by multi-level Gaussian noises:
• Initially use scores corresponding to large noise (σ₃): providing support for the entire regions.
• Gradually anneal down the noise level to σ₂ and σ₁.
• Smoothly transfer the benefits of large noise levels to low noise levels where the perturbed data are almost indistinguishable from the original ones.

Spring 2025

13

Noise Conditional Score Networks: Training

• Training NCSN via denoising score matching
• Choose noise distribution to be:

​

• Denoising score matching objective is to minimize:���
• Thus, the loss function is given by

​

• For all noise level :

Spring 2025

14

Noise Conditional Score Networks: Inference

• NCSN inference via annealed Langevin dynamics
• After NCSN is trained, sampling with annealed Langevin dynamics.
• Start by initializing the samples from some fixed prior distribution (e.g., Uniform noise).
• Continually run Langevin dynamics to sample with reducing step size.

Spring 2025

15

Noise Conditional Score Networks: Examples

• NCSN inference via annealed Langevin dynamics

Spring 2025

16

• 6/4, 6/8 Class
• 6/11 Final review
• 6/18 Final Exam

Spring 2025

17

Denoising Diffusion Probabilistic Models (DDPM)

Spring 2025

18

Forward Diffusion Process

Spring 2025

19

Forward Diffusion Process

Reverse Diffusion Process

Spring 2025

20

Introduction

• A diffusion model is a parameterized Markov chain trained using variational inference to produce samples matching the data after finite time.

Reverse process

Forward process

https://arxiv.org/pdf/2006.11239

Spring 2025

21

Forward (Diffusion) Process

• Sample a data point from a real data distribution:
• Add small Gaussian noise to the sample for T steps, producing a sequence of noisy samples .���
• β_t indicates the degree of added noise. (With β_t = 0, x_t = x_t-1; with β_t → 1, x_t gets independent of x_t-1.)

Spring 2025

22

Forward (Diffusion) Process

Reparameterization trick: let α_t = 1 - β_t, .

• From ,������

.�

​

When we merge two Gaussians with different variance, 𝓝(0, σ₁²I) and 𝓝(0, σ₂²I), the new distribution is 𝓝(0, (σ₁²+σ₂²)I).�

Spring 2025

23

Forward (Diffusion) Process

• Reparameterization trick: let α_t = 1 - β_t, .
• From ,������
• Thus, .�
• Through this reparameterization trick, we can noise the data by arbitrary time step in a closed form.
• Here, is called diffusion kernel.
• t can be any arbitrary time step, 1, …, T.

When we merge two Gaussians with different variance, 𝓝(0, σ₁²I) and 𝓝(0, σ₂²I), the new distribution is 𝓝(0, (σ₁²+σ₂²)I).�

Spring 2025

24

Reverse (Denoising) Process

• If we can reverse the forward process and sample from q(x_t-1 | x_t), we will be able to recreate the true sample from a Gaussian noise input, x_T ~ 𝓝(0, I).
• This is counter-intuitive, contradicting to the second law of thermodynamics.
• However, we can approximate q(x_t-1 | x_t) by a normal distribution if β_t is small enough!�
• Let’s model p_θ to approximate q(x_t-1 | x_t):�
• With this,

In short, our goal is to learn this conditional distribution.

Chain rule hold: iid, Markovian assumption

Spring 2025

25

Reverse (Denoising) Process

• Our goal is to learn p_θ by maximizing likelihood of real images x₀.
• Equivalently, we minimize the negative log-likelihood, -log p_θ(x₀).
• We apply variational lower bound (VLB) of this:

The blue term enforces p ≈ q.

Now, in order to minimize our desired negative log-likelihood -log p_θ(x₀), we minimize this variational bound term.

Spring 2025

26

Reverse (Denoising) Process

Reverse conditional probability becomes tractable when conditioned on x₀ ,(still mathematically equivalent due to Markovian assumption) and using Bayes’ rule, can be expressed as Gaussian distribution.

Spring 2025

27

Reverse (Denoising) Process

• To sum up, we would like to minimize

Prior matching term.

L_T is constant, since q has no learnable parameters and x_T is the input.

Reconstruction term.

This is modeled with a separate discrete decoder derived from

Consistency term.

KLD compares two Gaussian distributions, so we have a closed form solution.

​

For this, we will first find the single-Gaussian expression of the true distribution q(x_t-1 | x_t, x₀), and will model p_θ(x_t-1 | x_t) to approximate it.

Spring 2025

28

Reverse (Denoising) Process

• Let’s represent the consistency term as a single Gaussian:

From the reparameterization trick in page page 21:

Spring 2025

29

Reverse (Denoising) Process

• Let’s represent the consistency term as a single Gaussian (cont’d):

From the diffusion kernel in page 21:

Exercise!

Middle school math:

Find the μ and β to make this a perfect square.

Spring 2025

30

Reverse (Denoising) Process

• Coming back to our objective function L_VLB,

• Now, we model , so that it approximates the true distribution q(x_t-1 | x_t, x₀) by

Spring 2025

31

Reverse (Denoising) Process

Constant terms ignored.

Reparametrize the Gaussian noise term to make it predict ε from the input x_t.

This is often very large for small t and very small for large t.�Empirically, simply removing this improves the sample quality.

Spring 2025

32

DDPM: Model Architecture

• The U-Net architecture with ResNet blocks and self-attention layers are adopted.
• Time representations (sinusoidal positional embeddings) are fed to the residual blocks.

Spring 2025

33

DDPM: Summary

Spring 2025

34

DDPM: Examples

Spring 2025

35

DDPM: Limitations

• Slow sampling due to the iterative T steps.
• Takes around 20 hours to sample 50k images of size 32 × 32.
• cf. A GAN model takes less than 1 minute for the same task.

• The training process spends too much capacity and time on modeling unimportant details.

Spring 2025

36