1 of 29

Diffusion

Chris Gregg

CS109, Stanford University

Autumn 2026

2 of 29

Two whippets on a catamaran

2

What looks

convincing?

​

What looks

unusual?

Two whippets are sailing on a catamaran on a lake with one whippet on the trapeze. One whippet is black and white, and the other is a fawn brindle. The catamaran hull should be orange, and the sail should be rainbow. Both whippets should be on the windward side of the catamaran, which should be moderately heeled over. Both whippets should be facing the sail. The whippet not on the trapeze should be manning the tiller.

�

3 of 29

Another sample of the scene

3

What changed?

​

What stayed

consistent with

the request?

Two whippets are sailing on a catamaran on a lake on a windy day with one whippet on the trapeze. One whippet is a black and white female, and the other is a male light brindle. The catamaran hull should be orange, and the sail should be rainbow. Both whippets should be on the windward side of the catamaran, which should be moderately heeled over. Both whippets should be facing the sail. The whippet not on the trapeze should be manning the tiller.

�

4 of 29

Diffusion has Stanford roots

4

ICML 2015

Sohl-Dickstein, Weiss,

Maheswaranathan and Ganguli

2015

Sohl-Dickstein and Ganguli’s team

introduced diffusion generative models.

2019

Yang Song and Stefano Ermon

developed score-based generation.

2021: Song and collaborators unified

score-based and diffusion approaches.

5 of 29

The big question: How can a neural network learn to turn noise into an image?

5

6 of 29

Generation as conditional sampling

6

“A frog walking on stilts across a sandy desert”

Close-up

Wide view

Sunset profile

The prompt constrains the scene. Randomness supplies variation.

Logistic regression / classification: Given an image, predict a label.

Image generation: Given a description, sample a possible image.

MLE connection: learn parameters that make observed data plausible.

Here,

describes a distribution over whole images.

7 of 29

An image is a vector of numbers

7

Each pixel has red, green and blue values.

Natural images have strong structure across those values.

A generator must learn how these values vary together.

8 of 29

The forward process adds noise

8

Image structure fades as the noise level increases.

We choose the corruption process and can sample it.

Less noise

More noise

Published example: Yang Song (2021)

9 of 29

A simple noise recipe

9

We trade signal for noise while keeping the overall scale controlled

The “noise schedule” keeps the overall scale controlled.

Each component of

is an independent

draw.

t: the chosen noise level

10 of 29

Different inputs approach the same noise

10

One coordinate of the image vector, starting at three different values

Near the end, the original value matters very little.

Different draws still give different noise samples.

11 of 29

A noisy image has several explanations

11

Suppose A and B are

equally plausible.

​

What image should a predictor return?

12 of 29

Which image should we return?

12

Suppose A and B are equally plausible.

Squared error favors the average of A and B.

A random sample chooses A or B, each with probability ½.

13 of 29

The supervised learning task

13

Input: noisy image, noise level and caption

Network output: a prediction of the added noise

Target: the actual random noise we added

We create the training answer ourselves.

14 of 29

The central training task

14

For each exercise:

1. Start with an image and its caption.

2. Sample a noise level and a random noise pattern.

3. Mix the image and noise using the schedule.

4. Give the network the noisy image, noise level, and caption.

5. Compare its predicted noise with the actual noise we sampled, then update its weights.

The network never receives the actual noise as an input. We use it as the target when computing the loss.

The surprising payoff is that to predict noise well across many images, the network must learn about image structure. Those predictions then help the sampler turn fresh noise into a new image during generation.

15 of 29

One training iteration

15

1. Sample an image and its caption from the dataset.

2. Sample a noise level and a random noise array.

3. Construct the noisy image.

4. Predict the noise and compute the loss.

5. Update the network parameters.

Repeat with new images and new corruptions.

16 of 29

Checkpoint: where are the labels?

16

Why do we know the correct noise

for a training example?

A. A human annotates the noise.

B. The program sampled and added it.

C. The network supplies its own target.

17 of 29

Generation starts with fresh noise

17

Predict noise, then use a sampling rule to update the state.

Repeat at lower noise levels with the same trained network.

Random noise

Generated room

Published example: Yang Song (2021)

18 of 29

An estimate and a sample differ

18

Squared error favors an average of plausible explanations.

A sampler should end at one possible outcome.

19 of 29

One model, different random samples

19

Sample A

Sample B

Randomness produces variation even before we add text.

20 of 29

Text becomes a numerical representation

20

“A frog walking on stilts across a sandy desert”

CLIP is one way to learn text–image associations.

A text encoder represents

the tokens in context.

Training connects words

to visual patterns.

Relationships help specify

how the scene fits together.

21 of 29

One prompt describes a whole scene

21

“A frog walking on stilts across a sandy desert”

Prompt component

What the image should express

frog

The subject’s appearance and anatomy

walking

A pose suggesting movement

on stilts

Stilts supporting the frog, with appropriate

contact and placement

across a sandy desert

The surroundings and the subject’s

relationship to them

The relationships matter as much as the objects.

22 of 29

How text shapes the image

22

1. Encode the prompt in context.

A sequence of vectors represents the text and its relationships.

2. Combine text with the noisy image.

Attention lets image locations draw on relevant text information.

3. Predict noise using both inputs.

Training teaches the network how captions relate to image structure.

4. Update the image and repeat.

The sampler uses the prediction, then proceeds at a lower noise level.

23 of 29

The prompt guides each denoising step

23

“A frog walking on stilts across a sandy desert”

One description conditions

the whole scene.

The denoiser consults the text repeatedly as the image develops.

A different random seed can yield another scene that fits the same prompt.

24 of 29

Text guides the denoiser at each step

24

Different image regions

can use different parts

of the description.

​

The assignments

are learned.

Changing the prompt changes the denoiser’s predictions.

Noisy image

Noise level

Denoising network

with attention

Text-token vectors

Predicted noise

25 of 29

Guidance strengthens the text signal

25

Make a prediction with the prompt.

Compare with a prediction without the prompt.

Amplify the difference when forming the update.

Stronger guidance can improve prompt adherence,

but excessive guidance can introduce artifacts.

26 of 29

Familiar concepts, new combinations

26

A koala eating a BLT on a pink chaise lounge on Mars. There is a platypus eyeing the scene with skepticism.

The exact scene need not occur in the training set.

Learned patterns can generalize to new combinations.

Object counts and relationships can still go wrong.

An invented word needs a visual definition before it conveys a clear request.

27 of 29

Working in a compressed image space

27

Decoder

Denoise a smaller

latent array

Random noise

A smaller representation makes repeated updates cheaper.

During training, an encoder maps real images into this space.

Image latents and text embeddings play different roles.

28 of 29

Video adds consistency across time

28

An extra limb

holds the tiller

Frames must agree about anatomy and motion.

Visual plausibility does not guarantee physical consistency.

29 of 29

What did we learn?

29

Why can we create the noise labels automatically?

What changes during training? During generation?

Why can one prompt produce several different images?

A learned denoiser and a sampling procedure

turn simple randomness into structured samples.