Diffusion
Chris Gregg
CS109, Stanford University
Autumn 2026
Two whippets on a catamaran
2
What looks
convincing?
What looks
unusual?
Two whippets are sailing on a catamaran on a lake with one whippet on the trapeze. One whippet is black and white, and the other is a fawn brindle. The catamaran hull should be orange, and the sail should be rainbow. Both whippets should be on the windward side of the catamaran, which should be moderately heeled over. Both whippets should be facing the sail. The whippet not on the trapeze should be manning the tiller.
�
Another sample of the scene
3
What changed?
What stayed
consistent with
the request?
Two whippets are sailing on a catamaran on a lake on a windy day with one whippet on the trapeze. One whippet is a black and white female, and the other is a male light brindle. The catamaran hull should be orange, and the sail should be rainbow. Both whippets should be on the windward side of the catamaran, which should be moderately heeled over. Both whippets should be facing the sail. The whippet not on the trapeze should be manning the tiller.
�
Diffusion has Stanford roots
4
ICML 2015
Sohl-Dickstein, Weiss,
Maheswaranathan and Ganguli
2015
Sohl-Dickstein and Ganguli’s team
introduced diffusion generative models.
2019
Yang Song and Stefano Ermon
developed score-based generation.
2021: Song and collaborators unified
score-based and diffusion approaches.
The big question: How can a neural network learn to turn noise into an image?
5
Generation as conditional sampling
6
“A frog walking on stilts across a sandy desert”
Close-up
Wide view
Sunset profile
The prompt constrains the scene. Randomness supplies variation.
Logistic regression / classification: Given an image, predict a label.
Image generation: Given a description, sample a possible image.
MLE connection: learn parameters that make observed data plausible.
Here,
describes a distribution over whole images.
An image is a vector of numbers
7
Each pixel has red, green and blue values.
Natural images have strong structure across those values.
A generator must learn how these values vary together.
The forward process adds noise
8
Image structure fades as the noise level increases.
We choose the corruption process and can sample it.
Less noise
More noise
Published example: Yang Song (2021)
A simple noise recipe
9
We trade signal for noise while keeping the overall scale controlled
The “noise schedule” keeps the overall scale controlled.
Each component of
is an independent
draw.
t: the chosen noise level
Different inputs approach the same noise
10
One coordinate of the image vector, starting at three different values
Near the end, the original value matters very little.
Different draws still give different noise samples.
A noisy image has several explanations
11
Suppose A and B are
equally plausible.
What image should a predictor return?
Which image should we return?
12
Suppose A and B are equally plausible.
Squared error favors the average of A and B.
A random sample chooses A or B, each with probability ½.
The supervised learning task
13
Input: noisy image, noise level and caption
Network output: a prediction of the added noise
Target: the actual random noise we added
We create the training answer ourselves.
The central training task
14
For each exercise:
1. Start with an image and its caption.
2. Sample a noise level and a random noise pattern.
3. Mix the image and noise using the schedule.
4. Give the network the noisy image, noise level, and caption.
5. Compare its predicted noise with the actual noise we sampled, then update its weights.
The network never receives the actual noise as an input. We use it as the target when computing the loss.
The surprising payoff is that to predict noise well across many images, the network must learn about image structure. Those predictions then help the sampler turn fresh noise into a new image during generation.
One training iteration
15
1. Sample an image and its caption from the dataset.
2. Sample a noise level and a random noise array.
3. Construct the noisy image.
4. Predict the noise and compute the loss.
5. Update the network parameters.
Repeat with new images and new corruptions.
Checkpoint: where are the labels?
16
Why do we know the correct noise
for a training example?
A. A human annotates the noise.
B. The program sampled and added it.
C. The network supplies its own target.
Generation starts with fresh noise
17
Predict noise, then use a sampling rule to update the state.
Repeat at lower noise levels with the same trained network.
Random noise
Generated room
Published example: Yang Song (2021)
An estimate and a sample differ
18
Squared error favors an average of plausible explanations.
A sampler should end at one possible outcome.
One model, different random samples
19
Sample A
Sample B
Randomness produces variation even before we add text.
Text becomes a numerical representation
20
“A frog walking on stilts across a sandy desert”
CLIP is one way to learn text–image associations.
A text encoder represents
the tokens in context.
Training connects words
to visual patterns.
Relationships help specify
how the scene fits together.
One prompt describes a whole scene
21
“A frog walking on stilts across a sandy desert”
Prompt component | What the image should express |
frog | The subject’s appearance and anatomy |
walking | A pose suggesting movement |
on stilts | Stilts supporting the frog, with appropriate contact and placement |
across a sandy desert | The surroundings and the subject’s relationship to them |
The relationships matter as much as the objects.
How text shapes the image
22
1. Encode the prompt in context.
A sequence of vectors represents the text and its relationships.
2. Combine text with the noisy image.
Attention lets image locations draw on relevant text information.
3. Predict noise using both inputs.
Training teaches the network how captions relate to image structure.
4. Update the image and repeat.
The sampler uses the prediction, then proceeds at a lower noise level.
The prompt guides each denoising step
23
“A frog walking on stilts across a sandy desert”
One description conditions
the whole scene.
The denoiser consults the text repeatedly as the image develops.
A different random seed can yield another scene that fits the same prompt.
Text guides the denoiser at each step
24
Different image regions
can use different parts
of the description.
The assignments
are learned.
Changing the prompt changes the denoiser’s predictions.
Noisy image
Noise level
Denoising network
with attention
Text-token vectors
Predicted noise
Guidance strengthens the text signal
25
Make a prediction with the prompt.
Compare with a prediction without the prompt.
Amplify the difference when forming the update.
Stronger guidance can improve prompt adherence,
but excessive guidance can introduce artifacts.
Familiar concepts, new combinations
26
A koala eating a BLT on a pink chaise lounge on Mars. There is a platypus eyeing the scene with skepticism.
The exact scene need not occur in the training set.
Learned patterns can generalize to new combinations.
Object counts and relationships can still go wrong.
An invented word needs a visual definition before it conveys a clear request.
Working in a compressed image space
27
Decoder
Denoise a smaller
latent array
Random noise
A smaller representation makes repeated updates cheaper.
During training, an encoder maps real images into this space.
Image latents and text embeddings play different roles.
Video adds consistency across time
28
An extra limb
holds the tiller
Frames must agree about anatomy and motion.
Visual plausibility does not guarantee physical consistency.
What did we learn?
29
Why can we create the noise labels automatically?
What changes during training? During generation?
Why can one prompt produce several different images?
A learned denoiser and a sampling procedure
turn simple randomness into structured samples.