Introduction to Diffusion-based Generative Models

​

Biomolecular Diffusion Reading Group

Meeting 1: 09/13/2023

Meeting Outline

• Introduction to the Biomolecular Diffusion Reading Group (BDRG)!
• Logistics, schedule, and future topics

​

• Overview of diffusion-based generative models (DGMs)
• What is diffusion and how does it work?
• Some details: forward diffusion, variance schedule, reverse diffusion, example training and sampling algorithms, extensions
• Summary: key points, takeaways, and recommended readings

​

2

BDRG Logistics

• Meet every 2 weeks
• Next meeting planned for Wed, Sept 27^th at 11 am EDT (UTC-4)
• Topic (?): Practical considerations and more math
• Karras et al. (2022) Elucidating the Design Space of Diffusion-Based Generative Models. [paper] [code]

​

• Future meetings will look at specific papers, dive into the methods, and discuss the results
• If possible, include author(s) from paper in discussion

​

• Polls to vote on and select papers for discussion!

​

3

Potential protein-related papers to discuss

4

Why diffusion-based generative models?

5

Why diffusion-based generative models?

6

DGMs for proteins

7

¹Watson et al. (2022); ²Lin & AlQuraishi (2023)

What is diffusion? How does it work?

8

random prior

Sohl-Dickstein et al. (2015); Yang & Ermon (2019); Ho et al. (2020); Song et al. (2021)

Forward diffusion iteratively corrupts the data distribution towards a simple prior distribution

9

Sohl-Dickstein et al. (2015)

random prior

Forward diffusion injects Gaussian noise

Forward diffusion iteratively corrupts the data distribution towards a simple prior distribution

10

Sohl-Dickstein et al. (2015)

random prior

Forward diffusion injects Gaussian noise

11

The Diffusion Process

12

Sohl-Dickstein et al. (2015); Yang & Ermon (2019); Ho et al. (2020); Song et al. (2021)

random prior

Reverse diffusion denoises corrupted data to generate new samples from the data distribution

13

Forward Diffusion:

Reverse Diffusion:

Reverse diffusion denoises corrupted data to generate new samples from the data distribution

14

Reconstruction Term

Prior Term

Denoising Step Term

Forward Diffusion:

Reverse Diffusion:

Ho et al. (2020)

Example training and sampling algorithms for DGMs

15

Ho et al. (2020)

Compute predicted noise

Take a step to remove the noise

Get example from dataset

Get time

Get noise

Compute loss and backprop

(inference)

Extensions to DGMs

• Conditional generation
• Diffusion models can be guided towards producing samples of specific classes using either “classifier guided diffusion” or “classifier-free guidance”.

​

• Stochastic differential equations
• Rather a set of discrete noise distributions, a diffusion process formulated as a SDE with Brownian motion can evolve a continuum of noise distributions.

​

• Denoising diffusion implicit models (DDIMs)
• Sampling diffusion models can be slow, especially when there are thousands of time steps. Instead, with DDIMs sampling uses only a subset of the time steps

​

• Riemannian manifolds
• Many real-world data distributions do not live on Euclidean spaces (i.e., “flat geometry”). Diffusion processes on the Riemannian manifolds can better capture distributions of high-order data

16

Nichol & Dhariwal 2021; Dhariwal & Nichol 2021; Ho & Salimans 2021; Nichol, Dhariwal, & Ramesh 2022; Song et al. 2021; De Bortoli et al. 2022

Summary and Takeaways for DGMs

17

Recommended Readings

Papers:

• Sohl-Dickstein et al. (2015) Deep Unsupervised Learning using Nonequilibrium Thermodynamics.
• Song & Ermon (2019) Generative Modeling by Estimating Gradients of the Data Distribution.
• Ho et al. (2020) Denoising diffusion probabilistic models.
• Song et al. (2020) Denoising diffusion implicit models.
• Nichol & Dhariwal (2021). Improved denoising diffusion probabilistic models.
• Song et al. (2021) Score-Based Generative Modeling through Stochastic Differential Equations.
• Karras et al. (2022) Elucidating the Design Space of Diffusion-Based Generative Models.

​

Blogs:

• How diffusion models work: the math from scratch (2022). Karagiannakos & Adaloglou
• What are Diffusion Models? (2021). Lilian Weng
• Improving Diffusion Models as an Alternative To GANs (2022). Vahdat & Kreis
• Generative Modeling by Estimating Gradients of the Data Distribution (2021). Song

​

18