Introduction to Diffusion-based Generative Models
YANG Can
macyang@ust.hk
Department of Mathematics, HKUST
Fall, 2024
Review of Deep Generative models (GAN)
Review of Deep Generative models (VAE and flow-based)
Score-based methods
https://yang-song.net/blog/2021/score/
Connection with Tweedie’ formula
Tweedie’s formula
Herbert Robbins (1956) credits personal correspondence with Maurice Kenneth Tweedie for an extraordinary Bayesian estimation formula.
Naive score-based generative modeling and its pitfalls
1. The manifold hypothesis
The manifold hypothesis states that data in the real world tend to concentrate on low dimensional manifolds embedded in a high dimensional space (a.k.a., the ambient space)
2. Inaccurate score estimation with score matching
In regions of low data density, score matching may not have enough evidence to estimate score functions accurately, due to the lack of data samples.
Naive score-based generative modeling and its pitfalls
Naive score-based generative modeling and its pitfalls
3. Trouble in mode recovery
When two modes of the data distribution are separated by low density regions, Langevin dynamics will not be able to correctly recover the relative weights of these two modes in reasonable time, and therefore might not converge to the true distribution.
[Ref] Luo, C. (2022). Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208.11970.
Score-based generative modeling with multi-scale noise perturbations
Score-based generative modeling with multi-scale noise perturbations
https://yang-song.net/blog/2021/score/
Cifar10 and Celebra
https://yang-song.net/blog/2021/score/
Diffusion models
⌣
[Ref] Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33, 6840-6851.
[Ref] Vincent, P. (2011). A connection between score matching and denoising autoencoders. Neural computation, 23(7), 1661-1674.
Denoising diffusion probabilistic models (DDPM)
True Posterior mean (used for solving inverse problems)
Problem Setting
Linear inverse problem
Inpainting
Super Resolution
Non-uniform deblur (Non-linear)
[ref] Chung, H., Kim, J., Mccann, M. T., Klasky, M. L., & Ye, J. C. (2022). Diffusion Posterior Sampling for General Noisy Inverse Problems. arXiv preprint arXiv:2209.14687.
https://onlinelibrary.wiley.com/doi/10.1111/insr.12002
Reference
[1] Chung, H., Kim, J., Mccann, M. T., Klasky, M. L., & Ye, J. C. (2022). Diffusion Posterior Sampling for General Noisy Inverse Problems. arXiv preprint arXiv:2209.14687.
[2] Luo, C. (2022). Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208.11970.
[3] Song, Y., & Ermon, S. (2019). Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32.
[4] Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33, 6840-6851.
[5] Song, J., Meng, C., & Ermon, S. (2020). Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502.
[6] Bao, F., Li, C., Zhu, J., & Zhang, B. (2022). Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint arXiv:2201.06503.
[7] Chung, H., Sim, B., Ryu, D., & Ye, J. C. (2022). Improving Diffusion Models for Inverse Problems using Manifold Constraints. arXiv preprint arXiv:2206.00941.
[8] Song, Y., Shen, L., Xing, L., & Ermon, S. (2021). Solving inverse problems in medical imaging with score-based generative models. arXiv preprint arXiv:2111.08005.
[9] Chung, H., Kim, J., Mccann, M. T., Klasky, M. L., & Ye, J. C. (2022). Diffusion Posterior Sampling for General Noisy Inverse Problems. arXiv preprint arXiv:2209.14687.
[10] Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020). Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456.
[11] Choi, J., Kim, S., Jeong, Y., Gwon, Y., & Yoon, S. (2021). Ilvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.02938.
[12] Jalal, A., Arvinte, M., Daras, G., Price, E., Dimakis, A. G., & Tamir, J. (2021). Robust compressed sensing mri with deep generative priors. Advances in Neural Information Processing Systems, 34, 14938-14954.
[13] Wei, X., Fu, S., Li, H., Liu, Y., Wang, S., Feng, W., ... & Gu, Y. (2022). Single-cell Stereo-seq reveals induced progenitor cells involved in axolotl brain regeneration. Science, 377(6610), eabp9444.
[14] Vincent, P. (2011). A connection between score matching and denoising autoencoders. Neural computation, 23(7), 1661-1674.