1 of 74

1

EC 500 A1: �Generative Models - 2

2 of 74

Final project proposal (3%)

​

  • Project proposal: 11/9 (3%)

​

​

  • Baseline:
    • Something “simple” that you can implement in a day or two
    • Get a basic idea of the computation
    • Get a basic idea of the performance (e.g., classification accuracy)

3 of 74

Baseline/prototype is important!

  • Where is your data?
  • How are you going to separate your data?
  • How are you going to evaluate (what metric)?

​

  • What is your baseline approach?
    • What algorithm (e.g., YOLO)?
    • What neural network pre-trained backbone?
    • Anything simpler --- just to get a sense of whether the project is manageable

​

  • How long does it take to train and test?
    • Using what computational resource?

​

​

​

​

4 of 74

Today

  • Recap
  • GAN
  • Other fancier models
  • Variational auto-encoder (VAE)

​

​

​

4

5 of 74

Recap: Recognition models vs. generative models

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Generative models achieve “one-to-many” mapping by making the generator a “stochastic function”

latent variables

6 of 74

Recap: Unconditional generative models

  • Given a collection of images x without label y

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Gray: visible; White: latent

7 of 74

Recap: What is the “training” objective?

  • High-level: the output of the generator looks like real data

​

​

​

  • Different realization:
    • Match certain statistics: mean color, color variance, etc.
    • Synthetic data have high probability under a density model fit to real data
    • Synthetic data and real data are indistinguishable

​

8 of 74

Recap: What we have seen or will see

  • Density model
  • Autoregressive density model
  • Diffusion model
  • Generative adversarial net (GAN)
  • Variational auto-encoder (VAE)

9 of 74

Recap: Direct and indirect approaches

Direct “generator”

Indirect “generator”

    • Pros: easy to sample image (in testing)
    • Cons: hard to compute the density (in training)
    • Pros: easy to compute the density (in training) Cons: hard to sample image (in testing)

10 of 74

Recap: Maximum likelihood estimation (MLE)

similar

How likely the model can generate the true data?

11 of 74

Recap: Autoregressive model (where is z?)

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

12 of 74

Recap: Diffusion model (where is z?)

13 of 74

Training

14 of 74

Neural network model

15 of 74

Today

  • Recap
  • GAN
  • Other fancier models
  • Variational auto-encoder (VAE)

​

​

​

15

16 of 74

What is the objective?

  • High-level: the output of the generator looks like real data

​

​

​

  • Different realization:
    • Match certain statistics: mean color, color variance, etc.
    • Synthetic data have high probability under a density model fit to real data
    • Synthetic data and real data are indistinguishable

​

17 of 74

Generative adversarial networks (GAN)

18 of 74

Generative adversarial net (GAN)

18

Generator

Discriminator

REAL

FAKE

[Credits: Mengdi Fan and Xinyu Zhou, CSE 5539 course presentation]

 

 

 

19 of 74

Example results (by Style-GAN)

19

[A Style-Based Generator Architecture for Generative Adversarial Networks, CVPR 2019]

20 of 74

Today

  • Recap
  • GAN
  • Other fancier models
  • Variational auto-encoder (VAE)

​

​

​

20

21 of 74

Latent diffusion models

[High-Resolution Image Synthesis with Latent Diffusion Models, CVPR 2022]

You may also search: Stable Diffusion

22 of 74

Scalable Diffusion Models with Transformers

[Scalable Diffusion Models with Transformers, ICCV 2023]

23 of 74

Scalable Diffusion Models with Transformers

[Scalable Diffusion Models with Transformers, ICCV 2023]

24 of 74

Diffusion with Representation Autoencoders

[Diffusion Transformers with Representation Autoencoders, 2025]

25 of 74

Autoregressive image generation

[Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generation, 2025]

26 of 74

 Visual AutoRegressive modeling (VAR)

[Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction, 2024]

27 of 74

Today

  • Recap
  • GAN
  • Other fancier models
  • Variational Auto-encoder (VAE)
    • Concepts
    • AE vs. VAE
    • Math of VAE

​

​

27

28 of 74

Representation learning vs. generative modeling

“Unlabeled” data

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

29 of 74

Representation learning vs. generative modeling

“Unlabeled” data

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Map data to simple underlying representation (embedding)

Map simple base distribution (noise) to data

30 of 74

Representation learning vs. generative modeling

“Unlabeled” data

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

31 of 74

Representation learning vs. generative modeling

“Unlabeled” data

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

​

Latent variables are representations of the data

​

32 of 74

Technical setting for connecting them

  • Notation:

​

​

​

​

​

​

  • Goal:

Representation

Generation

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

33 of 74

Today

  • Recap
  • GAN
  • Other fancier models
  • Variational Auto-encoder (VAE)
    • Concepts
    • AE vs. VAE
    • Math of VAE

​

​

33

34 of 74

Question: difference from auto-encoder (AE)

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Now we introduce Pz: Gaussian

Auto-encoder (AE)

Variational

Auto-encoder (VAE)

35 of 74

Trick from AE to VAE

  • Regularize the latent distribution to squish into a Gaussian

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

36 of 74

Auto-encoder (AE) vs. Variational Auto-encoder (VAE)

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Auto-encoder (AE)

Variational

Auto-encoder (VAE)

37 of 74

Auto-encoder (AE) vs. Variational Auto-encoder (VAE)

Auto-encoder (AE)

Variational

Auto-encoder (VAE)

38 of 74

Today

  • Recap
  • Variational Auto-encoder (VAE)
    • Concepts
    • AE vs. VAE
    • Math of VAE

​

​

​

38

39 of 74

Objective function & Hypothesis space

  • Data log likelihood:

​

​

​

  • What is ?

​

Mixture model

Gaussian

Conditional Gaussian

40 of 74

Infinite mixture of Gaussian

  • VAE hypothesis space:

​

​

​

​

  • Standard Mixture of Gaussian:

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

41 of 74

Infinite mixture of Gaussian

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

42 of 74

Today

  • Recap
  • Variational Auto-encoder (VAE)
    • Concepts
    • AE vs. VAE
    • Math of VAE

​

​

​

42

43 of 74

Optimization (1)

  • Maximize the log likelihood

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

44 of 74

Optimization (2)

  • Integration approximated by Monte Carlo estimate (sampling + average)

45 of 74

46 of 74

47 of 74

48 of 74

Today

  • Recap
  • Variational Auto-encoder (VAE)
    • Concepts
    • AE vs. VAE
    • Math of VAE

​

​

​

48

49 of 74

Optimization (3)

  • Problem: inefficient usage of “z”

​

​

​

​

  • Decomposition

​

​

    • Only some of “z” will lead to high probability

50 of 74

Optimization (3)

  • Problem: inefficient usage of “z”

​

​

​

​

  • Decomposition

​

​

    • Only some of “z” will lead to high probability

51 of 74

Optimization (3)

  • Importance sampling

52 of 74

Optimization (3)

  • Importance sampling

“specific” distribution for sampling z

53 of 74

Optimization (3)

  • Importance sampling

54 of 74

Optimization (3)

  • Importance sampling

​

​

​

​

​

  • Optimal q:

55 of 74

Optimization (4)

  • Variational inference: approximate by another neural network

56 of 74

Today

  • Recap
  • Variational Auto-encoder (VAE)
    • Concepts
    • AE vs. VAE
    • Math of VAE

​

​

​

56

57 of 74

Optimization (4)

  • It could be hard

​

  • Variational inference: approximate by another conditional Gaussian

58 of 74

Put everything together

59 of 74

Put everything together

60 of 74

Put everything together

61 of 74

Put everything together

62 of 74

Today

  • Recap
  • Variational Auto-encoder (VAE)
    • Concepts
    • AE vs. VAE
    • Math of VAE

​

​

​

62

63 of 74

Overall training

64 of 74

Overall training

65 of 74

AE vs. VAE

Auto-encoder (AE)

Variational

Auto-encoder (VAE)

66 of 74

67 of 74

68 of 74

69 of 74

Today (33; skip 34)

  • Recap
  • Variational Auto-encoder (VAE)
    • Concepts
    • AE vs. VAE
    • Math of VAE
    • Examples

​

​

​

69

70 of 74

Examples

71 of 74

72 of 74

Gaussian encourages “disentanglement”

73 of 74

Recap: Generative models with “z” as Gaussian

[Credits: What are Diffusion Models?]

74 of 74