1 of 78

Generative Modelling

Sudeshna Sarkar

Professor, Computer Science & Engineering

Professor, Centre of Excellence in Artificial Intelligence

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

2 of 78

Slides source

  • 11-785 Introduction to Deep Learning
  • CMU Course Spring 2022
  • Instructors: Bhiksha Raj and Rita Singh

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

3 of 78

What is a generative model

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

seed

 

Question: how do we generate the random seeds…

4 of 78

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

  • From a large collection of face images, can a network learn to generate a new portrait
    • Generate samples from the distribution of “face” images

5 of 78

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

  • From a large collection of landscapes, can a network learn to generate new landscape pictures
    • Generate samples from the distribution of “landscape” images
      • How do we even characterize this distribution?

6 of 78

Generative Models

Training data ~ pdata(x)

Generated samples ~ pmodel(x)

 

Given training data, generate new samples from same distribution

​

Formulate as density estimation problems

  • Explicit density estimation: explicitly define and solve for pmodel(x)
  • Implicit density estimation: learn model that can sample from pmodel(x) w/o explicitly defining it

pmodel(x)

learning

sampling

7 of 78

The distribution of data

Hypothesis: The data are distributed about a non-linear manifold in high dimensional space

    • The principal components of all instances of the target class of data lie on this manifold
  • To generate data for this class, we must select a point on this manifold

Problems:

    • Characterizing the manifold
    • Having a good strategy for selecting points from it

​

​

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

 

8 of 78

Application of Generative Models

  • Realistic samples for artwork, super-resolution, colorization, etc
  • Learn useful features for downstream tasks such as classification
  • Training generative models can also enable inference of latent representations that can be useful as general features
  • Modeling physical world for simulation and planning (reinforcement learning applications, robotics)
  • Image augmentation
  • Natural language generation

​

​

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

9 of 78

Taxonomy of Generative Models

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

Generative models

Explicit density

Implicit density

Direct

Tractable density

Approximate density

Markov Chain

Variational

Markov Chain

Fully Visible Belief Nets

  • NADE
  • MADE
  • PixelRNN/CNN

Change of variables models (nonlinear ICA)

Variational Autoencoder

Boltzmann Machine

GSN

GAN

Figure copyright and adapted from Ian Goodfellow, Tutorial on Generative Adversarial Networks, 2017.

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

10 of 78

Taxonomy of Generative Models

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

Generative models

Explicit density

Implicit density

Direct

Tractable density

Approximate density

Markov Chain

Variational

Markov Chain

Fully Visible Belief Nets

  • NADE
  • MADE
  • PixelRNN/CNN

Change of variables models (nonlinear ICA)

Variational Autoencoder

Boltzmann Machine

GSN

GAN

Figure copyright and adapted from Ian Goodfellow, Tutorial on Generative Adversarial Networks, 2017.

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

11 of 78

Autoregressive generative models

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

Main principle for training:

each of these is just a softmax

Using autoregressive generative models:

12 of 78

Deep Generative Models

We will introduce two representative deep generative models:

  1. Variational Autoncoder (VAE) is a variational algorithm that infers the statistical characteristics of latent variables using an autoencoder neural network architecture.
  2. Generative Adversarial Networks (GAN) train two neural networks (a generative neural network and a discriminative neural network) contesting with each other in a zero-sum game framework.

12

20-Oct-22

13 of 78

Latent Variable Models

Autoencoders and VAE

Generative Adversarial Networks

13

20-Oct-22

14 of 78

Latent Variables and VAE

Can we learn the true explanatory factors (latent variables) from observed data

​

  • PixelCNNs define tractable density function, optimize likelihood of training data:

​

​

  • VAEs define intractable density function with latent z:

​

​

  • Cannot optimize directly, derive and optimize lower bound on likelihood instead

​

​

14

20-Oct-22

15 of 78

Variational Auto Encoders

VAEs are a combination of the following ideas:

  1. Auto Encoders
  2. Variational Approximation
    • Variational Lower Bound / ELBO
  3. Amortized Inference Neural Networks
  4. “Reparameterization” Trick

​

(C) Dhruv Batra

15

16 of 78

Autoencoder

A neural network that reconstructs its own input.

reproduces the input from a learned encoding.

​

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

Basic idea:

1. Train a network that encodes the input into some hidden state

2. decodes that input as accurately as possible from that hidden state

Hidden state

this is what we use for downstream tasks

encoder

decoder

The autoencoder captures the underlying manifold of the data

17 of 78

Forcing structure in Autoencoder

Forcing structure: something about the design of the model, or in the data processing or regularization, must force the autoencoder to learn a structured representation

​

  • Dimensionality: make the hidden state smaller than the input/output, so that the network must compress it
  • Sparsity: force the hidden state to be sparse (most entries are zero), so that the network must compress the input
  • Denoising: corrupt the input with noise, forcing the autoencoder to learn to distinguish noise from signal
  • Probabilistic modeling: force the hidden state to agree with a prior distribution

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

18 of 78

Autoencoders

Slide Credit: Fei-Fei Li, Justin Johnson, Serena Yeung, CS 231n

Reconstructed data

Input data

Encoder: 4-layer conv

Decoder: 4-layer upconv

Encoder

Input data

Features

Decoder

Reconstructed input data

L2 Loss function:

19 of 78

Bottleneck autoencoder

  • This has some interesting properties:
    • If both encoder and decoder are linear, this exactly recovers PCA
    • Can be viewed as “non-linear dimensionality reduction” – could be useful simply because dimensionality is lower and we can use various algorithms that are only tractable in low-dimensional spaces (e.g., discretization)

​

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

encoder

hidden

state

decoder

100 x 100 =

10,000 dimensions

128 dimensions

20 of 78

Sparse autoencoder

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

Idea: can we describe the input with a small set of “attributes”?

This might be a more compressed and structured representation

Pixel (0,0): #FE057D

Pixel (0,1): #FD0263

Pixel (0,2): #E1065F

NOT structured

“dense”: most values non-zero

has_ears: 1

has_wings: 0

has_wheels: 0

very structured!

“sparse”: most values are zero

there are many possible “attributes,” and most images don’t have most of the attributes

Idea: “sparse” representations are going to be more structured!

21 of 78

Sparse autoencoder

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

encoder

hidden state

decoder

sparsity loss

dimensionality might be very large, even larger than the input!

called overcomplete

“L1 regularization”

“L2 regularization”

22 of 78

Sparse Autoencoder

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

23 of 78

Contractive Autoencoders

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

24 of 78

Denoising autoencoder

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

Idea: a good model that has learned meaningful structure should “fill in the blanks”

encoder

hidden state

decoder

There are many variants on this basic idea, and this is one of

the most widely used simple autoencoder designs

Can train an autoencoder to learn to denoise input by giving input corrupted instance ˜x and targeting uncorrupted instance x

25 of 78

Denoising Autoencoder

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

26 of 78

The types of autoencoders: Forcing Structure

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

1. Dimensionality: make the hidden state smaller than the input/output, so that the network must compress it

+ very simple to implement

- simply reducing dimensionality often does not provide the structure we want

2. Sparsity: force the hidden state to be sparse (most entries are 0), so that the network must compress input

+ principled approach that can provide a “disentangled” representation

  • harder in practice, requires choosing the regularizer and adjusting hyperparameters

3. Denoising: corrupt the input with noise, forcing the autoencoder to learn to distinguish noise from signal

+ very simple to implement

  • not clear which layer to choose for the bottleneck, adhoc choicers (e.g., how much noise to add) �4. Probabilistic modeling: force the hidden state to agree with a prior distribution

27 of 78

AE for generation

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

Decoder

  • Train AE with the pictures…
  • The decoder can now be used to generate instances from the “faces” manifold

28 of 78

The problem with AEs

  • Improper choice of input to the decoder can result in incorrect generation
  • How do we know what inputs are reasonable for the decoder?
  • Solution : only choose input (z’s) that are typical of the class
    • I.e. drawn from the distribution of z’s for faces
    • But what is this distribution?

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

29 of 78

Variational Autoencoders

Probabilistic spin on autoencoders - will let us sample from the model to generate data!

Image Credit: https://jaan.io/what-is-variational-autoencoder-vae-tutorial/

 

 

30 of 78

Impose distribution on z

  •  

20-10-2022

DECODER

Encoder output must be Gaussian

ENCODER

31 of 78

Generation

To generate novel values, sample z from prescribed distribution N(0,1)

20-10-2022

DECODER

32 of 78

How to train the model

Problem: How does one train an AE to ensure that the hidden representation z has a specific distribution e.g., N(0,1)

​

20-10-2022

Encoder output must be Gaussian

DECODER

ENCODER

33 of 78

How to train the model

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

Encoder

Decoder

34 of 78

Training with statistical constraints

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

Encoder

Decoder

35 of 78

Training with statistical constraints

  •  

Encoder

Decoder

This formulation does not adequately capture the variation in the data

36 of 78

The actual model

  • The “decoder” is actually a “generative” model for the data
  • A K-dimensional vector is drawn from a standard K-dimensional Gaussian and passed through the decoder
    • This results in data lying on a K-dimensional non-linear surface in the data space
  • Then a full-rank, low-amplitude noise is added to it, to generate the final data

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

37 of 78

The distribution of the data

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

38 of 78

The distribution of the data

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

39 of 78

The distribution of the data

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

40 of 78

The distribution of the data

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

41 of 78

The distribution of the data

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

42 of 78

The variational autoencoder

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

Encoder

Decoder

43 of 78

The variational autoencoder

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

Encoder

Decoder

44 of 78

The variational autoencoder

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

Encoder

Decoder

45 of 78

The variational autoencoder

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

Encoder

Decoder

46 of 78

The variational autoencoder

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

Encoder

Decoder

47 of 78

 

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

 

48 of 78

The variational autoencoder

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

Decoder

encoder

49 of 78

Training the encoder

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

50 of 78

Training the encoder

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

2

2

51 of 78

Training the encoder

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

2

2

52 of 78

Training the encoder

20-10-2022

2

2

53 of 78

 

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

54 of 78

VAE Computation Graph

 

 

 

Problem: Cannot backpropagate gradients through sampling layers

55 of 78

Reparametrizing the sampling layer

  •  

55

20-Oct-22

56 of 78

VAE network with and without the “reparameterization” trick

56

20-Oct-22

 𝜙 representations the distribution the network is trying to learn.

The epsilon remains as a random variable (sampled from a standard normal distribution) with a very low value thereby not causing the network to shift away too much from the true distribution. 

The distribution 𝜙 (parameterized by the mean and log-variance vectors) is still being learned by the network. This idea actually allowed a VAE to train in an end-to-end manner 

57 of 78

Training the encoder

58 of 78

The variational autoencoder

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

Decoder

encoder

59 of 78

Training the decoder

20-10-2022

60 of 78

The variational autoencoder

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

Decoder

encoder

61 of 78

The constraint on P(z)

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

62 of 78

The constraint on P(z)

20-10-2022

63 of 78

The complete training pipeline

64 of 78

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

65 of 78

The VAE for generation

  • Once trained the encoder can be discarded
  • The rest of the network gives us a generative model for x
  • Generating data using this part of the model should (ideally) give us data similar to the training data

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

66 of 78

VAE Takeaways

  • An autoencoder with statistical constraints on the hidden representation
    • The encoder is a statistical model that computes the parameters of a Gaussian
    • The decoder converts samples from the Gaussian back to the input
  • The decoder is a generative model that, when excited by standard Gaussian inputs, generates samples similar to the training data

​

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

Sample z

67 of 78

The Variational AutoEncoder

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

68 of 78

VAE and latent spaces

  •  

20-10-2022

69 of 78

VAEs : Latent perturbation

69

20-Oct-22

70 of 78

VAEs : Latent perturbation

70

20-Oct-22

71 of 78

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

72 of 78

Variational Autoencoders

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

73 of 78

Variational Autoencoders�Latent Variables

  • Independence of z dimensions makes it easy to generate instances wrt complex distributions via decoder g
  • Latent variables can be thought of as values of attributes describing inputs
    • E.g., for MNIST, latent variables might represent “thickness”, “slant”, “loop closure”

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

74 of 78

Variational �Autoencoders�Architecture

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

75 of 78

Variational Autoencoders�Optimization

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

76 of 78

Variational Autoencoders�Reparameterization Trick

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

77 of 78

Variational Autoencoders�Example Generated Images: Random

  •  

Sudeshna Sarkar, IIT Kharagpur

20-10-2022

78 of 78

Variational Autoencoders�Example Generated Images: Manifold

  • Uniformly sample points in (2-dimensional) z space and decode

Sudeshna Sarkar, IIT Kharagpur

20-10-2022