The Six Families of�Deep Generative Models

Yannis Pantazis

Journal Club on AI @ FORTH

​

Friday, October 18^th

Logistics

• Biweekly meetings – Friday @ 14:00
• Location: Vassilis Dougalis Room
• JC list: jc-list@iacm.forth.gr
• Email Panos Evaggelidakis for subscribing το the JC list
• --> panosevangelidakis@gmail.com

November’s Presentations

Introduction – What is a Generative Model?

What is required:

• Families of Generative Models
• Algorithms to train these GMs
• Neural network architectures
• Loss functions & distances between probability density functions

Families of Generative Models – Taxonomy based on Likelihood Function

AutoRegressive Models (ARMs)

AutoRegressive Models (ARMs)

• Softmax when discrete

• Gaussian when continuous

Empirical observation:

• Neural networks perform better when the output is discrete.
• Thus, even when data are continuous, they are often quantized and treated as discrete variables.
• Current trend: Tokenize everything

AutoRegressive Models (ARMs)

• Dilated convolutions

• Recurrent Architectures

• Transformers (decoder only)

AutoRegressive Models (ARMs)

1542M

762M

345M

117M parameters

GPT released June 2018

GPT-2 released Nov. 2019 with 1.5B parameters

GPT-3: 175B parameters trained on 45TB texts

Normalizing Flows (NFs)

Many small steps adds up

to big results.

Normalizing Flows (NFs)

Normalizing Flows (NFs)

Normalizing Flows (NFs) – RealNVP 2016

Variational Autoencoders (VAEs) - Motivation

Variational Autoencoders (VAEs) - Motivation

Variational Autoencoders (VAEs)

z

Variational Autoencoders (VAEs)

Variational Autoencoders (VAEs)

Variational Autoencoders (VAEs)

Variational Autoencoders (VAEs)

Variational Autoencoders (VAEs)

• Training VAEs requires:
• Approximate the model evidence with a lower bound called ELBO (from Evidence Lower BOund) and maximize ELBO instead of the evidence.
• Reparametrization trick for efficient gradient estimation.

​

​

• Typically, the latent variable is continuous.�However, there are extensions to discrete latent variables (Vector Quantized VAE – VQ-VAE).

​

​

• Latent variables can be disentangled (β-VAE and InfoVAE)

Energy-based Models (EBMs)

Energy-based Models (EBMs)

• The first family of Deep Generative Models!
• Inspired by Statistical Physics and Boltzmann distribution

​

• Interesting algorithms for training have been proposed

​

• Contrastive divergence

​

• Score function

​

• Noise contrastive estimation

Energy-based Models (EBMs) – Product of Experts

Generative Adversarial Networks (GANs)

• Main Idea: Instead of using KLD minimization (ie, log-likelihood maximization) use a different “distance” to minimize.

Generative Adversarial Networks (GANs)

Generative Adversarial Networks (GANs)

Generative Adversarial Networks (GANs)

Generative Adversarial Networks (GANs)

Generative Adversarial Networks (GANs) – Cycle GAN 2017 – Unpaired Matching

Generative Adversarial Networks (GANs) – Cycle GAN 2017 – Unpaired Matching

Diffusion Probabilistic Models (DPMs)

Diffusion Probabilistic Models (DPMs)

Diffusion Probabilistic Models (DPMs)

Diffusion Probabilistic Models (DPMs)

Diffusion Probabilistic Models (DPMs)

Simplified training objective:

Diffusion Probabilistic Models (DPMs)

Diffusion Probabilistic Models (DPMs)

Palette: Image-to-Image Translation

Challenges in Generative Models

Challenges in Generative Models

• How to evaluate their performance?
• Likelihood-based
• Inception score for images
• But nothing conclusive and rigorous

​

• Avoid “spreading mis-information”
• On the usage of generative models - Ethics
• On their reliability

Reading on Generative Models

Introduction to Deep GMs Course

https://www.csd.uoc.gr/~hy673/