Special Course in Machine Learning:

CS236 Deep Generative Models

Mikhail Papkov

11.02.2020

Slides available here: eid.ee/4yo

Agenda

• Course organization
• Introducing ourselves
• Short discussion of Lecture 1 (Introduction and Background)
• Kahoot quiz

Logistics

Delta-1022, Tuesday, 16:15 - 17:45 (we can stay longer or leave earlier)

Do we need to discuss changing time and place?

(Wednesday or Thursday evening are also fine)

Prerequisites

• Probability theory, linear algebra, calculus
• Python (we’ll most probably use NumPy and PyTorch)
• MTAT.03.227 Machine Learning (lectures)

Resources

1. courses.cs.ut.ee/2020/scml/spring
2. piazza.com/ut.ee/spring2020/mtat03317 (invitations were sent)
3. deepgenerativemodels.github.io
4. deeplearningbook.org
1. Chapter 3. Probability and Information Theory
2. Chapter 20. Deep Generative Models
3. Other chapters that you find useful
5. Course videos (I hope so, sent an email to TA; meanwhile YouTube)

Organization

This is a reverse classroom seminar! I am also a student and present only today

1. Presentation based on lecture material (see schedule) — 45 minutes
• Preferably not only using published slides
• You are welcome to prepare a lecture based on related external source (YouTube lecture)
• I can help you to prepare (or at least try to)
2. Solve and discuss HW task — 30 minutes
• Solving every HW individually is not required (simply does not fit in 3 ECTS) but encouraged
3. Quiz — 10 minutes
• We’ll try Kahoot today and see how it goes
• You can prepare your own homework instead (possibly based on CS236 HW) with deadline next week
4. Discussion — some more minutes

How to pass?

1. Prepare and conduct one seminar: lecture, HW, quiz (sign up here)
2. Attend at least 9 times = solve 9 quizzes (60%)
3. Read/watch material at home, participate in the discussion

Do you need more motivation?

Questions?

About me

• Mikhail, 1st year PhD student in Health Image Analysis
• Research in image quality enhancement (denoising, super-resolution)
• Want to try applying GANs for cell microscopy images
• Data generation
• Domain adaptation

About you

(this would probably help us to define directions + I’ll try to remember you)

• What do you do?
• How could you apply generative models in your work?

(it’s fine if you do not yet have a clear perspective or take this course for fun!)

Background

Summary

1. Tasks for generative models
2. What is a distribution?
3. Conditional independence
4. Chain rule and Bayes rule
5. Bayes network
6. Naive Bayes (feature independence)
7. Logistic regression
8. Neural networks (more parameters)

Data

Hereinafter, CS236 materials were used (by Stefano Ermon and Aditya Grover, MIT License)

Generation

Tasks

Distribution

You kick the black box, it gives you some value according to certain rules:

• Bernoulli (coin flip), Gaussian, uniform etc

Can be joint (RGB)

Are all the pixels (parameters) independent? We can assume so, but most likely not

Conditional independence

Rules

Bayes rule terminology

P(A) — prior

P(A|B) — posterior

P(B|A) — likelihood

Bayes network

We can assume that some events are conditionally independent

Figure from Meelis Kull: Fall 2019 MTAT.03.227 Machine Learning Lecture 11, slide 32

Naive Bayes

Assume features to be conditionally independent given label

Generative vs Discriminative

Naive Bayes is a generative model (although we use it for classification)

Logistic regression (discriminative)

Parametrized with α

Logistic regression (discriminative)

Parametrized with α

From logistic regression to neural network

Why using generative models?

Kahoot!