Generative Modeling (1)
Atılım Güneş Baydin
gunes@robots.ox.ac.uk
based on slides by
Tom Rainforth (HT2020)
Outline
Outline
3
This lecture covers
Based on slides by Tom Rainforth https://www.robots.ox.ac.uk/~twgr/
for ATML HT2020
Why take a Bayesian approach?
Motivation: why take a Bayesian approach?
5
Motivation: why take a Bayesian approach?
6
Epidemiology (Wood et al. 2020)
Particle physics (Baydin et al. 2019)
Spacecraft collision avoidance (Acciarini et al. 2021)
Motivation: why take a Bayesian approach?
7
The Bayesian paradigm
Bayesian probability is all about belief
9
Bayesianism vs frequentism
10
Bayesianism vs frequentism
11
Bayesianism vs frequentism
12
Interesting discussions about this:
Basic laws of probability
13
Using the Bayes Rule
14
Thomas Bayes
(1701 – 1761)
Evidence
(Marginal likelihood)
Likelihood
Prior
Posterior
Multiple observations: using the posterior as prior
15
Example: positive cancer test
16
Example: positive cancer test
17
Example: positive cancer test
18
Example: positive cancer test
19
The reason: “base rate neglect”
https://en.wikipedia.org/wiki/Base_rate_fallacy
More examples by Julia Galef
(highly recommended!)
The generative viewpoint
20
The Bayesian pipeline
21
Making predictions
22
Making predictions
23
Generative models
25
What is a model?
26
Example model: poker players reasoning about each other
27
Example model: poker players reasoning about each other
Check out N. D. Goodman, J. B. Tenenbaum, et al. (2016). Probabilistic Models of Cognition https://probmods.org/
28
Example: scientific simulators
Climate science
Cosmology
Weather
Drug discovery
Nuclear physics
Material design
Particle physics
29
What is a Bayesian model?
Thomas Bayes
(1701 – 1761)
How might we write a system to break captchas?
30
Example Bayesian model: captcha simulator
31
Example Bayesian model: captcha simulator
Rendered image
Correct letters
32
p( | Letters )
Correct letters
Example Bayesian model: captcha simulator
p( | Observed image)
Can be solved if we have a generative model (simulator) of p(letters, image)
Rendered image
Example Bayesian model: captcha simulator
33
Le, Baydin, Wood, “Inference compilation and universal probabilistic programming”, AISTATS 2017
34
Example Bayesian model: neuron growth
35
Example Bayesian model: Gaussian mixture model
36
Example Bayesian model: Gaussian mixture model
37
A fundamental assumption
“All models are wrong,
but some are useful”
George Box
(1919 – 2013)
39
What is the purpose of a model?
40
Some models can be better than others
41
Some models can be better than others
42
Some models can be better than others
Bayesian modeling through the eyes of multiple hypotheses
44
Bayesian modeling as multiple hypotheses
45
Example: density estimation
46
Example: density estimation
47
Example: density estimation
48
Example: density estimation
49
Example: density estimation
50
Bayesian modeling allows us to express our prior beliefs
51
Posterior predictive averages over hypotheses
52
Posterior predictive averages over hypotheses
53
Posterior predictive averages over hypotheses
54
Posterior predictive averages over hypotheses
55
An important subtlety
Model comparison:
what makes a good model?
57
Example: polynomial regression
Linear regression probably isn’t the best choice
58
Example: polynomial regression
59
Example polynomials for each degree
60
Least squares estimate for each degree
61
Perfectly matching the data is not enough
62
Perfectly matching the data is not enough
−
63
What went wrong?
64
Marginal likelihoods
65
Marginal likelihoods
66
Marginal likelihoods for polynomial regression
67
Bayesian Occam’s razor
William of Ockham
(1285 – 1347)
68
Bayesian Occam’s razor
69
Bayesian Occam’s razor
Summary
70
This lecture covered
Next lecture
Bayesian model averaging
72
Bayesian model averaging
73
Bayesian model averaging is not model combination
74
Further reading