Bayesian Machine Learning

Bayesian Decision Theory

2

Linear Classification

• When we first studied classification, we began by assuming that the decision boundary is linear
• This assumption served as a starting point, allowing us to construct simple and interpretable models such as SVM and logistic regression

​

• However, at that stage, the linearity of the boundary was merely a modeling choice, not something rigorously derived from the underlying probabilistic structure of the data

​

• We will mathematically investigate when and why this assumption is valid

​

3

Binary Classification with Gaussian

4

Optimal Boundary for Classes

5

Minimum Error Rate Classification

• Minimize

​

​

​

• We take derivatives

6

Posterior Probabilities

7

Boundaries for Gaussian

8

Equal Covariance

9

Not Equal Covariance

10

Examples of Gaussian Decision Regions

• When the covariances are all equal, the separating surfaces are hyperplanes

​

• When the covariances are not equal, the separating surfaces are quadratic functions

​

​

11

Python Implementation in 1D

12

Bayesian Classifier for Four Scenarios

• Equal variance and equal prior

​

• Equal variance and not equal prior

​

• Not equal variance and equal prior

​

• Not equal variance and not equal prior

​

​

​

​

​

13

Case 1: Equal Variance and Equal Prior

14

The optimal decision boundary lies at the midpoint between the two means

Case 2: Equal Variance and Not Equal Prior

15

It shifts toward the less probable class (class 0), because we require more evidence to classify a sample as belonging to the less likely class

Case 1: Equal Variance and Equal Prior

16

Case 3: Not Equal Variance and Equal Prior

17

The classification boundary becomes quadratic in this case. In one dimension, this implies that the decision rule involves two thresholds

Back to Logistic Regression

18

Probability Density Estimation

19

Probability Density Estimation

• Probability Density Estimation refers to the task of estimating an unknown probability distribution from observed data.
• Instead of producing a single estimate (e.g., a mean), the goal is to recover the entire probability density function (pdf), allowing for a more complete representation of uncertainty

​

• There are two major approaches:
• Kernel Density Estimation
• Bayesian Density Estimation

​

20

Kernel Density Estimation

21

Kernel Density Estimation

22

Bayesian Density Estimation

23

Hidden State

24

Likelihood

25

Posterior

26

Combining Multiple Evidences

• Compute posterior probability
• Assume conditional independence

27

Recursive Bayesian Estimation

28

Recursive Bayesian Estimation

• Let us define the sequence of posterior distributions:

​

​

​

​

​

​

​

​

​

• Thus, each posterior can be recursively computed as:

29

Recursive Bayesian Estimation

• Recursive

​

​

​

• Core idea of recursive Bayesian estimation: 
• "the current posterior becomes the prior for the next update."

​

• Recursive Bayesian Estimation

30

Recursive Bayesian Estimation

31

Example 1: Bernoulli Model

32

Bernoulli Model

33

Recursive Bayesian Estimation

34

Recursive Bayesian Estimation

35

Example 2: Gaussian Model

36

Posterior Probability

37

Recursive Bayesian Estimation

38

Recursive Bayesian Estimation

39

Summary

• Bayesian Machine Learning

​

• Bayesian Classifier

​

• Bayesian Density Estimation

​

• Bayes’ Rule
• Prior
• Likelihood
• Posterior

40

Object Tracking in Computer Vision