1 of 39

ME 5990�Machine Learning for ME

�

Multivariate Gaussian Distribution

2 of 39

Outline

  • Review of Matrix
  • Multivariate Gaussian Distribution
    • Mean Vector, Covariance Matrix
    • Probability Density Function
  • Classification using multivariate Gaussian distribution

​

3 of 39

A Matrix

  • In mathematics, a matrix (plural matrices) is a rectangular array or table of numbers, symbols, or expressions, arranged in rows and columns (from Wiki)

​

​

​

​

  • There are multi-layer understanding and application of matrix in linear algebra

4 of 39

Matrix Multiplication

  •  

5 of 39

Square

  • A square is a matrix with same number of rows and column
  • Identity matrix is an example of square

6 of 39

Determinant of Square

  •  

7 of 39

Determinant

  •  

8 of 39

Inverse of a matrix

  •  

9 of 39

Inverse of a matrix

  •  

10 of 39

Outline

  • Review of Matrix
  • Multivariate Gaussian Distribution
    • Mean Vector, Covariance Matrix
    • Probability Density Function
  • Classification using multivariate Gaussian distribution

​

11 of 39

Gaussian Distribution

  •  

12 of 39

Gaussian Distribution in 2D

  •  

 

 

​

Shot 1

Shot 2

Shot 3

Dimension 1

1

-2

-2

Dimension 2

1

2

0

13 of 39

Gaussian Distribution in 2D

  •  

 

 

14 of 39

Gaussian distribution in 2D

  •  

 

 

15 of 39

Covariance Matrix

  •  

Variance

Covariance

16 of 39

Gaussian distribution in 2D

  •  

17 of 39

Gaussian distribution in N-D

  •  

18 of 39

Gaussian distribution in N-D

  •  

19 of 39

Gaussian Distribution

​

​

  • Variance: quantify how spread the data for each dimension
  • Covariance: quantify how data variance on 2 dimensions
  • Covariance is the area between an observation and the mean vector

 

 

Covariance area for (-2, 2)

​

Covariance area for (-2, 0)

20 of 39

Covariance visualization

  • Multi-variate gaussian distribution simulation (simulation code)

Image: https://builtin.com/data-science/covariance-matrix

​

21 of 39

Covariance visualization

  • Multi-variate gaussian distribution simulation

22 of 39

Outline

  • Review of Matrix
  • Multivariate Gaussian Distribution
    • Mean Vector, Covariance Matrix
    • Probability Density Function
  • Classification using multivariate Gaussian distribution

​

23 of 39

2D Gaussian distribution

  • Coordinate on x and y direction, both follow Gaussian distribution

 

 

24 of 39

Gaussian distribution density

  •  

25 of 39

Multivariate Gaussian Distribution Density

  •  

26 of 39

Density

  •  

27 of 39

Density

  •  

28 of 39

Irrelevant: Terrain map

From maps.google.com

29 of 39

Outline

  • Review of Matrix
  • Multivariate Gaussian Distribution
    • Mean Vector, Covariance Matrix
    • Probability Density Function
  • Classification using multivariate Gaussian distribution

​

30 of 39

Classification Problem

  • We can assume 1-dimension data follows Gaussian distribution for each category.
  • E.g. we can do probabilistic reasoning (classification/decision) based on posterior of height
  • But we also know weight and foot-size, how can we use it?

​

31 of 39

Review of Bayes Theorem

  •  

32 of 39

Multi-variate Gaussian Bayes classifier

  • Male or Female
    • If we consider everything
    • Assume multi-variate Gaussian distribution
    • If someone is 5.6 feet, 120 lb and 7 size foot, can we decide this sample is from a male or female?
    • Can we draw decision boundary?

​

33 of 39

Likelihood

  •  

34 of 39

Multi-variate Gaussian Bayes classifier

  •  

35 of 39

Multi-variate Gaussian Bayes classifier

  •  

36 of 39

Multi-variate Gaussian Bayes classifier

  •  

37 of 39

Posterior

  •  

38 of 39

Decision Boundary

  • We can visualize the Decision Boundary for 1D and 2D
  • Hard to visualize 3D or above

39 of 39

Summary

  • Likelihood of multi-dimensional data can be modeled by multi-variate Gaussian distribution
  • The covariance matrix is symmetric; the main diagonal values are all positive; the other values can be negative
  • We can apply multi-variate Gaussian model to achieve a Bayes classifier. Each category shall have its own mean vector and covariance matrix

​