Statistics for Machine Learning

Statistics and Probability

• Probability
• Probability deals with predicting the likelihood of future outcomes based on known conditions or assumptions
• Focus: Probability starts with a known model (e.g., theoretical distribution) and aims to predict possible outcomes.

​

​

​

​

​

​

​

• Statistics
• Statistics involves analyzing existing data to infer properties of the population or predict future outcomes
• Focus: Statistics starts with observed data and attempts to build models or draw conclusions about the underlying distribution.

2

Populations and Samples

3

Multivariate Statistics

4

Correlation of Two Random Variables

5

Correlation Coefficient

​

​

​

​

• Interpretation

​

​

​

​

• Indicate how close to a linear line, but
• No information on slope

​

• Correlation does not imply causation
• Even if two variables are strongly correlated, it doesn't necessarily mean one causes the other

6

Correlation Coefficient Plot

• Plots correlation coefficients among pairs of variables
• http://rpsychologist.com/d3/correlation/

​

7

Unsupervised Learning:�Dimension Reduction

Dimension

9

Dynamics/Robot kinematics

- Degree of freedom

- Generalized coordinate

Dimension Reduction

• Motivation:
• Can we describe high-dimensional data in a “simpler” way? 

​

• Dimension reduction without losing too much information

​

• Find a low-dimensional, yet useful representation of the data

10

Dimension Reduction

11

Example of Dimension Reduction

​

​

​

​

​

​

​

• Change variables

​

• Total amount of water

​

• Difference in height

​

​

​

​

​

​

​

​

​

​

​

​

​

• De-coupled

​

​

​

​

• Physical insights
• Data/statistics

12

Example of Dimension Reduction

13

Highly Correlated Data

• Idea: highly correlated data contains redundant features

14

Principal Component Analysis (PCA)

15

Principal Component Analysis (PCA)

16

Principal Component Analysis (PCA)

17

Principal Component Analysis (PCA)

18

Principal Component Analysis (PCA)

19

Principal Component Analysis (PCA)

20

Principal Component Analysis (PCA)

21

Principal Component Analysis (PCA)

22

Principal Component Analysis (PCA)

23

Principal Component Analysis (PCA)

​

​

​

​

​

​

​

​

​

​

• HOW?
• 1. Maximize variance (most separable)
• 2. Minimize the sum-of-squares (minimum squared error)

​

24

PCA Algorithm: Pre-processing

• Given data

​

​

​

​

• Shifting (zero mean) and rescaling (unit variance)
• 1) Shift to zero mean

​

​

​

• 2) [optional] Rescaling (unit variance)

​

25

PCA Algorithm: Maximize Variance

26

Maximize Variance

• In an optimization form

​

27

Minimize the Sum-of-Squared Error

28

Minimize the Sum-of-Squared Error

• In an optimization form

29

30

Linear Regression vs. PCA

31

Linear Regression

PCA

Linear Regression vs. PCA

32

Linear Regression

PCA

Linear Regression vs. PCA

33

Python Codes

34

Python Codes

35

Scikit-learn

36

PCA Example

• Multiple video camera records of spring and mass system
• Optimal data representation
• Find the most informative point of view

​

​

​

​

​

​

​

​

​

• source:
• https://www.cs.princeton.edu/picasso/mats/PCA-Tutorial-Intuition_jp.pdf

​

37

Multivariate Time Series

• System order can be inferred from
• Laws of physics or
• Data

​

38

Measured observations

Multivariate Time Series

39

Eigenvalues

40

Projection onto Principal Components

41

Singular Value Decomposition (SVD)

Geometry of Linear Maps

43

Geometry of Linear Maps

• An extremely important fact:

44

Singular Values and Singular Vectors

45

Thin Singular Value Decomposition

46

Full Singular Value Decomposition

47

Interpretation of SVD

48

SVD: Matrix factorization

49

PCA and SVD

50

PCA and SVD

51

Low Rank Approximation: Dimension Reduction

52

Full SVD

53

Economy SVD

54

Truncated SVD (Low Rank Approximation)

55

Expansion of SVD

56

Expansion of SVD

57

Expansion of SVD

58

Truncated SVD (Low Rank Approximation)

59

…

+

+

+

Example: Image Approximation

60

Example: Image Approximation

61

A Video of People Walking, Captured by a Surveillance Camera

• A set of sequential frames taken from a video of people walking

62

To Get Rid of People in the Pictures

• A set of sequential 20 pictures (or frames) taken from a video of people walking
• Only extract background from pictures

63

…

To Get Rid of People in the Pictures

• A set of sequential 20 pictures (or frames) taken from a video of people walking
• Only extract background from pictures

64

Question: How to Build a Matrix (Sequence of Images)

65

…

time

Question: How to Build a Matrix (Sequence of Images)

66

…

time

time

time

…

…

…

…

Eigenfaces: A SVD-Based Facial Recognition Technique

• Eigenfaces is a well-known technique in computer vision that applies PCA or SVD for facial recognition.

​

• The method is based on the idea that faces can be represented as combinations of key facial features, called eigenfaces.

​

• By leveraging SVD, Eigenfaces effectively reduces the dimensionality of face images while retaining the most important features for identification.

67

Eigenface

68

Eigenface

69

Eigenface or eigenmode from U

Data Compression

70

Face Recognition Using Eigenfaces

• The Eigenfaces method is a classical yet effective technique for face recognition

​

​

• Step 1: Project Image onto the Eigenface Subspace

​

​

• Step 2: Face Identification Using Distance Metrics

​

​

• Step 3: Classification Decision

71

Recognition and Reconstruction of Disguised Face

• Here, I present an interesting scenario: President John F. Kennedy is wearing sunglasses when he enters the White House

72

Recognition and Reconstruction of Disguised Face

73

Proper Orthogonal Decomposition (POD)

74

Proper Orthogonal Decomposition (POD)

75

Arrangement of Data

76

Full SVD

77

Economy SVD

78

Truncated SVD

79

Expansion of SVD

80

Expansion of SVD

81

POD

82

POD Example 1

• Consider data generated from the function

​

​

​

​

​

​

​

​

​

​

83

…

The Method of Snapshot, Sirovich (1987)

POD Example 1

• Consider data generated from the function

​

​

​

​

​

​

​

​

​

​

84

POD Example 1

• The leading two POD modes capture all of the energy/dynamics of the data,

​

85

POD Example 1

• The leading two POD modes capture all of the energy/dynamics of the data,

​

86

Fluid Flow Past a Circular Cylinder at Low Reynolds Number

87

Question: How to Build a Matrix (2D + time)

88

Singular Values and POD Modes

89

Low Rank Approximation (K = 30)

90

Ground Truth

K = 30

Low Rank Approximation (K = 10)

91

Ground Truth

K = 10

Low Rank Approximation (K = 5)

92

Ground Truth

K = 5

Where We Are At?

• Coordinate transformation
• With physical intuition
• Or data-driven approach

​

​

​

• Data-driven coordinate transformation
• Linear coordinate transformation

​

• Natural to think about nonlinear coordinate transformation
• Autoencoder

93

Idea from Prof. Steve Brunton, “Deep Learning to Discover Coordinates for Dynamics: Autoencoders & Physics Informed Machine Learning”

Heliocentrism

Geocentrism

Fisher Discriminant Analysis (FDA)

Dimensionality Reduction with Label

• Dimensionality reduction with label information (when the ultimate goal is classification/regression)
• PCA ignores label information even if it is available
• Only chooses directions of maximum variance

​

• Fisher Discriminant Analysis (FDA) takes into account the label information
• It is also called Linear Discriminant Analysis (LDA)

​

• FDA/LDA projects data while preserving class separation
• Examples from same class are put closely together by the projection
• Examples from different classes are placed far apart by the projection

​

95

Projection onto Line ω

96

Projection onto Line ω

97

Sample Statistics in Projected Space

98

Fisher Discriminant Analysis

99

Fisher Discriminant Analysis

100

Fisher Discriminant Analysis

101

Fisher Discriminant Analysis

• Why?
• Dot product of a unit vector and another vector is maximum when the two have the same direction.

​

102

Python Code

103

Histogram

104

105

106

Scikit-learn

107