Parameter Estimation

Probability Density Estimation

2

Probability Density Estimation

3

Drawn from a Gaussian Distribution

4

Drawn from a Gaussian Distribution

• You will often see the following derivation

​

​

​

​

​

​

​

​

​

​

• Log-likelihood

5

Drawn from a Gaussian Distribution

6

Maximum Likelihood Estimation (MLE)

7

Maximum Likelihood Estimation

8

Numerical Example for Gaussian

9

When Mean is Unknown

10

When Variance is Unknown

11

Data Fusion

12

Data Fusion

• In many real-world systems, measurements are collected from multiple sensors, each of which may be noisy or imprecise.

​

• An important question is: 
• How can we combine noisy measurements from multiple sensors to produce a more accurate estimate?

13

Data Fusion with Uncertainties

• Two noisy sensors

​

​

​

​

​

​

• In a matrix form 

14

Learning Theory (Prof. Reza Shadmehr, Johns Hopkins University)

http://courses.shadmehrlab.org/learningtheory.html

Data Fusion with Uncertainties

15

Analytical Evaluation

16

Variance of the Estimate

• This inequality reflects the benefit of combining information: the estimate is more precise than either of the individual sensors.

17

Data Fusion with Less Uncertainties

• Summary �

​

​

​

​

• Big lesson:
• Two sensors are better than one sensor ⟹ less uncertainties
• Accuracy or uncertainty information is also important in sensors

18

(weighted average)

Example: Two Rulers (1D Sensor Fusion in the Brain)

• Estimate the length of an object using two different sources of sensory information:
• Visual measurement: looking at the object
• Haptic measurement: feeling the object with eyes closed

​

​

• How brain works on human measurements from both haptic and visual channels

19

Example: Two Rulers (1D Sensor Fusion in the Brain)

​

​

​

​

​

​

​

​

​

​

• (Neuroscience Perspective) This model reflects how the human brain integrates sensory input:
• It doesn't rely solely on one modality
• It combines information in a statistically optimal way (approximately)
• Uncertainty matters: The brain gives more weight to more reliable sources

​

20

Data Fusion with 2D Example

21

Maximum a Posteriori (MAP) Estimation

22

Think Differently

23

Key Perspective

24

Maximum-a-Posteriori Estimation (MAP)

25

Maximum-a-Posteriori Estimation (MAP)

26

Maximum-a-Posteriori Estimation (MAP)

27

MAP for a Univariate Gaussian

28

MAP for a Univariate Gaussian

29

MAP for a Univariate Gaussian

30

MAP for a Univariate Gaussian

• MLE interpretation: 

​

​

​

​

• Big lesson: The prior acts like a virtual data point with unit variance

​

​

​

​

​

• Prior knowledge can come from various sources such as:
• Education: e.g., learning that volume is not always proportional to weight
• Experience: accumulated through interactions over time
• Aging: as people grow older, they refine priors based on repeated sensory outcomes

​

31

Example: Perceiving Object Weight via Multiple Senses

• You're presented with two objects (as in the image) and asked: "Which one is heavier?"
• We consider four different sensory conditions:
• Without Touching and with Eyes Closed
• Only Visual Inspection
• Only Haptic (Touch) Inspection
• Both Visual and Haptic Inspection

32

Example: Perceiving Object Weight via Multiple Senses

• You're presented with two objects (as in the image) and asked: "Which one is heavier?"
• We consider four different sensory conditions:
• Without Touching and with Eyes Closed
• Only Visual Inspection
• Only Haptic (Touch) Inspection
• Both Visual and Haptic Inspection

33

Example: Perceiving Object Weight via Multiple Senses

• Without Touching and with Eyes Closed
• No sensory input (neither visual nor haptic)
• You have to guess purely based on prior knowledge or random chance
• Only Visual Inspection
• You see the objects but do not touch them
• You likely assume the larger object is heavier
• This is guided by a prior based on daily experience (volume ∝ mass)
• Outcome: You likely say "the left one is heavier"
• Only Haptic (Touch) Inspection
• Eyes are closed, but you feel the objects
• No visual bias is introduced
• This engages the haptic sensory system, which may have higher precision for weight
• Outcome: You may correctly identify which is heavier if your sense of touch is reliable
• Both Visual and Haptic Inspection
• You integrate visual and haptic signals
• The brain uses a Bayesian strategy to weight each signal inversely to its uncertainty

​

34

MAP in Python

35

MAP in Python

36

MAP in Python

37

MAP in Python

38

Linear Measurement with Noise

39

Linear Measurement

40

Linear Measurement

41

42

Linear Measurement

43

Recursive Bayesian Estimation �from Two Sensors

44

Re-visit Two Sensors Problem

45

Different Perspective: Recursive Bayesian Update

46

Linear Measurement

47

Bayesian Kalman Filter

48

(1) Prediction Step (Prior Update)

49

(2) Correction Step (Posterior Update)

50

Balances trust between the model prediction and the new data

Covariance decreases over time as more information is accumulated

Object Tracking in Computer Vision

• Lecture: Introduction to Computer Vision by Prof. Aaron Bobick at Georgia Tech

​

51

Probabilistic Machine Learning

• Maximum Likelihood Estimation (MLE)
• Maximum a Posteriori Estimation (MAP)

​

• Probabilistic Machine Learning
• I personally believe this is a more fundamental way of looking at machine learning

​

• Probabilistic Regression
• Probabilistic Classification
• Probabilistic Clustering
• Probabilistic Dimension Reduction

52