1 of 151

1

Parameter Estimation

Chris Gregg

CS109, Stanford University

Autumn 2026

2 of 151

Where are we in CS109?

2

You are here

3 of 151

3

4 of 151

4

5 of 151

5

General “Inference”

6 of 151

6

Probabilistic Model

Fever

Tired

Flu

Undergrad

7 of 151

7

Probabilistic Model

Fever

Tired

Flu

Undergrad

If you know the probability of each random variables given the ones that directly cause it, you can joint sample!

8 of 151

8

Four Prototypical Trajectories

But where do those numbers come from?

9 of 151

9

Four Prototypical Trajectories

Suspense

10 of 151

10

Four Prototypical Trajectories

At this point, if you are given a model, with all the involved probabilities, you can make predictions

11 of 151

11

Four Prototypical Trajectories

But what if you want to learn the probabilities in the model?

12 of 151

12

Four Prototypical Trajectories

Machine Learning

13 of 151

13

AI and Machine Learning

ML: Rooted in probability theory

Artificial

Intelligence

Machine

Learning

Deep

Learning

Gen AI

14 of 151

14

Our Path

Parameter Estimation

Deep Learning

Core Algorithms

15 of 151

15

Our Path

Deep Learning

Core Algorithms

Unbiased

estimators

Maximizing

likelihood

16 of 151

16

17 of 151

17

Four Prototypical Trajectories

Review

18 of 151

Shorthand for Equality Events

18

Our shorthand notation

Is shorthand for the event

Is shorthand for the event

Full Notation

19 of 151

19

Four Prototypical Trajectories

End Review

20 of 151

20

21 of 151

21

Once upon a time…

22 of 151

22

…there was parameter estimation

23 of 151

23

What are Parameters?

24 of 151

  • Consider some probability distributions:
    • Ber(p)
    • Poi(λ)
    • Uni(⍺, β)
    • Normal(μ, σ2)
    • Y = mX + b
    • etc…
  • Call these “parametric models”
  • Given model, parameters yield actual distribution
    • Usually refer to parameters of distribution as θ
    • Note that θ that can be a vector of parameters

​

24

What are Parameters?

25 of 151

  • Consider some probability distributions:
    • Ber(p)
    • Poi(λ)
    • Uni(⍺, β)
    • Normal(μ, σ2)
    • Y = mX + b
    • etc…
  • Call these “parametric models”
  • Given model, parameters yield actual distribution
    • Usually refer to parameters of distribution as θ
    • Note that θ that can be a vector of parameters

​

25

What are Parameters?

26 of 151

  • Consider some probability distributions:
    • Ber(p)
    • Poi(λ)
    • Uni(⍺, β)
    • Normal(μ, σ2)
    • Y = mX + b
    • etc…
  • Call these “parametric models”
  • Given model, parameters yield actual distribution
    • Usually refer to parameters of distribution as θ
    • Note that θ that can be a vector of parameters

​

26

θ = p

θ = λ

θ = (α, β)

θ = (μ, σ2)

θ = (m, b)

​

What are Parameters?

27 of 151

27

Fever

Tired

Flu

Undergrad

Parameters

What are Parameters?

28 of 151

28

Why Do We Care?

Real World Problem

Formal Model 𝜃

Prediction

Function 𝜃*

Model the problem

Learning Algorithm

Testing

Data

Training Data

Evaluation

score

29 of 151

29

Modelling

Real World Problem

Formal Model 𝜃

Prediction

Function 𝜃*

Model the problem

Learning Algorithm

Testing

Data

Training Data

Evaluation

score

30 of 151

30

Real World Problem

Model the problem

Parameter Estimation (aka Training)

Formal Model 𝜃

Prediction

Function 𝜃*

Learning Algorithm

Testing

Data

Training Data

Evaluation

score

31 of 151

31

We have already seen some

parameter estimators!

32 of 151

32

9 Heads out of 10 Flips. What is your Belief in p?

33 of 151

33

Unbiased Estimators of Mean and Variance

34 of 151

  •  

34

 

Unbiased Estimators of Mean and Variance

35 of 151

  •  

35

 

 

Unbiased Estimators of Mean and Variance

36 of 151

  •  

36

 

 

Sample variance:

 

Unbiased Estimators of Mean and Variance

37 of 151

  •  

37

 

 

Sample variance:

 

Unbiased Estimators of Mean and Variance

38 of 151

38

Our Path

Deep Learning

Core Algorithms

Unbiased

estimators

Maximizing

likelihood

39 of 151

39

Unbiased Estimation is a limited tool:

how could we use that for fitting WebMD?

40 of 151

40

Great idea in Machine Learning

41 of 151

41

To the Course Reader!

42 of 151

42

We want to choose the parameter value that maximizes the probability of the data.

How to Choose the “Best” Parameters: MLE

43 of 151

43

We want to choose the parameter value that maximizes the probability of the data.

Maximum

Likelihood

Estimation!

How to Choose the “Best” Parameters: MLE

44 of 151

44

“I feel seen”

45 of 151

45

46 of 151

46

We want to choose the parameter value that maximizes the probability of the data.

Maximum

Likelihood

Estimation!

How do we quantify “probability of the data”?

How to Choose the “Best” Parameters: MLE

47 of 151

47

A generalized term for “PDF / PMF / Joint”

of data as a function of parameters

Wikipedia:

Likelihood Definition

48 of 151

48

θ is shorthand for parameter(s)

(if we have a Poisson, θ = λ)

Definition: The probability of our observed data if our parameters were θ.

The Likelihood Function

49 of 151

49

(in our example, this would just be the Poisson PMF)

If we had a single observation, X = x:

The Likelihood Function

Definition: The probability of our observed data if our parameters were θ.

50 of 151

50

For a list of observations, [x1, x2, …, xn]:

The Likelihood Function

Definition: The joint probability of our observed data if our parameters were θ.

​

51 of 151

51

For a list of observations, [x1, x2, …, xn]:

The Likelihood Function

Definition: The joint probability of our observed data if our parameters were θ.

​

We assume that data points are I.I.D.

52 of 151

52

For a list of observations, [x1, x2, …, xn]:

The Likelihood Function

Definition: The joint probability of our observed data if our parameters were θ.

​

We assume that data points are I.I.D.

53 of 151

53

For a list of observations, [x1, x2, …, xn]:

The Likelihood Function

Definition: The joint probability of our observed data if our parameters were θ.

​

We assume that data points are I.I.D.

We always use f for likelihood in MLE (even for discrete) 🙃

54 of 151

54

The Likelihood Function

n I.I.D. data points

We explicitly specify parameter θ of distribution

​

This is just a product since Xi are I.I.D.

55 of 151

55

Likelihood (of data given parameters):

Either the

PDF (continuous) or

PMF (discrete), or

joint if multiple variables per datapoint

56 of 151

56

We want to choose the parameter value that maximizes the probability of the data:

How to Choose the “Best” Parameters: MLE

Likelihood

57 of 151

57

We want to choose the parameter value that maximizes the probability of the data:

How to Choose the “Best” Parameters: MLE

Likelihood

To put words into math:

Our best estimate

58 of 151

58

We want to choose the parameter value that maximizes the probability of the data:

How to Choose the “Best” Parameters: MLE

Likelihood

To put words into math:

Our best estimate

Log Likelihood

59 of 151

59

Sidequest: What is argmax?

60 of 151

60

Argmax

61 of 151

61

Argmax

Maximum output

Maximizing input

62 of 151

62

63 of 151

63

 

64 of 151

64

But how do we compute argmax?

65 of 151

65

Option #1: Straight optimization

66 of 151

66

Finding the Argmax with Calculus

67 of 151

67

Differentiate w.r.t.�argmax’s argument

Finding the Argmax with Calculus

68 of 151

68

Differentiate w.r.t.�argmax’s argument

Finding the Argmax with Calculus

69 of 151

69

Differentiate w.r.t.�argmax’s argument

Finding the Argmax with Calculus

70 of 151

70

Differentiate w.r.t.�argmax’s argument

Set to 0 and solve

Finding the Argmax with Calculus

71 of 151

71

Differentiate w.r.t.�argmax’s argument

Set to 0 and solve

 

Finding the Argmax with Calculus

72 of 151

72

Differentiate w.r.t.�argmax’s argument

Set to 0 and solve

 

 

 

Finding the Argmax with Calculus

73 of 151

73

Argmax of Log

Claim:

Log is monotonic

x ≤ y ⇔ log(x) ≤ log(y) for all x, y > 0

74 of 151

74

Argmax of Log

75 of 151

75

Log I Love You

76 of 151

76

Natural Log

77 of 151

77

End Sidequest

78 of 151

78

Let’s go deeper!

79 of 151

79

MLE For Poisson

80 of 151

80

We observed the following samples:

[6, 1, 2, 1, 2, 3, 3, 2, 1, 3, 1, 3]

​

What is lambda, ?

λ

MLE For Poisson

81 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate λ
    • Probability mass function can be written as:

    • Likelihood:
    • Log-likelihood:

​

​

    • Differentiate w.r.t. λ, and set to 0:

81

1. What is the likelihood of one Xi

2. What is the likelihood of all the data

3. What is the log-likelihood all the data

4. Find the value of λ which maximizes log likelihood

MLE For Poisson

82 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate λ
    • Probability mass function can be written as:

    • Likelihood:
    • Log-likelihood:

​

​

    • Differentiate w.r.t. λ, and set to 0:

82

2. What is the likelihood of all the data

3. What is the log-likelihood all the data

4. Find the value of λ which maximizes log likelihood

MLE For Poisson

83 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate λ
    • Probability mass function can be written as:

    • Likelihood:
    • Log-likelihood:

​

​

    • Differentiate w.r.t. λ, and set to 0:

83

3. What is the log-likelihood all the data

4. Find the value of λ which maximizes log likelihood

MLE For Poisson

84 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate λ
    • Probability mass function can be written as:

    • Likelihood:
    • Log-likelihood:

​

​

    • Differentiate w.r.t. λ, and set to 0:

84

4. Find the value of λ which maximizes log likelihood

MLE For Poisson

85 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate λ
    • Probability mass function can be written as:

    • Likelihood:
    • Log-likelihood:

​

​

    • Differentiate w.r.t. λ, and set to 0:

85

4. Find the value of λ which maximizes log likelihood

MLE For Poisson

86 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate λ
    • Probability mass function can be written as:

    • Likelihood:
    • Log-likelihood:

​

​

    • Differentiate w.r.t. λ, and set to 0:

86

4. Find the value of λ which maximizes log likelihood

MLE For Poisson

87 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate λ
    • Probability mass function can be written as:

    • Likelihood:
    • Log-likelihood:

​

​

​

    • Differentiate w.r.t. λ, and set to 0:

87

MLE For Poisson

88 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate λ
    • Probability mass function can be written as:

    • Likelihood:
    • Log-likelihood:

​

​

​

    • Differentiate w.r.t. λ, and set to 0:

88

MLE For Poisson

89 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate λ
    • Probability mass function can be written as:

    • Likelihood:
    • Log-likelihood:

​

​

​

Differentiate w.r.t. λ, and set to 0:

89

MLE For Poisson

90 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate λ
    • Probability mass function can be written as:

    • Likelihood:
    • Log-likelihood:

​

​

​

    • Differentiate w.r.t. λ, and set to 0:

90

MLE For Poisson

91 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate λ
    • Probability mass function can be written as:

    • Likelihood:
    • Log-likelihood:

​

​

​

    • Differentiate w.r.t. λ, and set to 0:

91

MLE For Poisson

92 of 151

92

Isn’t that the same as

the sample mean?

93 of 151

93

Yes. For Poisson.

94 of 151

94

MLE For Poisson

95 of 151

95

MLE For Poisson

96 of 151

96

We know sand is distributed as a pareto with PDF

MLE For Pareto

97 of 151

97

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Xi ~ Pareto(α). Use Maximum Likelihood to estimate α.
    • Likelihood:

​

​

    • Log-likelihood:

​

​

    • Chose α to be the argmax of LL:

MLE for a Pareto

3. Find the value of α which maximizes log likelihood

1. What is the likelihood of all the data

2. What is the log-likelihood all the data

98 of 151

98

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Xi ~ Pareto(α). Use Maximum Likelihood to estimate α.
    • Likelihood:

​

​

    • Log-likelihood:

​

​

    • Chose α to be the argmax of LL:

MLE for a Pareto

3. Find the value of α which maximizes log likelihood

2. What is the log-likelihood all the data

99 of 151

99

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Xi ~ Pareto(α). Use Maximum Likelihood to estimate α.
    • Likelihood:

​

​

    • Log-likelihood:

​

​

    • Chose α to be the argmax of LL:

MLE for a Pareto

3. Find the value of α which maximizes log likelihood

100 of 151

100

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Xi ~ Pareto(α). Use Maximum Likelihood to estimate α.
    • Likelihood:

​

​

    • Log-likelihood:

​

​

    • Chose α to be the argmax of LL, set equal to 0, then solve for : :

MLE for a Pareto

101 of 151

101

Time to finish Medical Diagnosis in Seconds

MLE for Erlang

102 of 151

102

MLE for Erlang

103 of 151

103

 

104 of 151

104

Gradient Ascent

Walk uphill and you will find a local maxima

(if your step size is small enough)

105 of 151

105

Gradient Ascent

Motivation: We often want to choose parameters that maximize a function—for example, finding the probability of heads that makes observed coin flips most likely. What if we can’t solve for the best parameters directly?

Idea: Start with a guess, then repeatedly take a small step uphill.

Intuition: Like climbing a foggy hill, use the slope where you are to decide which way to move. You may reach a local peak rather than the highest peak.

106 of 151

106

Gradient Ascent

Motivation: We often want to choose parameters that maximize a function—for example, finding the probability of heads that makes observed coin flips most likely. What if we can’t solve for the best parameters directly?

Idea: Start with a guess, then repeatedly take a small step uphill.

107 of 151

107

Gradient Ascent

Walk uphill and you will find a local maxima

(if your step size is small enough)

Especially good if function is convex

108 of 151

108

Gradient Ascent

Repeat many times

Walk uphill and you will find a local maxima

(if your step size is small enough)

This is some profound life philosophy

Step size constant

109 of 151

Initialize: θj = random for all 0 ≤ j ≤ m

​

109

Gradient Ascent

Repeat many times:

Calculate all gradient[j]’s based on data

𝜃j += η * gradient[j] for all 0 ≤ j ≤ m

110 of 151

110

To the code!

111 of 151

111

Gradient Ascent for MLE of Erlang

112 of 151

112

Derived these partial derivatives of log likelihood

Gradient Ascent for MLE of Erlang

113 of 151

113

MLE For Bernoulli

114 of 151

114

Don’t we already have the Beta?

​

Yes! But this example is critical for developing towards deep learning.

115 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate p
    • Probability mass function can be written as:

​

​

115

MLE For Bernoulli

116 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate p
    • Probability mass function can be written as:

​

​

116

MLE For Bernoulli

117 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Xi ~ Ber(p)
    • Probability mass function,

117

Differentiable PMF for Bernoulli

118 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Xi ~ Ber(p)
    • Probability mass function,

118

PMF of Bernoulli

Differentiable PMF for Bernoulli

119 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Xi ~ Ber(p)
    • Probability mass function,

119

0

1

PMF of Bernoulli

Differentiable PMF for Bernoulli

120 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Xi ~ Ber(p)
    • Probability mass function,

120

0

1

p

PMF of Bernoulli

Differentiable PMF for Bernoulli

121 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Xi ~ Ber(p)
    • Probability mass function,

121

0

1

p

1 - p

PMF of Bernoulli

Differentiable PMF for Bernoulli

122 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Xi ~ Ber(p)
    • Probability mass function,

122

0

1

p

1 - p

PMF of Bernoulli

PMF of Bernoulli (p = 0.2)

Differentiable PMF for Bernoulli

123 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Xi ~ Ber(p)
    • Probability mass function,

123

0

1

p

1 - p

PMF of Bernoulli

PMF of Bernoulli (p = 0.2)

Differentiable PMF for Bernoulli

124 of 151

124

Bernoulli PMF

125 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate p
    • Probability mass function can be written as:

    • Likelihood:
    • Log-likelihood:

​

​

125

1. What is the likelihood of one Xi

2. What is the likelihood of all the data

3. What is the log-likelihood all the data

4. Find the value of p which maximizes log likelihood

Maximum Likelihood For Bernoulli

126 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate p
    • Probability mass function can be written as:

    • Likelihood:
    • Log-likelihood:

​

​

126

2. What is the likelihood of all the data

3. What is the log-likelihood all the data

4. Find the value of p which maximizes log likelihood

Maximum Likelihood For Bernoulli

127 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate p
    • Probability mass function can be written as:

    • Likelihood:
    • Log-likelihood:

​

​

127

3. What is the log-likelihood all the data

4. Find the value of p which maximizes log likelihood

Maximum Likelihood For Bernoulli

128 of 151

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate p
    • Probability mass function can be written as:

    • Likelihood:
    • Log-likelihood:

​

​

128

4. Find the value of p which maximizes log likelihood

Maximum Likelihood For Bernoulli

129 of 151

Take Derivative:

129

130 of 151

Take Derivative:

130

Take the derivative wrt p

131 of 151

Take Derivative:

131

Take the derivative wrt p

Derivative of a sum!

132 of 151

Take Derivative:

132

Take the derivative wrt p

Derivative of a sum!

Derivative of a sum!

133 of 151

Take Derivative:

133

Take the derivative wrt p

Derivative of a sum!

Derivative of a sum!

134 of 151

Take Derivative:

134

Take the derivative wrt p

Derivative of a sum!

Derivative of log p

Derivative of a sum!

135 of 151

Take Derivative:

135

Take the derivative wrt p

Derivative of a sum!

Derivative of log p

Derivative of a sum!

136 of 151

Take Derivative:

136

Take the derivative wrt p

Derivative of a sum!

Derivative of log p

Derivative of a sum!

Derivative of log (1-p)

137 of 151

Set to Zero:

137

138 of 151

Set to Zero:

138

139 of 151

Set to Zero:

139

Let

To make life easier

140 of 151

Set to Zero:

140

Let

And

To make life easier

141 of 151

Set to Zero:

141

Let

And

To make life easier

142 of 151

Set to Zero:

142

Let

And

To make life easier

143 of 151

Set to Zero:

143

Let

And

To make life easier

144 of 151

Set to Zero:

144

Let

And

To make life easier

145 of 151

Set to Zero:

145

Let

And

To make life easier

146 of 151

Set to Zero:

146

Let

And

To make life easier

147 of 151

147

Isn’t that the same as

unbiased estimator?

148 of 151

148

Yes. For Bernoulli.

149 of 151

149

MLE for Bernoulli is Sample Mean

150 of 151

150

The medicine is tried on 20 patients. It “works” for 14 and “doesn’t work” for 6. What is your new belief that the drug works?

​

In other words I have 20 IID samples from a Bernoulli. Estimate p. The data is [1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0]

MLE estimate:

Prior

Posterior

mode

Beta estimate:

MLE vs. Beta

151 of 151

151

Think about the difference between a point estimate and a distribution

p = 0.75

p =