1 of 177

Logistic Regression

Chris Gregg

CS109, Stanford University

Autumn, 2026

2 of 177

2

3 of 177

3

4 of 177

Where are we in CS109?

5 of 177

Earlier the start, the better, for PSet #6

5

6 of 177

Earlier the start, the better, for PSet #6

6

7 of 177

Earlier the start, the better, for PSet #6

7

8 of 177

What will be on Pset #7?

Review and Applications!

9 of 177

Background

10 of 177

10

Background: Sigmoid Function

The sigmoid function squashes z to be a number between 0 and 1

Caution:

different use of sigma than normal std

11 of 177

11

Background: Key Notation

Sigmoid function

Weighted sum

(aka dot product)

Sigmoid function of

weighted sum

12 of 177

12

Background: Chain Rule

Who knew calculus would be so useful?

Aka decomposition of composed functions

13 of 177

Machine Learning

(aka Applied Probability)

14 of 177

14

Machine Learning in CS109

Parameter Estimation

Logistic

Regression

Neural Networks

Theory

Core Algorithms

Great Idea

MLE

15 of 177

MLE Idea: Chose params that make the data look likely

15

16 of 177

Likelihood Definition

16

A generalized term for “PDF / PMF / Joint”

of data as a function of parameters

Wikipedia:

17 of 177

17

Define likelihood, use independence.

Define the loglikelihood

Use LL to chose params

Chose the params

That maximize likelihood

Maximum Likelihood

18 of 177

MLE for a Pareto

18

We know sand is distributed as a pareto with PDF

19 of 177

19

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Xi ~ Pareto(α). Use Maximum Likelihood to estimate α.
    • Likelihood:

​

​

    • Log-likelihood:

​

​

    • Chose α to be the argmax of LL:

MLE for a Pareto

3. Find the value of α which maximizes log likelihood

1. What is the likelihood of all the data

2. What is the log-likelihood all the data

20 of 177

20

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Xi ~ Pareto(α). Use Maximum Likelihood to estimate α.
    • Likelihood:

​

​

    • Log-likelihood:

​

​

    • Chose α to be the argmax of LL:

MLE for a Pareto

3. Find the value of α which maximizes log likelihood

2. What is the log-likelihood all the data

21 of 177

21

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Xi ~ Pareto(α). Use Maximum Likelihood to estimate α.
    • Likelihood:

​

​

    • Log-likelihood:

​

​

    • Chose α to be the argmax of LL:

MLE for a Pareto

3. Find the value of α which maximizes log likelihood

22 of 177

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Xi ~ Pareto(α). Use Maximum Likelihood to estimate α.
    • Likelihood:

​

​

    • Log-likelihood:

​

​

    • Chose α to be the argmax of LL:

22

MLE for a Pareto

Argmax Option #1: set the derivative to 0, and solve for alpha

23 of 177

 

24 of 177

24

Gradient Ascent

Walk uphill and you will find a local maxima

(if your step size is small enough)

Especially good if function is convex

25 of 177

25

Gradient Ascent

Repeat many times

Walk uphill and you will find a local maxima

(if your step size is small enough)

This is some profound life philosophy

Step size constant

26 of 177

Initialize: θj = random for all 0 ≤ j ≤ m

​

26

Gradient Ascent

Repeat many times:

Calculate all gradient[j]’s based on data

𝜃j += η * gradient[j] for all 0 ≤ j ≤ m

27 of 177

MLE of Erlang

27

Time to finish Medical Diagnosis in Seconds

28 of 177

End Review

Core Algorithms

29 of 177

Warmup

Core Algorithms

30 of 177

MLE for Bernoulli

30

31 of 177

Don’t we already have the Beta?

​

Yes! But this example is critical for developing towards deep learning.

32 of 177

Maximum Likelihood with Bernoulli

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate p
    • Probability mass function can be written as:

​

​

32

33 of 177

Differentiable PMF for Bernoulli

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Xi ~ Ber(p)
    • Probability mass function,

33

0

1

p

1 - p

PMF of Bernoulli

PMF of Bernoulli (p = 0.2)

34 of 177

34

Bernoulli PMF

35 of 177

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate p
    • Probability mass function can be written as:

    • Likelihood:
    • Log-likelihood:

​

​

35

35

1. What is the likelihood of one Xi

2. What is the likelihood of all the data

3. What is the log-likelihood all the data

4. Find the value of p which maximizes log likelihood

Maximum Likelihood For Bernoulli

36 of 177

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate p
    • Probability mass function can be written as:

    • Likelihood:
    • Log-likelihood:

​

​

36

2. What is the likelihood of all the data

3. What is the log-likelihood all the data

4. Find the value of p which maximizes log likelihood

Maximum Likelihood For Bernoulli

37 of 177

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate p
    • Probability mass function can be written as:

    • Likelihood:
    • Log-likelihood:

​

​

37

37

3. What is the log-likelihood all the data

4. Find the value of p which maximizes log likelihood

Maximum Likelihood For Bernoulli

38 of 177

  • Consider I.I.D. random variables X1, X2, ..., Xn
    • Use Maximum Likelihood to estimate p
    • Probability mass function can be written as:

    • Likelihood:
    • Log-likelihood:

​

​

38

38

4. Find the value of p which maximizes log likelihood

Maximum Likelihood For Bernoulli

39 of 177

Take Derivative:

39

40 of 177

Take Derivative:

40

Take the derivative wrt p

41 of 177

Take Derivative:

41

Take the derivative wrt p

Derivative of a sum!

42 of 177

Take Derivative:

42

Take the derivative wrt p

Derivative of a sum!

Derivative of a sum!

43 of 177

Take Derivative:

43

Take the derivative wrt p

Derivative of a sum!

Derivative of a sum!

44 of 177

Take Derivative:

44

Take the derivative wrt p

Derivative of a sum!

Derivative of log p

Derivative of a sum!

45 of 177

Take Derivative:

45

Take the derivative wrt p

Derivative of a sum!

Derivative of log p

Derivative of a sum!

46 of 177

Take Derivative:

46

Take the derivative wrt p

Derivative of a sum!

Derivative of log p

Derivative of a sum!

Derivative of log (1-p)

47 of 177

Set to Zero:

47

48 of 177

Set to Zero:

48

49 of 177

Set to Zero:

49

Let

To make life easier

50 of 177

Set to Zero:

50

Let

And

To make life easier

51 of 177

Set to Zero:

51

Let

And

To make life easier

52 of 177

Set to Zero:

52

Let

And

To make life easier

53 of 177

Set to Zero:

53

Let

And

To make life easier

54 of 177

Set to Zero:

54

Let

And

To make life easier

55 of 177

Set to Zero:

55

Let

And

To make life easier

56 of 177

Set to Zero:

56

Let

And

To make life easier

57 of 177

Isn’t that the same as

unbiased estimator?

58 of 177

Yes. For Bernoulli.

The ML estimate of p of a Bernoulli is the same as the sample mean.

59 of 177

MLE vs Beta

59

The medicine is tried on 20 patients. It “works” for 14 and “doesn’t work” for 6. What is your new belief that the drug works?

​

In other words I have 20 IID samples from a Bernoulli. Estimate p. The data is [1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0]

MLE estimate:

Prior

Posterior

mode

Beta estimate:

60 of 177

60

Think about the difference between a point estimate and a distribution

p = 0.75

p =

61 of 177

Param Estimation With a Prior == Inference

61

We know sand is distributed as a pareto with PDF

Prior:

62 of 177

62

Maximum A Posteriori (MAP)

1

Infer the posterior belief in params

2

Return the argmax

(aka the mode)

63 of 177

You need to know MLE and Inference.

MAP is something you should recognize!

64 of 177

It is time….

65 of 177

Today a very special MLE problem

66 of 177

Classification

67 of 177

67

Example Datasets

Heart

Ancestry

Netflix

68 of 177

68

Training Data

Assume IID data:

n training datapoints

Each datapoint has m features and a single output

Training Data: assignments all random variables X and Y

69 of 177

69

Single Feature Value

User 1

User 2

User n

Movie 1

Movie 2

Movie m

Output

1

0

1

1

1

1

0

0

0

0

1

1

…

…

…

70 of 177

70

Single Feature Value

1

0

1

1

1

1

0

0

0

0

1

1

…

…

…

User 1

User 2

User n

Movie 1

Movie 2

Movie m

Output

71 of 177

71

Single Feature Value

1

0

1

1

1

1

0

0

0

0

1

1

…

…

…

User 1

User 2

User n

Movie 1

Movie 2

Movie m

Output

72 of 177

72

Single Feature Value

1

0

1

1

1

1

0

0

0

0

1

1

…

…

…

User 1

User 2

User n

Movie 1

Movie 2

Movie m

Output

73 of 177

73

Single Feature Value

1

0

1

1

1

1

0

0

0

0

1

1

…

…

…

In general:

In this case:

User 1

User 2

User n

Movie 1

Movie 2

Movie m

Output

74 of 177

74

Healthy Heart Classifier

Heart 1

Heart 2

Heart n

ROI 1

ROI 2

ROI m

Output

0

1

1

0

1

1

1

0

0

0

0

1

…

…

…

75 of 177

75

Healthy Heart Classifier

0

1

1

0

1

1

1

0

0

0

0

1

…

…

…

Heart 1

Heart 2

Heart n

ROI 1

ROI 2

ROI m

Output

76 of 177

76

Healthy Heart Classifier

0

1

1

0

1

1

1

0

0

0

0

1

…

…

…

Heart 1

Heart 2

Heart n

ROI 1

ROI 2

ROI m

Output

77 of 177

77

Healthy Heart Classifier

0

1

1

0

1

1

1

0

0

0

0

1

…

…

…

Heart 1

Heart 2

Heart n

ROI 1

ROI 2

ROI m

Output

78 of 177

78

Healthy Heart Classifier

0

1

1

0

1

1

1

0

0

0

0

1

…

…

…

Heart 1

Heart 2

Heart n

ROI 1

ROI 2

ROI m

Output

79 of 177

79

Ancestry Classifier

User 1

User 2

User n

SNP 1

SNP 2

SNP m

Output

1

0

1

0

0

0

1

1

1

1

0

1

…

…

…

80 of 177

80

Aside Predicting Real Numbers is Regression

Game 1

Game 2

Game n

Opposing team

ELO

Points in last game

At Home?

Output

84

105

1

120

90

102

0

95

74

120

0

115

…

…

…

# Points

81 of 177

81

Ancestry Classifier

User 1

User 2

User n

SNP 1

SNP 2

SNP m

Output

1

0

1

0

0

0

1

1

1

1

0

1

…

…

…

82 of 177

82

Still Classification

User 1

User 2

User n

SNP 1

SNP 2

SNP m

Output

15

0

2

0

0

0

1

1

7

20

0

1

…

…

…

83 of 177

Classification

84 of 177

84

Healthy Heart Classifier

Heart 1

Heart 2

Heart n

ROI 1

ROI 2

ROI m

Output

0

1

1

0

1

1

1

0

0

0

0

1

…

…

…

85 of 177

85

Classification is Building a Harry Potter Hat

86 of 177

Machine Learning for Classification

86

(inputs)

(prediction)

(model)

87 of 177

Machine Learning for Classification

87

(inputs)

(prediction)

(model)

88 of 177

88

Logistic Regression

89 of 177

Chapter 1: Big Picture

90 of 177

  • In classification we care about

​

90

From Naïve Bayes to Logistic Regression

Let’s build a machine that can you can put x into, which then spits out

91 of 177

​

  • Could we compute via a machine?
  • Welcome our friend: logistic regression!

91

0.81

[1, 1, 0]

Logistic Regression Assumption

92 of 177

​

  • Could we compute via a machine?
  • Welcome our friend: logistic regression!

92

0.81

[1, 1, 0]

Logistic Regression Assumption

93 of 177

93

Logistic Regression Assumption

94 of 177

94

Logistic Regression

+

z = 2.1

σ(z) = 0.7

95 of 177

95

Logistic Regression

96 of 177

96

+

z = 2.1

σ(z) = 0.7

Logistic Regression Cartoon

97 of 177

97

+

Inputs x = [0, 1, 1]

z = 2.1

σ(z) = 0.7

98 of 177

98

+

Inputs

z = 2.1

σ(z) = 0.7

99 of 177

99

+

Inputs + Output

z = 2.1

σ(z) = 0.7

100 of 177

100

+

Inputs

z = 2.1

σ(z) = 0.7

101 of 177

101

+

Weights

z = 2.1

σ(z) = 0.7

102 of 177

102

+

Weighed Sum

z = 2.1

σ(z) = 0.7

103 of 177

103

+

Squashing Function

z = 2.1

σ(z) = 0.7

104 of 177

104

+

Prediction

z = 2.1

σ(z) = 0.7

105 of 177

105

+

z = 2.1

σ(z) = 0.7

Parameters Affect Prediction

106 of 177

106

+

z = -1.5

σ(z) = 0.4

Parameters Affect Prediction

107 of 177

107

+

z = 2.1

σ(z) = 0.7

Parameters Affect Prediction

108 of 177

108

+

Different Predictions for Different Inputs

z = 2.1

σ(z) = 0.7

109 of 177

109

+

Different Predictions for Different Inputs

z = 2.1

σ(z) = 0.7

110 of 177

110

+

Different Predictions for Different Inputs

z = -1.9

σ(z) = 0.3

111 of 177

  • Model conditional likelihood
  • with logistic function:

​

​

    • For simplicity define so
    • Since P(Y = 0 | X = x) + P(Y = 1 | X = x) = 1:

​

​

111

Recall:

Sigmoid function

Handling the Intercept

112 of 177

112

Logistic Regression Assumption:

Big Assumption

113 of 177

Note: inflection point at z = 0. f (0) = 0.5

113

Want to distinguish y = 1 (blue) points from y = 0 (red) points

The Sigmoid Function

114 of 177

114

What is in a Name

Classification Algorithms

Regression Algorithms

Linear Regression

Logistic Regression

If Chris could rename it he would call it: Sigmoidal Classification

Awesome classifier, terrible name

Decision Tree Classifier

115 of 177

115

What makes for a “smart”

logistic regression algorithm?

116 of 177

116

Logistic regression gets its intelligence from its thetas (aka its parameters)

117 of 177

117

+

z = -1.5

σ(z) = 0.4

Let’s say that:

y = 1

Data looks unlikely

How Do We Learn Parameters?

118 of 177

118

+

z = -1.5

σ(z) = 0.4

How Do We Learn Parameters?

Let’s say that:

y = 1

Data looks unlikely

119 of 177

119

+

z = 2.1

σ(z) = 0.9

How Do We Learn Parameters?

y = 1

Data is much more likely!

Let’s say that:

120 of 177

120

Maximum Likelihood Estimation

Remember this?

121 of 177

121

Let’s show you the big picture,

then we can derive it!

122 of 177

122

Math for Logistic Regression

1

2

3

Make logistic regression assumption

Calculate the log likelihood for all data

Get derivative of log likelihood with respect to thetas

Often call this

1

123 of 177

123

Gradient Ascent

Walk uphill and you will find a local maxima

(if your step size is small enough)

Logistic regression LL function is convex

124 of 177

124

Gradient Ascent Step

Do this

for all

thetas!

𝜃1

𝜃2

LL(𝜃)

125 of 177

What does this look like in code?

126 of 177

Initialize: θj = 0 for all 0 ≤ j ≤ m

​

126

Real Code!!!

Repeat many times:

For each training example (x, y):

For each parameter j:

𝜃j += η * gradient[j] for all 0 ≤ j ≤ m

gradient[j] = 0 for all 0 ≤ j ≤ m

gradient[j]

127 of 177

Step by Step

128 of 177

Initialize: θj = 0 for all 0 ≤ j ≤ m

​

128

Logistic Regression Training

Calculate all θj

129 of 177

Initialize: θj = 0 for all 0 ≤ j ≤ m

​

129

Logistic Regression Training

Repeat many times:

Calculate all gradient[j]’s based on data

𝜃j += η * gradient[j] for all 0 ≤ j ≤ m

gradient[j] = 0 for all 0 ≤ j ≤ m

130 of 177

Initialize: θj = 0 for all 0 ≤ j ≤ m

​

130

Logistic Regression Training

Repeat many times:

For each training example (x, y):

For each parameter j:

𝜃j += η * gradient[j] for all 0 ≤ j ≤ m

gradient[j] = 0 for all 0 ≤ j ≤ m

Update gradient[j] for current training example (x, y)

131 of 177

Initialize: θj = 0 for all 0 ≤ j ≤ m

​

131

Logistic Regression Training

Repeat many times:

For each training example (x, y):

For each parameter j:

𝜃j += η * gradient[j] for all 0 ≤ j ≤ m

gradient[j] = 0 for all 0 ≤ j ≤ m

gradient[j]

132 of 177

132

+

Training

Dataset likelihood:

Likelihood

Training iterations

133 of 177

133

+

Training

Dataset likelihood:

Training iterations

Likelihood

134 of 177

134

+

Training

Dataset likelihood:

Training iterations

Likelihood

135 of 177

135

+

Training

Dataset likelihood:

Training iterations

Likelihood

136 of 177

136

+

Training

Dataset likelihood:

Training iterations

Likelihood

137 of 177

137

+

Training

Dataset likelihood:

Training iterations

Likelihood

138 of 177

138

Don’t forget:

xj is j-th input variable and x0 = 1.

​

Allows for θ0 to be an intercept.

139 of 177

139

+

z = 2.1

σ(z) = 0.7

Prediction

0

x:

1

1

140 of 177

140

+

z = 2.1

σ(z) = 0.7

0

x:

1

1

Prediction

141 of 177

141

+

z = 2.1

σ(z) = 0.7

0

x:

1

1

Prediction

142 of 177

142

+

z = 2.1

σ(z) = 0.7

0

x:

1

1

Prediction

0.817

143 of 177

143

+

z = 2.1

σ(z) = 0.7

0.817

0

x:

1

1

Prediction

144 of 177

144

Live Demo!!

145 of 177

Chapter 2: How Come?

146 of 177

146

Logistic Regression

1

2

3

Make logistic regression assumption

Calculate the log probability for all data

Get derivative of log probability with respect to thetas

Often call this

1

147 of 177

147

How did we get that LL function?

148 of 177

​

Probability mass function:

148

Recall: PMF of Bernoulli

0

1

p

1 - p

PMF of Bernoulli

PMF of Bernoulli (p = 0.2)

149 of 177

149

Implies

For IID data

Take the log

1

Recall:

150 of 177

150

How did we get that gradient?

151 of 177

151

Sigmoid has a Beautiful Slope

True fact about sigmoid functions

152 of 177

152

Sigmoid has a Beautiful Slope

Sigmoid, you should be a ski hill

Chain rule!

Plug and chug

153 of 177

153

Sigmoid has a Beautiful Slope

154 of 177

ARE YOU READY???

155 of 177

155

I think I’m Ready…

Where

156 of 177

156

Think About Only One Training Instance

We only need to calculate the gradient for one training example!

We will pretend we only have one example

We can sum up the gradients of each example to get the correct answer

157 of 177

157

First, imagine only one example

Where

CHAIN RULE

Already did that one

Derive this one

Simplify

158 of 177

158

Now, all the data

See last slide

Derivative of sum…

Some people don’t like hats…

159 of 177

159

Logistic Regression

1

2

3

Make logistic regression assumption

Calculate the log probability for all data

Get derivative of log probability with respect to thetas

1

160 of 177

160

The Hard Way

161 of 177

Phew!

162 of 177

Chapter 3: Philosophy

(if time)

163 of 177

Neuron

164 of 177

Neuron

165 of 177

Neuron

166 of 177

Neuron

167 of 177

Neuron

168 of 177

Some inputs are more important

169 of 177

Artificial Neurons

170 of 177

  • A neuron

​

​

​

  • Your brain

​

​

​

​

x1

x2

x3

x4

θ1

θ2

θ3

θ4

y

x1

x2

x3

x4

Biological Basis for Neural Networks

171 of 177

Deep learning is (at its core) many logistic regression pieces stacked on top of each other.

(aka Neural Networks)

172 of 177

Computer Vision

173 of 177

Alpha GO

174 of 177

Revolution in AI

175 of 177

Computers Making Art

176 of 177

176

177 of 177

Basically just many logistic regression cells

And lots of chain rule…