Logistic Regression
Chris Gregg
CS109, Stanford University
Summer, 2026
2
3
Where are we in CS109?
Where are we in CS109?
Earlier the start, the better, for PSet #6
6
Earlier the start, the better, for PSet #6
7
Earlier the start, the better, for PSet #6
8
What will be on Pset #7?
Review and Applications!
Background
11
Background: Sigmoid Function
The sigmoid function squashes z to be a number between 0 and 1
Caution:
different use of sigma than normal std
12
Background: Key Notation
Sigmoid function
Weighted sum
(aka dot product)
Sigmoid function of
weighted sum
13
Background: Chain Rule
Who knew calculus would be so useful?
Aka decomposition of composed functions
Machine Learning
(aka Applied Probability)
15
Machine Learning in CS109
Parameter Estimation
Logistic
Regression
Neural Networks
Theory
Core Algorithms
Great Idea
MLE
MLE Idea: Chose params that make the data look likely
16
Likelihood Definition
17
A generalized term for “PDF / PMF / Joint”
of data as a function of parameters
Wikipedia:
18
Define likelihood, use independence.
Define the loglikelihood
Use LL to chose params
Chose the params
That maximize likelihood
Maximum Likelihood
MLE for a Pareto
19
We know sand is distributed as a pareto with PDF
20
MLE for a Pareto
3. Find the value of α which maximizes log likelihood
1. What is the likelihood of all the data
2. What is the log-likelihood all the data
21
MLE for a Pareto
3. Find the value of α which maximizes log likelihood
2. What is the log-likelihood all the data
22
MLE for a Pareto
3. Find the value of α which maximizes log likelihood
23
MLE for a Pareto
Argmax Option #1: set the derivative to 0, and solve for alpha
25
Gradient Ascent
Walk uphill and you will find a local maxima
(if your step size is small enough)
Especially good if function is convex
26
Gradient Ascent
Repeat many times
Walk uphill and you will find a local maxima
(if your step size is small enough)
This is some profound life philosophy
Step size constant
Initialize: θj = random for all 0 ≤ j ≤ m
27
Gradient Ascent
Repeat many times:
Calculate all gradient[j]’s based on data
𝜃j += η * gradient[j] for all 0 ≤ j ≤ m
MLE of Erlang
28
Time to finish Medical Diagnosis in Seconds
End Review
Core Algorithms
Warmup
Core Algorithms
MLE for Bernoulli
31
Don’t we already have the Beta?
Yes! But this example is critical for developing towards deep learning.
Maximum Likelihood with Bernoulli
33
Differentiable PMF for Bernoulli
34
0
1
p
1 - p
PMF of Bernoulli
PMF of Bernoulli (p = 0.2)
35
Bernoulli PMF
36
36
1. What is the likelihood of one Xi
2. What is the likelihood of all the data
3. What is the log-likelihood all the data
4. Find the value of p which maximizes log likelihood
Maximum Likelihood For Bernoulli
37
2. What is the likelihood of all the data
3. What is the log-likelihood all the data
4. Find the value of p which maximizes log likelihood
Maximum Likelihood For Bernoulli
38
38
3. What is the log-likelihood all the data
4. Find the value of p which maximizes log likelihood
Maximum Likelihood For Bernoulli
39
39
4. Find the value of p which maximizes log likelihood
Maximum Likelihood For Bernoulli
Take Derivative:
40
Take Derivative:
41
Take the derivative wrt p
Take Derivative:
42
Take the derivative wrt p
Derivative of a sum!
Take Derivative:
43
Take the derivative wrt p
Derivative of a sum!
Derivative of a sum!
Take Derivative:
44
Take the derivative wrt p
Derivative of a sum!
Derivative of a sum!
Take Derivative:
45
Take the derivative wrt p
Derivative of a sum!
Derivative of log p
Derivative of a sum!
Take Derivative:
46
Take the derivative wrt p
Derivative of a sum!
Derivative of log p
Derivative of a sum!
Take Derivative:
47
Take the derivative wrt p
Derivative of a sum!
Derivative of log p
Derivative of a sum!
Derivative of log (1-p)
Set to Zero:
48
Set to Zero:
49
Set to Zero:
50
Let
To make life easier
Set to Zero:
51
Let
And
To make life easier
Set to Zero:
52
Let
And
To make life easier
Set to Zero:
53
Let
And
To make life easier
Set to Zero:
54
Let
And
To make life easier
Set to Zero:
55
Let
And
To make life easier
Set to Zero:
56
Let
And
To make life easier
Set to Zero:
57
Let
And
To make life easier
Isn’t that the same as
unbiased estimator?
Yes. For Bernoulli.
The ML estimate of p of a Bernoulli is the same as the sample mean.
MLE vs Beta
60
The medicine is tried on 20 patients. It “works” for 14 and “doesn’t work” for 6. What is your new belief that the drug works?
In other words I have 20 IID samples from a Bernoulli. Estimate p. The data is [1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0]
MLE estimate:
Prior
Posterior
mode
Beta estimate:
61
Think about the difference between a point estimate and a distribution
p = 0.75
p =
Param Estimation With a Prior == Inference
62
We know sand is distributed as a pareto with PDF
Prior:
63
Maximum A Posteriori (MAP)
1
Infer the posterior belief in params
2
Return the argmax
(aka the mode)
You need to know MLE and Inference.
MAP is something you should recognize!
It is time….
Today a very special MLE problem
Classification
68
Example Datasets
Heart
Ancestry
Netflix
69
Training Data
Assume IID data:
n training datapoints
Each datapoint has m features and a single output
Training Data: assignments all random variables X and Y
70
Single Feature Value
User 1
User 2
User n
Movie 1
Movie 2
Movie m
Output
1
0
1
1
1
1
0
0
0
0
1
1
…
…
…
71
Single Feature Value
1
0
1
1
1
1
0
0
0
0
1
1
…
…
…
User 1
User 2
User n
Movie 1
Movie 2
Movie m
Output
72
Single Feature Value
1
0
1
1
1
1
0
0
0
0
1
1
…
…
…
User 1
User 2
User n
Movie 1
Movie 2
Movie m
Output
73
Single Feature Value
1
0
1
1
1
1
0
0
0
0
1
1
…
…
…
User 1
User 2
User n
Movie 1
Movie 2
Movie m
Output
74
Single Feature Value
1
0
1
1
1
1
0
0
0
0
1
1
…
…
…
In general:
In this case:
User 1
User 2
User n
Movie 1
Movie 2
Movie m
Output
75
Healthy Heart Classifier
Heart 1
Heart 2
Heart n
ROI 1
ROI 2
ROI m
Output
0
1
1
0
1
1
1
0
0
0
0
1
…
…
…
76
Healthy Heart Classifier
0
1
1
0
1
1
1
0
0
0
0
1
…
…
…
Heart 1
Heart 2
Heart n
ROI 1
ROI 2
ROI m
Output
77
Healthy Heart Classifier
0
1
1
0
1
1
1
0
0
0
0
1
…
…
…
Heart 1
Heart 2
Heart n
ROI 1
ROI 2
ROI m
Output
78
Healthy Heart Classifier
0
1
1
0
1
1
1
0
0
0
0
1
…
…
…
Heart 1
Heart 2
Heart n
ROI 1
ROI 2
ROI m
Output
79
Healthy Heart Classifier
0
1
1
0
1
1
1
0
0
0
0
1
…
…
…
Heart 1
Heart 2
Heart n
ROI 1
ROI 2
ROI m
Output
80
Ancestry Classifier
User 1
User 2
User n
SNP 1
SNP 2
SNP m
Output
1
0
1
0
0
0
1
1
1
1
0
1
…
…
…
81
Aside Predicting Real Numbers is Regression
Game 1
Game 2
Game n
Opposing team
ELO
Points in last game
At Home?
Output
84
105
1
120
90
102
0
95
74
120
0
115
…
…
…
# Points
82
Ancestry Classifier
User 1
User 2
User n
SNP 1
SNP 2
SNP m
Output
1
0
1
0
0
0
1
1
1
1
0
1
…
…
…
83
Still Classification
User 1
User 2
User n
SNP 1
SNP 2
SNP m
Output
15
0
2
0
0
0
1
1
7
20
0
1
…
…
…
Classification
85
Healthy Heart Classifier
Heart 1
Heart 2
Heart n
ROI 1
ROI 2
ROI m
Output
0
1
1
0
1
1
1
0
0
0
0
1
…
…
…
86
Classification is Building a Harry Potter Hat
Machine Learning for Classification
87
(inputs)
(prediction)
(model)
Machine Learning for Classification
88
(inputs)
(prediction)
(model)
89
Logistic Regression
Chapter 1: Big Picture
91
From Naïve Bayes to Logistic Regression
Let’s build a machine that can you can put x into, which then spits out
92
0.81
[1, 1, 0]
Logistic Regression Assumption
93
0.81
[1, 1, 0]
Logistic Regression Assumption
94
Logistic Regression Assumption
95
Logistic Regression
+
z = 2.1
σ(z) = 0.7
96
Logistic Regression
97
+
z = 2.1
σ(z) = 0.7
Logistic Regression Cartoon
98
+
Inputs x = [0, 1, 1]
z = 2.1
σ(z) = 0.7
99
+
Inputs
z = 2.1
σ(z) = 0.7
100
+
Inputs + Output
z = 2.1
σ(z) = 0.7
101
+
Inputs
z = 2.1
σ(z) = 0.7
102
+
Weights
z = 2.1
σ(z) = 0.7
103
+
Weighed Sum
z = 2.1
σ(z) = 0.7
104
+
Squashing Function
z = 2.1
σ(z) = 0.7
105
+
Prediction
z = 2.1
σ(z) = 0.7
106
+
z = 2.1
σ(z) = 0.7
Parameters Affect Prediction
107
+
z = -1.5
σ(z) = 0.4
Parameters Affect Prediction
108
+
z = 2.1
σ(z) = 0.7
Parameters Affect Prediction
109
+
Different Predictions for Different Inputs
z = 2.1
σ(z) = 0.7
110
+
Different Predictions for Different Inputs
z = 2.1
σ(z) = 0.7
111
+
Different Predictions for Different Inputs
z = -1.9
σ(z) = 0.3
112
Recall:
Sigmoid function
Handling the Intercept
113
Logistic Regression Assumption:
Big Assumption
Note: inflection point at z = 0. f (0) = 0.5
114
Want to distinguish y = 1 (blue) points from y = 0 (red) points
The Sigmoid Function
115
What is in a Name
Classification Algorithms
Regression Algorithms
Linear Regression
Logistic Regression
If Chris could rename it he would call it: Sigmoidal Classification
Awesome classifier, terrible name
Decision Tree Classifier
116
What makes for a “smart”
logistic regression algorithm?
117
Logistic regression gets its intelligence from its thetas (aka its parameters)
118
+
z = -1.5
σ(z) = 0.4
Let’s say that:
y = 1
Data looks unlikely
How Do We Learn Parameters?
119
+
z = -1.5
σ(z) = 0.4
How Do We Learn Parameters?
Let’s say that:
y = 1
Data looks unlikely
120
+
z = 2.1
σ(z) = 0.9
How Do We Learn Parameters?
y = 1
Data is much more likely!
Let’s say that:
121
Maximum Likelihood Estimation
Remember this?
122
Let’s show you the big picture,
then we can derive it!
123
Math for Logistic Regression
1
2
3
Make logistic regression assumption
Calculate the log likelihood for all data
Get derivative of log likelihood with respect to thetas
Often call this
1
124
Gradient Ascent
Walk uphill and you will find a local maxima
(if your step size is small enough)
Logistic regression LL function is convex
125
Gradient Ascent Step
Do this
for all
thetas!
𝜃1
𝜃2
LL(𝜃)
What does this look like in code?
Initialize: θj = 0 for all 0 ≤ j ≤ m
127
Real Code!!!
Repeat many times:
For each training example (x, y):
For each parameter j:
𝜃j += η * gradient[j] for all 0 ≤ j ≤ m
gradient[j] = 0 for all 0 ≤ j ≤ m
gradient[j]
Step by Step
Initialize: θj = 0 for all 0 ≤ j ≤ m
129
Logistic Regression Training
Calculate all θj
Initialize: θj = 0 for all 0 ≤ j ≤ m
130
Logistic Regression Training
Repeat many times:
Calculate all gradient[j]’s based on data
𝜃j += η * gradient[j] for all 0 ≤ j ≤ m
gradient[j] = 0 for all 0 ≤ j ≤ m
Initialize: θj = 0 for all 0 ≤ j ≤ m
131
Logistic Regression Training
Repeat many times:
For each training example (x, y):
For each parameter j:
𝜃j += η * gradient[j] for all 0 ≤ j ≤ m
gradient[j] = 0 for all 0 ≤ j ≤ m
Update gradient[j] for current training example (x, y)
Initialize: θj = 0 for all 0 ≤ j ≤ m
132
Logistic Regression Training
Repeat many times:
For each training example (x, y):
For each parameter j:
𝜃j += η * gradient[j] for all 0 ≤ j ≤ m
gradient[j] = 0 for all 0 ≤ j ≤ m
gradient[j]
133
+
Training
Dataset likelihood:
Likelihood
Training iterations
134
+
Training
Dataset likelihood:
Training iterations
Likelihood
135
+
Training
Dataset likelihood:
Training iterations
Likelihood
136
+
Training
Dataset likelihood:
Training iterations
Likelihood
137
+
Training
Dataset likelihood:
Training iterations
Likelihood
138
+
Training
Dataset likelihood:
Training iterations
Likelihood
139
Don’t forget:
xj is j-th input variable and x0 = 1.
Allows for θ0 to be an intercept.
140
+
z = 2.1
σ(z) = 0.7
Prediction
0
x:
1
1
141
+
z = 2.1
σ(z) = 0.7
0
x:
1
1
Prediction
142
+
z = 2.1
σ(z) = 0.7
0
x:
1
1
Prediction
143
+
z = 2.1
σ(z) = 0.7
0
x:
1
1
Prediction
0.817
144
+
z = 2.1
σ(z) = 0.7
0.817
0
x:
1
1
Prediction
145
Live Demo!!
Chapter 2: How Come?
147
Logistic Regression
1
2
3
Make logistic regression assumption
Calculate the log probability for all data
Get derivative of log probability with respect to thetas
Often call this
1
148
How did we get that LL function?
Probability mass function:
149
Recall: PMF of Bernoulli
0
1
p
1 - p
PMF of Bernoulli
PMF of Bernoulli (p = 0.2)
150
Implies
For IID data
Take the log
1
Recall:
151
How did we get that gradient?
152
Sigmoid has a Beautiful Slope
True fact about sigmoid functions
153
Sigmoid has a Beautiful Slope
Sigmoid, you should be a ski hill
Chain rule!
Plug and chug
154
Sigmoid has a Beautiful Slope
ARE YOU READY???
156
I think I’m Ready…
Where
157
Think About Only One Training Instance
We only need to calculate the gradient for one training example!
We will pretend we only have one example
We can sum up the gradients of each example to get the correct answer
158
First, imagine only one example
Where
CHAIN RULE
Already did that one
Derive this one
Simplify
159
Now, all the data
See last slide
Derivative of sum…
Some people don’t like hats…
160
Logistic Regression
1
2
3
Make logistic regression assumption
Calculate the log probability for all data
Get derivative of log probability with respect to thetas
1
161
The Hard Way
Phew!
Chapter 3: Philosophy
(if time)
Neuron
Neuron
Neuron
Neuron
Neuron
Some inputs are more important
Artificial Neurons
x1
x2
x3
x4
θ1
θ2
θ3
θ4
y
x1
x2
x3
x4
Biological Basis for Neural Networks
Deep learning is (at its core) many logistic regression pieces stacked on top of each other.
(aka Neural Networks)
Computer Vision
Alpha GO
Revolution in AI
Computers Making Art
177
Basically just many logistic regression cells
And lots of chain rule…