1 of 36

Logistic Regression 2

Chris Gregg

CS109, Stanford University

Autumn, 2026

2 of 36

Logistic Regression: A Refresher

1

Predict a probability

Example: Is this email spam?

Features x: links, words, etc. Label y: 1 = spam, 0 = not spam

2

Compute a weighted score

z = θᵀx

The weights θ determine how features affect the score.

3

Use the sigmoid to get a probability

p = σ(z) = 1 / (1 + e⁻ᶻ)

Large negative score: p near 0. Zero: p = 0.5.

Large positive score: p near 1.

We can turn the probability into a class using a threshold.

3 of 36

Learning the Weights

1

Learning means choosing θ

Choose weights that assign high probability

to the labels we actually observed.

2

Score the probability of the observed label

Suppose the model predicts p = 0.9 for spam.

If y = 1: observed-label probability is 0.9.

If y = 0: observed-label probability is 0.1.

3

Today’s math

Write the likelihood of all observed labels.

Take its log, then differentiate to improve θ.

4 of 36

Chapter 2: How Come?

5 of 36

5

Logistic Regression

1

2

3

Make logistic regression assumption

Calculate the log probability for all data

Get derivative of log probability with respect to thetas

Often call this

1

6 of 36

6

How did we get that LL function?

7 of 36

​

Probability mass function:

7

Recall: PMF of Bernoulli

0

1

p

1 - p

PMF of Bernoulli

PMF of Bernoulli (p = 0.2)

8 of 36

8

Implies

For IID data

Take the log

1

Recall:

9 of 36

9

How did we get that gradient?

10 of 36

10

Sigmoid has a Beautiful Slope

True fact about sigmoid functions

11 of 36

11

Sigmoid has a Beautiful Slope

Sigmoid, you should be a ski hill

Chain rule!

Plug and chug

12 of 36

12

Sigmoid has a Beautiful Slope

13 of 36

ARE YOU READY???

14 of 36

14

I think I’m Ready…

Where

15 of 36

15

Think About Only One Training Instance

We only need to calculate the gradient for one training example!

We will pretend we only have one example

We can sum up the gradients of each example to get the correct answer

16 of 36

16

First, imagine only one example

Where

CHAIN RULE

Already did that one

Derive this one

Simplify

17 of 36

17

Now, all the data

See last slide

Derivative of sum…

Some people don’t like hats…

18 of 36

18

Logistic Regression

1

2

3

Make logistic regression assumption

Calculate the log probability for all data

Get derivative of log probability with respect to thetas

1

19 of 36

19

The Hard Way

20 of 36

Phew!

21 of 36

Chapter 3: Philosophy

(if time)

22 of 36

Neuron

23 of 36

Neuron

24 of 36

Neuron

25 of 36

Neuron

26 of 36

Neuron

27 of 36

Some inputs are more important

28 of 36

Artificial Neurons

29 of 36

  • A neuron

​

​

​

  • Your brain

​

​

​

​

x1

x2

x3

x4

θ1

θ2

θ3

θ4

y

x1

x2

x3

x4

Biological Basis for Neural Networks

30 of 36

Deep learning is (at its core) many logistic regression pieces stacked on top of each other.

(aka Neural Networks)

31 of 36

Computer Vision

32 of 36

Alpha GO

33 of 36

Revolution in AI

34 of 36

Computers Making Art

35 of 36

35

36 of 36

Basically just many logistic regression cells

And lots of chain rule…