Logistic Regression (2)
Lecture 09
Logistic Regression Optimization, Decisions, and Evaluation
EECS 189/289, Fall 2026 @ UC Berkeley
Joseph E. Gonzalez and Narges Norouzi
Join at slido.com�#1643396
The Slido app must be installed on every computer you’re presenting from
Do not edit�How to change the design
1643396
Roadmap
1643396
The Logistic Regression Model
1643396
The S-shaped curve is formally known as the sigmoid function.
Range
Reflection/ Symmetry
Domain
Derivative
Inverse
Logit Function
1643396
Which of the following are properties of the sigmoid function, σ(x)?
The Slido app must be installed on every computer you’re presenting from
Do not edit�How to change the design
1643396
Logistic Regression
Model:
Linear Model
Generalized Linear Model:
Non-linear transformation of a linear model.
1643396
Logistic Regression and Decision Boundary
8
1643396
Reading the Weights: Odds
1643396
Logistic Regression Ignores Easy Points
1643396
Logistic Regression Objective
1643396
The Logistic Regression Objective (MLE)
1643396
The Logistic Regression Objective (MLE)
1643396
The Logistic Regression Objective (MLE)
We can define an error function by taking the negative logarithm of the likelihood, which gives the cross-entropy error function.
1643396
Cross-Entropy: Two Error Functions In One!
The Cross-Entropy (CE) function is often written more compactly as the product of the label and the log probability, summed over both classes.
15
Unbounded for wrong-way certainty
1643396
Convexity By Picture
16
Squared Error Surface
Cross-Entropy Error Surface
A straight line crosses the curve Non-convex
Convex!
1643396
Logistic Regression Optimization
1643396
1643396
Need to apply chain rule
1643396
Separating the terms
1643396
We Cannot Solve Directly
We minimize E(w) numerically instead.
1643396
Regularization with Logistic Regression
1643396
1
0
-1
1
(1,0)
(-1,1)
The Data
1643396
What is the best value for w?
The Slido app must be installed on every computer you’re presenting from
Do not edit�How to change the design
1643396
1
0
-1
1
(1,0)
(-1,1)
The Data
1643396
1
0
-1
1
(1,0)
(-1,1)
The Data
For the point (-1,1):
Objective:
1643396
1
0
-1
1
(1,0)
(-1,1)
The Data
For the point (1,0):
Objective:
1643396
1
0
-1
1
(1,0)
(-1,1)
The Data
Degenerate Solution!
Overly confident!
For the point (-1,1):
For the point (1,0):
1643396
Adding Regularization to �Logistic Regression
Prevents weights from diverging on linearly separable data.
Earlier �Example
Without Regularization
1643396
1643396
Model Evaluation
1643396
Classifier Performance
| 0 | 1 |
0 | True Negative (TN) | False Positive (FP) |
1 | False Negative (FN) | True Positive (TP) |
Actual
Prediction
“Positive” means a prediction of 1.�“Negative” means a prediction of 0.
“True” means correct prediction.�“False” means incorrect prediction.
1643396
Classifier Performance
A confusion matrix plots TP, TN, FP, FN for a particular classifier, dataset, and threshold (!).
Our 0/1 predictions depend on our choice of probability threshold, so the confusion matrix can also change with a new threshold!
1643396
Classifier Accuracy
The most basic evaluation metric for a classifier is accuracy.
1643396
Pitfalls of Accuracy: A Case Study
Suppose we’re trying to build a classifier to filter spam emails.
Each email is spam (1) or ham (0).
We have 100 emails: 5 of them are spam, and the remaining 95 are real (i.e., ham).
Classifier 1:
Classify every email as ham (0).
What is the accuracy of Classifier 1?
1643396
Pitfalls of Accuracy: A Case Study
Suppose we’re trying to build a classifier to filter spam emails.
Each email is spam (1) or ham (0).
We have 100 emails: 5 of them are spam, and the remaining 95 are real (i.e., ham).
Classifier 1:
Classify every email as ham (0).
Classifier 2:
Classify every email as spam (1).
While widely used, the accuracy metric is is problematic if there is class imbalance.
High accuracy…�…but we detected none ⚠️ of the spam!!!
Low ⚠️ accuracy…�…but we detected all of the spam!!!
1643396
Suppose I increase the probability threshold for converting my estimated probabilities from logistic regression into 0/1 predictions.��I do not change my fitted model or test data.��What can happen to the number of true positives (TP) in the test data? Select all that apply.�
The Slido app must be installed on every computer you’re presenting from
Do not edit�How to change the design
1643396
Effect of changing the probability threshold on TP
I increase the probability threshold. What happens to the count of true positives (TP)?
Estimated probabilities | 0.21 | 0.63 | 0.85 |
Actual class | 0 | 1 | 1 |
Threshold=0.3
Threshold=0.6
Threshold=0.8
No change!
TP = 1
TP = 2
If we increase the threshold, it becomes harder to make a positive prediction.
The number of positives (P) will stay the same or decrease.�So, TP and FP will either go down or stay the same, too.
Prediction (T=0.6) | 0 (TN) | 1 (TP) | 1 (TP) |
Prediction (T=0.8) | 0 (TN) | 0 (FN) | 1 (TP) |
Prediction (T=0.3) | 0 (TN) | 1 (TP) | 1 (TP) |
1643396
Choosing an Accuracy Threshold
The choice of threshold T impacts classification performance.
Do we get max accuracy when T = 0.5? Not always the case…
Here (validation): 0.868 at T = 0.5, max 0.895 at T = 0.58.
1643396
Accuracy, Precision, and Recall
Actual
Prediction
Of all predicted positives, what proportion were correct (TP)?�"What % of spam predictions were correct?"
Actual
Prediction
Of all actual 1s, what proportion did our model detect?�What % of spam emails were detected?"
| 0 | 1 |
0 | TN | FP |
1 | FN | TP |
| 0 | 1 |
0 | TN | FP |
1 | FN | TP |
| 0 | 1 |
0 | TN | FP |
1 | FN | TP |
Actual
Prediction
What proportion of all points were correctly classified?�"What % of emails did I classify correctly?"
Recall is also known as sensitivity.
These metrics are threshold-dependent.
1643396
Back to the Spam
Suppose we’re trying to build a classifier to filter spam emails.
Each email is spam (1) or ham (0).
Let’s say we have 100 emails, of which only 5 are truly spam, and the remaining 95 are ham.
Classifier 1:
Classify every email as ham (0).
Classifier 2:
Classify every email as spam (1).
Many false positives!
No false negatives!
| 0 | 1 |
0 | TN: 0 | FP: 95 |
1 | FN: 0 | TP: 5 |
1643396
Precision vs. Recall
FPs and FNs can have different costs based on the Loss matrix.
There is a tradeoff between precision and recall; they are often inversely related.
Precision penalizes FPs.
Recall penalizes FNs.
1643396
Precision-Recall Curves
To construct a precision-recall curve, we:
We often choose a threshold that keeps both precision and recall high. But, if the cost of FPs and FNs differs a lot, we may not.
1643396
Decisions = Posteriors + Loss
Input x
=
1643396
Decisions = Posteriors + Loss
Input x
=
1643396
From a Loss Matrix to a Threshold
1643396
Computing an F1 Score
One way to balance precision and recall is to maximize the F1 Score:
How do you use it?
Note: If FPs and FNs have different costs, we may not want to balance precision + recall.
1643396
Maximizing F1 Score
We can maximize the F1 score by evaluating the F1 score �for many threshold values.
1643396
True and False Positive Rates
Two more performance metrics worth knowing!
True Positive Rate (TPR): Out of all actual 1s, how many did we classify correctly? Same as recall.
False Positive Rate (FPR): Out of all actual 0s, how many did we classify incorrectly?
| 0 | 1 |
0 | TN | FP |
1 | FN | TP |
Actual
Prediction
Lots of classification metrics out there! All based on TP, TN, FP, FN.
1643396
ROC Curves
We can perform a similar process with FPR and TPR.
ROC = “receiver operating characteristic” (comes from radar in WWII)
Threshold is low: FPR high, TPR high
(many positive predictions)
Threshold is high: FPR low, TPR low
(few positive predictions)
1643396
ROC Curves
A perfect predictor has TPR = 1 and FPR = 0
The best Area Under the ROC Curve (AUC-ROC) is 1.
Because we want our classifier to be as close as possible to the perfect predictor, we aim to maximize the AUC.
1643396
ROC Curves
A predictor that predicts randomly has an AUC-ROC of 0.5
Useful classifiers have AUC between 0.5 and 1. Ours: 0.940 on validation. Below 0.5 means worse than random.
1643396
Multi-Class Classification
1643396
Multiple Classes
One-versus-the-rest classifier
One-versus-one classifier
1643396
Better Option: Multi-class Discriminant Solution
1643396
How?
Softmax Function
1643396
Which of the following are true about the softmax function
The Slido app must be installed on every computer you’re presenting from
Do not edit�How to change the design
1643396
Logistic Regression (2)
Lecture 09
Credit: Joseph E. Gonzalez and Narges Norouzi
Reference Book Chapters: Chapter 5 Sections 5.1.2, 5.1.3, 5.2