LESSON 01

Machine

Learning

CLASSIFICATION

SERIES

Logistic

Regression

Basics of Classification

Understanding how machines learn to categorize

data into discrete classes using probabilities

Introduction to Machine Learning | Logistic Regression

What We'll Cover Today

01

What is Classification?

Binary vs multi-class, real-world examples

02

Linear vs Logistic Regression

Key differences & why linear fails for classification

03

The Sigmoid Function

Mapping outputs to probabilities [0, 1]

04

Decision Boundary

How the model separates classes

05

Cost Function & Training

Log-loss and gradient descent basics

06

Evaluation Metrics

Accuracy, precision, recall, confusion matrix

What is Classification?

Classification: Predicting which category (class) a data point belongs to.

Binary Classification Examples

📧

Email

Spam

Not Spam

Output: 0 or 1

🏥

Diagnosis

Disease

Healthy

Output: 0 or 1

💳

Transaction

Fraud

Legit

Output: 0 or 1

✅

Loan

Approved

Denied

Output: 0 or 1

💡 Key Insight: Unlike regression (predicting a number), classification predicts a discrete label.

Linear vs Logistic Regression

Linear Regression

Logistic Regression

Output Type

Any real number (−∞ to +∞)

Probability (0 to 1)

Goal

Predict continuous values

Predict class probability

Output Function

y = mx + b (linear)

y = σ(mx + b) (sigmoid)

Example Use

Predict house price ($)

Predict spam? (yes/no)

Decision

No threshold needed

Threshold at 0.5 → class

Loss Function

Mean Squared Error (MSE)

Binary Cross-Entropy

⚠️ Why linear fails for classification: It can predict values outside [0,1], which can't represent probabilities!

The Sigmoid Function

σ(z) =

1

1 + e

-z

where z = wx + b

Key Properties:

• Output always between 0 and 1
• S-shaped (sigmoidal) curve
• σ(0) = 0.5 (decision boundary)
• As z → +∞, σ → 1
• As z → −∞, σ → 0

​

Probability → Class

P ≥ 0.5 → Class 1 ✓

P < 0.5 → Class 0 ✗

Threshold can be adjusted

based on use case

Decision Boundary

Feature Space Visualization

Feature X₁ →

Feature X₂

Decision

Boundary

σ(z) = 0.5

Class 0 (Negative)

Class 1 (Positive)

How It Works

1

Model learns weights

(w) and bias (b)

2

Computes z = w·x + b

for each data point

3

Applies sigmoid:

P = σ(z)

4

If P ≥ 0.5 → Class 1

If P < 0.5 → Class 0

Cost Function & Training

❌ Why not use MSE (Mean Squared Error)? With sigmoid, MSE creates a non-convex loss — prone to many local minima!

Binary Cross-Entropy (Log Loss):

J(w,b) = −(1/m) Σ [ y·log(ŷ) + (1−y)·log(1−ŷ) ]

y = 1 (positive class)

Loss = −log(ŷ)

Push ŷ → 1 to minimize loss

y = 0 (negative class)

Loss = −log(1−ŷ)

Push ŷ → 0 to minimize loss

Gradient Descent

w := w − α · ∂J/∂w

Iteratively find optimal weights

Evaluation Metrics

Confusion Matrix

Predicted →

Predicted 0

Predicted 1

Actual ↓

Actual 0

Actual 1

TN

True Neg

FP

False Pos

FN

False Neg

TP

True Pos

Key Metrics

Accuracy

(TP + TN) / Total

Overall correctness

Precision

TP / (TP + FP)

Of predicted positives, how many are correct

Recall

TP / (TP + FN)

Of actual positives, how many detected

F1-Score

2 × (P × R) / (P + R)

Harmonic mean of precision & recall

Visual Comparison: Linear vs Logistic

❌ Linear Regression for Classification

Problem: values go outside [0,1]!

✅ Logistic Regression for Classification

Always stays between 0 and 1 ✓

Key Takeaways

Logistic Regression — Lecture 05 Summary

🎯

#01

Logistic regression predicts class probabilities (0 to 1), not continuous values.

〽️

#02

The sigmoid function maps any real number to a probability between 0 and 1.

⚠️

#03

Linear regression fails for classification — it can output values outside [0, 1].

📏

#04

Decision boundary at threshold 0.5 separates the two classes in feature space.

📉

#05

Binary cross-entropy (log loss) is the correct loss function for logistic regression.

📊

#06

Accuracy, precision, recall & F1-score are used to evaluate classification models.

Next Lesson → Multiclass Classification & Softmax | Practice: Implement logistic regression from scratch!