Machine Learning Mechanics – �Terminology and Techniques

Lecture 3

Framing ML Problems and ML Techniques and Terminologies

EECS 189/289, Fall 2025 @ UC Berkeley

Joseph E. Gonzalez and Narges Norouzi

EECS 189/289, Fall 2025 @ UC Berkeley

Joseph E. Gonzalez and Narges Norouzi

EECS 189/289, Fall 2025 @ UC Berkeley

Joseph E. Gonzalez and Narges Norouzi

Join at slido.com�#2826097

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

2826097

Goals For Today

• Introduce major concepts in using machine learning.
• Provide a high-level understanding but not rigorous
• We will revisit all the topics more formally later in the semester
• Show you how to do basic machine learning in Python.
• Visit each step of the Machine Learning Lifecycle.
• Prepare you for Homework 1.

​

2826097

Today We Introduce Scikit-Learn

Widely used python �package for:

• Data Prep
• Feature Engineering
• Classic Models
• Evaluation

in Machine Learning.

2826097

Which of these libraries have you used before?

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

2826097

ML Lifecycle

L

M

P

O

LEARNING PROBLEM

PREDICT & EVALUATE

MODEL DESIGN

OPTIMIZATION

2826097

ML Lifecycle

L

LEARNING PROBLEM

• Target:
• What do I want to predict?
• What is the machine learning task?
• Objective:
• How would I evaluate success?
• What loss should I use?
• Data:
• What data do I have?
• Data representation?
• Training/Test split

​

2826097

Example: FashionHub

We are launching a new fashion trading website where people can upload pictures of clothing they want to trade.

• We want to automatically tag the clothing into categories based on what sellers upload.
• We have some example clothing pictures with category labels.

What do we want to predict?

What data do we have?

How would we evaluate success?

Type of Clothing

Labeled Pairs

Overall Accuracy?

2826097

ML Lifecycle

L

LEARNING PROBLEM

• Target:
• What do I want to predict?
• What is the machine learning task?
• Objective:
• How would I evaluate success?
• What loss should I use?
• Data:
• What data do I have?
• Data representation?
• Training/Test split

​

Understand the Data!

(Look at the data!)

2826097

How to Look at the Data

The Data

Labels

(Optional)

Features

2826097

Demo

Understanding the

Machine Learning Problem

and

Looking at the Data

2826097

Taxonomy of�Machine Learning

Supervised Learning

Labeled Data

Quantitative�Label

Categorical�Label

2826097

Supervised Learning�Learning from examples (demonstrations)

The “training data” consists of examples of a functional relationship.

• Most commonly-used learning process.
• The “fastest” way to learn – requires �the least data.
• Requires examples of the relationship.
• May not always be easy to obtain.

​

​

Supervised Learning

Quantitative�Label

Categorical�Label

2826097

Classification Problems

The labels are discrete classes or categories.

• Example: what type of clothing is in the image

Discrete labels are often “encoded” as numbers (but should be treated as classes)

• Example: 0=T-shirt/top, 1=Trousers, 2=Pullover

Two main types of classification problems:

• Binary Classification: two classes (spam vs. not spam)
• Multi-class Classification: more than two classes (e.g., a food prediction* – Hotdog, Pizza, Burrito, …).

*Funny scene in silicon valley where a binary classifier was trained for food prediction.

2826097

ML Lifecycle

L

LEARNING PROBLEM

• Target:
• What do I want to predict?
• What is the machine learning task?
• Objective:
• How would I evaluate success?
• What loss should I use?
• Data:
• What data do I have?
• Data representation?
• Training/Test split

​

Understand the Data

✅

Learning�Setting

✅

Train-Test

Split

2826097

How will we evaluate the model?

We are about to train a model using data.

How will we know if the model has “learned” from our data?

• Could we measure how well our model fits the data?

​

​

​

​

​

Why is memorizing the data (exam) not good?

​

The Exam Analogy:

What would happen if we gave everyone access to the exam and solutions (the data) to study (train) for the exam?

• Would everyone do well?
• Does this mean they learned the material in the class?

2826097

Generalization

Generalization in machine learning is the ability of a model to perform well on new, unseen data sampled from the same distribution as its training data.

​

To evaluate generalization,�we need a method to evaluate�the model’s performance on �new data not used for �training.

2826097

Evaluating generalization using the�Train-Test Split

The train-test split is the standard technique we use to evaluate generalization in machine learning:

1. Shuffle the training data
2. Split into two parts:
• Larger Training Part (~80%): used to �develop and train the model.
• Smaller Testing Part (~20%): used to �evaluate generalization performance.

Data

Test

You should only use the test dataset once �after developing and training the model.

Why?

If you use the test data to tune the model,�the test data no longer measures generalization.

2826097

Evaluating generalization using the�Train-Test Split

The train-test split is the standard technique we use to evaluate generalization in machine learning:

1. Shuffle the training data
2. Split into two parts:
• Larger Training Part (~80%): used to �develop and train the model.
• Smaller Testing Part (~20%): used to �evaluate generalization performance.

Data

Test

You should only use the test dataset once �after developing and training the model.

Why?

If you use the test data to tune the model�the test data no longer measures generalization.

But what if I want to peek at the test data�to tune for better generalization?

2826097

The Validation Split

The validation dataset is used to evaluate generalization performance during the model development process.

Data

Test

Train - Val.

Split

Train

Val.

Fit (train) the model using the training data.

Tune the model design using the validation data.

2826097

The Train-Validation-Test Split Analogy

You can think of the train, validation, and test splits as how you might study for an exam.

Test

Train

Val.

Practice question and answer pairs that you use to study for the exam. (Go to discussion!)

Practice exam evaluates if your studying process is working (or if you need to study more).

The exam is how we evaluate if you should pass the class.

2826097

Demo

Train-Test-Validation Splits

2826097

ML Lifecycle

L

M

LEARNING PROBLEM

MODEL DESIGN

• Feature Engineering
• Model family/Architecture
• Hypothesis space
• Inductive biases / Assumptions

• Target
• Objective
• Data

2826097

Feature Engineering

Feature engineering is the process of selecting and encoding input features from the raw features.

• Selecting Features: including the right features can help improve model performance.
• Adding new features from other data sources can improve performance
• Too many features can be harmful … when data is limited
• Soon we will explore techniques to automatically select features
• Encoding Features: some features may need to be transformed into the appropriate numeric representation.
• Categorical data, text, images, … often require transformations
• The core innovation in deep learning is learning features encodings.

2826097

Encoding Numerical Data

Numerical features (numbers) are often used without modification.

However, there are a few important exceptions:

• Categorical Features: ZIP Code, a product SKU, or a numerical coding of a string (e.g., “red” = 1, “blue” = 2, …)
• These are typically one-hot-encoded

2826097

One-Hot Encoding

One-hot Encoding takes a categorical feature (e.g., color, shape) and generates multiple binary features (one for each possible value) and assigns 1 to the feature column with the original value.

2826097

Encoding Numerical Data

Numerical features (numbers) are often used without modification.

However, there are a few important exceptions:

• Categorical Features: ZIP Code, a product SKU, or a numerical coding of a string (e.g., “red” = 1, “blue” = 2, …)
• These are typically one-hot-encoded
• Heavily Skewed Features: click counts, user content, pricing, or other situations where features can have extreme values.
• Often apply log transformations
• Feature Standardization: features with different magnitudes and variability can complicate modeling and optimization
• Typically apply standardization

2826097

Feature Standardization

2826097

Encoding Text Data

There are several methods for encoding a string of text:

• One-hot-Encoding: For categorical strings like color, state, name, etc. we often use one-hot-encodings.
• Bag-of-Words Encoding: The classic method for encoding multi-word strings (e.g., email, messages, etc.) is to use a bag-of-words.
• Demonstrate in a moment.
• Learned Vector Embeddings: Today we often use large language models to convert strings of text to fixed vectors.
• We will see how to do this and how these methods encode text as tokens later in the course.

All techniques produce a high-dimensional vector representation of a string.

​

​

2826097

Bag-of-words Encoding

Each word in the vocabulary is encoded as a separate column and the occurrence or count of that word is stored in each column:

​

​

​

​

​

Stop words (e.g., the, is, of…) that contain minimal information are often dropped from the vocabulary.

​

2826097

In graduate school, Prof. Gonzalez moved into a new building that had no art.

​

So, he secretly installed this “art piece” on the ML floor of the new building.

​

Do you see the stop word?

​

There used to be a dustbin and broom

… but the janitors got confused …

New buildings need

ML inspired art!

2826097

Encoding Image Data

Images can be thought of as 3 dimensional tensors:

• Flatten the Image Tensor: Probably the most �naïve representation but can be effective for �smaller monochromatic images. (homework 1)
• Transformations: Color space and pixel normalization
• Hand Craft Features: Edge detectors, texture descriptors
• Deep Learning Representations: Use neural networks to learn embeddings
• More on this later in the course

All techniques produce a high-dimensional vector representation of an image.

Color

Channel

Width

Height

2826097

Featurization with Scikit-Learn

Scikit-Learn has a large collection �of data transformations to aid in �the feature engineering process.

​

Many of them have the form:

from sklearn.preprocessing import OneHotEncoder

ohe = OneHotEncoder() # Constructor

ohe.fit(df[["color"]]) # Fit

ohe.transform(df[["color"]]) # Transform

https://scikit-learn.org/stable/data_transforms.html

2826097

Demo

Feature Engineering

2826097

ML Lifecycle

L

M

LEARNING PROBLEM

MODEL DESIGN

• Feature Engineering
• Model family/Architecture
• Hypothesis space
• Inductive biases / Assumptions

• Target
• Objective
• Data

Feature

Engineering

✅

Model

Family

2826097

Machine Learning�as Function Approximation

Feature

Vectors

Output (Y)

Input (X)

Learned�Function

(Learned?)

Featurization

2826097

Machine Learning�as Function Approximation with �(Learned) Features

Feature

Vectors

Output (Y)

Input (X)

Learned�Function

(Learned?)

Featurization

2826097

Generic Function Approximation

Input (X)

Feature

Vectors

Output (Y)

Learned�Function

(Learned?)

Featurization

2826097

The Linear Regression Model Family

The hypothesis space is all lines parametrized by real-valued slopes and intercepts.

2826097

Inductive Biases

The set of assumptions made in the model design to enable generalization beyond the training data.

No Free Lunch Theorem in ML: there is no universally best model – need to choose the model with the right inductive biases.

Example:

Infinitely many models fit the data

• Any of them could be correct!

Choosing the model family (e.g., linear) is introducing an inductive bias.

2826097

Inductive Biases in �Features and the Model

Feature

Vectors

Output (Y)

Input (X)

Featurization

2826097

What inductive bias assumptions are made with the bag-of-words encoding?

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

2826097

Complexity:�Linear Models vs Non-Linear Models

Nonlinear models are more expressive and can represent more complex relationships.

Classification (Decision Boundary)

Decision Boundary

Are non-linear models better?

2826097

Complexity and Overfitting

“Sweet Spot”

Overfitting

Underfitting

Raw data

Fit models to samples of data

More complex isn’t always better.

2826097

Regularization

Regularization is the process of adding constraints or penalties to the learning process to improve generalization.

​

​

​

​

​

�

Many models and learning algorithms have methods to tune the regularization during the training process.

• We will see this in the optimization phase of the ML Lifecycle.

​

Overfitting

Underfitting

More Regularization

“Sweet Spot”

2826097

Parametric vs. Non-Parametric Models

Training Data

2826097

Choosing the Model Family

1. Determine the learning problem: Classification, Regression, Clustering, Dimensionality Reduction
2. Start with linear models and good features
• Linear models + features eng. is a common way to encode inductive biases
3. Trying increasingly complex models and check validation performance

​

​

• Today’s lecture focuses on models in the Scikit-learn package.
• Future lectures we will explore deep learning model design.

2826097

Logistic Regression Model Family

2826097

ML Lifecycle

L

M

O

LEARNING PROBLEM

MODEL DESIGN

OPTIMIZATION

• Target
• Objective
• Data

• Model family/Architecture
• Hypothesis space
• Inductive biases / Assumptions

• Iterative Opt. Algorithms
• Hyperparameter Tuning

2826097

Iterative Optimization Algorithms

Lecture 5

Lecture 6

Lectures 10-11

2826097

Hyperparameters

for hp1 in [0.1, 1, 10]:

for hp2 in [0.1, 1, 10]:

model = MyModel(hp1=hp1, hp2=hp2)

model.fit(X_train, y_train)

error[hp1, hp2] = error(y_val, model.predict(X_val))

hp1_best, hp2_best = arg_min(error)

2826097

Training vs Validation Accuracy

When tuning regularization hyperparameters against the validation dataset, it is common to see plots where

• training acc. is greater than validation accuracy
• acc. on the training dataset continues to increase.
• acc. on the validation dataset increases and then decreases.

Better Fit

More Regularization

Sweet Spot

Sweet Spot

2826097

ML Lifecycle

L

M

P

O

LEARNING PROBLEM

PREDICT & EVALUATE

MODEL DESIGN

• Target
• Objective
• Data

• Model family/Architecture
• Hypothesis space
• Inductive biases / Assumptions

OPTIMIZATION

• Iterative Opt. Algorithms
• Hyperparameter Tuning

• Making predictions
• Accuracy Metrics

2826097

Making Predictions

Inference is the process of making predictions with a model.

• Label Prediction (e.g., the class or regressor value) �represent the most likely value given the data but �doesn’t reflect the uncertainty in the prediction.
• sklearn: model.predict()

​

• Predicted Distribution (e.g., the class probabilities �or mean and variance for regression) encodes the �uncertainty in the predicted value.
• sklearn: model.predict_proba() �# for classification models

​

Many modern techniques predict distributions (e.g., ChatGPT).

​

2826097

Evaluation Metrics

How do we measure success of a model at test time?

• Accuracy or some other error metric on the test data

Are all types of error equal?

​

​

​

• Falsely classifying something as spam has risks

Often use decision theory to make decisions from probabilities.

• Need to quantify the costs of decisions not just the model.

​

​

​

​

​

​

2826097

Demo

Scikit Learn Classification

2826097

Machine Leaning Mechanics – �Terminology and Techniques

Lecture 3

Credit: Joseph E. Gonzalez and Narges Norouzi

Reference Book Chapters: Chapter 1

Homework!

Berkeley's favorite pastime

2826097

Homework Outlines

• Written questions (similar to previous years)
• Coding: implementing concepts from lecture, setting up datasets used in part 2

• Paper reading + written questions
• We will teach you how to read papers before HW2
• Paper implementation + extensions of the paper

​

Part 1: Lecture application

Part 2: Paper implementation

2826097

Homework in the age of AGI

• Use AI – it is incredibly helpful
• Rule of thumb:
• Use AI when you can easily verify
• Visualizations (more about this in HW2)
• Documentation lookup
• When you cannot verify – do not use AI
• Writing functions or applications that you don’t know how to approach
• If you cant write tests or a design doc, don’t vibe code it
• Be careful with git/functions which have write access to your machine

​

​

2826097

Downloading Homeworks

• All content (discussions, lectures, homeworks) are in the BerkeleyML/fa25-student repo
• Content downloader notebook will download the content for you (either locally or in colab)
• Content will also be in the shared Google Drive (including downloader notebook)

2826097

Homework 1

Part 1 - due Sep 19th

• Written questions – prereqs 
• linear algebra, calculus, and probability
• Recommend not using AI – you will use these techniques throughout the class
• Coding: basic ML pipelines
• Pandas, plotly, scikit learn, image transformations
• Do not procrastinate – the later problems are much harder than the earlier problems

Part 2: due Sep26th

• No paper this week
• we will provide some fun papers to look at on Ed
• Coding: improving models 
• Your model from part 1 is now failing on a secret test set - how can we improve performance without any fancy model techniques?

2826097

Fun paper about train/test splits

ImageNetV2 was collected in the same way as ImageNet but ImageNet models achieve a lower accuracy

∗Authors ordered alphabetically. Ben did none of the work.

2826097

Fun paper about train/test splits

∗Authors ordered alphabetically. Ben did none of the work.

ImageNetV2 was collected in the same way as ImageNet but ImageNet models achieve a lower accuracy

2826097

Have Fun!!!