Meme of the Day
CSE 30124�Linear Regression
Previous Models
k-Nearest Neighbors | Decision Trees | Naive Bayes
Class 1
Class 2
Class 3
Classification Tasks
A classification task is one in which we try to assign a _________________ to a sample.
Class 1
Class 2
Class 3
?
Regression Tasks
In regression tasks, we try and assign a _____________________ to each sample.
Regression is another form of ____________ ______________
Is a regression task supervised or unsupervised learning? (Old Exam Question)
Linear Regression
A linear regression is a technique to fit a line to our data, allowing us to make predictions
Regression specific Vocab
Unfortunately, because linear regression is so old, it has a lot of vocab specific to it
ŷ = 𝛽1X1 + 𝛽0
ŷ:
𝛽1, 𝛽0:
X1:
NOTE: We’re going to replace 𝛽N with ⍵ today
Matrix Notation
We can represent basically everything as matrices and matrix operations.
Multiple Features
It’s rare that we have datasets with only a single feature. The iris dataset, a toy dataset, has 4!
When we perform a linear regression on a dataset with more than one feature, we call this a _________________________
Hyperplanes
y =
What would the equation look like for a multiple linear regression with three features? (Old Exam Question)
Fitting our line (exercise)
How do we actually find the line that “fits” the best?
Optimization Problems
We need some way to measure how well our model fits our data. We can call this a ______ __________
For linear regression we often use SSE or ____________________
Finding our weights
So given a loss function, our goal is to _____________ it
When would a loss function be minimized? When the ____________ is ______________
Visualizing Loss
We can plot the loss function with respect to w
Finding our weights (OLS)
We can solve for the gradient equaling 0
This is called Ordinary Least Squares (OLS)
Representing Curves (exercise)
How could you use a linear regression to model a sine wave?
A Possible Issue
The weight calculation relies on computing the inverse, which can be extremely computationally expensive and sometimes just literally impossible
Maximum Likelihood Estimation (MLE)
Maximum likelihood estimation (MLE) is a method of estimating the parameters of an ________________________________, given some observed data
Overfitting
Unfortunately, Linear Regressions (a form of MLE), are prone to overfitting
Regularization - L1 and L2
The goal of regularization is to encourage models to learn the broader patterns within the data rather than memorizing it.
To help prevent overfitting when using linear regressions, we can perform a technique called regularization
L2 Regularization - Ridge
L2 regularization (called ridge regularization) encourages ___________________________ weights
L1 Regularization - Lasso
L1 regularization (also called LASSO) leads to __________________________ and sparse models
MAP Estimation
L1 and L2 regularization are a form of MAP Estimation.
MAP stands for ________________________ which is a Bayesian approach for applying a prior to our weights
Underfitting
Underfitting is when your model is “too weak” to capture all of the underlying patterns in the data.
Bias and Variance
the bias–variance tradeoff describes the relationship between a model's complexity, the accuracy of its predictions, and how well it can make predictions on unseen data that were not used to train the model
Given the dataset above, answer the following questions (Old exam question):
“Direct Fit” Models
All of the models we’ve seen so far could be called “direct fit” models. There’s not really a true “learning” loop occurring here, we’re really just performing a single pass.
Next class we’re going to look at a model that “learns” - The Perceptron