STATS / DATA SCI 315

Lecture 02

Basic Elements of Linear Regression

Kinds of ML problems

• Supervised learning: have labeled data
• Regression: Label is a real number
• Classification: Label is chosen from a finite set
• Unsupervised learning: have only unlabeled data
• Reinforcement learning: interact with an environment and receive rewards

Regression

• Modeling the relationship between input variables and a real-valued output
• Input variables also called:
• Features
• Covariates
• independent variables
• Output variable also called:
• Label
• Target
• Dependent variable
• ``Regress y on x” means “Run a regression with x as input, y as output”

Examples

• predicting prices (of homes, think of Zillow’s Z-estimate)
• predicting length of stay (for patients in the hospital)
• demand forecasting (for retail sales)

Linear regression

• A special but important case of regression
• Model the relationship of y, the output variable, as linear in x
• Wish to estimate the prices of houses (in dollars) based on their area (in square feet) and age (in years)
• Linearity assumption: target (price) can be expressed as a weighted sum of the features (area and age):� price = 𝑤_area ⋅ area + 𝑤_age ⋅ age + 𝑏

Weights and bias

• 𝑤_area and 𝑤_age are called weights
• 𝑏 is called a bias (also called an offset or intercept)
• Strictly speaking, our model for price involves an affine transformation
• Affine = Linear + bias
• What makes a model good?
• How do we find good values for the weights and bias?

Training Dataset

• Need a dataset where we know the sale price, area, and age for each home
• This is called a training dataset or training set
• Put one sale info on each row
• Each row is called an example (or data point, data instance, sample)
• Each example has
• A label (price)
• Features (area, age)

Choosing weights and bias based on training data

• Choose the weights and the bias such that our model predictions best fit the true prices observed in the data
• Long form of our linear model:� price = 𝑤_area ⋅ area + 𝑤_age ⋅ age + 𝑏
• If we had d features instead of just two:� 𝑦̂ = 𝑤₁𝑥₁ + … + 𝑤_𝑑𝑥_𝑑 + 𝑏
• The “hat” on top of y denotes that it is an estimate

More compact notation

• Collect all features into a vector 𝐱∈ℝ^𝑑 and all weights into a vector 𝐰∈ℝ^𝑑
• Use dot product to express model compactly:� 𝑦̂ = 𝐰^⊤𝐱 + 𝑏
• Entire dataset of 𝑛 examples is referred to as the design matrix 𝐗∈ℝ^𝑛×𝑑
• 𝐗 contains one row for every example and one column for every feature
• Prediction vector 𝐲̂ ∈ℝ^𝑛 can be expressed via the matrix-vector product:� 𝐲̂ = 𝐗𝐰 + 𝑏