Inference for Modeling

Re-visiting many of the ideas in this class with an inferential perspective.

Data 100, Summer 2020 @ UC Berkeley

Suraj Rampure, John DeNero, Sam Lau, Ani Adhikari

LECTURE 21

Agenda

• Inference.
• Estimators and their bias and variance.
• Bootstrap resampling and confidence intervals.
• Review from Data 8.
• The regression model.
• We were making a set of assumptions when doing regression before.
• Bootstrapping model parameters.
• What do estimators and bootstrapping have anything to do with regression?
• Should tie a lot of ideas in this class together.
• Multicollinearity.
• Not exactly a new idea.

Inference

Prediction vs. inference

Prediction is the task of using our model to make predictions for the response of unseen data.

Inference is the task of using our model to draw conclusions about the underlying true relationship(s) between our features and response.

​

For example, suppose we are interested in studying the relationship between the value of a home and crime rates, a view of a river, school districts, size, income level of community, etc.

• Prediction: Given the attributes of some house, how much is it worth?
• Care more about making accurate predictions, don’t care so much about how.
• Inference: How much extra will a house be worth if it has a view of the river?
• Care more about having model parameters that are interpretable and meaningful.

What is statistical inference?

• There is some fact we want to know about the population. This is called a population parameter.
• Formally, a parameter is a numerical function of a population.
• Accessing the entire population is infeasible (too expensive, too time-consuming), so we collect a random sample of the population.
• We can compute a statistic of the random sample.
• A statistic is a numerical function of a sample.
• However, that sample could have come out differently.
• For example: when we estimate the population mean given the sample mean, our guess is almost always going to be somewhat wrong.
• Inference is all about drawing conclusions about population parameters, given only a random sample.

Terminology

Useful terminology:

• Parameter: Some function of a population (or “data generating process”).
• Denoted with .
• Example: Population mean.
• Estimator: Some function of a sample, whose goal is to estimate a population parameter.
• Denoted with .
• Remember, random variables were functions of random samples.
• Hence, since we sample at random, estimators are random variables.
• Example: sample mean.
• Sampling distribution: The distribution of estimator values, across all possible samples.
• This is unknown: we don’t have access to the population, so we don’t know what all possible samples look like.

​

Bias and variance of an estimator

Bias of an estimator: the difference between the estimator’s expected value and the true value of the parameter being estimated.

• Zero bias (unbiased): on average, our estimate is correct.
• Non-zero bias: on average, our estimate is consistently too large / too small.

​

​

Variance of an estimator: the expected squared deviation of an estimator from its mean.

• The larger the variance of an estimator is, the more it varies from its own average.

​

​

Example: sample mean estimator

What’s the variance of the sample mean estimator?

​

​

​

​

• Note: We use hats to denote estimates.
• Think about how this relates to the “optimal thetas” that we also denoted with hats!
• Where’s the population mean, ?
• Variance isn’t about the population parameter, it’s about the estimator itself.
• We can’t usually compute the expected value or variance of an estimator exactly!
• We’d need access to the true population.

​

​

​

​

If the sample were different

The sample mean would be different

Example: estimating an estimator’s variance

What’s the variance of the sample mean estimator?

​

​

​

​

Impractical approach that would work:

• Draw m different random samples of size n from the population.
• For each sample, apply the estimator (i.e., compute the sample mean).
• Estimate the variance of the estimator using the empirical variance of these estimates.

​

Example: estimating an estimator’s variance

Why is this so impractical?

(demo)

4.3

Bootstrapping

Note

• This section should be review from Data 8.
• The Data 8 textbook does a fantastic job of teaching bootstrapping if you’ve never seen it before.

Bootstrap resampling

• To determine the properties (e.g. variance) of the sampling distribution of an estimator, we’d need to have access to the population.
• We would have to consider all possible samples, and compute an estimate for each sample.
• But we don’t, we only have one random sample from the population.

​

Idea: Treat our random sample as a “population”, and resample from it.

• Intuition: a random sample resembles the population, so a random resample resembles a random sample.

Bootstrap resampling

Bootstrap resampling is a technique for estimating the sampling distribution of an estimator.

(demo)

Impractical:

Sample drawn from the �true sampling distribution of f

A proxy for a sample from the sampling distribution of f

Bootstrapping pseudocode

collect random sample of size n (called the bootstrap population)

initiate list of estimates

repeat 10,000 times:

resample with replacement n times from bootstrap population

apply estimator f to resample

store in list

list of estimates is the bootstrapped sampling distribution of f

Why must we resample with replacement?

Bootstrap discussion

• The bootstrapped sampling distribution of an estimator does not exactly match the sampling distribution of that estimator.
• The center and spread are both wrong (but often close).
• The center of the bootstrapped distribution is the estimator applied to our original sample.
• We have no way of recovering the estimator’s true expected value.
• The variance of the bootstrapped distribution is often close to the true variance of the estimator.
• The quality of our bootstrapped distribution depends on the quality of our original sample.
• If our original sample was not representative of the population, bootstrap is next to useless.

​

Bootstrap confidence intervals

Confidence intervals

• Intuition: Estimate an interval where we think the population parameter is, based on the center and variance of the estimator.
• What does a P% confidence interval mean?
• Imagine the following procedure:
• Take a sample from the population.
• Compute P% confidence interval for the true population parameter, somehow.
• If we repeat this procedure many times, the population parameter will be in our interval P% of the time, in the long run.

​

​

Confidence intervals

How do we compute ci(s,f,P)?

• Approximate the sampling distribution of f using the sample s.
• Choose the middle P% of samples from this approximate distribution.

ci( , f, P) =

About P% of these intervals contain θ*

An estimator f exists in order to guess the value of an unknown parameter θ*.

An estimator ci for a P% confidence interval for f is a function that takes a sample and returns an interval. This interval will (ideally) contain θ* for P% of samples.

Bootstrap confidence intervals

An estimator ci for a P% confidence interval for f is a function that takes a sample and returns an interval. This interval will (ideally) contain θ* for P% of samples.

​

Resample with replacement to approximate the sampling distribution of f

(Demo)

Confidence intervals

• The confidence level is a statement about the procedure used to create our interval.
• It is not the case that a 95% confidence interval means “there is a 95% chance that the population parameter is in our interval”.
• The population parameter is fixed.
• Our interval is fixed. The population parameter is either in it, or it isn’t.
• Nothing random here!
• The confidence intervals we’ve created are sometimes called percentile bootstrap confidence intervals.
• There are other methods of creating confidence intervals.
• E.g. assume that the sampling distribution of your estimator is normal.

The regression model

The regression model

• When fitting a linear regression model, we are assuming there is some underlying true relationship between our features and response.
• But, we never get to see the true relationship.
• We instead see a noisy version of it.

observed response

design matrix

true parameters

errors (assumed to be i.i.d. across observations)

Example: simple linear regression

true linear relationship

random noise

true underlying relationship

observed response

The regression model

• When fitting a linear regression model, we are assuming there is some underlying true relationship between our features and response.
• But, we never get to see the true relationship.
• We instead see a noisy version of it.

​

​

​

​

​

We can observe the quantities in blue. The quantities in red are unobservable.

Our goal is to estimate .

​

observed response

design matrix

errors (assumed to be i.i.d. across observations)

true parameters

Least squares estimation

• We called the optimal model parameter, because it minimized MSE for our training data.
• is an estimate of . Specifically, it is the one that minimizes the training MSE (or some regularized version of it).

​

​

​

• This is nothing new – just a more rigorous statistical treatment of what we’ve already seen.
• Still make predictions as .

Estimator! Takes in a sample, returns an estimate for a population parameter.

Bootstrapping model parameters

Parameter estimates

• Our estimate for depends on what our training data was.
• Different training data, different !
• We want to think about all of the different ways that our training data, and hence our parameter estimate, could have come out.
• Easy!
• Bootstrap our training data.
• Fit a linear model to each resample.
• Look at the resulting distribution of bootstrapped parameter estimates.

Assessing the quality of our model

• Suppose we fit a linear regression model with p features, plus an intercept term.

​

​

​

​

​

• If the true is 0, then the feature x1 has no effect on the response.
• How can we test whether or not ?

how we make predictions

assumed underlying model

Confidence interval for true slope

• We want to test whether is 0.
• We get one estimate from our sample.
• But we must imagine all the other ways the random sample could have come out.
• If the sample is large – bootstrap it!
• Estimate each time.
• Make a confidence interval for and see if 0 is in the interval.
• If yes: is not significantly different than 0.
• If no: is significantly different than 0.
• Can formalize with the language of hypothesis testing, but won’t do so here.
• Works for linear (and logistic!) regression models with any number of features.

(demo)

Multicollinearity

The meaning of “slope”

Consider the equation

​

​

• The slope a1 measures the change in y per unit change in x1, assuming all other variables are held constant.
• We use an equation of the above form for linear regression.
• But what if we can’t hold all other variables constant?

​

​

Multicollinearity

• If features are related to each other, it might not be possible to have a change in one of them while holding the others constant.
• Then, the individual slopes will have no meaning.
• Multicollinearity: when a feature can be predicted fairly accurately by a linear combination of other features.
• Slopes can’t be interpreted.
• Small changes in the data can lead to big changes in the slopes.
• Doesn’t impact the predictive capability of our model – only impacts interpretability.
• Perfect multicollinearity: one feature can be written exactly as a linear combination of other features.
• Design matrix isn’t full rank! Can’t find unique .
• For instance, one-hot encoding with an intercept term.

​

Case study in multicollinearity

(demo)

Summary

Summary

• Estimators are functions that provide estimates of true population parameters.
• We can bootstrap to estimate the sampling distribution of an estimator.
• Using this bootstrapped sampling distribution, we can compute a confidence interval for our estimator.
• This gives us a rough idea of how uncertain we are about the true population parameter.
• Only valid if the original random sample is representative.
• The assumption when performing linear regression is that there is some true parameter theta that defines a linear relationship between features X and response y.
• We can use the bootstrap to determine whether or not an individual feature is significant.
• Multicollinearity arises when features are correlated with one another.

What’s next

• This lecture was a (brief) diversion from the “ML” perspective.
• ML: Make accurate predictions.
• Statistics: Infer about a population.
• So far, we’ve covered supervised learning in great detail.
• Linear regression, logistic regression, decision trees / random forests.
• We will now spend some time talking about unsupervised learning.
• Dimensionality reduction, PCA.
• Clustering.

from Joseph Gonzalez