Lecture 07: Diving into Machine Learning on Big Data

Cross-Validation, Python for Linear Regression, and Support Vector Machines

​

Reminder as always: Pull the latest code (I probably made some minor updates)

Upcoming schedule

• Big pivot coming today. Shifting to ML!
• Do the readings, especially, New Tricks for Economists
• Reminder, the last day of class will be virtual.

ML in the news

• Relies on what we will learn today

Cross-validation

In my opinion, this is the most important advance in all of Machine Learning from the perspective of an applied economist.

Cross-validation vs model performance

• In either econometrics or ML, there are two tasks in building a model
1. Estimating parameters of the model
2. Evaluating how well that model does
• Different approaches for these steps between Econometrics and ML:
• Econometrics the above steps involve:
1. t- and p-statistics, hypothesis testing, analyzing specific coefficients
2. R-values, AIC/BIC, etc
• In ML, the emphasis is different. The above steps in ML are:
• Mostly absent in isolation, but included in step 2.
• Determined by cross-validation of the model.

​

Returning to the complexity tradeoff.

• Recall: Overfitting a model is making the model overly complex to that accuracy falls on the test data.
• We will talk about ways to methodologically hit the “sweet spot” of model complexity.
• How do we find this sweet-spot? Cross validation
• First though, let’s illustrate why overly-complex models can UNDER-perform.

Image source: statquest.org

The complex model is super accurate on the training data.

With new data, the complex model is much worse. Notice that the simple model performs about the same.

Simple model

Complex model

The source of the “sweet spot” in model complexity.

We’re going to prove this to ourselves

• Using OLS in ols_overfitting_example.ipynb
• But first, we need a little language on Cross Validation, Splitting of Data, and how ML folks code.

Cross-Validation in Scikit Learn

Splitting data and Cross-Validation

• In this course, we will use �scikit-learn to illustrate this.
• Scikit-learn has nice built-in functions to split our data into training and test data.
• This is the first step of cross-validation approaches.
• We are going to set aside the data and make sure we never use it again until the very end.

​

We train the model on a second split of the data

• To train our model, cross-validation creates a second split of the training data.
• ML algorithms will iteratively try different models/coefficients on this second spit, using whichever performs best on the training-test data.

​

• But we can do MORE than that!

Splitting into MANY Splits and Folds

Final model performance analysis

• Once the best set of parameters are found, the model is compared against the test data from the first split.
• Final performance assessment then is done with calculating the MSE of the model prediction of Test_X for Test_Y
• This method is SUPER FLEXIBLE
• It could compare totally dissimilar models in a rigorous way.
• Will helps us choose “tuning-” or “hyper-parameters”

Switch to VS Code

Open machine_learning/Linear_Regression.ipynb