CSE 519: Data Science
Steven Skiena
Stony Brook University
Lecture 13: Building and Validating Models
The Data Science Analysis Pipeline
Modeling is the process of encapsulating information into a tool which can make forecasts/predictions.
The key steps are building, fitting, and validating the model.
Which is Best?
There are many ways to model any given data set.
How can we decide which approach is better?
Philosophies of Modeling
We need to think in some fundamental ways about modeling to build them in sensible ways.
Occam’s Razor
This philosophical principle states that “the simplest explanation is best”.
With respect to modeling, this often means minimizing the parameter count in a model.
Machine learning methods like LASSO/ridge regression employ penalty functions to minimize features, but also do a “sniff test”.
Bias-Variance Tradeoffs
“All models are wrong, but some models are useful.”
– George Box (1919-2013)
First-principle models likely to suffer from bias, with data-driven models in greater danger of overfitting.
What would Nate Silver do?
Principles of Nate Silver
Live Models
A model is live if it continually updating predictions in response to new information.
Presidential Election Forecast, 2016
Look for Consensus
Boosting is a machine learning technique which explicitly combines an ensemble of classifier.
Google Flu Trends
Predicted flu outbreaks using query frequency of illness terms.
The model failed after Google added search suggestions
Modeling Methodologies
Good models are typically a mixture of both.
Baseline Models
“A broken clock is right twice a day.”
The first step to assess whether your model is any good is to build baselines: the simplest reasonable models to compare against.
Only after you decisively beat your baselines can your models be deemed effective.
Representative Baseline Models
Baseline models must be fair: they should be simple but not stupid.
How Good is Your Model?
After you train a model, you need to evaluate it on your testing data.
What statistics are most meaningful for:
Evaluating Classifiers
There are four possible outcomes for a binary classifier:
Threshold Classifiers
Identifying the best threshold requires deciding on an appropriate evaluation metric.
Accuracy
The accuracy is the ratio of correct predictions over total predictions:
The monkey would randomly guess positive with p=0.5, with accuracy 50%.
Picking the biggest class yields >=50%.
Precision
When |P|<<|N|, accuracy is a silly measure.
If only 5% of tests say cancer, are we happy with a 50% accurate monkey?
The monkey would achieve 5% precision, as would a sharp always saying cancer.
Recall
In the cancer case, we would tolerate some false positive (scares) to identify real cases:
Recall measures being right on positive instances.
Saying everyone has cancer gives perfect recall!
F-Score
To get a meaningful single score balancing precision and recall use F-score:
The harmonic mean is always less than/equal to the arithmetic mean, making it tough to get a high F-score.
Take Away Lessons
Receiver-Operator (ROC) Curves
Varying the threshold changes recall/precision.
Area under ROC is a measure of accuracy.
Evaluating Multiclass Systems
Classification gets harder with more classes.
The confusion matrix shows where the mistakes are being made: 5->3, 8->2
Confusion Matrix: Dating Documents
What periods are most often confused with each other?
The main diagonal is not exactly where the heaviest weight always is.
Summary Statistics: Numerical Error
For numerical values, error is a function of the delta between forecast f and observation o:
These can be aggregated over many tests:
Evaluation Data
The best way to assess models involve out-of-sample predictions, results on data you never saw (or even better did not exist) when you built the model.
Partitioning the input into training (60%), testing (20%) and evaluation (20%) data works only if you never open evaluation data until the end.
Sins in Evaluation
Formal evaluation metrics reduce models to a few summary statistics.
But many problems can be hidden by statistics:
Revealing such errors requires understanding the types of errors your model makes.
Building an Evaluation Environment
You need a single-command program to run your model on the evaluation data, and produce plots/reports on its effectiveness.
Input: evaluation data with outcome variables.
Embedded: function coding current model
Output: summary statistics and distributions of predictions on data vs. outcome variables.
Evaluation Environment Architecture
Designing Good Evaluation Systems
Error Histograms: Dating Documents
Performance of Random vs. Naive Bayes models
Evaluation Environment: Results Table
Stratifying cases by topic and difficulty (length)
The Veil of Ignorance
A joke is not funny the second time because you already know the punchline.
Good performance on data you trained models on is very suspect, because models can easily be overfit.
Out of sample predictions are the key to being honest, if you have enough data/time for them.
Cross-Validation
Often we do not have enough data to separate training and evaluation data.
Train on (k-1)/k th of the data, evaluate on rest, then repeat, and average.
The win here is that you get a variance as to the accuracy of your model!
The limiting case is leave one out validation.
Amplifying Small Evaluation Sets
Scoring Hard Problems Easier
Too low a classification rate is discouraging and often misleading with multiple classes.
The top-k success rate gives you credit if the right label would have been one of your first k guesses.
It is important to pick k so that real improvements can be recognized.
Probability Similarity Measures
There are several measures of distance between probability distributions
The KL-Divergence or information gain measures information lost replacing P with Q:
Entropy is a measure of the information in a distribution.
Evaluation Statistics (Projects)
Blackbox vs. Descriptive Models
Ideally models are descriptive, meaning they explain why they are making their decisions.
Linear regression models are descriptive, because one can see which variables are weighed heaviest.
Neural network models are generally opaque.
Lesson: “Distinguishing cars from trucks.”
Deep Learning Models are Blackbox
Deep learning models for computer vision are highly-effective, but opaque as to how they make decisions.
They can be badly fooled by images which would never confuse human observers.
Correlation Does Not Imply Causation
Levels of Modeling
Interesting problems usually exist on several different levels, each of which require independent submodels.
Predicting the future price for a stock should involve submodels for analyzing (a) the general state of the economy, (b) its balance sheet, (c) the performance of its industrial sector, ...
Hierarchical Decomposition
Imposing a hierarchical structure on the model permits it to be built and evaluated in a logical and transparent way, instead of as a black box.
Often subproblems lend themselves to theory-based, first-principle models, which can then be used as features in a data-driven general model.
Simulation Models
“What I cannot create, I do not understand” (Feynman)
Monte Carlo simulation is the key to modeling systems of discrete events.
Our jai-alai betting system simulated games using
Levels of Modeling (Projects)