Why should I trust you?

Explaining the Predictions of Any Classifier

Trusting a model

Widespread adoption of machine learning requires more trust in a model:

• Trusting a prediction - when using model directly
• Trusting a model - when deploying

LIME

Local

Interpretable

Model-Agnostic

Explanations

Explanations make model’s output more reliable

Why not just the test set?

• Test data may not correspond to real-world data - accuracy is often overestimated
• Data leakages
• Validation metric may not be indicative of the model’s goal - it’s hard to evaluate model without the prior knowledge

Two models that for specific input give the same output, may do so for very different reasons.

​

​

​

​

​

​

It’s easy to see the difference and pick better model when given explanations (due to prior knowledge).

Intuition

General framework

To use LIME for a specific model, we need to specify:

• Class of potentially interpretable models G (linear/logistic regression, decision trees, …)
• Measure of model complexity Ω(g) (number of non-zero weights, depth of the tree, …)
• What is the interpretable representation (superpixels, presence of a specific word, ...)
• Proximity measure (weights of training samples for G)
• Loss function (often imposed by choice of G)

Concrete example - sparse linear explanations

Explaining Inception prediction

Explaining model globally

Explaining model globally

Experiments

Are explanations faithful to the model?

Each model (sparse LR and decision tree) is trained to use at most 10 features.

We test how many of these features are recovered by:

• Random
• Parzen
• Greedy
• LIME

Should I trust this prediction?

25% of the features are randomly selected as untrustworthy. Prediction of a model is considered untrustworthy if the prediction changes when untrustworthy features are removed. We test precision and recall of finding untrustworthy predictions.

Can I trust this model?

We add 10 “noisy” features and train pair of random forests with:

• Similar validation accuracy
• Significantly different test accuracy

We test if user can identify the better classifier based on explanations from the validation set. Simulated user picks the model with fewer untrustworthy predictions.

MTurk experiments

Can users select the best classifier?

Users (with no knowledge of machine learning) need to decide between two models:

• SVM trained on the original 20 newsgroups dataset (test accuracy 94%)
• SVM trained on a “cleaned” dataset (test accuracy 88.6%)

Second model is better but it’s not evident from the test score.

​

Can non-experts improve a classifier?

Users (who are unfamiliar with feature engineering) need to identify which words from the explanations are unimportant and should be removed. Then new classifiers are trained.

Do explanations lead to insights?

We train classifier distinguishing between wolves and huskies. In the training set each wolf had snow in the background while huskies did not. Users were asked three questions:

• Do they trust this algorithm to work well in the real world?
• Why?
• What features do they think the algorithm is using?

tinyurl.com/WhyShouldITrustYou