Right for the Right Reasons:

​

Training Differentiable Models by Constraining their Explanations

​

Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Velez

August 24, 2017, IJCAI, Melbourne

​

doi.org/10.24963/ijcai.2017/371

Code & data: github.com/dtak/rrr

These slides: goo.gl/fMZiRu

​

Models don’t always learn what you think they learn

[Ribiero et al., ACM 2016]

Models don’t always learn what you think they learn

​

​

[Ribiero et al., ACM 2016]

The picture in my head

Parameter space

​

Loss function

Easily-found

local minimum

as snow detector

Much subtler

global minimum

as husky vs. wolf

face distinguisher

What are explanations?

What are explanations?

(let me try to explain...)

What are explanations?

(let me try to explain...)

[Keil, 2006] [Miller, 2017]

[Doshi-Velez and Kim, 2017]

[Biran and Cotton, 2017]

​

​

What are explanations?

One approach: interpretable surrogates

(let me try to explain...)

[Ribiero et al., ACM 2016, again]

[Keil, 2006] [Miller, 2017]

[Doshi-Velez and Kim, 2017]

[Biran and Cotton, 2017]

​

​

What are explanations?

One approach: interpretable surrogates

Another: gradients of output probabilities

with respect to input features

[Baehrens et al., JMLR 2010]

(let me try to explain...)

[Ribiero et al., ACM 2016, again]

[Keil, 2006] [Miller, 2017]

[Doshi-Velez and Kim, 2017]

[Biran and Cotton, 2017]

​

​

What are explanations?

[Baehrens et al., JMLR 2010]

[Ribiero et al., ACM 2016, again]

One approach: interpretable surrogates

Another: gradients of output probabilities

with respect to input features

Actually quite similar!

(let me try to explain...)

[Keil, 2006] [Miller, 2017]

[Doshi-Velez and Kim, 2017]

[Biran and Cotton, 2017]

​

​

Input gradients for image classifications

[Smilkov et al. 2017]

​

This kind of works!

​

So, we’re done, right?

​

This kind of works!

​

So, we’re done, right?

...what do we do if the explanations are wrong?

Optimizing for the right reason

Optimizing for the right reason

X, y

Training Examples X, y

Loss Function

θ

Model Parameters θ

θ′

L(θ|X, y)

Model that predicts y given X

Traditional ML

Optimizing for the right reason

X, y

Training Examples X, y

Loss Function

θ

Model Parameters θ

θ′

L(θ|X, y, A)

Model that predicts y given X because A

Explanation Annotations A

A

Traditional ML + explanation regularization

(Ideally, only need a small subset of X to have extra annotations)

Case 1: Annotations are given

How we encode domain knowledge

Features

​

Examples

Signifies that second feature of first example should be irrelevant to model’s prediction

Annotation Example

 A = 1 

 A = 0 

Our loss function

Our loss function

Our loss function

With some overall strength,

Our loss function

With some overall strength,

if a particular example

Our loss function

With some overall strength,

if a particular example’s feature

Our loss function

With some overall strength,

if a particular example’s feature

is marked irrelevant,

Our loss function

With some overall strength,

if a particular example’s feature

is marked irrelevant,

then penalize,

Our loss function

With some overall strength,

if a particular example’s feature

is marked irrelevant,

then penalize,

when the feature changes,

Our loss function

With some overall strength,

if a particular example’s feature

is marked irrelevant,

then penalize,

how much the prediction changes.

when the feature changes,

Experiments

Basic philosophy:

• use or create datasets that we know can by classified with qualitatively different rules
• see if we can use explanations to “select” which implicit rule the model learns

Models we used:

• 2 hidden layer fully connected network, but method works for CNNs and larger models

Experiments: Toy Colors

Class 1

Class 2

All colors

shared

At least two different

Experiments: Toy Colors

Class 1

Class 2

All colors

different

At least two shared

Learning an otherwise-unreachable rule

By default, model appears to learn corner rule.

If we penalize corners, model discovers top-mid rule!

Pixels w/ largest magnitude gradients

How regularization strength affects what we learn

Smooth transition between model learning each rule!

(Can transition with 10s of annotations if we oversample in minibatches)

Experiments: Decoy MNIST

Swatch shades a simple function of y in train, but not in test.

Experiments: Decoy MNIST

Normal model has low accuracy; gradients focus on swatches

+

= increasing pixel increases predicted label prob

​

​

-

= increasing pixel decreases predicted label prob

​

​

Experiments: Decoy MNIST

Model with gradient regularization recovers baseline accuracy!

+

= increasing pixel increases predicted label prob

​

​

-

= increasing pixel decreases predicted label prob

​

​

Case 2: What if we don’t have annotations?

Find-another-explanation

Find-another-explanation

Overall goal: obtain an ensemble of models that are all accurate but for different reasons.

Find-another-explanation

X, y

Training examples

Overall goal: obtain an ensemble of models that are all accurate but for different reasons.

Find-another-explanation

X, y

Training examples

θ₁

X, y

Overall goal: obtain an ensemble of models that are all accurate but for different reasons.

Find-another-explanation

• Take 1st model gradients of full training set
• Set A_n,d = 1 where |∂f/∂X_nd| is relatively large

X, y

Training examples

θ₁

X, y

Overall goal: obtain an ensemble of models that are all accurate but for different reasons.

A₁

Find-another-explanation

Penalize 2nd model from having large gradients where 1st model did!

• Take 1st model gradients of full training set
• Set A_n,d = 1 where |∂f/∂X_nd| is relatively large

X, y

Training examples

θ₁

θ₂

X, y

X, y

A₁

Overall goal: obtain an ensemble of models that are all accurate but for different reasons.

A₁

Find-another-explanation

Penalize 2nd model from having large gradients where 1st model did!

• Take 1st model gradients of full training set
• Set A_n,d = 1 where |∂f/∂X_nd| is relatively large

X, y

Training examples

θ₁

θ₂

X, y

X, y

A₁

Overall goal: obtain an ensemble of models that are all accurate but for different reasons.

A₂

A₁

Find-another-explanation

Penalize 2nd model from having large gradients where 1st model did!

• Take 1st model gradients of full training set
• Set A_n,d = 1 where |∂f/∂X_nd| is relatively large

3rd model should be dissimilar from both 1st and 2nd!

X, y

Training examples

θ₁

θ₂

θ₃

X, y

X, y

X, y

A₁

Overall goal: obtain an ensemble of models that are all accurate but for different reasons.

A₁ ∪ A₂

A₂

A₁

Find-another-explanation

Penalize 2nd model from having large gradients where 1st model did!

• Take 1st model gradients of full training set
• Set A_n,d = 1 where |∂f/∂X_nd| is relatively large

3rd model should be dissimilar from both 1st and 2nd!

X, y

Training examples

θ₁

θ₂

θ₃

X, y

X, y

X, y

A₁

A₁ ∪ A₂

A₂

Overall goal: obtain an ensemble of models that are all accurate but for different reasons.

And so on...

A₁

Back to the picture in my head

Parameter space

​

Loss function

Iterate through local minima

Find-another-explanation: Toy Colors

Model initially learns corner rule, falls back to top-three rule, then fails to learn anything.

Find-another-explanation: Decoy MNIST

Models initially learn decoy rule, then use other features.

​

Accuracy falls, but very slowly (MNIST is redundant)

Summary / Contributions

For when learning from X, y alone is insufficient:

• Introduced a novel method of injecting domain knowledge into NN training
• Works for any differentiable model, no need to modify architecture
• Can start using it with a small number of annotated examples
• Demonstrated how it can be used to obtain otherwise unreachable models
• If we have domain knowledge, we can use it to avoid fitting to spurious correlations
• If we don’t, we can obtain a diverse ensemble of models

Summary / Contributions

For when learning from X, y alone is insufficient:

• Introduced a novel method of injecting domain knowledge into NN training
• Works for any differentiable model, no need to modify architecture
• Can start using it with a small number of annotated examples
• Demonstrated how it can be used to obtain otherwise unreachable models
• If we have domain knowledge, we can use it to avoid fitting to spurious correlations
• If we don’t, we can obtain a diverse ensemble of models

May be more common than we think!

Future Work

• Human-in-the-loop
• Interactively select the best explanations, train new models
• Bridging features and concepts
• E.g. for images, “concepts” are only emergent at upper layers
• If we can identify concepts like in [Bau et al. CVPR 2017], regularize wrt concepts?
• Explore more options for loss functions and annotations
• Use non-binary A, L1 regularization, class specific positive/negative penalties rather than sum
• Much bigger networks
• Have already validated the approach for mid-size CNNs, but I’m a newbie
• Defending against adversarial perturbations
• Have results that setting A=1 universally builds robustness to FGSM and JSMA attacks
• Applications to medical domain
• Many types of medical knowledge are easily encodable as annotations

Future Work

• Human-in-the-loop
• Interactively select the best explanations, train new models
• Bridging features and concepts
• E.g. for images, “concepts” are only emergent at upper layers
• If we can identify concepts like in [Bau et al. CVPR 2017], regularize wrt concepts?
• Explore more options for loss functions and annotations
• Use non-binary A, L1 regularization, class specific positive/negative penalties rather than sum
• Much bigger networks
• Have already validated the approach for mid-size CNNs, but I’m a newbie
• Defending against adversarial perturbations
• Have results that setting A=1 universally builds robustness to FGSM and JSMA attacks
• Applications to medical domain
• Many types of medical knowledge are easily encodable as annotations

These slides again: goo.gl/fMZiRu

Learning with less data?

Best if “right answers” term ≈ “right reasons” term

Gradients are consistent with LIME but less sparse

Setting A = 1 for all features (“certainty regularization”)

...for certainty-regularized CNN:

Sum-of-log-prob input gradients for normal CNN:

Certainty-regularized CNN also much more resistant to adversarial perturbations

Input images:

Information-Theoretic Interpretation of Loss Function

= K * Cross-entropy of prediction w/ uniformly random guess

​

“distance from total uncertainty”