Chandan Singh

Useful interpretations for machine learning

in science & medicine

Advisor: Bin Yu

1. Defining interpretability
2. Interpreting neural nets
3. Cell-biology problem
4. Interpretable distillation
5. Rule-based models

Interpretable ml extracts useful, relevant knowledge

Murdoch*, CS*, Kumbier, Abbasi-Asl, & Yu

​

PNAS, 2019

Problem, data, audience

Model

Post hoc analysis

Iterate

Relevancy

• Defining interpretability
• Interpreting neural nets
• Cell-biology problem
• Interpretable distillation
• Rule-based models

​

​

Existing methods don’t capture interactions well (e.g. LIME, SHAP)....

Interpreting + improving neural networks

Hierarchical interactions: ICLR ‘19

CS*, Murdoch*, & Yu

Interaction regularization: ICML ‘20

Rieger, CS, Murdoch, & Yu

Transformation importances: ICLR Workshop ‘21

CS*, Ha*, Lanusse, Boehm, Liu, & Yu

Pred

Original input

Relevant part

Irrelevant part

CD importance of very good

Relevant (at layer i)

​

​

Linear / conv

Irrelevant (at layer i)

Relu

Pooling

Importances work for

transformations via a reparameterization trick

Evaluation 1: human accuracy at identifying a corrupted model

​

Human accuracy

Evaluation 2: penalizing importances improves performance

• Defining interpretability
• Interpreting neural nets
• Cell-biology problem
• Interpretable distillation
• Rule-based models

Molecular-partner prediction

with interpretable neural networks

​

​

CS*, Li*, Ruan, Song, Dang, He, Upadhyayula, & Yu

Clathrin-mediated endocytosis

Tracking molecular partners is a central challenge

...but it’s experimentally difficult

Fitting a network for spike prediction

Prediction

Interpreting individual predictions with CD

Positive

Negative

• What is interpretability?
• Interpreting neural nets
• Cell-biology problem
• Interpretable distillation
• Rule-based models

DALLE-2

Ha, CS, Lanusse, Upadhyayula, & Yu NeurIPS `21

Our interpretation is incomplete

Thousands of parameters….

Solution: distillation

Prediction

<20 parameters!

Wavelet transform

z

Wavelet function

Wavelet function can vary

x

z

Orthonormal basis under these conditions (mallat, 1998):

Haar

Mexican hat

db4

sym4

Adaptive wavelet

x

z

Adaptive wavelet + distillation

AWD is accurate and interpretable

R²

AWD works on a completely different problem

• Defining interpretability?
• Interpreting neural nets
• Cell-biology problem
• Interpretable distillation
• Rule-based models

Clinical-decision rule modeling

Kornblith*, CS*, Devlin, Addo, Streck, Holmes,

Kuppermann, Grupp-Phelan, Fineman, Butte, & Yu

FIGS: Fast interpretable greedy-tree sums

​

Tan*, Singh*, Nasseri, Agarwal, & Yu

Number of rules

FIGS

Trees compete with each other to predict the outcome

+

+

Singh*, Nasseri*, Tan, Tang, & Yu

JOSS, ‘21

: a python library for interpretability

Acknowledgements

Acknowledgements

Thesis committee

Internship advisors

Undergrad advisors

Yu Group

Berkeley friends

Old friends

Family

Thank you

• Defining interpretability
• Interpreting neural nets
• Cell-biology problem
• Interpretable distillation
• Rule-based models

Interpreting individual predictions with CD

Positive

Negative

Positive

Negative

Unsure

Time

Random stuff

w/ Alejo Rico-Guevara, Xin Cheng

w/ Gang-Yu Liu, Jiali Zhang

​

​

w/ Mike Eickenberg, Reza Abbasi-Asl, Mike Oliver

​

​

w/ Summer Devlin, Jamie Murdoch

w/ Raaz Dwivedi, Martin Wainwright

w/ Yu-group

w/ James Duncan, Rush Kapoor, Sahil Saxena

Hierarchical shrinkage for trees

Agarwal*, Tan*, Ronen, Singh, & Yu

arXiv, submitted to ICML ‘22

Covid county-level

forecasting / data curation

Yu Group + Response4Life, HDSR 2021

Evaluating interpretation: simulations

Training

Importance

Error (%)

FIGS can potentially improve clinical decision rules

• What is interpretability?
• Interpreting neural nets
• Molecular partner prediction
• Distilling interpretable models
• Rule-based models

Ha, Singh, Lanusse, Upadhyayula, & Yu, NeurIPS `21

• What is interpretability?
• Interpreting neural nets
• Molecular partner prediction
• Distilling interpretable models
• Rule-based models

DALLE-2

bin yu

jamie murdoch

karl kumbier

reza

abbasi-asl

aaron kornblith

françois lanusse

​

wooseok ha

vanessa boehm

gokul

upadhyayula

​

xiongtao

ruan

xiao

li

yu group

yan shuo tan

Raaz Dwivedi

laura rieger

Definitions

Interpretability is the “ability to explain or to present in understandable terms to a human” (doshi-velez & kim, 2017)

Related to trust, causality, transferability, informativeness, fairness (lipton 2017)

“Explanations are... the currency in which we exchanged beliefs” (lombrozo 2006)

​

Interpretations should be useful

Improve predictive accuracy

Uncertainty

Make causal recommendations

Original input

Baseline importance of very good

Molecular partners underlie biological processes

Images by DALLE 2

Predicting via peak counts

Predict using nearest-neighbor

Filtered values

at the local maxima

Binning into histogram

Ex: Pneumonia-death prediction

​

​

​

​

​

Caruana et al. 2015

Spurious correlation: asthma correlated with lower risk in the training data

cell biology

science &

medicine

cosmological inference

neuroscience

clinical decision

rules

Back to biology: interpreting many predictions

Back to biology: interpreting a DNN prediction

transformation importance

singh, ha, lanusse, boehm, & yu

ICLR 2020 workshop

evaluating (hierarchical) CD importance

qualitative

human experiments

improving models

recovering groundtruth

and more...

Evaluating interpretation: penalizing scores improves performance