1 of 51

Interpretable and Explainable Machine Learning

Unraveling the Complexities

AML Spring 2026

3 of 51

COMPAS, ProPublica

COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) is a risk assessment tool used in US the criminal justice system

Predicts the likelihood of recidivism for individual defendants
Influences decisions on bail, sentencing, and parole
130+ factors
Might include socio-economic factors
expensive

Propublica

Founded in 2007 by Paul Steiger, the former managing editor of The Wall Street Journal
investigative journalism in the public interest
Has won several Pulitzer Prizes and numerous other journalism awards

4 of 51

COMPAS vs ProPublica

In 2016, a ProPublica investigation found that the COMPAS algorithm was biased against African-American defendants
Black defendants were more likely to be falsely labeled as high risk, while white defendants were more likely to be falsely labeled as low risk

5 of 51

COMPAS vs ProPublica

6 of 51

COMPAS vs CORELS

CORELS (Certifiably Optimal RulE ListS) Angelino et al., KDD 2017 & JMLR 2018
Model (Rule List) for prediction of recidivism within 2 years
Free, transparent

7 of 51

COMPAS vs CORELS

Simple CORELS rule list is more accurate than COMPASS for prediction of recidivism in 2 years
There’s no benefit from complicated models for re-arrest prediction in
criminal justice.
Perhaps we are using complicated models when we don't need them?

8 of 51

Outline

Motivation
Interpretable Machine Learning
Explainable Machine Learning (XAI)
Interactions and nonlinearities
Reliability
Contradiction

9 of 51

Interpretable Models

In a full data science process, one interprets the results and tunes the processing of the data, the loss function, the evaluation metric, or anything else that is relevant. How can one do this without understanding how the model works?
Avoid catastrophic consequences
Black-box models often predicts the right answer for the wrong reason
In cases where the underlying distribution of data changes (domain shift), problems arise if users cannot troubleshoot the model in real-time

10 of 51

Interpretable vs Explainable Models

Interpretable Models:

Models that are inherently easy to understand and grasp by humans.
Simpler models like linear regression, decision trees

Explainable Models:

Tools to explain decision of black box models
LIME, SHAP, Feature Importances

11 of 51

Interpretation vs Explanation

Could make the situation worse by providing misleading or false characterizations or adding unnecessary authority to the model

12 of 51

The Mythos of Model Interpretability

Lipton (2018) — The Mythos of Model Interpretability. Queue, 16(3).

“Interpretability” is not a single well-defined concept — the term encompasses simulation, decomposability, and algorithmic transparency
Desiderata for interpretability are often in tension: a model can be simulatable but not decomposable
Linear models are not always more interpretable than neural networks (e.g. high-dimensional sparse regression)
Post-hoc rationalization may be unfaithful to the actual computation performed by the model

Key question:

For whom is the model interpretable? A domain expert, a regulator, and an ML engineer have different needs

13 of 51

Stop Explaining Black Boxes (Rudin, 2019)

Rudin (2019) — Nature Machine Intelligence, 1(5), 206–215.

For high-stakes decisions (criminal justice, medicine, credit), post-hoc explanations are insufficient — they cannot be trusted to faithfully represent the model
When the Rashomon set is large, a simple interpretable model exists with comparable accuracy — there is no accuracy-interpretability tradeoff (COMPAS vs CORELS)
Black-box explanations add a second layer of approximation on top of an already-approximate model

Counter-arguments to consider:

Interpretable models may encode harmful proxies transparently — transparency is necessary but not sufficient for fairness
For unstructured data (images, text), a black-box with faithful explanations may outperform a constrained interpretable model

14 of 51

General Principles

15 of 51

Interpretability constraints

Sparsity of the model
Monotonicity with respect to the variable
Decomposability into sub-models
Ability to perform case based-reasoning
Disentanglement of certain types of information within the model reasoning process
Generative constraints (laws of physics)
Preferences among choice of variables

16 of 51

General Principles

17 of 51

Rashomon set of good models

Set of almost equally accurate models

Rashomon effect occurs there are multiple descriptions of the same event with possible no ground truth
Seen in credit score estimation, medical imagining, health record analysis, recidivism prediction
It has been argued that when Rashomon set is large, it must contain a simple model within

18 of 51

Rashomon set

19 of 51

Rashomon set

20 of 51

Difficulties in creation of the model

Solving the optimization problem may hard (i.e. finding the right decision tree)
When one does create an interpretable model, on invariably realizes that the data are problematic and require troubleshooting, which slows down development
It might not be initially clear which definition of interpretability use

21 of 51

Algorithms for data types

22 of 51

Logical Models

Decision tree
Decision list
Decision set

23 of 51

Decision Tree

Current SOTA optimal decision tree methods can handle medium-sized datasets (thousands of samples, tens of binary variables) within 10 minutes when appropriate sparsity constraints are used
Scale exponentially with dimension of data
Handle categorical variables and complicated interactions better than e.g. linear models
When fully optimized, single trees can be as accurate as ensembles of trees or NN

24 of 51

Scoring Systems

Linear classification models models that require users to add, subtract and multiply only a few small numbers
Do not handle interactions
Good for counterfactual reasoning

25 of 51

Scoring Systems

Optimization problem

Practical implementation: round real coefficients -> loss of information
Frameworks to allow Computer-aided exploration, human in the loop
Risk scores

Scoring systems that have a conversion table to probabilities (1 point -> 15%, 2->33%)

26 of 51

Generalized Additive Models (GAMs)

Link function g

Identity -> regression
Logistic -> classification

Component functions f

Step functions
Splines

27 of 51

Generalized Additive Models (GAMs)

we can impose the prior belief that predictive relationships are inherently smooth in nature, even though the dataset at hand may suggest a more noisy relationship
If the researcher could control the sparsity, smoothness, and monotonicity of the component functions, she might be able to design a model that not only predicts well but also reveals interesting relationships between observed variables and outcomes
Could be used to troubleshoot complex datasets (raw medical data), find counterintuitive patterns
GA2Ms

28 of 51

Case-Based Reasoning

Solving a new problem using known solutions to similar past problems.
Emulation of how humans reason
Two types

Nearest neighbour-based techniques
Prototype-based techniques

29 of 51

Prototype-Based Techniques

Learn, from the training data, a set of prototypical cases for comparison
Given a previously unseen test instance, they make a decision by finding prototypical cases that most closely resemble the particular test instance
Part based prototypes compare parts of observations to parts of other observations
Current methods do not take into account prior knowledge or expert opinions

30 of 51

Prototype-Based Techniques

Whole vs part-based prototypes

31 of 51

Disentanglement of neural networks

Refers to the way information travels through the network: all information about a specific concept traverse through one part of the network
Contains information about bed and room -> classify image as bedroom
Supervised vs unsupervised ()

32 of 51

Explainable AI (XAI) Techniques

Global XAI Techniques:

Methods that aim to explain the overall behavior of a model across all data points.
Provide insights into the general decision-making process of the model.
Methods

Feature Importance
Partial Dependence Plots (PDP)

Local XAI Techniques:

Methods that focus on explaining specific individual predictions made by the model.
Offer insights into the model's decision-making process for a particular instance.
Methods

LIME (Local Interpretable Model-agnostic Explanations)
SHAP (SHapley Additive exPlanations)
Counterfactual Explanations:

33 of 51

Permutation Feature Importance

34 of 51

Partial Dependence Plot

A visualization technique that shows the marginal effect of one or two features on the predicted outcome of a machine learning model.
Reveals the relationship between the target and a feature: linear, monotonic, or complex.
Plot of partial dependence function, for regression
isolate the effect of the feature(s) of interest by averaging the model output over the distribution of other features.

35 of 51

Partial Dependence Plot

36 of 51

Local interpretable model-agnostic explanations (LIME)

Explain individual predictions of black box machine learning models using interpretable local surrogate models.

L…loss, G… family of possible explanations, π … proximity measure for neigh. definition

LIME Process:

Select an instance of interest.
Perturb the dataset and obtain the black box predictions for the new points.
Weight the new samples based on their proximity to the instance of interest.
Train a weighted, interpretable model on the perturbed dataset.
Explain the prediction by interpreting the local model.

37 of 51

LIME

Depends strongly of the proximity measure (kernel)

38 of 51

Shapley Values for Explaining Predictions

Fairly distribute the contribution of each feature to a model's prediction using Shapley values from coalitional game theory.
Coalition:combination of feature values working together to produce a specific prediction
Algorithm (example on appartement price)

Determine all possible coalitions of feature values.
Compute the prediction with and without the feature value of interest for each coalition.
Calculate the marginal contribution as the difference between the predicted apartment prices.
Compute the (weighted) average of marginal contributions to obtain the Shapley value.

39 of 51

SHAP (SHapley Additive exPlanations)

Additive feature attribution method represents the Shapley value explanation as a linear model (of coalitions).

Connecting Shapley value and surrogates (LIME)

40 of 51

TreeSHAP

Reduces complexity from O(TL2^M) to O(TLD²)
traversing the decision tree recursively. At each node j, the algorithm calculates the contribution of the split feature and updates the Shapley values accordingly. The update rule for the Shapley values is:

42 of 51

SHAP vs PFI on Simulated Data

All features are random and has no relation to the target
PFI can detect it, SHAP not

43 of 51

SHAP vs LIME

44 of 51

Counterfactual Explanations

Wachter, Mittelstadt & Russell (2017) — Harvard J. Law & Technology, 31(2).

Definition: a counterfactual explanation answers “What is the smallest change to the input that would flip the model’s decision?”
Example: “Your loan was denied. If your income were 5k higher and debt 2k lower, it would be approved.”
Model-agnostic and directly actionable — aligns with GDPR “right to explanation”

Desiderata (Mothilal et al., 2020):

Proximity (small change), sparsity (few features changed), diversity (multiple options), feasibility (realistic values)

Limitation:

Does not explain why the model made its decision — only what would change it (Rashomon problem: many valid counterfactuals exist)

45 of 51

Saliency Maps

Saliency maps are visual representations that highlight important regions or features in an input image that contribute to a model's prediction.
Recipe

Perform a forward pass of the image of interest.
Compute the gradient of class score of interest with respect to the input pixels:
Visualize the gradients. You can either show the absolute values or highlight negative and positive contributions separately.

46 of 51