Experimental design and evaluation for graph machine learning
Oct 15th, 2026
These slides, excluding third-party material, are licensed under CC BY-NC 4.0 by Sushmita Roy, Anthony Gitter, and Raison Dsouza
Topics in this section
Goals for today
3
ML development cycle
4
Bernett et al. Nature Methods 2024 doi:10.1038/s41592-024-02362-y
Motivating example 1: drug discovery
5
Protein Kinase A
Inhibitor: 7-[(3S,4R)-4-(3-chlorophenyl)carbonylpyrrolidin-3-yl]-3H-quinazolin-4-one
Motivating example 1: drug discovery
6
Mattsson et al. bioRxiv 2026 doi:10.64898/2026.06.29.735309
Motivating example 2: protein-protein interaction prediction
7
Bernett et al. Briefings in Bioinformatics 2024 doi:10.1093/bib/bbae076
Motivating example 2: protein-protein interaction prediction
8
Sledzieski et al. Cell Systems 2021 doi:10.1016/j.cels.2021.08.010
D-SCRIPT
Motivating example 2: protein-protein interaction prediction
9
Bernett et al. Briefings in Bioinformatics 2024 doi:10.1093/bib/bbae076
2.Model learned shortcuts (e.g. node degree)
1.Model learned sequence patterns that govern binding probability (e.g. matching domains)
3.Model learned protein sequence similarity, finds similar training pair p1 p2 when predicting edge for test pair p1’ p2’
Motivating example 2: protein-protein interaction prediction
10
Bernett et al. Briefings in Bioinformatics 2024 doi:10.1093/bib/bbae076
Some pitfalls that arise when applying supervised ML
they can be interrelated
therefore might share mitigation strategies
11
Whalen, Sean, et al. "Navigating the pitfalls of applying machine learning in genomics." Nature Reviews Genetics 23.3 (2022): 169-181.
i.i.d: independent & identically distributed
heads/tails
12
online search results
Failure of i.i.d assumption
13
performance will be higher in cross validation (same setting) than on the prediction set (different setting)
standard k-fold cross validation
Samples from the same source can be split between train and test sets across iterations
14
Examples are randomly assigned to folds.
How to find i.i.d failures?
15
Whalen, Sean, et al. "Navigating the pitfalls of applying machine learning in genomics." Nature Reviews Genetics 23.3 (2022): 169-181.
Simple approach : projecting the data into two dimensions and making scatter plots or by comparing histograms of feature values
Sophisticated approach : use statistical tests to detect when distributions differ
for example,
for univariate continuous features
Possible solutions:
16
Whalen, Sean, et al. "Navigating the pitfalls of applying machine learning in genomics." Nature Reviews Genetics 23.3 (2022): 169-181.
batch correction methods
are commonly used, such as quantile normalization,
empirical Bayes adjustment for measured variables
with ComBat
More elaborate approaches from domain
adaptation and transfer learning include unsupervised feature transformations and supervised learning
of robust feature representations.
Causes of distributional shift in Omics
Population structure (ascertainment bias)
Training genomic risk models on European-ancestry cohorts creates distributional shifts when applied to diverse global population
17
*Accounting for distributional differences is still an area of open research*
Biological context (In vitro vs. In vivo)
when a model is trained and applied in different biological contexts
Technical factors (Mass spec / Sequencing depth)
In proteomics, distributional differences across mass spectrometers mean that reproducibility is higher on the same instrument than across instruments
2. Dependent examples : where it occurs?
like drug discovery example as before
18
Standard machine learning algorithms depend entirely on the assumption of independence - that observing example A tells you nothing about example B
Card Analogy: Drawing cards without replacement alters conditional probabilities for all subsequent draws
Dependent examples : where it occurs?
Standard machine learning algorithms depend entirely on the assumption of independence -
that observing example A tells you nothing about example B
19
Omics Analogy:
When genomic data is arranged into a standard matrix with one example per row, standard ML packages treat every row as independent.
But in biology, rows often share underlying genes, evolutionary lineages, or spatial genomic loci, violating the core mathematical assumptions of cross-validation
Dependent examples : where it occurs?
20
The independence and identical distribution assumptions are entangled. For instance, genotyping results for family members are dependent and also may differ distributionally from other families.
Unfortunately, dependence relationships are not always known.
Even known dependencies have a tendency to be ignored in supervised ML analyses.
When data are formatted as a table with one example in each row, the standard for most ML toolkits, it is easy to proceed with model fitting and cross validation without checking whether examples are independent.
Dependent example : symptoms
Failing to account for dependencies between examples
can lead to biased models and overly optimistic
estimates of model performance.
21
Randomized cross validation does not protect against this problem and overestimates performance because examples in the test set can be correlated with training examples and bring information into the test set that should not be there.
Possible solutions
The best solution may be to acknowledge dependence and mitigate overfitting at the model evaluation stage. For example, using Group k fold cross validation.
22
Or using methods that explicitly model the covariance between examples, such as mixed effects models from biostatistics, time-series models, and autocorrelation models from spatial statistics
Group-k Cross validation
When splitting into k-folds, all examples from the same group are assigned to the same fold.
23
Examples that belong to the same group cannot cross the train-test divide.
Better data splitting for biological data
24
Joeres, Roman, David B. Blumenthal, and Olga V. Kalinina. "Data splitting to avoid information leakage with DataSAIL." Nature communications 16.1 (2025): 3337.
Causes of Data leakage
26
A protein’s evolutionary family strongly predicts its cellular function, tertiary structure, and interaction preferences
Randomized cross-validation assigns paralogs or orthologs (e.g., 80% sequence identity) across train and test folds..
Result: The model acts as a family lookup table, memorizing shared domain motifs rather than learning generalizable biophysical rules
Possible solutions
Pipeline encapsulation:
Never pass pre-transformed feature matrices into cross-validation functions
27
Connection to structure prediction & co-fold
28
Proteins can have conserved structure without conserved sequence
Myoglobin proteins with sequence identity to human
Similarity within AlphaFold Protein Structure Database
African elephant (80%)
Human
Black fin tuna (45%)
Pigeon (25%)
Barrio-Hernandez et al. Nature 2023 doi:10.1038/s41586-023-06510-w
Top Hat Question
29
Take away points
30