1 of 1

References:

  1. Kaufman, S et al. (2012) Leakage in data mining: Formulation, detection, and avoidance. ACM TKDD
  2. Kapoor, Narayanan (2023) Leakage and the reproducibility crisis in machine-learning-based science. Patterns
  3. Bernett et al. (2024) Guiding questions to avoid data leakage in biological machine learning applications. Nature Methods
  4. National Center for Advancing Translational Sciences (2014) The Tox21 data challenge
  5. Wu et al. (2018) MoleculeNet: a benchmark for molecular machine learning. Chemical Sciences
  6. Li, et al. (2023) Leak Proof PDBBind: A Reorganized Dataset of Protein-Ligand Complexes for More Generalizable Binding Affinity Prediction. arXiv
  7. Durairaj et al. PLINDER: The protein-ligand interactions dataset and evaluation resource. bioRxiv

 

 

Data Splitting Against Information Leakage with DataSAIL

Roman Joeres 1,2,3,4, David B. Blumenthal 5, Olga V. Kalinina 1,2,6

1 Helmholtz Institute for Pharmaceutical Research Saarland, Saarbrücken, Germany

2 Center for Bioinformatics, Saarland University, Saarbrücken, Germany

3 Department for Chemistry and Molecular Biology, University of Gothenburg, Gothenburg, Sweden

4 Wallenberg Center for Translational Medicine, University of Gothenburg, Gothenburg, Sweden

5 Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

6 Medical Faculty, Saarland University, Homburg, Germany

The Problem: Similarity-induced Information Leakage, especially in Biomedical Machine Learning

What is Information Leakage in Machine Learning?

  • A mismatch between test and inference data
  • A model might have illegitimate access to information about data at test time that is not available at inference time
  • Therefore leads to performance overestimation of models
  • This may have several reasons (see [1-3])
  • DataSAIL focuses on similarity-induced information leakage
  • Biomolecules have complex, hidden inter-sample similarities:
    • sequence and/or structure similarities
    • phylogenetic relations
  • DataSAIL splits datasets while accounting for this

read any type of input data

cluster the data

“solve” optimization problem

map results

The Solution: Data Splitting against Information Leakage with DataSAIL

Clustering

  • Constraint optimization problem becomes infeasible
  • reduce the problem size
  • reduce the runtime of solvers.
  • DataSAIL splits the data based on inter-cluster similarities.

​

Other algorithms, e.g., Butina Splitting, also use clustering, but treat clusters as uni-distant.

​

Evaluation of DataSAIL on

  • Molecular property prediction on Tox21[4] from MoleculeNet[5]
  • Combining stratification and leakage reduction on SR-ARE assay of Tox21
  • Drug-target affinity prediction on LP-PDBBind[6]
  • Established machine learning models perform worse on leakage-reduced �splits and especially on DataSAIL splits�
  • Leakage-reduction on the protein-ligand interaction dataset PLINDER
  • DataSAIL reduced leakage better than the manually designed algorithm

Conclusion:

  • DataSAIL computes a variety of data-splits for ML in different inference scenarios
  • Clustering as heuristic to solve large problem instances without reducing the quality of splits
  • First-of-its-kind working on 2D data and combining stratification with leakage-reduction
  • Effectiveness experimentally validated on state-of-the-art biological dataset and various tasks

Different biological data-splits leak more or less information. Train in blue, validation in yellow, test in red.

github.com/kalininalab/DataSAIL

Available from:

Machine Learning Life Cycle

The Heart of DataSAIL

The idea behind DataSAIL:

  • formulate an integer linear program (ILP) based on this constrained optimization problem (COP). This COP is NP-hard
  • solve it using SCIP, MOSEK, or GUROBI (users choice)
  • the result is a leakage-reduced split

​

This allows DataSAIL to

  • split multi-dimensional data together
  • combine leakage reduction with stratification, i.e., balancing classes
  • split any type of data, e.g., proteins, chemicals, genomic and custom data

weights of

data points

Minimize the inter-split similarities

are in the same split?

similarity metric between data points

relative error for split sizes

subset of 1 �with to class 1

relative error for class ratios

relative size

of split i

Objective:

subject to:

split function

​

subset of 1 with type 1

Respect the requested �split sizes

Keep the ratios between class

(a.k.a. stratification)

Tox21 Challenge

All Assays

Stratification on SR-ARE

LP-PDBBind (DTI)

DataSAIL is usable

  • in the commandline,
  • or as a Python Package.

​

It is actively used, tested, maintained, improved, and extended.

www.helmholtz-hzi.de