1 of 26

Juan C. Caicedo Ph.D.

HTC 2024

Madison, WI, July 8 2024

Advancing Biology through �High throughput computing

Modeling cellular morphology using machine learning

2 of 26

Imaging reveals cellular phenotypic variation

https://micro.magnet.fsu.edu/cells/fluorescencemitosis/index.html

Eitaki, M., et al. Vincristine enhances amoeboid-like motility via GEF-H1 / RhoA / ROCK / Myosin light chain signaling in MKN45 cells. BMC Cancer 12, 469 (2012).

Compound treatments

Cell-cycle stages

3 of 26

Screen for specific phenotypes using images

Clinical trials underway for Alisertib in adults with AMKL.

Wen Q, et al. (2012). Cell 150(3):575-89

DNA stain with outlines identifying the nuclei

DMSO

SU6656

Treatment for AMKL (leukemia)

4 of 26

1. Raw images

2. Segmented images

3. Single-cell feature matrices

4. Population profiles of treatments

Image-based cell profiling

5. Downstream statistical analysis

Are treatments

significantly different / effective?

Caicedo, et al. 2017 Nature Methods

5 of 26

Imaging data informs biomedical research

Chemical structures (CS)

Yeast

Fungal

Cell-based

Biochemical

Bacterial

Parasite

Worm

MO

CS

GE

11

14

8

4

3

0

Morphology (MO)

Gene expression (GE)

Gene Expression

Morphology

Same allele across platforms

EGFR_p.S645C

KRAS_p.G12V

CCA space

1.0

0.4

-0.2

EGFR Wild Type

Control

EGFR Mutant

Functional Genomics Research

Drug Discovery

6 of 26

Cell segmentation

7 of 26

Nucleus segmentation problem

Segmentation

Otsu’s thresholding method

The problem of identifying single cells can be formulated as an image segmentation problem, where the goal is to identify regions of the image or groups of pixels that belong to the same object of interest. Single cells can be identified by their nuclei. And here is an example of how to segment an input image to generate the desired output, which is single nuclei identified with a different number or color in this case.

There are several classical algorithms for accomplishing this task, which can be configured and tuned to solve some cell segmentation problems. But in general, these have limitations and cannot produce the desired output correctly all the time. In this case, we see a simple thresholding algorithm failing to separate a few objects of interest correctly. The good news is this can be solved with machine learning algorihtms.

8 of 26

Problem: where are the micronuclei?

Very small compared to nucleus
Low intensity (dim)
Confused with noise
Difficult to detect by eye
Difficult to segment with existing tools

Nucleus

Micronucleus

Example input

Manual annotation

9 of 26

A transformer-based segmentation model

Linear projection of patches

Output Tokens

1

2

3

4

5

6

7

8

9

0

Vision Transformer Network

Self-attention layers

Input

Tokenized image

Sequence of patches

Segmentation head

32x32

64x64

128x128

256x256

Probability map

32x32

256x256

10 of 26

Experimental setup

18 manually annotated images

Image size: 2,960x2,960 pixels

Each image has ~80 objects of interest

Randomly sample crops of 256x256 pixels

Leave-one-out cross validation

Train on 17 images
Evaluate on held-out image

Densely sample all crops of 256x256

Explore hyper-parameters

Architecture
Training settings

11 of 26

Baseline model

Precision: 0.81465 Recall: 0.42912

12 of 26

Experiments

Precision: 0.81465 Recall: 0.42912

Precision: 0.68104 Recall: 0.32303

Precision: 0.42653 Recall: 0.64992

A: Frozen backbone

13 of 26

Experiments

Precision: 0.81465 Recall: 0.42912

Precision: 0.8352 Recall: 0.3367

Precision: 0.6884 Recall: 0.56416

A: Frozen backbone

B: Fine tuning transformer

14 of 26

Experiments

Precision: 0.81465 Recall: 0.42912

Precision: 0.6922 Recall: 0.6525

Precision: 0.7469 Recall: 0.591

A: Frozen backbone

B: Fine tuning transformer

C: Output resolution 128

15 of 26

Experiments

Precision: 0.81465 Recall: 0.42912

Precision: 0.7024 Recall: 0.71113

A: Frozen backbone

B: Fine tuning transformer

C: Output resolution 128

D: Training with more data

16 of 26

Experiments

Precision: 0.81465 Recall: 0.42912

Precision: 0.77526 Recall: 0.78932

A: Frozen backbone

B: Fine tuning transformer

C: Output resolution 128

D: Training with more data

E: Adding LayerNorm

17 of 26

Final result

Precision: 0.81465 Recall: 0.42912

Precision: 0.8157 Recall: 0.8352

A: Frozen backbone

B: Fine tuning transformer

C: Output resolution 128

D: Training with more data

E: Adding LayerNorm

F: Screen images

18 of 26

Cell phenotyping

19 of 26

Can we create a generalist bioimage analysis model?

20 of 26

Adapted from (a) https://bmcmolcellbiol.biomedcentral.com/articles/10.1186/1471-2121-9-42/ (b) https://www.kaggle.com/c/human-protein-atlas-image-classification/ � (c) https://www.broadinstitute.org/ (d) https://www.nature.com/articles/s41587-019-0207-y/

RGB natural images

Cell Painting^(c)

Highly multiplexed immunofluorescence^(d)

Human protein atlas^(b)

High-content screening^(a)

3 channels

4 channels

5 channels

10+ channels

ImageNet pretrained models are designed for 3 channel, RGB images

21 of 26

Learning from images at scale

Millions of images

High-throughput computing

Image-based discoveries

22 of 26

Data sources

30 studies ~5M images

10 TB

75 TB

120 TB

70 studies ~20M images

12 laboratories ~40M images

Powered by

23 of 26

Self-supervised representation learning

Linear projection of patches

Output

1

2

3

4

5

6

7

8

9

0

Vision Transformer Network

Self-attention layers

Input cell

Tokenized image

Sequence of patches

Goal: discover biologically relevant cellular features without human supervision

Teacher Network

Student Network

Features

Global views

Local views

Self-supervision

Single-cell

image

Self-distillation with no labels

Doron, et al. "Unbiased single-cell morphology with self-supervised vision transformers." bioRxiv (2023).

24 of 26

A single model to recapitulate all studies

Compound treatments

Cell-cycle stages

Protein localization

Replicate findings from about 100 publicly available imaging studies

Use the model to power biological discoveries at Morgridge and UW

25 of 26

Summary

Imaging

Imaging encodes rich phenotypic information

Representation

Machine learning identifies useful biological variation

Generalization

From specialized methods to foundation model

Applications

Unbiased features have many biological applications

25

26 of 26

Broad Institute

University of Helsinki

Columbia University

Meta AI

Morgridge Institute

CHTC Team

Biomedical Imaging

Boston University

Research Computing

Brian Bockelman�

Justin Hiemstra

�Andrew Maier

Gratitude

Caicedo Lab

Nikita Moshkov

John Peters

Other past and future members

Tyler Thompson

Zayn

Kayali

Yifan

Ren

Aditya Pillai