Machine and Deep Learning on the cloud: Segmentation
1
Ikerbasque Research Associate
Computer Science and Artificial Intelligence Department
University of the Basque Country
Ignacio Arganda-Carreras, PhD.
Defragmentation:
Bringing BioIMage Analysts to the cloud!
Day 2 (September 30th 2022)
neubiasacademy.org
Outline
���
2
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Image Segmentation
3
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Image Segmentation
4
Semantic
segmentation
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Image Segmentation
5
Instance
segmentation
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Evaluation metrics
6
�How to measure how well the objects are found?
Intersection over Union (IoU)
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Evaluation metrics: IoU
7
�How to measure how well the objects are found?
Intersection over Union (IoU)
0.0 means no overlap and 1.0 means perfect overlap
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Evaluation metrics: mAP
8
�What about multiple object and classes?
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Evaluation metrics: mAP
9
�What about multiple object and classes?
Consider each object positive or negative based on and IoU threshold (e.g., 0.50, 0.75)
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Evaluation metrics: mAP
10
�What about multiple object and classes?
Calculate precision-recall curves and average precision (AP) per class
Consider each object positive or negative based on and IoU threshold (e.g., 0.50, 0.75)
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Evaluation metrics: mAP
11
�What about multiple object and classes?
Calculate precision-recall curves and average precision (AP) per class
Consider each object positive or negative based on and IoU threshold (e.g., 0.50, 0.75)
And provide the mean (mAP)!
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Top-down approach
12
CNN
...
Classification loss
Bounding-box regression loss
Classification loss
Bounding-box regression loss
Object
Detection
Instance
Segmentation
DOG, DOG, CAT
DOG, DOG, CAT
Mask
prediction
Top-down: break a big problem (multiple object segmentation) into smaller ones (region proposals, bounding box locations, single-object classification and mask prediction)
He et al, “Mask R-CNN”, ICCV 2017
Mask R-CNN
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Top-down limitations
13
Input Image
(e.g. 1 x 1024 x 768)
Lucchi et al, “Supervoxel-based segmentation of mitochondria in EM image stacks with learned shape features”, TMI, 2011
Process directly all instances in the field of view (e.g., with a Mask R-CNN)
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Top-down limitations
14
Input Image
(e.g. 1 x 1024 x 768)
Lucchi et al, “Supervoxel-based segmentation of mitochondria in EM image stacks with learned shape features”, TMI, 2011
Process directly all instances in the field of view (e.g., with a Mask R-CNN)
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Top-down limitations
15
Input Image
(e.g. 1 x 1024 x 768)
Lucchi et al, “Supervoxel-based segmentation of mitochondria in EM image stacks with learned shape features”, TMI, 2011
100 x 4096 x 4096
Wei et al, “MitoEM dataset: Large-scale 3D mitochondria instance segmentation from EM images”, MICCAI, 2020
Problem: processing images with very large field of view (typical in Bioimage Analysis)
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Top-down limitations
16
CNN
...
Classification loss
Bounding-box regression loss
Classification loss
Bounding-box regression loss
Object
Detection
Instance
Segmentation
DOG, DOG, CAT
DOG, DOG, CAT
Mask
prediction
Problem: field of view defined by the input size of the CNN used as backbone
Q: how do we process images larger than the input size of the CNN?
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Top-down limitations
17
Input Image
(e.g. 1 x 1024 x 768)
Lucchi et al, “Supervoxel-based segmentation of mitochondria in EM image stacks with learned shape features”, TMI, 2011
100 x 4096 x 4096
Wei et al, “MitoEM dataset: Large-scale 3D mitochondria instance segmentation from EM images”, MICCAI, 2020
Intuitive idea: divide dataset in patches that fit into the backbone net
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Top-down limitations
18
Input Image
(e.g. 1 x 1024 x 768)
Lucchi et al, “Supervoxel-based segmentation of mitochondria in EM image stacks with learned shape features”, TMI, 2011
Wei et al, “MitoEM dataset: Large-scale 3D mitochondria instance segmentation from EM images”, MICCAI, 2020
Problem: instances may be placed along different patches
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Top-down limitations
19
Input Image
(e.g. 1 x 1024 x 768)
Lucchi et al, “Supervoxel-based segmentation of mitochondria in EM image stacks with learned shape features”, TMI, 2011
Q: How do we merge instances across patches?
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Bottom-up approach
20
Bottom-up: focus on solving the smaller problems and then integrate them into a complete solution
Full Image
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Bottom-up approach
21
Bottom-up: focus on solving the smaller problems and then integrate them into a complete solution
Full Image
Object probabilities
Small problem 1: calculate pixel probabilities of objects of interest
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Bottom-up approach
22
Bottom-up: focus on solving the smaller problems and then integrate them into a complete solution
Full Image
Object probabilities
Instance Segmentation
Small problem 1: calculate pixel probabilities of objects of interest
Small problem 2: extract individual instances from the object probabilities
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Bottom-up approach
23
Design an image processing pipeline to first identify pixels/voxels belonging to objects, and then extract instances from them
Full Image
Object probabilities
Instance Segmentation
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Bottom-up approach
24
Design an image processing pipeline to first identify pixels/voxels belonging to objects, and then extract instances from them
Full Image
Object probabilities
Instance Segmentation
Object probabilities:
Semantic Segmentation�network
Instance extraction:
???
???
???
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Which networks can we use?
25
Effective deep learning architectures
O. Çiçek, et al., 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation, MICCAI 2016
O. Ronneberger, et al., U-Net: Convolutional Networks for Biomedical Image Segmentation, MICCAI 2015
User friendly libraries
F. Chollet, et al., https://keras.io 2015
Deep Learning toolbox, Matlab 2015
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
U-Net (2D)
26
O. Ronneberger, et al., “U-Net: Convolutional Networks for Biomedical Image Segmentation”, MICCAI 2015
Contracting path: extracts high dimension features
Expanding path: refines the processing
Skip connections
Bottleneck
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
3D U-Net
27
Çiçek, Özgün, et al. 3D “U-Net: learning dense volumetric segmentation from sparse annotation”, MICCAI, 2016
3 x 3 x 3 convolutions
2 x 2 x 2 max pooling
2 x 2 x 2 up-convolutions
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Bottom-up approach
28
Design an image processing pipeline to first identify pixels/voxels belonging to objects, and then extract instances from them
Full Image
Object probabilities
Instance Segmentation
Object probabilities:
Semantic Segmentation�network
Instance extraction:
???
U-Net-like model (2D or 3D)
???
O. Ronneberger, et al., “U-Net: Convolutional Networks for Biomedical Image Segmentation”, MICCAI 2015.
Çiçek, Özgün, et al. 3D “U-Net: learning dense volumetric segmentation from sparse annotation”, MICCAI 2016.
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Instance extraction
29
Object probabilities
Instance Segmentation
Binary masks
An intuitive idea: extract most likely object regions by applying a threshold (0.5?), and identify connected regions on top.
Q: how do we extract the connected regions from a binary image?
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Connected component labeling
30
See for instance: skimage.measure.label
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Connected component labeling
31
See for instance: skimage.measure.label
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Connected component labeling
32
See for instance: skimage.measure.label
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Possible issues: splits and mergers
33
Object probabilities
Binary masks
Connected regions
Ground truth
Problem: results are too sensitive to the binarization process
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Possible issues: splits and mergers
34
Object probabilities
Binary masks
Connected regions
Ground truth
Possible solution: post-process using Watershed
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Watershed Segmentation
35
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Watershed Segmentation
36
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Watershed Segmentation
37
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Watershed limitations
38
Original image
Watershed segmentation
Local minima
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Watershed with markers
39
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Solutions to over-segmentation
40
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Watershed segmentation of contrasted objets
41
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Separation of binary particles
42
Input image
Touching nuclei
Separated nuclei
Distance map
Watershed on inverse of distance map
Watershed lines
Sometimes known as Distance Transform Watershed
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Bottom-up approach
43
Design an image processing pipeline to first identify pixels/voxels belonging to objects, and then extract instances from them
Full Image
Object probabilities
Instance Segmentation
Object probabilities:
Semantic Segmentation�network
Instance extraction:
Connected components,
Watershed transforms,
...
U-Net-like model (2D or 3D)
O. Ronneberger, et al., “U-Net: Convolutional Networks for Biomedical Image Segmentation”, MICCAI 2015.
Çiçek, Özgün, et al. 3D “U-Net: learning dense volumetric segmentation from sparse annotation”, MICCAI 2016
Post-
processing
method
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Bottom-up approach
44
Problem: Too sensitive to object predictions (mergers & splits)
Q: How can we help the post-processing?
Full Image
Object probabilities
Instance Segmentation
U-Net-like model (2D or 3D)
O. Ronneberger, et al., “U-Net: Convolutional Networks for Biomedical Image Segmentation”, MICCAI 2015.
Çiçek, Özgün, et al. 3D “U-Net: learning dense volumetric segmentation from sparse annotation”, MICCAI 2016
Post-
processing
method
First step: Run once per patch
Second step: Run once per full image
Two-step pipeline!
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Fighting mergers
Splits or artifacts are “easy” to correct in post-processing.
Mergers can be caused by a single wrong prediction in an object border.
45
Yan et al., “A deep model with shape-preserving loss for gland instance segmentation”, MICCAI 2018
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Fighting mergers
Splits or artifacts are “easy” to correct in post-processing.
Mergers can be caused by a single wrong prediction in an object border.
Idea: predict objects and borders
46
Yan et al., “A deep model with shape-preserving loss for gland instance segmentation”, MICCAI 2018
Adapt loss for multi-task and weight regions to preserve shape
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Bottom-up approach
47
Full Image
Object + boundary probabilities
Instance Segmentation
U-Net-like model (2D or 3D)
Yan et al., “A deep model with shape-preserving loss for gland instance segmentation”, MICCAI 2018
Post-
processing
method
First step: Run once per patch
Second step: Run once per full image
Two-step pipeline!
Q: Can we help more the post-processing?
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Common source of mistakes
48
Noisy images + Crowded cells = Common source of segmentation errors
GT Segmentation
Dense Segmentation
(e.g. U-Net)
�Problem: Merging of touching cells
Bounding box based methods
(e.g. Mask-RCNN)��Problem: suppression of valid cell instances due to large overlap of
bounding box localization
Slide adapted from: Martin Weigert, EPFL Lausanne
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Segmentation with star-convex cell priors (StarDist)
49
Dense Polygon Prediction
(e.g. U-Net, ResNet)
Polygon Selection
Non-Maximum Suppression (NMS)
Idea: calculate distances (r) to the object boundary along a fixed set of rays and object probabilities (d)
Post-processing: NMS of polygons
Schmidt et al., "Cell detection with star-convex polygons”, MICCAI 2018
Together produce an overcomplete set of candidate polygons
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Segmentation with star-convex cell priors (StarDist)
50
Input image
Schmidt et al., "Cell detection with star-convex polygons”, MICCAI 2018
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Segmentation with star-convex cell priors (StarDist)
51
Object probabilities
Schmidt et al., "Cell detection with star-convex polygons”, MICCAI 2018
Single channel for the object probability output
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Segmentation with star-convex cell priors (StarDist)
52
Radial distances
Schmidt et al., "Cell detection with star-convex polygons”, MICCAI 2018
The polygon distance output layer has as many channels as there
are radial directions
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Segmentation with star-convex cell priors (StarDist)
53
Radial distances
Schmidt et al., "Cell detection with star-convex polygons”, MICCAI 2018
The polygon distance output layer has as many channels as there
are radial directions
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Segmentation with star-convex cell priors (StarDist)
54
Radial distances
Schmidt et al., "Cell detection with star-convex polygons”, MICCAI 2018
The polygon distance output layer has as many channels as there
are radial directions
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Segmentation with star-convex cell priors (StarDist)
55
Star-convex polygon candidates
Schmidt et al., "Cell detection with star-convex polygons”, MICCAI 2018
Only consider polygon proposals from pixels with sufficiently high object probability
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Segmentation with star-convex cell priors (StarDist)
56
Final instances
Schmidt et al., "Cell detection with star-convex polygons”, MICCAI 2018
Perform non-maximum suppression (NMS) to arrive at the final set of polygons, each representing an instance
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Segmentation with star-convex cell priors (StarDist)
57
Weigert et al., "Star-convex Polyhedra for 3D Object Detection and Segmentation in Microscopy”, WACV 2020
Similar approach for 3D objects
Competitive with Mask R-CNN with many less parameters
Distance (2D and 3D) is only well-defined for (non-background) pixels that are contained within an object.
Only convex objects are segmented.
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Cellpose: generalist cellular segmentation
58
Not all cells are blob-like objects!
Idea: simulate smooth topological map.
Stringer et al., "Cellpose: a generalist algorithm for cellular segmentation", Nature Methods 2021
Manual annotation
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Cellpose: generalist cellular segmentation
59
Not all cells are blob-like objects!
Idea: simulate smooth topological map.
Stringer et al., "Cellpose: a generalist algorithm for cellular segmentation", Nature Methods 2021
Manual annotation
Simulated diffusion
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Cellpose: generalist cellular segmentation
60
Not all cells are blob-like objects!
Idea: simulate smooth topological map.
Stringer et al., "Cellpose: a generalist algorithm for cellular segmentation", Nature Methods 2021
Manual annotation
Simulated diffusion
Spatial gradients
X and Y gradients
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Cellpose: generalist cellular segmentation
61
Not all cells are blob-like objects!
Idea: simulate smooth topological map.
Stringer et al., "Cellpose: a generalist algorithm for cellular segmentation", Nature Methods 2021
Manual annotation
Simulated diffusion
Spatial gradients
Flow representation
X and Y gradients
Normalized direction from 0° to 360°
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Cellpose: generalist cellular segmentation
62
Not all cells are blob-like objects!
Idea: simulate smooth topological map.
Stringer et al., "Cellpose: a generalist algorithm for cellular segmentation", Nature Methods 2021
Normalized direction from 0° to 360°
Adaptable to more diverse cell shapes
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Cellpose: generalist cellular segmentation
63
Stringer et al., "Cellpose: a generalist algorithm for cellular segmentation", Nature Methods 2021
CNN predicts gradients and object mask
Predictions are converted into single flow field
Pixels that converge to the same fixed point are assigned to the same mask
U-Net with:
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Cellpose: generalist cellular segmentation
64
Stringer et al., "Cellpose: a generalist algorithm for cellular segmentation", Nature Methods 2021
Q: How to force generalization?
Idea: train on very diverse dataset
Dataset includes images of cells from:
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Cellpose: generalist cellular segmentation
65
Stringer et al., "Cellpose: a generalist algorithm for cellular segmentation", Nature Methods 2021
Segmentation in 3D without 3D labels!
Apply 2D network to each plane XY, YZ, XZ
Six predicted flow maps are pairwise averaged into a single 3D flow field XYZ
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Generic bottom-up approach
66
Full Image
Object + boundary probabilities + distance
Instance Segmentation
U-Net-like model (2D or 3D)
Wei et al., "MitoEM dataset: Large-scale 3d mitochondria instance segmentation from EM images", MICCAI 2020.
Lin et al., "NucMM Dataset: 3D Neuronal Nuclei Instance Segmentation at Sub-Cubic Millimeter Scale", MICCAI 2021.
Post-
processing
method
Idea: Add a distance based output to our generic 2D/3D pipeline
Q: How to combine the outputs?
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Instance extraction with watershed
67
Idea: combine network outputs to get watershed mask and markers
Predictions
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Instance extraction with watershed
68
Threshold
Idea: combine network outputs to get watershed mask and markers
Predictions
Mask: medium-high object probability
Mask
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Instance extraction with watershed
69
Threshold
Threshold
+ Closing
Threshold
Idea: combine network outputs to get watershed mask and markers
Predictions
Mask: medium-high object probability
Markers: high object probability and no boundary + closing
Mask
Markers
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Instance extraction with watershed
70
Threshold
Threshold
+ Closing
Threshold
- Distance
Input: negative distance
Idea: combine network outputs to get watershed mask and markers
Predictions
Mask: medium-high object probability
Markers: high object probability and no boundary + closing
Mask
Markers
Input
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Instance extraction with watershed
71
Threshold
Threshold
+ Closing
Threshold
- Distance
Input: negative distance
Idea: combine network outputs to get watershed mask and markers
Predictions
Mask: medium-high object probability
Markers: high object probability and no boundary + closing
Mask
Markers
Input
Marker-controlled watershed
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Open Source Frameworks
72
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Instance Segmentation Challenges!
73
Wei et al., "MitoEM dataset: Large-scale 3d mitochondria instance segmentation from EM images", MICCAI 2020.
Lin et al., "NucMM Dataset: 3D Neuronal Nuclei Instance Segmentation at Sub-Cubic Millimeter Scale", MICCAI 2021.��Wei et al., "AxonEM Dataset: 3D Axon Instance Segmentation of Brain Cortical Regions", MICCAI 2021.
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Hands-on Tutorial
Go to: https://colab.research.google.com/drive/1iOyDymnPFFlWrw_obfcM3FlHeGanpL59?usp=sharing
74
More Instance Segmentation notebooks
75
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation
Slide credits and references
76
Defragmentation Training School - Day 2: Machine and Deep Learning on the cloud: Segmentation