Fully Automated

Multi-heartbeat Echocardiography

Video Segmentation and Motion Tracking

Yida Chen, Xiaoyan Zhang,

Christopher M. Haggerty, Joshua V. Stough

Outline

• Echocardiography
• Video-based Segmentation
• CLAS-FV
• R2+1D Convolution
• EchoNet-Dynamic Dataset
• Test-time EF Estimation
• Results

2

Echocardiography

• Non-invasive ultrasonic test
• Common imaging modality
• Allow the derivation of left-ventricular ejection fraction
• Labor-intensive process
• Deep learning-based segmentation methods have been extensively studied

3

Ouyang et al. Video-based AI for beat-to-beat assessment of cardiac function, https://doi.org/10.1038/s41586-020-2145-8

​

​

Sparse Annotation

• Most deep learning methods rely on frame-based analysis
• Only ED/ES frames have manual annotations
• No manual label for other frames
• CLAS provides a solution for generating pseudo labels on intermediate frames

ED Frame

ES Frame

4

LeClerc et al., Deep Learning for Segmentation Using an Open Large-Scale Dataset in 2D Echocardiography, https://doi.org/10.1109/TMI.2019.2900516

​

​

​

Wei et al., Temporal-consistent Segmentation of Echocardiography with Co-learning from Appearance and Shape, https://doi.org/10.1007/978-3-030-59713-9_60

CLAS Video Segmentation

• Take in a sampled 10-frame ED-ES Half-heartbeat clip
• A shared 3D-UNet is used for motion tracking and segmentation
• Output frame-level segmentation and pixel-wise bi-directional motion estimation

5

Wei et al., Temporal-consistent Segmentation of Echocardiography with Co-learning from Appearance and Shape, https://doi.org/10.1007/978-3-030-59713-9_60

Pseudo Labels

• Use motion estimation to spatially transformed the manual labels.
• On a ED-ES echo clip:
• Forward transform ED label
• Backward transform ES label

6

Forward Deformed Pseudo Labels

Deformed with forward motions from ground truth ED label

Advantages & Limitations

• Temporally-consistent Segmentation
• Better EF estimation�
• Confine analysis to systole videos
• Require humans or other frameworks to identify ED-ES half-heartbeats
• Undermined applicability

7

CLAS segmented ED-ES video clip

Chen et al. Assessing the Generalizability of Temporally-Coherent Echocardiography Video Segmentation, https://doi.org/10.1117/12.2580874

Outline

• Echocardiography
• Video-based Segmentation
• CLAS-FV
• R2+1D Convolution
• EchoNet-Dynamic Dataset
• Test-time EF Estimation
• Results

8

CLAS-FV Full Video Segmentation

• Extend the CLAS framework for full video segmentation on multi-heartbeat echocardiogram
• Joint video segmentation and motion tracking network

9

CLAS-FV Full Video Segmentation

• Trained on clips with one or more cardiac cycles
• Leverage the efficient R2+1D spatiotemporal convolutions for feature extraction

10

R2+1D Convolution

• Decompose a 3D convolution into a spatial 2D convolution followed by a 1D temporal convolution
• Used as feature encoder for EF regression in Ouyang et al.’s work

11

Tran et al. A Closer Look at Spatiotemporal Convolutions for Action Recognition, https://doi.org/10.1109/CVPR.2018.00675

Compare with 3D-UNet

• 10-Fold CV on CAMUS
• 450 patients
• Compare R2+1D ResNet (~32M params) with:
• 3D-UNet (~19M)
• Deeper 3D-UNet (~34M)
• Under same experimental setup

12

Cicek et al. 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation, https://doi.org/10.1007/978-3-319-46723-8_49

10-Fold CV Results

• CLAS with R2+1D ResNet achieves highest Dice scores and accurate EDV, ESV, EF estimation

13

10-Fold CV Results

• CLAS with R2+1D ResNet achieves highest Dice scores and accurate EDV, ESV, EF estimation
• Mean Absolute Error
• EDV: 8.2 ml vs. 8.7 ml

​

14

10-Fold CV Results

• CLAS with R2+1D ResNet achieves highest Dice scores and accurate EDV, ESV, EF estimation
• Mean Absolute Error
• EDV: 8.2 ml vs. 8.7 ml
• ESV: 5.6 ml vs. 6.2 ml

​

15

10-Fold CV Results

• CLAS with R2+1D ResNet achieves highest Dice scores and accurate EDV, ESV, EF estimation
• Mean Absolute Error
• EDV: 8.2 ml vs. 8.7 ml
• ESV: 5.6 ml vs. 6.2 ml
• EF: 4.1% vs. 4.5%

​

16

Full Video Segmentation

• In full video segmentation, we use the R2+1D ResNet as feature extractor and train the network on EchoNet-Dynamic dataset

17

EchoNet-Dynamic Dataset

• 10,030 Echocardiogram
• AP4 View
• Multi-heartbeat video
• Annotated LV_endo in ED/ES

18

Ouyang et al. Video-based AI for beat-to-beat assessment of cardiac function, https://doi.org/10.1038/s41586-020-2145-8

​

​

Training Setup

• We train network on a 32-frame video clip that starts randomly in an echocardiogram but covers a clinically identified ED-ES which contains manual annotations.

19

Training Setup

• We train network on a 32-frame video clip that starts randomly in an echocardiogram but covers a clinically identified ED-ES which contains manual annotations.

20

Random 32-Frame Clip

• For the same echocardiogram, we may find multiple 32-frame clips that covers ED-ES but with different starting points.

21

Data Augmentation

• As a data augmentation, one of possible clips will be used in training during each epoch.

​

22

Extending from CLAS

• The network outputs frame-level segmentation and bi-directional motion estimation.
• Adapting CLAS’s method, we transform true ED/ES label fully backward and forward to generate the pseudo labels for all other frames.

23

Forward Deformed Pseudo Labels

Start from the true ED label to the end of clip

Outline

• Echocardiography
• Video-based Segmentation
• CLAS-FV
• R2+1D Convolution
• EchoNet-Dynamic Dataset
• Test-time EF Estimation
• Results

24

Test-time Full Video Segmentation

• Divide multi-heartbeat video into 32-frame non-overlapping consecutive clips

25

Test-time Full Video Segmentation

​

• Analysis is discontiguous at boundaries between clips

26

Test-time Full Video Segmentation

​

​

• Analysis is discontiguous at boundaries between clips
• Shifted video 1 frame forward for 4 times

27

Label Fusion

• Redo the segmentation on each shifted video
• Apply SIMPLE label fusion to merge multi-segmentation

28

Langerak et al. Label Fusion in Atlas-Based Segmentation Using a Selective and Iterative Method for Performance Level Estimation (SIMPLE) https://doi.org/10.1109/TMI.2010.2057442

​

Label Fusion

• Redo the segmentation on each shifted video
• Apply SIMPLE label fusion to merge multi-segmentation

29

Langerak et al. Label Fusion in Atlas-Based Segmentation Using a Selective and Iterative Method for Performance Level Estimation (SIMPLE) https://doi.org/10.1109/TMI.2010.2057442

​

Improved Smoothness

30

Multi-heartbeat Echocardiogram

• When inferring EF, we identify all ED-ES half-heartbeats using the sizes of fused LV_endo segmentation.
• Use the average of derived EFs

​

31

Results

​

• Trained and tested on EchoNet-Dynamic
• Use the original train test split provided by Ouyang et al.
• 7332 Train
• 1258 Valid
• 1276 Test

​

32

Ouyang et al. Video-based AI for beat-to-beat assessment of cardiac function, https://doi.org/10.1038/s41586-020-2145-8

​

​

Results

​

• 12 test patients have clinic ES frame before clinic ED frame (no clinic ED-ES)

​

33

Results

​

• 12 test patients have clinic ES frame �before clinic ED frame (no clinic ED-ES)
• CLAS failed to report Dice score on these 12 test patients

34

Results

​

• 12 test patients have clinic ES frame before clinic ED frame (no clinic ED-ES)
• Over the rest 1264 test patients
• Dice Similarity (LV ED/ES):
• CLAS-FV: 0.935/0.907
• CLAS: 0.930/0.903

35

Results

​

• 12 test patients have clinic ES frame before clinic ED frame (no clinic ED-ES)
• Over the rest 1264 test patients
• Dice Similarity (LV ED/ES):
• CLAS-FV: 0.935/0.907
• CLAS: 0.930/0.903
• Paired Wilcoxon signed-rank test:
• ED: p = 1.52e-26
• ES: p = 8.42e-05

36

Ejection Fraction

• The EF is derived using 4CH segmentation with Simpson’s Monoplane Method
• Fail to identify ED-ES half-heartbeat in 7 out of 1276 test patients when inferring EF

37

Ejection Fraction

• The EF is derived using 4CH segmentation with Simpson’s Monoplane Method
• Fail to identify ED-ES half-heartbeat in 7 out of 1276 test patients when inferring EF
• Over the rest 1269 patients
• Mean Absolute Error (± std.):
• CLAS-FV: 5.25% ± (4.45)
• CLAS: 5.82% ± (5.08)

38

Ejection Fraction

• On rest 1269 test patients:
• Bias ± 1.96 std.
• -2.1% ± 12.9

39

• Cross-correlation:
• 0.85

​

Conclusion

• Fully automated framework
• Achieve comparable segmentation accuracy and EF estimation on a large clinical dataset
• Provide dense frame-level segmentation for full cardiac cycle
• Motion outputs could be useful in strain analysis of heart muscles

40

Thank You

• Xiaoyan Zhang, PhD
• Christopher M. Haggerty, PhD
• Joshua V. Stough, PhD

​

41

• Ciffolillo Healthcare Technology Inventors Program

CAMUS

LeClerc et al., IEEE TMI 2019,

https://doi.org/10.1109/TMI.2019.2900516

EchoNet-Dynamic

Ouyang et al., Nature 2020,

https://doi.org/10.1038/s41586-020-2145-8

Loss Structure

• (SGA) Segmentation on appearance level

42

Loss Structure

• (SGA) Segmentation on appearance level�
• (OTA) Object tracking on appearance level�

43

Original Echo Frame

Forward Deformed

Backward Deformed

Loss Structure

• (SGS) Segmentation on shape level�

​

44

Auto Segmentation

Forward Deformation

Backward Deformation

Loss Structure

• (SGS) Segmentation on shape level�
• (OTS) Object tracking on shape level

45