1 of 20

Quantifying the Value of Lateral Views in Deep Learning for Chest X-rays

Mohammad Hashir¹², Hadrien Bertrand¹, and Joseph Paul Cohen¹²

¹Mila, Quebec AI Institute ²University of Montreal

https://arxiv.org/abs/2002.02582

1

2 of 20

The lateral view

The L view contains information

missing in the PA view that is

relevant for diagnosis [1].

Most chest X-ray datasets have only the PA view, but some recent ones have also the L view.

2

Postero-anterior (PA)

Lateral (L)

QUANTIFYING THE VALUE OF LATERAL VIEWS IN DEEP LEARNING FOR CHEST X-RAYS

Medical Imaging with Deep Learning

Montréal, 6 ‑ 9 July 2020

3 of 20

Task

3

Single view model

Multi-view model

Predictions

Pneumonia 0.82

Mass 0.81

Hernia 0.79

Predictions

Pneumonia 0.84 ↑

Mass 0.80 ↓

Hernia 0.82 ↑

Evaluate the contribution of a paired lateral view in chest X-ray prediction and find the best multi-view model

QUANTIFYING THE VALUE OF LATERAL VIEWS IN DEEP LEARNING FOR CHEST X-RAYS

Medical Imaging with Deep Learning

Montréal, 6 ‑ 9 July 2020

4 of 20

Our work

We explore the two questions

Does a paired lateral view help in prediction? If so, for which labels specifically?

Instead of having a paired lateral view, is it better to increase training set with more PA samples and use a single view model?

4

QUANTIFYING THE VALUE OF LATERAL VIEWS IN DEEP LEARNING FOR CHEST X-RAYS

Medical Imaging with Deep Learning

Montréal, 6 ‑ 9 July 2020

5 of 20

Materials and methods

5

6 of 20

Dataset and preprocessing

PadChest [2]

160k images from 67k Spanish patients.

Multiple labels per image from total 194.

Preprocessing

Keep patients with paired PA and L views: total 31k
Keep labels affecting 50+ patients: total 64.
Images resized to 224x224 and pixels rescaled to [-1, 1]

6

QUANTIFYING THE VALUE OF LATERAL VIEWS IN DEEP LEARNING FOR CHEST X-RAYS

Medical Imaging with Deep Learning

Montréal, 6 ‑ 9 July 2020

7 of 20

Models

7

Based on DenseNet blocks [3]. Baseline is single view DenseNet-121

Havaei et al., 2016 [4]

Rubin et al., 2018 [5]

Our contribution

QUANTIFYING THE VALUE OF LATERAL VIEWS IN DEEP LEARNING FOR CHEST X-RAYS

Medical Imaging with Deep Learning

Montréal, 6 ‑ 9 July 2020

As for our multiview models, we test different ways of joining the views

The first is stacked in which the lateral view is concatenated as a second channel of the PA image.

The second is HeMis from Havaei et al. First each view is processed by its own independent set of convolotuional layers The two views are then combined by taking the mapwise mean and variance. These are then propagated further through more layers

The third is Dualnet from Rubin et al. In this model, each view has its own independent densenet (4 blocks) but the output of the respective pooling layers is concatenated before input to the classification layer

We propose a modification of this model, where we add specific classification layers for each view and optimize all losses jointly. %This is intended to regularize the model from relying too much on both views to be available.

8 of 20

Experiments and results

8

9 of 20

Performance of multiview models

9

All joint view models perform better than single view models.

QUANTIFYING THE VALUE OF LATERAL VIEWS IN DEEP LEARNING FOR CHEST X-RAYS

Medical Imaging with Deep Learning

Montréal, 6 ‑ 9 July 2020

10 of 20

Utilization of the lateral view

10

Change in AUC as proportion of patients with paired lateral views increase

QUANTIFYING THE VALUE OF LATERAL VIEWS IN DEEP LEARNING FOR CHEST X-RAYS

Medical Imaging with Deep Learning

Montréal, 6 ‑ 9 July 2020

11 of 20

Label-wise increase with L view

11

32/64 labels see an improvement in AUC with AuxLoss

QUANTIFYING THE VALUE OF LATERAL VIEWS IN DEEP LEARNING FOR CHEST X-RAYS

Medical Imaging with Deep Learning

Montréal, 6 ‑ 9 July 2020

12 of 20

More PA samples

We add 18k patients to the training set that have a PA view but no L view.

12

QUANTIFYING THE VALUE OF LATERAL VIEWS IN DEEP LEARNING FOR CHEST X-RAYS

Medical Imaging with Deep Learning

Montréal, 6 ‑ 9 July 2020

13 of 20

Conclusion

13

14 of 20

Takeaways

Multi-view models significantly better than single view overall
32 labels improve with multi-view model
Doubling PA images in training set -> change in AUC not significant

14

QUANTIFYING THE VALUE OF LATERAL VIEWS IN DEEP LEARNING FOR CHEST X-RAYS

Medical Imaging with Deep Learning

Montréal, 6 ‑ 9 July 2020

15 of 20

Thank you

arxiv.org/abs/2002.02582

15

16 of 20

References

[1] Raoof, Suhail, et al. "Interpretation of plain chest roentgenogram." Chest 141.2 (2012): 545-558.

[2] Bustos, Aurelia, et al. "Padchest: A large chest x-ray image dataset with multi-label annotated reports." arXiv preprint arXiv:1901.07441 (2019).

[3] Huang, Gao, et al. "Densely connected convolutional networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

[4] Havaei, Mohammad, et al. "Hemis: Hetero-modal image segmentation." International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2016.

[5] Rubin, Jonathan, et al. "Large scale automated reading of frontal and lateral chest x-rays using dual convolutional neural networks." arXiv preprint arXiv:1804.07839 (2018).

16

QUANTIFYING THE VALUE OF LATERAL VIEWS IN DEEP LEARNING FOR CHEST X-RAYS

Medical Imaging with Deep Learning

Montréal, 6 ‑ 9 July 2020

17 of 20

Appendix

17

18 of 20

Why AuxLoss

Advantages of AuxLoss

Uses both views productively
Robust to missing views
Lowest variance across multiview models
Less sensitive to hyperparameter changes

18

Multiview models at test time perform similarly when given both views but diverge significantly when given only one view

Figure 4: Distributions of AUC for a 40 combination hyperparameter search for each model. Some models are much more robust to hyperparameter changes than others.

QUANTIFYING THE VALUE OF LATERAL VIEWS IN DEEP LEARNING FOR CHEST X-RAYS

Medical Imaging with Deep Learning

Montréal, 6 ‑ 9 July 2020

19 of 20

Training details

Hyperparameters found through extensive search

40 epochs, batch size of 8 and Adam optimizer
Early stopping on validation AUC
Loss weighted by class frequency (clamped at 5.0 max)
Learning rate scaled by 0.1 every 10 epochs but initial LR different for every model
Curriculum learning: views dropped randomly for Hemis and AuxLoss
Dropout of 0.1-0.2

19

QUANTIFYING THE VALUE OF LATERAL VIEWS IN DEEP LEARNING FOR CHEST X-RAYS

Medical Imaging with Deep Learning

Montréal, 6 ‑ 9 July 2020

20 of 20

Label-wise increase with more PA samples

32 labels

22 overlap with AuxLoss

20

QUANTIFYING THE VALUE OF LATERAL VIEWS IN DEEP LEARNING FOR CHEST X-RAYS

Medical Imaging with Deep Learning

Montréal, 6 ‑ 9 July 2020