1 of 25

There has been recent work in medical image translation via distribution matching

1

MR->CT

CT-> PET

Synthesized H&E staining

2 of 25

Image translation (via distribution matching) should not be used for direct interpretation.

2

Our statement:

3 of 25

Losses like in CycleGAN just match distributions

3

Training Data

Model Output

4 of 25

They are very good at distribution matching

4

[CycleGAN, Zhu 2017]

[Karras, 2018]

5 of 25

But a bias in training data can lead to incorrect translation

5

T1 Transformed

Everyone is so healthy!

T1 Real

Real Image

Image

Translation/

Synthesis

Source Image

T1 Real

6 of 25

Even with all the training data in the world today.

6

Time t

Time t+1

There will be new diseases tomorrow that are out of distribution.

7 of 25

What is image translation via distribution matching?

7

The generator learns to match the target distribution to fool the discriminator

Generator/

Translator

Fake

Discriminator

Real or Fake ?

Fake

Real

Real

8 of 25

Model Breakdown

8

9 of 25

  • should produce examples in �
  • can be anything non-finite, like a Gaussian�
  • No guarantee mapping maintains phenotypes

9

Optimizing

GAN

CycleGAN

CondGAN

L1

Optimizing

10 of 25

  • Add a reconstruction loss regularizer for the function�
  • Loss term still matches distribution �
  • No guarantee mapping maintains phenotypes

10

Optimizing

Optimizing

GAN

CycleGAN

CondGAN

L1

Optimizing

Cycle Loss!

a

b

11 of 25

  • is given paired examples allowing detection of what to preserve�
  • still plays a role in what learns �
  • No guarantee mapping maintains phenotypes

​

11

Optimizing

GAN

CycleGAN

CondGAN

L1

Optimizing

12 of 25

  • should produce examples in �
  • Pixel-wise loss�
  • No guarantee mapping maintains phenotypes

​

​

12

Optimizing

GAN

CycleGAN

CondGAN

L1

13 of 25

Visual Evaluation

13

14 of 25

14

CycleGAN results

% training data with tumor

15 of 25

15

Real Flair

Real T1

Biased Transformations

16 of 25

16

Example with a tumor from the holdout set

17 of 25

Quantitative Evaluation

17

18 of 25

18

healthy

tumor

Predicted as a:

Test data is healthy

Test data has tumor

19 of 25

19

healthy

tumor

Predicted as a:

Test data is healthy

Test data has tumor

20 of 25

Image translation (via distribution matching) should not be used for direct interpretation.

20

Our statement:

21 of 25

Where do go from here?

  1. How to guarantee image translation? (I doubt it)

�

  • Where should distribution matching be used in medical imaging?
    1. Data augmentation (for classification, segmentation, registration)
    2. Better features (for unsupervised learning)
    3. To correct model predictions [Zhang MICCAI 2017]

21

22 of 25

Limitations

  • We test only a subset of loss terms which compose most methods�
  • The synthetic BRATS 2013 data had tumors added to healthy brains (in real data the entire brain is sick)

22

23 of 25

Pr. Yoshua Bengio, PhD

Francis Dutil

Martin Weiss

Tristan Sylvain

Margaux Luck, PhD

Assya Trofimov

Vincent Frappier, PhD

Joseph Paul Cohen, PhD

Shawn Tan

Sina Honari

Geneviève Boucher

24 of 25

Thanks!

24

25 of 25

Distribution Matching Losses Can Hallucinate Features

in Medical Image Translation

25

See us at poster M-60

https://arxiv.org/abs/1805.08841

Joseph Paul Cohen

Margaux Luck

Sina Honari