Approaches and Challenges for using Artificial Intelligence in Medical Imaging

The 2nd Swiss Deep Learning Days

Kevin Mader

22nd of September 2017

There’s Waldo!

WHAT DOES STAGING MEAN?

Categorizing the severity of the disease

• 5 levels for the tumor
• 4 levels for the lymph nodes
• 2 levels for metastases
• 8 different stages for treatment planning
• 40 different possible categories

American Cancer Society 2016

WHY AUTOMATE STAGING?

• Mis-staging patients can have serious consequences as both under and over treating can drastically affect patient health
• Currently two physicians take over an hour to perform the initial assessment
• Reports are often incomplete or missing key information

JUST DEEP LEARN!

• Everybody is doing it!
• Google → Inception, Show and Tell, Retina Disease
• Facebook → DeepFace
• Apple → Deploy DNN models on your iPhone
• Baidu → Recognize complicated sentences and synthesize realistic speech
• Watson → Win Jeopardy and read
• Academia as well
• Stanford → ImageNet, Visual Genome
• U Washington → YOLO

WHAT TO LEARN?

Input

700 x 512 x 512 x 2 x 32bit [367MPixels]

Output

TNM Stage [3 values]

JUST CLASSIFICATION?

• Physicians do a lot more then just check a box
• Detailed Report for each scan (50-1000 words)
• Covers imaging technique, patient information, questions to be answered, other findings
• Very lightly structured, very personalized content

JUST CLASSIFICATION?

• Physicians examine images and look for lesions (detection)
• Image[float] -> Image[boolean]
• Each lesion is then classified into a lesion type (lesion classification)
• Image[boolean] -> (Image[int], List[string])
• Features are generated for each lesion
• Image[int] -> List[FeatureVectors]
• The collection of lesions is then used to classify the patient
• (List[FeatureVectors], List[string]) -> (T_stage, N_stage, M_stage)

IS DEEP LEARNING OUR SAVING GRACE?

Deep Learning a spiral pattern from noise-free data: 6 layers, 30+ neurons, 3000+ epochs of training

Programming a custom feature for spirals (1 line of code)

IS DEEP LEARNING OUR SAVING GRACE?

GPU

• ~16-24GB of memory
• SIMD very well, thousands of cores
• Branching very poorly

CPU

• >1TB of memory per node
• Memory mapping
• A few cores, that can run fast and branch well

​

InceptionV3 for medical volumes would need 338GB for a batch size of 1

TRAINING DATA?

ImageNet

• 59K pixels per image
• 1.2M images
• 0.04 variables (pixels) per example
• 1200 examples per category

​

Lung Cancer

• 157M pixels per patient
• 2,000 patients
• 78,000 variables (pixels) per example (3.9M times worse)
• 50 examples per category (24 times worse)

TRAINING DATA?

We have gotten really far by learning to

• stir big piles (Stochastic Gradient Descent on Deep Neural Networks)
• really fast (GPU)
• With lots of data to pour

https://xkcd.com/1838/

STARTING WITH DEEP LEARNING

A few models that even work well for medical images / limited training data

• UNET
• 3D CNN
• PyraMid LSTM

https://www.kaggle.com/kmader/simple-nn-with-keras

WELL….

Using UNET to segment bone requires around

• 7M parameters to be fit
• > 2,000 manually labeled test images
• Hours of training
• Execution time on standard CPU systems measured in hours for a single volume

WELL….

A well adapted approach with standard techniques requires around

• 15 parameters
• All of which have anatomical / physiological significance
• 1% of the computational time
• 2% higher accuracy

LEARNING, FORGETTING, RELEARNING

• Using Deep Learning to segment lungs works really well on training data (>99%)
• Fails simple sanity checks
• Rotating 90
• Noisy scanners
• Images without lungs
• Retraining causes it to forget how to do it’s original job

https://www.kaggle.com/kmader/simple-nn-with-keras

LITTLE DIFFERENCES

• Every patient is slightly differently positioned on the scanner
• Every scanner has slightly different artifacts
• Every patient has at least a few hundred (thousand) completely benign abnormalities

https://www.kaggle.com/kmader/simple-nn-with-keras

https://xkcd.com/1831/

JUST CLASSIFICATION?

• Physicians examine images and look for lesions (detection)
• Image[float] -> Image[boolean]
• Each lesion is then classified into a lesion type (lesion classification)
• Image[boolean] -> (Image[int], List[string])
• Features are generated for each lesion
• Image[int] -> List[FeatureVectors]
• The collection of lesions is then used to classify the patient
• (List[FeatureVectors], List[string]) -> (T_stage, N_stage, M_stage)

WE NEED TO LEARN SMART!

Incorporate things we already know

• Lung tumors don’t show up in the brain (context)
• Lungs are supposed to be full of air (normality)

Train networks more efficiently

• Standard loss functions like
• MSE, MAE (regression)
• DICE, IOU (segmentation)
• Accuracy and cross-entropy (classification)
• Do not apply well to many medical problems, one or two pixels of metastases COMPLETELY change treatment planning and outcome likelihoods (<0.001% of the image)

INCORPORATING WHAT WE ALREADY KNOW

• The body and anatomy are fairly well understood
• We can use Bayesian models to incorporate all of the prior information better and establish
• Bayesian Networks
• Selectively incorporate information outside of simple patient images
• Bayesian Conditional Random Fields
• Begin to incorporate larger ideas about shape and structure

INCORPORATING WHAT WE ALREADY KNOW

What is normal?

• Have the network imagine how the patient should look
• Simulating normality (Generative Models)
• Find regions which don’t match this well

TRAINING MORE EFFICIENTLY

• Neural Networks that won image net found categories of images
• A cat in the corner is the same as a cat in the middle
• What does Deep Learning give us?
• Using Convolutional Neural Networks
• fully position independent
• Few parameters (few thousand per layer)
• Using Fully Connected Layers
• Fully position dependent (every pixel counts)
• 512x512 image has 68 billion free parameters
• Volume has 1.8e16 parameters (512^6)
• Medicine is different
• A bulge in the lung isn’t the same as a bulge in your knee
• A spot on your arm could be a life-threatening metastases or an FDG tracer injection site

TRAINING MORE EFFICIENTLY

• Decision boundaries are not linear
• A lot of tumor might not always be a problem
• A tiny bit of metastases always is a problem
• Class distributions are very skewed
• A lot of cancer is still less than 1% of the image
• Adversarial techniques allow us to ‘learn the loss metric’ rather than trying to manually define and tune it
• Adapt to get better results

HIGH QUALITY, TARGETED ANNOTATIONS

• Focus the algorithms on the pixels that are important
• Most of them aren’t
• < 0.02%

​

40 different categories of annotation

Incorporating targeted pieces of information

ANNOTATIONS ARE HARD

INCORPORATING OTHER INFORMATION IS DIFFICULT

The golden dagger of back-propagation doesn’t apply to many existing algorithms

• How to incorporate
• Atlases
• Known segmentations
• Other image data
• Registration
• Different views

​

Using approaches like synthetic gradients allows many of these pieces to be decoupled

​

DeepMind: Synthetic Gradients

SMART LEARNING

ImageNet

• 59K pixels per image
• 1.2M images
• 0.04 variables per example
• 1200 examples per category

​

LungStage (+Annotations)

• ~4000 pixels per patient
• 2000 patients
• 2 variables (pixels) per example (50 times worse)
• 50 examples per category (24 times worse)

Other Challenges

CURRENT STATUS

~ 1000 patients

> 3,000 lesions

EVALUATING RESULTS

• Nobody really knows what works best so gotta try them all.
• A good cross validation method is essential for figuring out which models learn well

TRANSFER LEARNING

• Medical images are black and white
• Pretrained models are color
• We can learn a pseudo-color mapping by prepending layers to the model

EVALUATING RESULTS

• Each network can take days to train on distributed HPC systems
• > 10,000 different network structures
• Finding the best will still take some time, but in the meantime, good enough will work

> 4 billion calculations per patient

VISUALIZING RESULTS

• TSNE can be a useful visualization
• We can focus on the more difficult lesions

VISUALIZING RESULTS

• Looking at mistakes
• Resample classes which perform poorly to improve them

COMMUNICATING RESULTS

As important as making a classification is delivering the confidence in the classification. Neural networks are naturally very bad at this, but Bayesian Networks and simpler models can help

CONSTRUCTING SIMPLE RULES

Our Results

CONSTRUCTING SIMPLE RULES

Our Results

CONSTRUCTING SIMPLE RULES

Our Results

INTERACTIVE TOOLS

UNROLLING RESULTS TO TEXT

LungStage found 20 suspicious tumor regions and decided on T4 because of the shown lesions which are invasive in the mediastinum

NEXT STEPS

• We are looking to involve other hospitals and health-systems
• We want to further improve the diagnosis of lung cancer.
• We want to let radiologists make our models better (make code better by giving feedback and using the tools)
• Unsupervised approaches for finding lesions better (RSNA 2017)

CEO & Co-Founder

Joachim Hagger

Master of Science in Physics

TEAM

CTO & Co-Founder

Dr. Kevin Mader

Doctor of Sciences ETHZ

CFO & Co-Founder

Flavio Trolese

Dipl. Ing. FH Informatik

​

Scientific Advisor & Co-Founder

Prof. Dr. Marco Stampanoni

Inst. f. Biomedizinische Technik

CMO

Bram Stieltjes, MD, PhD

Research Group Leader

Dr. Thomas J Re, MD, MSEE

Research / Radiology Insights

Dr. med. Dipl. Phys. Gregor Sommer

Stv. Oberarzt Kardiale und

Thorakale Diagnostik

PD Dr. Tobias Heye

Oberarzt und stv. Leiter Kardiale

und Thorakale Diagnostik

PD Dr. Alexander Sauter

Facharzt Radiologie,

Assistenzarzt Nuklearmedizin

Joshy Cyriac

Machine Learning / Software Engineering

Prof. Dr. Elmar M. Merkle

Chefarzt und Leiter Klinik für

Radiologie und Nuklearmedizin

Partner

Interdisciplinary and experienced team

4Quant�BIG IMAGE ANALYTICS

Thank you for your attention!

​

Yes, we’re hiring!

Explore our data

​

Detect lung nodules in CT images

https://www.kaggle.com/kmader/lungnodemalignancy

​

Look for cancer in PETCT Images

https://www.kaggle.com/4quant/soft-tissue-sarcoma

​