1 of 97

1

1/4/25

Lectures 1 & 2

Introduction to Medical Imaging and Analysis

​

BME 495:

Deep Learning for Medical Imaging

​

​

​

Ulas Bagci, Ph.D.,

​

​

​

Director of Machine & Hybrid Intelligence Lab,�Northwestern University, Chicago

Machine & Hybrid Intelligence Lab

2 of 97

Outline of Lectures 1 & 2

Overview of medical image analysis and its importance in healthcare

Types of medical images (X-rays, CT scans, MRI scans, etc.)

Challenges & Software Basics

Machine & Hybrid Intelligence Lab

3 of 97

3

Medical AI

Image Processing

Computer Vision

Machine Learning

Imaging Sciences (Radiology, Biomedical)

The NYT recently ranked biomedical jobs as the number one fastest growing career field in the nation and listed bio-medical imaging as the primary reason for the growth.

Machine & Hybrid Intelligence Lab

4 of 97

Medical Imaging

  • The most direct way to see inside the human (or animal) body is cut it open (i.e., surgery)
  • We can see inside the human body in ways that are less invasive or (completely non-invasive)
  • We can even see metabolic/functional/molecular activities which are not visible to naked eye

​

Image Processing

Image quality improvement

Machine Learning

Tissue types

Image Understanding

Semantic description & content understanding

Machine & Hybrid Intelligence Lab

5 of 97

Where do radiologists interpret scans?

5

  • Dedicated light source
  • Darkened environment
  • Limited distraction

Machine & Hybrid Intelligence Lab

6 of 97

Radiologists are great, but they can miss tumors!

6

Machine & Hybrid Intelligence Lab

7 of 97

Radiologists are great, but they can miss tumors!

7

(tiny/ small tumors, similar to normal tissues, and other biases)

Machine & Hybrid Intelligence Lab

8 of 97

Radiologists are great, but they can miss tumors!

8

(tiny/ small tumors, similar to normal tissues, and other biases)

Machine & Hybrid Intelligence Lab

9 of 97

Machine & Hybrid Intelligence Lab

10 of 97

Machine & Hybrid Intelligence Lab

11 of 97

human error (visual search error) remains a significant problem to detect abnormalities.�

    • 35% of lung nodules are typically missed during the screening process.2�
    • Over-diagnosis is another significant bias leading to unnecessary treatment.

Machine & Hybrid Intelligence Lab

12 of 97

human error (visual search error) remains a significant problem to detect abnormalities.�

    • 35% of lung nodules are typically missed during the screening process.2�
    • Over-diagnosis is another significant bias leading to unnecessary treatment.

"20 (out of 24) radiologists who did not

report the gorilla, 12 looked directly at the gorilla’s location when it was visible.”

Drew et al., 2013

Machine & Hybrid Intelligence Lab

13 of 97

Medical AI is important because …

  • Improved Diagnostic Accuracy
    • AI algorithms can analyze complex medical data with high precision, often detecting nuances that may be missed by humans.

13

Six radiologists missed the tumor while AI captured!

​

https://www.nature.com/articles/s41586-019-1799-6

​

Machine & Hybrid Intelligence Lab

14 of 97

Medical AI is important because …

  • Efficiency & Speed
    • AI can process and analyze large volumes of data much faster than human practitioners, leading to quicker diagnoses and treatment plans.

14

https://www.frontiersin.org/journals/digital-health/articles/10.3389/fdgth.2021.671015/full

While an AI model can segment (label) a pancreas within a few seconds only, for radiologists, this can be an hour per 3D pancreas, even more difficult For MRI !

Machine & Hybrid Intelligence Lab

15 of 97

Medical AI is important because …

  • Predictictive Analytics
    • AI can predict disease progression and outcomes, allowing for proactive healthcare interventions.

15

A deep learning algorithm trained to analyze images from MRI scans predicts the presence of an IDH1 gene mutation in brain tumors.

​

​

​

​

​

Credit: CA Cancer J Clin March/April 2019. doi: 10.3322/caac.21552. CC BY 4.0.

Machine & Hybrid Intelligence Lab

16 of 97

Medical AI is important because …

  • Predictictive Analytics
    • AI can predict disease progression and outcomes, allowing for proactive healthcare interventions.

16

https://www.nature.com/articles/s41571-020-0417-8

Machine & Hybrid Intelligence Lab

17 of 97

Medical AI is important because …

  • Cost Reduction
    • Automating routine tasks and improving diagnostic workflows with AI can lead to significant cost savings in healthcare.

17

https://gloriumtech.com/ai-reducing-healthcare-costs/

Machine & Hybrid Intelligence Lab

18 of 97

Medical AI is important because …

  • Personalized Medicine
    • AI supports the analysis of individual genetic makeup, contributing to customized treatment regimens.
  • Reducing Human Errors
    • By providing decision support, AI can reduce errors due to human factors such as fatigue and cognitive biases.
  • Accessibility
    • AI can extend medical expertise to underserved or remote areas, improving healthcare access.
  • Help Discovery Sciences
    • AI can lead discoveries by exploring the data and finding hidden patterns in the data associated with the outcome
  • …

18

Machine & Hybrid Intelligence Lab

19 of 97

Knowledge Check

  • X-Ray ?
  • CT ?
  • MRI ?
  • fMRI ? (functional MRI)
  • Diffusion MRI / Diffusion Tensor Imaging
  • EEG ?
  • MEG ?
  • MRS?
  • MPI ?

19

Machine & Hybrid Intelligence Lab

20 of 97

20

ELECTROMAGNETIC SPECTRUM (P. Suetens)

Machine & Hybrid Intelligence Lab

21 of 97

X-Ray Imaging / Radiography

  • The first published medical image was a radiograph of the hand of Wilhelm Conrad Roentgen’s wife in 1895. Nobel Prize in Physics 1901.

21

routine diagnostic radiography (2D images):

chest x-rays, fluoroscopy, mammography, motion tomography, angiography, …

Machine & Hybrid Intelligence Lab

22 of 97

X-Ray Imaging / Radiography

22

D=Optical density

E=exposure (Iin/Iout)

Iin=incoming light intensity

Iout=outgoing light intensity

Machine & Hybrid Intelligence Lab

23 of 97

X-Ray Imaging / Radiography-Sensitometric Curve

23

Linear part (useful!)

  • Maximum slope of the curve is

known as the gamma of the film.

​

  • A larger slope implies a higher

contrast at the cost of a smaller

useful exposure range

​

  • In low and high density areas, contrast is low and little information available.

​

  • In linear part, slope characterize

Contrast of the film. Max slope is known as Gamma of the film.

Defn. Contrast: is the intensity difference in adjacent regions of the image.

Machine & Hybrid Intelligence Lab

24 of 97

Basics Use of X-Rays

  • Dental examinations
  • Surgical markers prior to invasive procedures
  • Mammography
  • Orthopedic evaluations
  • Chest examination (Tuberculosis)
  • Age estimation (forensic, left hand)

24

Machine & Hybrid Intelligence Lab

25 of 97

Clinical Examples – X-Rays

25

PELVIS

ELBOW

Machine & Hybrid Intelligence Lab

26 of 97

How Radiologists Search Abnormal Patterns in Chest X-Rays?

26

Patterns belonging to Potentially Benign Lesions

Patterns belonging to Potentially Malignant Lesions

Machine & Hybrid Intelligence Lab

27 of 97

How Radiologists Search Abnormal Patterns in Chest X-Rays?

27

Radiologists often report the following

  • Size, dimension, volume
  • Pattern description,
  • Location,
  • Interaction with Nearby structures,
  • Intensity distribution
  • Shape
  • …

Difficulties

  • Noise
  • vessels can be seen as small nodules
  • radiologists may miss the pattern
  • patterns may not be diagnostic
  • CT often required for better diagnosis
  • size estimation is done manually in 2D
  • Shadowing
  • total lung capacity computation

Machine & Hybrid Intelligence Lab

28 of 97

28

Radiologists often report the following

  • Size, dimension, volume
  • Pattern description,
  • Location,
  • Interaction with Nearby structures,
  • Intensity distribution
  • Shape
  • …

Difficulties

  • Noise
  • vessels can be seen as small nodules
  • radiologists may miss the pattern
  • patterns may not be diagnostic
  • CT often required for better diagnosis
  • size estimation is done manually in 2D
  • Shadowing
  • total lung capacity computation

Computer algorithms can solve/simplify these problems for improved healthcare

How Radiologists Search Abnormal Patterns in Chest X-Rays?

Machine & Hybrid Intelligence Lab

29 of 97

Another Example for X-ray Imaging

29

Benign

Malignant

Machine & Hybrid Intelligence Lab

30 of 97

Ultrasound Imaging

  • US is defined as any sound wave above 20KHz

30

1794-Lazzaro Spallanzani - Physiologist

First to study US physics by deducing bats

used to US to navigate by echolocation.

1826-Jean Daniel Colladon - Physicist

Uses church bell (early transducer) under water to calculate speed of sound through water prove sound traveled faster through water than air.

1880-Pierre&Jacques Curie

discover the Piezo-Electric Effect (ability of certain materials to generate an electric charge in response to applied mechanical stress.

Machine & Hybrid Intelligence Lab

31 of 97

31

1942-Karl Dussik - Neurologist

First physician to use US for medical diagnosis

1948-George Ludwig - MD

First described the use of US to diagnose gallstones

1958-Ian Donald

Pioneers in OB-GYN

US Imaging Technology

Machine & Hybrid Intelligence Lab

32 of 97

Principle of US Imaging

32

US equipment assumes that sound velocity is constant in the body.

Machine & Hybrid Intelligence Lab

33 of 97

Features of US Imaging

  • Resolution:
    • direction of pulse propagation, pulse width 1-2mm
    • direction of scanning: beam width 2-3mm
    • low resolution and low SNR in deep region
  • Ability of imaging soft tissue
  • imaging in real time
  • Doppler image
  • Artefacts

33

Color flow mapping shows simultaneous amplitude (US) and velocity information (doppler)

Machine & Hybrid Intelligence Lab

34 of 97

Clinical Use of US Imaging

34

Machine & Hybrid Intelligence Lab

35 of 97

Clinical Use of US Imaging

35

Renal Artery Blood Flow

manual measurements?

can computer help calculating

all blood flow and identify

automatically the abnormal regions?

(See Next Lecture, afternoon)

stenosis is seen

eca: external carotid artery

cca: common carotid artery

ica: internal carotid artery

Machine & Hybrid Intelligence Lab

36 of 97

Clinical Use of US Imaging

36

Bone, fat, and physical length

Measurements –unborn babies

(Image Credit: S. Rueda, Oxford Univ.)

Machine & Hybrid Intelligence Lab

37 of 97

Computed Tomography (CT)

37

Tomo: slice/level (Greek)

Graphe: draw

Machine & Hybrid Intelligence Lab

38 of 97

CT Imaging (continue)

38

C-arm

CT

Micro-CT

~CAT Scan

(computerized

Axial tomography)

Machine & Hybrid Intelligence Lab

39 of 97

3D Nature of CT

39

Machine & Hybrid Intelligence Lab

40 of 97

3D View Terminology

40

Machine & Hybrid Intelligence Lab

41 of 97

3D Images

41

x

y

z

I: Image

I(x,y,z) denotes intensity value at pixel location x,y,z

Note also that whatever you see on the left is right part of the body!

Machine & Hybrid Intelligence Lab

42 of 97

Clinical Use of CT Imaging

  • Standard imaging technique in many organs, particularly gold standard for lung imaging
  • Fast
  • Radiation exposure
  • Often used in surgery rooms
  • Show anatomy and pathology
  • Intensity values are (more-or-less) fixed, read as HU (Hounsfield Unit)

42

Machine & Hybrid Intelligence Lab

43 of 97

CT Imaging Example: Tumor

43

2D manual measurement of tumor size (short and long axis of tumor)

Machine & Hybrid Intelligence Lab

44 of 97

CT Imaging Example: Lung

44

Machine & Hybrid Intelligence Lab

45 of 97

CT Imaging Example: Cardiac

45

how to calculate the amount of fluid?

​

Fluid

Machine & Hybrid Intelligence Lab

46 of 97

Magnetic Resonance Imaging (MRI)

  • 1882-Nichola Tesla
  • Discovered rotating magnetic field
  • 1971-Paul Lauterbur NOBEL PRIZE
  • First invented MRI
  • Late 1970-Sir Peter Mansfield (Nottingham) NOBEL PRIZE
  • Developed mathematical techniques to create clearer images and also in minutes rather than hours as Lauterbur did.

​

​

  • CT is more widely used than MRI.
  • MRI does not have ionizing-radiation.
  • MRI has excellent soft tissue contrast, while CT is preferred for lung and bone imaging.
  • CT is fast (few seconds), while MRI is slow (sparse MRI ~5-10 mins, abdomen or brain may take 30-40 mins).

46

Machine & Hybrid Intelligence Lab

47 of 97

Brief History of MRI

47

Machine & Hybrid Intelligence Lab

48 of 97

Nobel Prizes for MRI

​

  • 1944: Rabi

Physics (Measured magnetic moment of nucleus)

​

  • 1952: Felix Bloch and Edward Mills Purcell

Physics (Basic science of NMR phenomenon)

​

  • 1991: Richard Ernst

Chemistry (High-resolution pulsed FT-NMR)

​

  • 2002: Kurt Wüthrich

Chemistry (3D molecular structure in solution by NMR)

​

  • 2003: Paul Lauterbur & Peter Mansfield

Physiology or Medicine (MRI technology)

​

48

Machine & Hybrid Intelligence Lab

49 of 97

MRI Hardware Setup - Details

49

Machine & Hybrid Intelligence Lab

50 of 97

MRI Basics

50

Machine & Hybrid Intelligence Lab

51 of 97

51

Machine & Hybrid Intelligence Lab

52 of 97

MRI Basics

52

No magnetization

Machine & Hybrid Intelligence Lab

53 of 97

The Magnet

53

Machine & Hybrid Intelligence Lab

54 of 97

54

Machine & Hybrid Intelligence Lab

55 of 97

Types of MRI

55

Machine & Hybrid Intelligence Lab

56 of 97

1/4/25

TR

Long

Short

Short

Long

TE

Proton

Density

T1

poor!

Image contrast summary: TR, TE

T2

Machine & Hybrid Intelligence Lab

57 of 97

Brain MRI

57

Machine & Hybrid Intelligence Lab

58 of 97

The brain of a volunteer is imaged using a 3-T (left) and 9.4-T (right) magnetic resonance imaging machine.Credit: Rolf Pohmann/Max-Planck-Institute for Biological Cybernetics�(https://www.nature.com/articles/d41586-018-07182-7)

58

Machine & Hybrid Intelligence Lab

59 of 97

59

Machine & Hybrid Intelligence Lab

60 of 97

60

Machine & Hybrid Intelligence Lab

61 of 97

Safety in MRI

61

Machine & Hybrid Intelligence Lab

62 of 97

Diffusion Tensor Imaging (DTI)

62

  • MRI (sub-)modality
  • measures random Brownian motion of water molecules.
  • useful for tumor characterization (densely cellular tissues exhibit lower diffusion).

Machine & Hybrid Intelligence Lab

63 of 97

Diffusion Weighted Imaging (DWI)

63

Glioblastoma Tumor

Machine & Hybrid Intelligence Lab

64 of 97

Clinical Use: Example

64

Machine & Hybrid Intelligence Lab

65 of 97

Clinical Use: Example

65

Myocardial Infarction Detection

Machine & Hybrid Intelligence Lab

66 of 97

Clinical Use: Example

66

rectal tumor

Machine & Hybrid Intelligence Lab

67 of 97

Human brain mapped in unprecedented detail�(https://www.nature.com/news/human-brain-mapped-in-unprecedented-detail-1.20285)

67

Now, neuroscientists have charted an equivalent map of the brain’s outermost layer — the cerebral cortex — subdividing each hemisphere's mountain- and valley-like folds into 180 separate parcels.

Machine & Hybrid Intelligence Lab

68 of 97

Functional MRI (fMRI)

  • measures brain activity through oxygen concentration in the blood flow.
  • relies on the fact that cerebral blood flow and neuronal activation are coupled.
  • when area of the brain is active (in use), blood flow to that area also increases.

​

  • which part/location of the brain is activated when reading?
  • which part/location of the brain is activated when listening music?
  • which part/location of the brain is activated when searching puzzle?

68

Machine & Hybrid Intelligence Lab

69 of 97

fMRI Settings

69

Active Regions

Machine & Hybrid Intelligence Lab

70 of 97

Nuclear Medicine Imaging – PET/SPECT

  • Scint: Scintigraphy, two-dimensional images
  • PET: Positron Emission Tomography
  • SPECT: Single Photon Emission Tomography

70

Machine & Hybrid Intelligence Lab

71 of 97

Nuclear Medicine Imaging – PET/SPECT

71

Machine & Hybrid Intelligence Lab

72 of 97

Basics of PET Imaging

  • uses short-lived positron emitting isotopes (produced by collimators)
  • two gamma rays are produced from the annihilation of each positron and can be detected by specialized gamma cameras
  • resulting image show the distribution of isotopes
  • an agent is used to bind into isotopes (glucose, …)

72

Late 1950s, David L. Kuhl

concept of emission and transmission

​

molecular activity is measured.

Machine & Hybrid Intelligence Lab

73 of 97

PET/CT and MRI/PET (Hybrid Imaging)

73

PET/CT

-choice of modality for oncological applications(yet)

MRI/PET

-superior soft tissue

contrast resolution

-minimized radiation

Machine & Hybrid Intelligence Lab

74 of 97

What to Measure in PET?

  • SUV (standardized uptake value: voxel-wise or region-wise) (SUVpeak, SUVmax, SUVlbm)

​

  • Metabolic lesion/tumor volume (MTV)

​

  • Shape information of (functional) lesion (spiculated vs focal)

​

  • Texture information of lesion (heterogeneous vs homogeneous)

​

  • Number and distribution of the lesions (focal, multi-focal)

74

Machine & Hybrid Intelligence Lab

75 of 97

Clinical Use of PET: Example

75

Machine & Hybrid Intelligence Lab

76 of 97

Clinical Use of PET: Example

76

Machine & Hybrid Intelligence Lab

77 of 97

Serial and Simultaneous MRI/PET

77

Past

Now!

Machine & Hybrid Intelligence Lab

78 of 97

Shallow Comparison of Imaging Methods

78

�

Chest

Abdomen

Head/Neck

Cardiovascular

Skeletal/muscular

CT

gold standard

Need contrast for excellency, widely used

Good for trauma

Gold standard

Gold standard

US

no use except heart or P.Effusion

Problems with gas

Poor

Poor

Elastography

Nuclear

Extensive use in heart and therapy in lung

CT or MRI is merged

PET

Perfusion

bone marrow

MRI

growing cardiac applications

Increased role of MRI

Gold standard

Will replace ct in near future

Excellent

Machine & Hybrid Intelligence Lab

79 of 97

Medical Image Formats

  • Dicom
  • Nifti
  • Analyze (img/hdr)
  • Raw data
  • …

79

Machine & Hybrid Intelligence Lab

80 of 97

Machine & Hybrid Intelligence Lab

81 of 97

Digital Images

81

What computer sees!

Machine & Hybrid Intelligence Lab

82 of 97

Digital Images

  • Definition: A digital image is defined by integrating and sampling continuous (analog) data in a spatial domain [Klette, 2014].

82

Machine & Hybrid Intelligence Lab

83 of 97

Picture Elements (Pixels), Volume Elements (Voxels)

83

PIXELS are ATOMIC ELEMENTS of an image.

In late 1960s, terminology ‘pixel’ was introduced by a group of scientist at JPL in California!

Machine & Hybrid Intelligence Lab

84 of 97

Image Types

  • A scalar image has integer values

​

a: level (bit)

​

Ex. If 8 bit (a=8), image spans from 0 to 255

0 black

255 white

Ex. If 1 bit (a=1), it is binary image, 0 and 1 only.

84

Machine & Hybrid Intelligence Lab

85 of 97

Image Types-Color

  • Image has three channels (bands), each channel spans a-bit values.
  • RGB, Hue-Saturation-Brightness

85

Machine & Hybrid Intelligence Lab

86 of 97

3D Visualization

86

3D SLICER

Machine & Hybrid Intelligence Lab

87 of 97

The interlace deposition method of bone equivalent material extrusion 3D printing for imaging in radiotherapy

Machine & Hybrid Intelligence Lab

88 of 97

Free Software to use in this course

  • ImageJ (and/or FIJI)
  • MONAI (NVIDIA) (Deep Learning)
  • ITK-Snap
  • SimpleITK
  • MITK
  • FreeSurfer (Brain applications)
  • SLICER (tutorial is provided in this course)
  • OsiriX (Mac – Vis program)
  • An extensive list of software: www.idoimaging.com

​

blue: will be frequently used in this course

88

Machine & Hybrid Intelligence Lab

89 of 97

DeepMedic (from Imperial College, BiomedIA)

89

https://biomedia.doc.ic.ac.uk/software/deepmedic/

Machine & Hybrid Intelligence Lab

90 of 97

DeepMedic (from Imperial College, BiomedIA)

90

The system has been shown to yield excellent performance (winner of the ISLES 2015 competition) on challenging lesion segmentation tasks, including traumatic brain injuries, brain tumors, and ischemic stroke lesions.

Machine & Hybrid Intelligence Lab

91 of 97

91

Machine & Hybrid Intelligence Lab

92 of 97

92

Machine & Hybrid Intelligence Lab

93 of 97

FIJI (or ImageJ)

93

Machine & Hybrid Intelligence Lab

94 of 97

Coding

  • TensorFlow is OK too, PyTorch is preferable!
    • Ease of debugging, Dynamic computation graph, intuitive syntax, strong community support, comprehensive ecosystem, and etc!
  • www.paperswithcode.com

​

94

https://www.learnpytorch.io/01_pytorch_workflow/

Machine & Hybrid Intelligence Lab

95 of 97

Conferences and Journals to Follow for this Course

  • The top-tier conferences (double blind, acceptance rates are below ~30%, high quality technical articles):
    • MICCAI (medical image computing & computer assisted intervention)
    • IPMI (Information Processing in Medical Imaging)
    • Other conferences: MIDL, IEEE ISBI, EMBC and SPIE Med Imaging
    • Clinical Conferences: RSNA (>80.000 attendances), ISMRM, SNM
      • Vision and ML conferences: CVPR, NeurIPS, ICML, ICLR, ECCV, ICCV, WACV, …
  • The top-tier technical journals:
    • IEEE TMI, TBME, PAMI, Nature’s Journals, Medical Image Analysis, and NeuroImage
  • The top-tier clinical journals relevant to MIC:
    • Radiology, Lancet Digital Health, Nature Journals, Radiology AI, …
  • ArXiv, BioRxiv…

95

Machine & Hybrid Intelligence Lab

96 of 97

References and Slide Credits

​

96

Machine & Hybrid Intelligence Lab

97 of 97

97

Thank you for listening!

Ulas Bagci, Ph.D.,

Associate Professor, 

Director of Machine & Hybrid Intelligence Lab,

Department of Radiology, Feinberg School of Medicine,

Department of Biomedical Engineering (Courtesy), 

Department of Electrical and Computer Engineering (Courtesy)

Northwestern University,

737 N. Michigan Avenue Suite 1600,

Chicago, IL 60611, USA

Phone: +1 312-694-4951�Cell: +1 240 383 8587�

Web: https://www.bagcilab.com

​

​

Machine & Hybrid Intelligence Lab