Reproducible Radiology and Pathology Imaging Analysis Applications in MHub

Curtis Lisle, Ph.D.

KnowledgeVis, LLC

clisle@knowledgevis.com

Rhabdomyosarcoma tissue segmentation (binary and probability predictions)

Agenda

• We will discuss the use of MHub in two projects today:

​

​

• Analysis of a Non-small cell lung cancer (NSCLC) cohort from Orlando Health
• CT and PET/CT radiology analysis
• CT segmentation, PET segmentation, foundation model features

​

• An Example Pathology model to identify Rhabdomyosarcoma subtypes
• RMS is a rare pediatric cancer where outcome varies according to the subtype
• DICOM-WSI pathology image analysis
• Tissue type segmentation, genetic mutation prediction, survival prediction
• MHub Pipeline options for incorporating models

Lung Cancer Cohort Description

• A plot of pre-planning scans, planning CTs, and followup CT or PET scans for the majority of a cohort of 230 patients (some of the later follow-ups not shown)

Lung Cancer Cohort Description

• Radiotherapy treatment information was also shared to show geometry of delivered doses
• Dose delivery and delineated structures are used to calculate the dose for the lesion and the nearby organs-at-risk

MHub Model Card for Total Segmentator

Some additional information was cut off to fit on this slide

Running TotalSegmentator

• All MHub models come with one or more workflows, each described by a configuration file
• To run the model, invoke docker with input_data and output_data directories mapped and specify a particular workflow

Order of Operations in this Workflow

Optional arguments for steps in the workflow

Output specification

MHub Models: Total Segmentator on the Lung Cohort

• Automatic Segmentation of 167 CT Planning Scans
• Approximately 80 structures discovered from each scan (out of 104 possible structures)

Volume distribution of selected AI-segmented structures

Jakob Wasserthal, Hanns-Christian Breit, Manfred T. Meyer, Maurice Pradella, Daniel Hinck, Alexander W. Sauter, Tobias Heye, Daniel T. Boll, Joshy Cyriac, Shan Yang, Michael Bach, and Martin Segeroth (2023). TotalSegmentator: Robust Segmentation of 104 Anatomic Structures in CT Images. Radiology Artificial Intelligence, 5:5

MHub Models: Total Segmentator on the Lung Cohort

• Total Segmentator “fast mode” took an average of 90 seconds per patient using 20 virtual cores (no GPU) and approximately 32GB memory
• Total Segmentator “full mode” took an average of 75 seconds per patient on an RTX-5000 (16GB) Mobile and approximately 12GB memory

Total Segmentator “FAST”

Total Segmentator “FULL” mode

Total Segmentator Full Mode GPU profile

• Processing for four patients are shown at right
• Load, processing, unload cycle
• Different memory profile are due to patients have different size images

95%

(MBytes)

Elapsed time (in 3 second intervals)

~75 secs

One patient CT

Comparing Clinical and AI-delineated structures

• Compared TotalSegmentator-detected structures with those delineated by the treating radiation oncologist
• Our clinician focused only on Organs-at-risk (OARs), so clinical segmentation structures abruptly stop when outside the treatment region and clipping is necessary for fair volume comparisons
• Segmented structures were inconsistent (e.g. “heart” vs. left ventricle, right ventricle, myocardium, etc.)
• Data cleaning - clinical naming makes it harder to compare (e.g. ‘Lung_LT’, ‘LT_Lung’, ‘lung_LT’ are all found in the clinical dataset). TotalSegmentator names structures consistently.

Clinical Heart vs. AI Heart volume comparison (not clipped)

Clinical heart (cyan) vs. AI heart substructures

Clinical Esophagus vs. AI Esophagus volume (not clipped)

Workflow Comparing Clinical and AI Generated Segmentations

• TotalSegmentator structures need to be clipped to the bounding box of the clinical segmentations
• There is a Slicer Plugin for running MHub models, so the workflow could be implemented all in Slicer

Thoracic CT scans with RTStruct segmentation records

TotalSegmentator

MHUB.ai platform

3D Slicer

Label maps from clinical structures (NRRD)

DICOM segmentation object

Extract volume

Highdicom

Compare structure volumes

Clip volume to clinical BBOX

Batch Slicer script

MHub Models: Foundation Model on Lung Cohort

• It took only a few hours to run a foundation model and train a target-specific classifier!
• We used the MHub Foundation Model features (trained on 11670+ CT scans)
• We trained a predictor for 3-year overall survival
• Preliminary results show validation AUC of 0.71 (without model fine-tuning)

CT Lesion Foundation Model

Logistic Regression Model

ROI regions from lung cohort

4096 image features

Pai, S., Bontempi, D., Hadzic, I. et al. Foundation model for cancer imaging biomarkers. Nat Mach Intell (2024). https://doi.org/10.1038/s42256-024-00807-9

Target: 3Y survival

MHub Models: Foundation Model on Lung Cohort

• Examining the latent space of the inferenced features from the Foundation Model
• T-SNE plot of full 4096 features of 167 patients from the cohort vs. PCA first to reduce to 50 features before running T-SNE projection
• No obvious clustering noted for the 3Y survival case (yellow=survived > 3yr); tried multiple hyper parameters

T-SNE plot of features from 167 ROI regions

T-SNE plot after PC down to 50 features

PCA plot of first two of 50 features

MHub Models: PET Lesion Detection

• PET scans are used to screen for metastatic cancer progression
• Cancerous regions consume energy, so glucose detection identifies cancer (and sugar-hungry body structures like the brain)
• This model earned Second place in AutoPET challenge at the MICCAI’22

Y. Peng, J. Kim, D. Feng and L. Bi, 'Automatic Tumor Segmentation via False Positive Reduction Network for Whole-Body Multi-Modal PET/CT Images', In arXiv:2209.07705, doi: 10.48550/arXiv:2209.07705.

Lesion as delineated by hand by the radiation oncologist

Lesion as auto-detected by the DL model

Auto PET GPU profile

• Single patient processing is shown at right
• This is a two-stage algorithm:
• Global segmentation
• Local Refinement- remove false positive regions

97%

(MBytes)

Stage 1: global segmentation; 2 channels: (CT, PET)

Stage 2: local refinement; 5 channels: (CT, PET, segmentation prediction, probability map)

MHub Models: PET Lesion Detection

• MHub contains a flexible file importer that can be directed to scan according to configuration options
• Below is an excerpt from the config file for the FDG-PET lesion detection model, indicating how the data is structured

FileStructureImporter:

    input_dir: 'input_data'

    structures:

      - $sid@instance/ct@dicom:mod=ct

      - $sid/pet@dicom:mod=pt

    import_id: sid

docker run —gpus=all —rm -it

-v /media/clisle/LungProcess/LungCancer/OrlandoHealthData/PET-scans:/input_data

-v /media/clisle/LungProcess/LungCancer/OH_Processing/MHub/PET-results:/output_data

mhubai/gc_autopet_fpr

Semantic Queries in MHub allow search for only modality = CT or PET files and ignore anything else

Pathology in MHub segmentation

• MHub started with a radiology focus: because digital radiology is more mature
• Pathology is known for proprietary formats and relatively large run-time resource needed
• DICOM-WSI is an emerging format for conveying multi-resolution pathology images (separate file for each layer of the multi resolution pyramid)
• Pathology models are now being added to MHub to further benefit to the user community

Patho Model

e.g. runs at 10x

• Previously, many models had to traverse the pyramid to find an optimal resolution
• openslide is a popular prerequisite for these algorithms
• MHub offers a way to select a resolution reduction. More information shortly…

(150,000 x 100,000 pixels) is typical

Rhabdomyosarcoma Model Development

• Rhabdomyosarcoma (called RMS) is a rare pediatric cancer
• There are two major subtypes: Alveolar (ARMS) and Embryonal (ERMS)
• ERMS is usually less serious, except for when the MYOD1 gene is mutated
• Clinical benefit is RMS subtype identification from just an H&E biopsy image
• The red box’ed models are currently being imported into MHub

Pathology downstream analysis in MHub

• The Rhabdomyosarcoma tissue subtype segmentation is used by downstream models
• MYOD1 Mutation: Determine if the tissue is a virulent subtype of ERMS that possesses a MYOD1 genetic mutation
• Survival Risk: Determine overall survival risk (compared with the training cohort) from H&E image

Pathology images in DICOM-WSI

DicomImporter

RMS Segmentation

MYOD1 mutation risk

Survival risk

Mutation Risk estimate (JSON or CSV)

Survival Risk estimate (JSON or CSV)

Select resolution for pathology model

• Treat pathology cases same as radiology with standard DicomImporter
• Resolution selection algorithm implemented as a MHub processor module to reduce data movement and computation
• Tested: Works on the RMS-Mutation-Prediction collection publicly available from the Imaging Data Commons
• downloads are a huge directory with obscure filenames. MHub’s dicomSorter separates automatically by studyID (different patients)

Pathology images in DICOM-WSI

DICOM SEG objects

First RMS Pipeline Implemented in MHub

• The original RMS model implementation reads and writes only PNG images, and requires non-trivial image pre-processing
• The RMS model was ported to read DICOM-WSI natively and generate a DICOM Segmentation object as output.
• Porting required several weeks of effort, mostly devoted to learning DICOM and the highdicom python library
• MHub treats pathology cases same as radiology, using the standard DicomImporter
• Below is the MHub pipeline using the ported model

Pathology images in DICOM-WSI

DICOM SEG objects

Porting the Reference RMS Pipeline into MHub

• Demonstrates the value of MHub by converting the original model in its native form
• It took only 6-7 hours to adapt the reference RMS model to run inside an MHub pipeline
• Switched the model to read and write TIFF images and doesn’t have to embrace DICOM formats

Pathology images in DICOM-WSI

DICOM SEG object

Segmentation as TIFF image

Segmentation as TIFF image

Processing Pipeline managed by MHub

Segmentation as TIFF image

Conclusions

• MHub can ease the effort required to re-use state of the art machine learning models in medical imaging research.
• We discussed using MHub models in both radiology and pathology applications
• MHub is a step towards clinical use because the prerequisites and models are now configuration managed, tested, and stable
• Thank you for the opportunity to share this work today
• Any questions?

Curtis Lisle, KnowledgeVis, LLC; clisle@knowledgevis.com