by Tran Minh Trieu
Overall Survival Prediction in
Glioblastoma Patients via Local
Context of Brain Tumor MRI
Ph.D. Dissertation Defense
May 10th, 2022
Committee
Prof. Lee Guee Sang(Advisor) (Chonnam National University)
Prof. Kim Soo Hyung (Chonnam National University)
Prof. Yang Hyung Jeong (Chonnam National University)
Prof. Oh In Jae (Chonnam National University Hwasun Hospital)
Prof. Kang Sae Ryung (Chonnam National University Hwasun Hospital)
Motivation
2
[1] Chapter 4 cancer stem-like cells in glioblastoma, Glioblastoma, Codon Publications, Brisbane (AU), 2017
Motivation
3
[1] Chapter 4 cancer stem-like cells in glioblastoma, Glioblastoma, Codon Publications, Brisbane (AU), 2017
🡪 Improving the accurate survival prediction is essential.
Problem Statement
4
Local context
Measure tumor distribution
Imbalance between accuracy and MSE results
Problem Statement
5
Local context
Most recent researches 🡪 whole MRI brain image, only the tumor area for radiomics analysis
However, several studies exhibit the importance of the local area around the tumors.
Problem Statement
6
Local context
If the tumor region is close to or contacts the ventricle, it has critical impact on the survival probability [2].
Also, in Ref. [3], the mass effect around the tumor region is important in the analysis of survival prediction.
[2] Ventricle contact is associated with lower survival and increased peritumoral perfusion in glioblastoma, Journal of neurosurgery, 2018.
[3] Mass effect deformation heterogeneity (medh) on gadolinium-contrast t1-weighted mri is associated with decreased survival in patients with right cerebral hemisphere glioblastoma: a feasibility study, Scientific reports, 2019.
Problem Statement
7
Local context
if the tumor region is close to or contacts the ventricle, it has critical impact on the survival probability [2].
Also, in Ref. [3], the mass effect around the tumor region is important in the analysis of survival prediction.
[2] Ventricle contact is associated with lower survival and increased peritumoral perfusion in glioblastoma, Journal of neurosurgery, 2018
[3] Mass effect deformation heterogeneity (medh) on gadolinium-contrast t1-weighted mri is associated with decreased survival in patients with right cerebral hemisphere glioblastoma: a feasibility study, Scientific reports, 2019
🡪 Focus on the usefulness of the local area around the tumor including the tumor itself
🡪 Extract more effective features for the deep learning
Problem Statement
8
Figure 1: The comparison between ventricle contact and ventricle non-contact patients. �
Problem Statement
9
Measure tumor distribution
Most recent researches 🡪 only the largest tumor, or the total number of tumors.
However, in many cases, finding connected components of the tumors becomes very difficult or inappropriate, because the labeled tumors can include extremely small ones of only a small number of pixels, or even a single pixel, in the image.
Problem Statement
10
Measure tumor distribution
🡪 We propose a new scheme for approximating the measure of tumor distribution, and we observe a considerable improvement in the final performance.
Problem Statement
11
Figure 2:
(A) The distribution of the number of connected components in the segmentation label of the BraTS 2020 dataset
(B) a sample of segmentation labels with many connected components inside.
Problem Statement
12
Imbalance between accuracy and MSE results
Most of the existing works focus on either one of these evaluation measures, and sometimes lead to unconvincing results, because of the imbalance between accuracy and MSE results.
In other words, with high accuracy, a large MSE is observed, or vice versa
Problem Statement
13
Imbalance between accuracy and MSE results
Most of the existing works focus on either one of these evaluation measures, and sometimes lead to unconvincing results, because of the imbalance between accuracy and MSE results.
In other words, with high accuracy, a large MSE is observed, or vice versa
Table 1. The performance of top-ranking methods
on the BraTS 2020 validation Dataset
Problem Statement
14
Imbalance between accuracy and MSE results
--
🡪 Balance these two measures by using a multi-task network
Our Contributions
15
16
Content
Introduction
Related Works
Proposed Method
Results
Limitations
Conclusions
17
Introduction
Brain tumors usually appear in three kinds:
Necrotic tumor is the tumor with dead cell
Enhancing tumor is in active progress
Edema tumor is the tumor region that has swelled.
Original Image
Image with annotated tumor
Green: Edema Tumor
Yellow: Enhancing Tumor
Red: Necrotic Tumor
Figure 3: Illustration of original MR image with and w/o annotated tumor.
18
Introduction
Brain cancer
one of the most dangerous public health problems
leading reason for millions of deaths every year, high mortality.
Brain tumor patient survival prognosis with high performance is a critical task for therapy planning
Only less than 3-10% of patients with glioblastoma (GBM) survive more than 5 years while most of them usually have overall survival days lower than 20 months.
Tumors in the brain are graded according to how fast the tumor grows, and how they look after medical treatment.
Tumor grades into two groups:
low-graded tumor group (I, II).
high-graded (III, IV) tumor group.
Glioblastoma is the high-graded (IV) tumor.
19
Introduction
Improved therapy treatments are urgently needed through survivor classification and overall survival days predicted.
Classifying the patient with high accuracy can bring more benefits to patients, because oncologists can make precise decisions on medicine plan treatment.
Thanks to the overall survival days predicted, we can know the effect of treatments.
Due to the characteristic of each person being different from others, the direct observation of each person is very important.
The overall survival days of patients in the same survivor class are different, so with more precise days prediction, we will have a clear analysis of treatment for each person.
Overall, the classification task is valuable for the overview of health conditions, while regression is needed to take care of each person in treatment.
20
Introduction
Although magnetic resonance imaging is widely used as a standard non-invasive technique, manual medical analysis in MRI is still a challenging task, due to the complexity of the images.
With the brain tumor, the prediction of survival days from an MR image takes oncologists much time and energy.
Additionally, the survival prediction is not stable, due to the different levels of individual experience, heterogeneity in the dataset, and the tumor class imbalance problem.
🡪 Finding an automated high-accuracy survival prediction in brain tumor patients plays a crucial role in therapy or treatment planning.
21
Related Works
In recent years, a lot of researches investigate the issue of overall survival prediction in GBM patients.
Divided into two groups:
22
Related Works
[4] Overall survival prediction for gliomas using a novel compound approach, Frontiers in Oncology, 2021.
[5] Brain tumor segmentation using an ensemble of 3d u-nets and overall survival prediction using radiomic features, Frontiers in computational neuroscience, 2020.
[6] Automatic brain tumour segmentation and biophysics-guided survival prediction, MICCAI, 2019.
[7] Deep learning of imaging phenotype and genotype for predicting overall survival time of glioblastoma patients, IEEE transactions on medical imaging, 2020.
[8] Brain tumor segmentation and survival prediction using multimodal MRI scans with deep learning, Frontiers in neuroscience, 2019.
[9] Multi-channel 3D deep feature learning for survival time prediction of brain tumor patients using multi-modal neuroimages, Scientific Reports, 2019.
[10] Deep learning radiomics algorithm for gliomas (drag) model: a novel approach using 3d unet based deep convolutional neural network for predicting survival in gliomas, MICCAI, 2018.
[11] Brain tumor segmentation and survival prediction, MICCAI, 2019.
[12] Brain tumor segmentation with uncertainty estimation and overall survival prediction, MICCAI, 2019.
[13] Brain tumor segmentation and survival prediction using 3D attention Unet, MICCAI, 2019.
[14] Glioma prognosis: Segmentation of the tumor and survival prediction using shape, geometric and clinical information, MICCAI, 2018.
[15] Tumor grade and overall survival prediction of gliomas using radiomics, Scientific Programming, 2021.
[16] Predicting overall survival time in glioblastoma patients using gradient boosting machines algorithm and recursive feature elimination technique, Cancers, 2021.
[17] Brain tumor segmentation and tractographic feature extraction from structural mr images for overall survival prediction, MICCAI, 2018.
MRI 🡪 segmented tumors 🡪 extract the radiomics features 🡪 Network/machine learning🡪 OS days�
23
Related Works
[4] Overall survival prediction for gliomas using a novel compound approach, Frontiers in Oncology, 2021.
[5] Brain tumor segmentation using an ensemble of 3d u-nets and overall survival prediction using radiomic features, Frontiers in computational neuroscience, 2020.
[6] Automatic brain tumour segmentation and biophysics-guided survival prediction, MICCAI, 2019.
[7] Deep learning of imaging phenotype and genotype for predicting overall survival time of glioblastoma patients, IEEE transactions on medical imaging, 2020.
[8] Brain tumor segmentation and survival prediction using multimodal MRI scans with deep learning, Frontiers in neuroscience, 2019.
[9] Multi-channel 3D deep feature learning for survival time prediction of brain tumor patients using multi-modal neuroimages, Scientific Reports, 2019.
[10] Deep learning radiomics algorithm for gliomas (drag) model: a novel approach using 3d unet based deep convolutional neural network for predicting survival in gliomas, MICCAI, 2018.
[11] Brain tumor segmentation and survival prediction, MICCAI, 2019.
[12] Brain tumor segmentation with uncertainty estimation and overall survival prediction, MICCAI, 2019.
[13] Brain tumor segmentation and survival prediction using 3D attention Unet, MICCAI, 2019.
[14] Glioma prognosis: Segmentation of the tumor and survival prediction using shape, geometric and clinical information, MICCAI, 2018.
[15] Tumor grade and overall survival prediction of gliomas using radiomics, Scientific Programming, 2021.
[16] Predicting overall survival time in glioblastoma patients using gradient boosting machines algorithm and recursive feature elimination technique, Cancers, 2021.
[17] Brain tumor segmentation and tractographic feature extraction from structural mr images for overall survival prediction, MICCAI, 2018.
MRI 🡪 Segmented tumors 🡪 Extract the radiomics features 🡪 Network/machine learning🡪 OS days�
1. All radiomics features
2. Feature selection approach
24
Related Works
The number of extracted features can be from several hundred to several thousand while the size of datasets remain small.
Recommend: 10 samples for each feature [21]
🡪 Low-sample size situations [18,19, 20] often make model will be an overfitting phenomenon. �Large number of features
[18] Reproducibility and generalizability in radiomics modeling: possible strategies in radiologic and statistical perspectives, Korean Journal of Radiology, 2019.
[19] Statistical pattern recognition: a review, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000.
[20] Radiomics: The process and the challenges, Magn. Reason. Imaging, 2012.
[21] Radiomics: Images are more than pictures, they are data, Radiology, 2016.
25
Related Works
2. Using Non-Radiomics Features Approach
[22] Context aware deep learning for brain tumor segmentation, subtype classification, and survival prediction using radiology images, Scientific Reports, 2020.
[23] Uncertainty-driven refinement of tumor-core segmentation using 3dD-to-2d networks with label uncertainty, MICCAI, 2020.
[24] Post-hoc Overall Survival Time Prediction from Brain MRI, ISBI 2021.
[25] 3d u-net based brain tumor segmentation and survival days prediction, MICCAI, 2019.
[26] Impact of spherical coordinates transformation pre-processing in deep convolution neural networks for brain tumor segmentation and survival prediction, MICCAI, 2020.
[27] Modified mobilenet for patient survival prediction, MICCAI, 2020.
[28] A deep learning-based method for glioblastoma survival prediction using local context, International SMA conference, 2021.
1. MRI 🡪 Segmented tumors🡪 Extract deep feature 🡪 Network/machine learning🡪 OS days
2. MRI 🡪 Network 🡪 OS days �
26
Related Works
2. Using Non-Radiomics Features Approach
[22] Context aware deep learning for brain tumor segmentation, subtype classification, and survival prediction using radiology images, Scientific Reports, 2020.
[23] Uncertainty-driven refinement of tumor-core segmentation using 3dD-to-2d networks with label uncertainty, MICCAI, 2020.
[24] Post-hoc Overall Survival Time Prediction from Brain MRI, ISBI 2021.
[25] 3d u-net based brain tumor segmentation and survival days prediction, MICCAI, 2019.
[26] Impact of spherical coordinates transformation pre-processing in deep convolution neural networks for brain tumor segmentation and survival prediction, MICCAI, 2020.
[27] Modified mobilenet for patient survival prediction, MICCAI, 2020.
[28] A deep learning-based method for glioblastoma survival prediction using local context, International SMA conference, 2021.
1. MRI 🡪 Segmented tumors🡪 Extract deep feature 🡪 Network/machine learning🡪 OS days
2. MRI 🡪 Network 🡪 OS days �
Ignore meaningful surrounding region (deficiency), more dependent on tumor segmented quality
A whole-brain MR image contains large normal brain regions, leading to having many non-effective features for learning (redundant)
27
Proposed Method
We propose a multiple stages method with three phases.
1. First, we segment the tumor from the combination of four modalities in brain MR images.
2. Then, we extract the local context feature and the approximating tumor distribution feature of tumors.
3. Finally, we put these features into a multi-task-based neural network, to predict the overall survival days of GBM patients.
Figure 4: We propose a multiple stages method with three phases.
28
Proposed Method
1. An ensemble framework for brain tumor segmentation
Include two sub-networks: DMNet [29] + DKNet [30].
Input: T1, T1ce, T2, and FLAIR.
main task 🡪 the survival problem 🡪 experiment with some lightweight, effective segmentation models such as DMFNet and DKNet.
Segmentation result is better when using an ensemble model of DMFNet and DKNet.
Our ensemble approach has Dice and HD scores in brain tumor segmentation outperformed than DMFNet and DKNet.
[29] C. Chen, L. Xiaopeng, D. Meng, Z. Junfeng, L. Jiangyun, 3d dilated multifiber network for real-time brain tumor segmentation in mri, MICCAI, 2019.
[30] N. Duc-Ky, T. Minh-Trieu, K. Soo-Hyung, Y. Hyung-Jeong, L. Guee-Sang, Multi-task learning for small brain tumor segmentation from mri, Applied Sciences, 2020.
29
Proposed Method
Figure 5: Illustration of qualitative comparison in segmentation phase between different segmentation method.
(A) column is the the input images.
(B) column is our segmented results.
(C) column is the segmented results from DKNet.
(D) column is the segmented results from DMFNet.
1. An ensemble framework for brain tumor segmentation
30
Proposed Method
Figure 6: The quantitative comparison in segmentation and survival prediction stages between different segmentation networks.
Note that with dice score and accuracy score, the higher, the better.
With Hausdorff distance, the lower, the better.
1. An ensemble framework for brain tumor segmentation
31
Proposed Method
2. Multiple Type of Features Extraction
After obtaining the segmented tumor map, we extend it to local context by finding the area around tumor.
We extract the features from local context by a variant of VGG pre-trained network.
Beside the deep local context feature, we also employ clinical feature (Age) and we propose a new scheme to approximate tumor distribution, or simple call as semantic feature.
32
Proposed Method
2. Multiple Type of Features Extraction
Figure 5: The illustration of having a local window from the original image and a segmented tumor image
A) the input original image
B) Image with segmented tumor
C) Red box indicates the local area
D) Extracted local window.
33
Proposed Method
2. Multiple Type of Features Extraction
Figure 6: The illustration of new scheme for approximating the measure of tumor distribution.
1 means that slice has tumor,
0 means that slice does not have tumor.
34
Proposed Method
3. Overall Survival via Multi-task Learning
We proposed a regression model includes eight fully connected (FC) layers.
This model predicts the overall survival days and the class of survivor at the same time.
35
Proposed Method
Loss Function
In the survival prediction stage, we use mean absolute error (MAE), which is also called L1 loss for training overall survival days prediction.
s is the number of subject
y is the predicted survival days
x is actual survival days�
We also used a loss function for the auxiliary task of predicting the survivor class, which is categorical cross entropy.�
Our final loss for survival prediction stage is summarized as follows: �
In the experiment, we set weights ε1 = ε2 = 1 to serve the purpose of learning ability in regression and classification equally.
36
Proposed Method
Dataset
Our experiments evaluate in the BraTS 2018, BraTS 2019 and BraTS 2020 datasets.
There are four modalities, such as T1, T2, T1ce, and FLAIR.
Size: 240 x 240 x 155
Dataset | Training Set | Validation Set (provide/evaluate) |
BraTS 2018 | 285 | 66/28 |
BraTS 2019 | 335 | 125/29 |
BraTS 2020 | 369 | 125/29 |
Table 2. Number of sample in training set and validation set of BraTS datasets
37
Proposed Method
Dataset
Our experiments evaluate in the BraTS 2018, BraTS 2019 and BraTS 2020 datasets.
There are four modalities, such as T1, T2, T1ce, and FLAIR.
Size: 240 x 240 x 155
Dataset | Training Set | Validation Set (provide/evaluate) |
BraTS 2018 | 285 | 66/28 |
BraTS 2019 | 335 | 125/29 |
BraTS 2020 | 369 | 125/29 |
Table 2. Number of sample in training set and validation set of BraTS datasets
Figure 7: Example of image modalities with some provided information in the BraTS
training dataset
38
Results
Evaluation Metrics
All results from BraTS validation sets are scored by the challenge website https://ipp.cbica.upenn.edu.
The subjects were grouped into three classes of survival comprising short survivors for who survived less than ten months, medium survivors for who survived between 10 and 15 months, and long survivors for who survived more than 15 months
39
Results
Table 3. Our overall survival prediction results in BraTS validation datasets
40
Results
Table 4. Comparison in accuracy metric between our method with other methods in BraTS validation datasets
41
Results
Table 5. The comparison results in MSE and SpearmanR metrics between our method with other methods in BraTS 2020 Validation dataset.
42
Results
Table 6. The comparison results in MSE, SpearmanR metrics between our method with other methods in BraTS 2018, BraTS 2019 Validation datasets.
43
Results
The effect of local context on overall survival days prediction framework
Figure 8: The illustration of multiple type of contexts
(A) global context from whole image,
(B) context from whole image without tumor region,
(C) local context without tumor region,
(D) the context inside tumor,
(E) local context with tumor region,
(F) performance of using only each kind of context feature in BraTS 2020 Validation dataset.
44
Results
The effect of local context on overall survival days prediction framework
Figure 9: The affection of margin adjustment to the local context area.
The blue line shows the accuracy of our method when we modify the margin (full features).
The orange line shows the accuracy of employing only local context feature when we modify the margin (only local feature).
45
Results
The effect of employing multiple dimensionality reduction methods
Table 7. The effect of employing multiple dimensionality reduction methods in BraTS 2020 Validation dataset.
Method | ACC |
Projection in 1D vector | 0.172 |
Gaussian Random Projection | 0.207 |
Principle Component Analysis | 0.310 |
Ours | 0.345 |
Table 8. The effect of projection plane in survival prediction results.
Method | ACC |
Sagittal | 0.379 |
Coronal | 0.483 |
Axial (Ours) | 0.621 |
Sum 3 features from 3 planes | 0.552 |
Concatenate 3 features from 3 planes | 0.448 |
46
Results
The effect of employing multiple dimensionality reduction methods
Table 9. The comparison between using our proposed method with using number of connected components feature in BraTS 2020 Validation dataset.
Method | Accuracy | MSE |
Ours (Use tumor distribution feature) | 0.621 | 93,951.386 |
Ours (Use Exactly total number of connected components) | 0.379 | 119,985.44 |
When we use exactly total number of connected components for training, the performance is so low, because some labeled tumors can include
- extremely small ones of only a small number of pixels
- or even a single pixel, in the image
🡪 could lead to improper or even wrong features
47
Results
The effect of learning-based methods on overall survival days
prediction framework
Figure 10: The quantitative comparison between single task and multiple tasks learning approaches in the BraTS 2020 Validation dataset
(C) is the comparison in the mean square error (MSE) metric.
48
Results
The effect of feature extractors on overall survival days prediction framework
Figure 11: The effection of using different feature extractors in survival prediction results in the BraTS 2020 Validation dataset
49
Results
Figure 12: Illustration of border-threshold patient and the performance based on multi-value of threshold.
(A) The definition of border-threshold survivor.
(B) The quantitative accuracy comparison in multiple border thresholds subjects between our method and others.
The blue line is the performance of random forest method,
The grey line is the performance of light gradient boosting machine method,
The yellow is the performance of gradient boosting method
our performance shows in red line color.
The efficacy of overall survival days prediction framework
50
Results
Figure 13: Illustration of qualitative comparison in overall survival prediction phase in BraTS 2020 sub validation set.
(A) shows the distribution of overall survival days predicted results of gradient boosting method.
(B) shows the distribution of overall survival days predicted results of light gradient boosting machine method.
(C) shows the distribution of overall survival days predicted results of random forest method.
(D) shows the distribution of overall survival days predicted results of our method.
The efficacy of overall survival days prediction framework
The red line is the ideal line with the predicted and ground truth of overall survival days are the same value. The method that predicts more closer points to the red line, the better method. Note that our method generates the results mostly closer to the red line than others.
51
LIMITATIONS
Firstly, local context images need to resize to a fixed size, to fit into the feature extractor in the feature extraction step. That may be the cause of the degradation of the extracted feature.
Secondly, because of using 2D slicing images, the 3D structure is not utilized.
Finally, until now, this research only works for MR images, while there are many kinds of medical images, such as PET, CT, PET/CT, and ultrasound.
52
Conclusions
Our research proposed a novel approach for survival prediction in brain tumor patients via a multi-task learning approach.
While most previous approaches focus on radiomics features for survival tasks, our method can optimize the local context features, approximating tumor distribution, and age as meaningful features for model learning.
The network leveraged the knowledge of survival class and the overall survival days simultaneously.
Therefore, our model predicts more accurately in border-threshold subjects.
Results prove that our method outperform other top-ranking and state of the art methods in Glioblastoma overall survival prediction task of most BraTS challenges (2018, 2019, 2020).
53
Publication
Related to thesis topic:
Non-Related to thesis topic:
54
I would like to express my gratitude to my Advisor Professor, Professor Lee Guee Sang, who always supports, encourages me.
My sincere thanks to all Professors of the review committee for spending time on my presentation. �
THANK YOU
Appendix I: How the result will be if we try to use radiomics features with proposed framework ?
55
Method | Accuracy | MSE |
Ours (w/o using radiomics features) | 0.621 | 93,951.386 |
Ours (with using radiomics features) | 0.207 | 355,954.963 |
The performances of accuracy and MSE are decreased a lot due to the overfitting problem (The number of extracted features is nearly 1300 features/sample while the size of datasets remain small which only around 300 samples)
Table 10. The comparison between using our proposed method with / without using radiomics features in BraTS 2020 Validation dataset.
Appendix 2: C-index between tumor distribution features with overall survival days.
56
Tumor Distributed Feature | C-index |
Edema | 0.497564 |
Enhancing | 0.500054 |
Necrotic | 0.503314 |
Table 11. C-index between tumor distributed features with overall survival days in BraTS 2020 training dataset.