| A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z | AA | AB | AC | AD | AE | AF | AG | AH | AI | AJ | AK | AL | AM | AN | AO | AP | AQ | AR | AS | AT | AU | AV | AW | AX | AY | AZ | BA | BB | BC | BD | BE | BF | BG | BH | BI | BJ | BK | BL | BM | BN | BO | BP | BQ | BR | BS | BT | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | Title | DOI | Author | Journal/Conference | Publication Year | Data Origin | Data Source | Dataset Name (if Public) | Link | Sample Size | Class Distribution | Gini index | Balanced/ Imbalanced/ Highly Imbalanced | Deal with imbalance (Yes/No) | Deal with imbalance (Method) | Multimodal Dimension | Type of modalities - before | Type of modalities - after | Specific Modality | Missing Modality | Feature Extraction | Unimodal Architecture | Homogeneity | Input size | Marginal Representation Size | Distribution | Dimensionality Transformation | How many? | Multiple Type | # of Fusions | # of Fusion Flows | Sync/Async | Concatenation | Attention | Tensor-operation | Calibration | Knowledge sharing | # of different fusion types | Arichitecture | Size | Learning Task | Specific Task | Medical Task | Disease Area | Number of Tasks | Training From Scatch | Pretraining Module | Pretraining Supervision | Pretraining Task | Pretraining Domain | Finetuning Module | Robustness To Missing Modalities | Multimodal Optimization | Unimodal XAI | Multimodal XAI | Unimodal vs Multimodal | Early vs Intermediate vs Late | Resampling Technique | External Validation | Statistical Test | Avg(Multimodal) > Avg(Unimodal) | Avg(Multimodal) = Avg(Unimodal) | Std(Multimodal) < Std(Unimodal) | Std(Multimodal) > Std(Unimodal) | Std(Multimodal)=Std(Unimodal) | Avg(Joint) > (Avg(Early) & Avg(Late)) | Avg(Early) > (Avg(Joint) & Avg(Late)) | Avg(Late) > (Avg(Joint) & Avg(Early)) | Std(Joint) < (Std(Early) & Std(Late)) | Std(Joint) = Std(Early) = Std(Late) | Code Link | ||
2 | A Bi-level representation learning model for medical visual question answering | https://doi.org/10.1016/j.jbi.2022.104183 | Yong Li | Journal of Biomedical Informatics 134 | 2022 | Real | Public | VQA-Rad | https://osf.io/89kps/ | 3515 | imbalanced declared | Yes | Label-Distribution-Smooth Margin loss | bimodal | imaging text | \ | m1: X-Ray m2: question-answer | No | m1: raw m2: raw | m1: CNN m2: RNN | Heterogeneous | m1: 65536 | m1: 512 | m1: 0.0078 | Multiple | Multiflow | 7 | 2 | Async | 2 | 2 | 3 | 0 | 0 | 3 | FC | Unimodal > Multimodal | Supervised | Classification | Question Answering | Other (visual question answering on head, chest, abdomen) | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | ||||||||||||||||||||||
3 | Public | PathVQA | https://github.com/UCSD-AI4H/PathVQA | 32799 | imbalanced declared | m1: Pathology images m2: question-answer | 1 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
4 | A dynamic multi-modal fusion network for ovarian tumor differentiation | 10.1109/BIBM55620.2022.9995556 | Yang Li | IEEE International Conference on Bioinformatics and Biomedicine (BIBM) | 2022 | Real | Private | \ | \ | 532 | trimodal | imaging imaging tabular | \ | m1: MRI (T1c) m2: MRI (T2WI) m3: clinical | No | m1: raw m2: raw m3: raw | m1: CNN m2: CNN m3: FC | Heterogeneous | Multiple | Sudden | 5 | 1 | Async | 1 | 0 | 4 | 0 | 0 | 2 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Oncology | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | Yes | \ | ||||||||||||||||||||||||||
5 | AATSN: Anatomy Aware Tumor Segmentation Network for PET-CT volumes and images using a lightweight fusion-attention mechanism | https://doi.org/10.1016/j.compbiomed.2023.106748 | Ibtihaj Ahmad, Yong Xia, Hengfei Cui, Zain Ul Islam | Computers in Biology and Medicine | 2023 | Real | Public | from the HECKTOR challenge | https://www.aicrowd.com/challenges/miccai-2020-hecktor | 224 | \ | \ | \ | \ | \ | bimodal | imaging imaging | \ | m1: CT m2: PET | No | m1: raw m2: raw | m1: CNN m2: CNN | Homogeneous | Multiple | Sudden | 3 | 1 | Async | 0 | 3 | 0 | 0 | 0 | 1 | Conv | Unimodal > Multimodal | Supervised | Segmentation | Detection | Oncology | 1 | Yes | \ | \ | \ | \ | \ | No | \ | m1: guided backprojection m2: guided backprojection | Yes | Yes | Yes | Yes | Yes | \ | Yes | \ | \ | ||||||||||||||
6 | Public | from TCIA | https://www.cancerimagingarchive.net/collection/head-neck-radiomics-hn1/ | 28918 | \ | \ | \ | \ | \ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
7 | Attention-like multimodality fusion with data augmentation for diagnosis of mental disorders using MRI | https://doi.org/10.1109/TNNLS.2022.3219551 | Rui Liu , Zhi-An Huang | IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS | 2022 | Real, Simulated | Public | ABIDE I | http://fcon_1000.projects.nitrc.org/indi/abide | 1035 | Autism Spectrum Disorder: 505 Typical Control: 530 | 0.50 | balanced | \ | \ | trimodal | imaging imaging tabular | \ | m1: MRI (fMRI) m2: MRI (sMRI) m3: clinical | No | m1: handcrafted m2: handcrafted m3: raw | m1: CNN m2: CNN m3: raw | Heterogeneous | Multiple | Gradual | 8 | 1 | Async | 5 | 0 | 0 | 3 | 0 | 2 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Mental Health | 1 | Yes | \ | \ | \ | \ | \ | No | \ | m1:CAM | Yes | Yes | Yes | Yes | \ | Yes | \ | \ | |||||||||||||||
8 | Public | ADHD-200 | http://fcon_1000.projects.nitrc.org/indi/adhd200/ | 947 | Attention Deficit Hyperactivity Disorder: 362 Typical Control: 585 | 0.47 | imbalanced | Yes | MCGAN | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
9 | Public | COBRE | http://fcon_1000.projects.nitrc.org/indi/retro/cobre.html | 146 | Schizophrenia: 72 Typical Control: 74 | 0.50 | balanced | \ | \ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
10 | AviPer: assisting visually impaired people to perceive the world with visual‑tactile multimodal attention network | https://doi.org/10.1007/s42486-022-00108-3 | Xinrong Li, Meiyu Huang, Yao Xu, Yingze Cao, Yamei Lu, Pengfei Wang, Xueshuang Xiang | CCF Transactions on Pervasive Computing and Interaction | 2022 | Real | Private | \ | \ | 1650 | balanced | \ | \ | bimodal | video time series | \ | m1: RGB video (hand motion video) m2: tactile signals | No | m1: raw m2: raw | m1: CNN m2: CNN | Homogeneous | Single | Sudden | 1 | 1 | Async | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Detection | Other (visual impairment) | 1 | Yes | \ | \ | \ | \ | \ | No | \ | m1:GradCAM | Yes | Yes | \ | ||||||||||||||||||||||
11 | Private | \ | \ | 660 | balanced | \ | \ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
12 | Private | \ | \ | 220 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
13 | Comparative assessment of text-image fusion models for medical diagnostics | 10.31799/1684-8853-2020-5-70-79 | A. А. Lobantsev | Information and Control Systems | 2020 | Real | Public | MIMIC-CXR | https://physionet.org/content/mimic-cxr/2.0.0/ | 377.110 | bimodal | imaging text | imaging tabular | m1: X-Ray (Chest X-Ray) m2: Free-Text Reports | No | m1: raw m2: learned | m1: CNN m2: RNN | Heterogeneous | Single | Sudden | 1 | 1 | Async | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Pneumonia | 1 | Yes No | m1 m2 | supervised | same | related | multimodal | No | \ | Yes | Yes | Yes | \ | \ | Yes | \ | ||||||||||||||||||||||
14 | Computer-aided diagnosis of hepatocellular carcinoma fusing imaging and structured health data | https://doi.org/10.1007/s13755-021-00151-x | AB Menegotto | Health Information Science and Systems | 2021 | Real | Public | TCGA-LIHC | https://www.cancerimagingarchive.net/collection/tcga-lihc/ | 41.584 | Confirmed Hepatocellular Carcinoma: 20.792 Negative Hepatocellular Carcinoma: 20.792 | 0.50 | balanced | \ | \ | bimodal | imaging tabular | \ | m1: CT m2: clinical | No (imputed) | m1: raw m2: raw | m1: CNN m2: raw | Heterogeneous | Single | Sudden | 1 | 1 | Async | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Oncology | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | https://github.com/amenegotto/pyLiver | ||||||||||||||||||||||
15 | Public | TCGA-STAD | https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=19039400 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
16 | Public | TCGA-KIRP | https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=11829555 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
17 | Public | CPTAC-PDA | https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=33948258 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
18 | Computer-Aided Hepatocarcinoma Diagnosis Using Multimodal Deep Learning | https://doi.org/10.1007/978-3-030-24097-4_1 | Alan Baronio Menegotto | International Symposium on Ambient Intelligence | 2019 | Real | Public | TCGA-LIHC | https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=6885436 | 46832 | Positive: 20.792 Negative: 26.040 | 0.49 | balanced | \ | \ | bimodal | imaging tabular | \ | m1: CT m2: clinical | No (imputed) | m1: raw m2: raw | m1: CNN m2: FC | Heterogeneous | Single | Sudden | 1 | 1 | Async | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Oncology | 1 | No | m1 | supervised | different | different | entire net | No | \ | Yes | |||||||||||||||||||||||
19 | Public | TCGA-STAD | https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=11829555 | No | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
20 | Public | TCGA-KIRP | https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=19039400 | No | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
21 | Public | CPTAC-PDA | https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=33948258 | No | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
22 | Deep learning approach for predicting lymph node metastasis in non-small cell lung cancer by fusing image–gene data | https://doi.org/10.1016/j.engappai.2023.106140 | Guojie Hou, Liye Jia, Yanan Zhang, Wei Wu, Lin Zhao, Juanjuan Zhao, Long Wang, Yan Qiang | Engineering Applications of Artificial Intelligence | 2023 | Real | Public | NSCLC-Radiogenomics | https://wiki.cancerimagingarchive.net/display/Public/NSCLC+Radiogenomics | 124 | Metastasis: 23 Non-metastasis: 101 | 0.30 | imbalanced declared | Yes | Random oversampling | bimodal | imaging tabular | \ | m1: CT m2: genomics | No | m1: raw m2: raw | m1: CNN m2: FC | Heterogeneous | Multiple | Sudden | 7 | 1 | Async | 0 | 0 | 5 | 2 | 0 | 2 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Oncology | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | Yes | \ | |||||||||||||||||||||
23 | Deep Learning Based Data Fusion Methods for Multimodal Emotion Recognition | https://doi.org/10.7840/kics.2022.47.1.79 | Njoku et al. | The Journal of Korean Institute of Communications and Information Sciences '22-01 Vol.47 No.01 | 2022 | Real | Public | RAVDESS | https://www.kaggle.com/datasets/uwrfkaggler/ravdess-emotional-speech-audio | 1440 | Positive: 384 Neutral: 288 Negative: 768 | 0.44 | imbalanced | No | \ | bimodal | video time series | tabular tabular | m1: RGB video m2: electrical (EEG) | No | m1: learned m2: handcrafted | m1: FC m2: FC | Homogeneous | m1: 2400 m2: 8190 | m1: 500 m2: 500 | Equal | m1: 0.2083 m2: 0.0611 | Single | Sudden | 1 | 1 | Sync | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal < Multimodal | Supervised | Classification | Detection | Mental health | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | \ | \ | Yes | ||||||||||||||||
24 | Public | EEG dataset | https://scholar.google.com/scholar?hl=it&as_sdt=0%2C5&q=J.+J.+Bird%2C+et+al.%2C+%E2%80%9CMental+emotional+sentiment+classification+with+an+eeg-based+brain-machine+interface%2C%E2%80%9D+in+Proc.+Digital+Image+and+Sign.+Process.%2C+Oxford%2C+UK%2C+Apr.+2019.&btnG= | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
25 | Deep learning model integrating positron emission tomography and clinical data for prognosis prediction in non-small cell lung cancer patients | https://doi.org/10.1186/s12859-023-05160-z | Seungwon Oh | BMC Bioinformatics | 2023 | Real | Private | \ | \ | 2687 | \ | \ | \ | \ | \ | bimodal | imaging tabular | \ | m1: PET m2: clinical | No | m1: raw m2: raw | m1: CNN m2: FC | Heterogeneous | Single | Sudden | 1 | 1 | Async | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Regression | Prognosis | Oncology | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | \ | Yes | \ | \ | Yes | ||||||||||||||||||
26 | Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction | 10.3389/fmolb.2023.1136071 | K Niu | Frontiers in Molecular Biosciences | 2023 | Real | Public | MIMIC III | https://physionet.org/content/mimiciii/1.4/ | 18904 | \ | \ | \ | \ | \ | bimodal | text time series | tabular time series | m1: Free-Text Reports m2: clinical time series | No (imputed) | m1: learned m2: raw | m1: Transformer m2: RNN | Heterogeneous | Single | Sudden | 1 | 1 | Sync | 1 | 0 | 0 | 0 | 0 | 1 | RNN-FC | Unimodal = Multimodal | Supervised | Classification | Prognosis | Other (overall survival from discharge) | 1 | Yes | \ | \ | \ | \ | \ | No | \ | data ablation | Yes | Yes | \ | ||||||||||||||||||||
27 | Deep multimodal fusion for subject-independent stress detection | 10.1109/Confluence51648.2021.9377132 | K Radhika | International Conference on Cloud Computing, Data Science & Engineering (Confluence) | 2021 | Real | Public | ASCERTAIN | https://ascertain-dataset.github.io/ | 2088 | imbalanced declared | Yes | Synthetic Minority Oversampling Technique | bimodal | time series time series | tabular tabular | m1: electrical (ECG) m2: electrical (EDA) | No | m1: handcrafted m2: handcrafted | m1: CNN m2: CNN | Homogeneous | Single | Sudden | 1 | 1 | Sync | 1 | 0 | 0 | 0 | 0 | 1 | Conv-FC | Unimodal = Multimodal | Supervised | Classification | Diagnosis | Mental health | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | Yes | \ | |||||||||||||||||||||||
28 | K Radhika | Public | CLAS | https://ieee-dataport.org/open-access/database-cognitive-load-affect-and-stress-recognition | 944 | imbalanced declared | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
29 | Deep multimodal predictome for studying mental disorders | 10.1002/hbm.26077 | Md Abdur Rahaman | Human Brain Mapping | 2023 | Real | Public | fBIRN | https://www.nitrc.org/projects/fbirn/ | 437 | Schizophrenia: 162 Healthy Control: 275 | 0.47 | imbalanced | No | \ | trimodal | imaging imaging tabular | tabular tabular tabular | m1: MRI (fMRI) m2: MRI (sMRI) m3: genomics | No | m1: handcrafted m2: handcrafted m3: raw | m1: AE m2: FC m3: RNN | Heterogeneous | Single | Sudden | 1 | 1 | Async | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Mental health | 1 | Yes | \ | \ | \ | \ | \ | No | Multimodal regularization | m1:saliency maps m2:saliency maps m3:saliency maps | modality-wise attention correlation of interaction | Yes | Yes | Yes | Yes | \ | Yes | \ | \ | ||||||||||||||
30 | Public | COBRE | https://fcon_1000.projects.nitrc.org/indi/retro/cobre.html | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
31 | Public | MRPC | https://onlinelibrary.wiley.com/doi/full/10.1002/hbm.24723?casa_token=lZwao4xN4usAAAAA%3AjHtnhy56Dtc0nVR70A6x6B12W9bUAniIB_gXsFGlBpW5txcZtmhAtyaRMgL31iDMybQdZbQu0h0 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
32 | DyHealth: Making Neural Networks Dynamic for Effective Healthcare Analytics | 10.14778/3554821.3554835 | Kaiping Zheng | Proceedings of the VLDB Endowment | 2022 | Real | Private | \ | \ | 16700 | trimodal | tabular time series time series | \ | m1: clinical m2: clinical time series (categorical data) m3: clinical time-series (numerical data) | No | m1: raw m2: raw m3: raw | m1: FC m2: RNN m3: RNN | Heterogeneous | Single | Sudden | 1 | 1 | Async | 0 | 1 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Prognosis | Other (kidney injury) | 1 | Yes | \ | \ | \ | \ | \ | Yes | \ | m2: feature importance m3:feature importance | modality-wise attention | Yes | Yes | Yes | \ | Yes | \ | \ | ||||||||||||||||||||
33 | End-to-End Learning of Fused Image and Non-Image Features for Improved Breast Cancer Classification from MRI | 10.1109/ICCVW54120.2021.00368 | Gregory Holste | ICCV 2021 | 2021 | Real | Private | \ | \ | 17046 | Malignant: 3421 Benign: 13625 | 0.31 | highly imbalanced | No | \ | bimodal | imaging tabular | \ | m1: MRI m2: clinical | No (imputed) | m1: raw m2: raw | m1: CNN m2: FC | Heterogeneous | m1: 50000 m2: 18 | m1: 512 m2: 512 | Equal | m1: 0.0102 m2: 28.444 | Single | Sudden | 1 | 1 | Async | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Oncology | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | Yes | Yes | \ | Yes | \ | \ | |||||||||||||
34 | Exploring multimodal fusion for continuous protective behavior detection | https://doi.org/10.1109/ACII55700.2022.9953851 | G Cen | 10th International Conference on Affective Computing and Intelligent Interaction (ACII) | 2022 | Real | Public | EmoPain | https://wangchongyang.ai/EmoPainChallenge2020/ | 7629 | Protective behaviour: 1012 No protective behaviour: 6617 | 0.23 | highly imbalanced | No | \ | bimodal | video time series | graph graph | m1: RGB video (body movement) m2: electrical (EMG) | No | m1: handcrafted m2: handcrafted | m1: GNN m2: GNN | Homogeneous | Multiple | Sudden | 12 | 1 | Sync | 8 | 0 | 4 | 0 | 0 | 2 | RNN-FC | Unimodal > Multimodal | Supervised | Classification | Detection | Other (chronic pain) | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | Yes | Yes | Yes | Yes | \ | Yes | \ | \ | https://github.com/EnTimeMent/Hierarchical_HAR-PBD | ||||||||||||||
35 | GMRLNet: A graph-based manifold regularization learning framework for placental insufficiency diagnosis on incomplete multimodal ultrasound data | 10.1109/TMI.2023.3278259 | Jing Jiao | IEEE Transactions on Medical Imaging | 2023 | Real | Private | \ | \ | 179 | Placental insufficiency: 31 Normal placentas: 148 | 0.29 | highly imbalanced | Yes | balanced batching | bimodal | imaging imaging | \ | m1: Ultrasound m2: MicroFlow Imaging | No | m1: raw m2: raw | m1: GNN m2: GNN | Homogeneous | Multiple | Sudden | 5 | 1 | Sync | 4 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Other (placenta insufficency) | 1 | Yes | \ | \ | \ | \ | \ | Yes | Triplet Loss | m1:CAM,UMAP m2:CAM,UMAP | CAM | Yes | Yes | Yes | Yes | \ | Yes | \ | \ | Yes | \ | \ | \ | Yes | |||||||||
36 | Private | \ | \ | 635 | Gestional diabetes mellitus: 119 Non gestional diabetes mellitus: 516 | 0.30 | highly imbalanced | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
37 | Hierarchical-order multimodal interaction fusion network for grading gliomas | https://doi.org/10.1088/1361-6560/ac30a1 | Man He1,2, Kangfu Han | Physics in Medicine & Biology | 2021 | Real | Public | No name. Multimodal brain MRI data of glioma patients from TCIA | https://cancerimagingarchive.net | 214 | High-grade gliomas: 106 Low-grade gliomas: 108 | 0.50 | balanced | \ | \ | bimodal | imaging imaging | \ | m1: MRI (T1ce) m2: MRI (T2-FLAIR) | No | m1: raw m2: raw | m1: CNN m2: CNN | Homogeneous | Multiple | Sudden | 10 | 1 | Sync | 6 | 0 | 1 | 3 | 0 | 3 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Oncology | 1 | Yes | \ | \ | \ | \ | \ | \ | Diverse learning Loss | Yes | Yes | Yes | Yes | Yes | Yes | \ | Yes | \ | \ | Yes | \ | \ | Yes | \ | |||||||||
38 | Public | BraTS2017 | https://www.med.upenn.edu/sbia/brats2017/registration.html | 285 | High-grade gliomas: 210 Low-grade gliomas: 75 | 0.39 | highly imbalanced | No | \ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
39 | Improving detection of prostate cancer foci via information fusion of MRI and temporal enhanced ultrasound | 10.1007/s11548-020-02172-5 | Alireza Sedghi | International Journal of Computer Assisted Radiology and Surgery | 2020 | Real | Private | \ | \ | 145 | Cancer: 51 Benign: 94 | 0.46 | imbalanced | No | \ | bimodal | imaging video | imaging imaging | m1: MRI (ADC) m2: TeUS | No | m1: raw m2: learned | m1: CNN m2:CNN | Homogeneous | Single | Sudden | 1 | 1 | Sync | 0 | 0 | 0 | 0 | 1 | 1 | Conv | Unimodal = Multimodal | Supervised | Segmentation Detection | Detection Diagnosis | Oncology | 2 | Yes | \ | \ | \ | \ | \ | No | \ | m1:masking m2:masking | Yes | Yes | \ | ||||||||||||||||||||
40 | Improving knee osteoarthritis classification using multimodal intermediate fusion of X-ray, MRI, and clinical information | https://doi.org/10.1007/s00521-023-08214-8 | Carmine Guida1 • Ming Zhang2 • Juan Shan1 | Neural Computing and Applications | 2023 | Real | Public | Osteoarthritis Initiative database | https://nda.nih.gov/oai/ | 1100 | Severity class 1: 220 Severity class 2: 220 Severity class 3: 220 Severity class 4: 220 Severity class 5: 220 | 0.50 | balanced | \ | \ | trimodal | imaging imaging tabular | \ | m1: MRI m2: X-Ray m3: clinical | No | m1: raw m2: raw m3: raw | m1: CNN m2: CNN m3: raw | Heterogeneous | m1: 3072000 m2: 132000 m3: 4 | m1: 512 m2: 512 m3: \ | Not equal | m1: 0.0002 m2: 0.0039 m3: \ | Multiple | Gradual | 2 | 1 | Async | 2 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Other (osteoartritis) | 1 | No | m1 m2 | supervised | same | related | entire net | No | \ | Yes | Yes | Yes | Yes | \ | Yes | \ | \ | Yes | \ | \ | |||||||||
41 | iTCep: a deep learning framework for identification of T cell epitopes by harnessing fusion features | https://doi.org/10.3389/fgene.2023.1141535 | Y Zhang | Frontiers in Genetics | 2023 | Real | Public | No name. Dataset built from the combination of various public datasets | http://biostatistics.online/iTCep/#/download | 21518 | Positive: 10759 Negative: 10759 | 0.50 | balanced | \ | \ | bimodal | tabular tabular | \ | m1: genomics (peptide sequences) m2: genomics (CDR3 sequences) | No | m1: raw m2: raw | m1: CNN m2: CNN | Homogeneous | Single | Sudden | 1 | 1 | Async | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Oncology | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | https://github.com/kbvstmd/iTCep/ | ||||||||||||||||||||||
42 | Liver Tumor Detection Via A Multi-Scale Intermediate Multi-Modal Fusion Network on MRI Images | 10.1109/ICIP42928.2021.9506237 | Pan, Chao and Zhou, Peiyun and Tan, Jingru and Sun, Baoye and Guan, Ruoyu and Wang, Zhutao and Luo, Ye and Lu, Jianwei | 2021 IEEE International Conference on Image Processing (ICIP) | 2021 | Real | Private | \ | \ | 732 | Malignant: 159 Benign: 573 | 0.34 | highly imbalanced | No | \ | bimodal | imaging imaging | \ | m1: MRI (T1A) m2: MRI (T1V) | No | m1: raw m2: raw | m1: CNN m2: CNN | Homogeneous | Multiple | Sudden | 11 | 1 | Sync | 6 | 0 | 1 | 4 | 0 | 3 | Attention-FC | Unimodal < Multimodal | Supervised | Object Detection | Detection | Oncology | 2 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | Yes | Yes | \ | Yes | \ | \ | |||||||||||||||||
43 | Long-term cognitive decline prediction based on multi-modal data using Multimodal3DSiameseNet: transfer learning from Alzheimer’s disease to Parkinson’s disease | https://doi.org/10.1007/s11548-023-02866-6 | Cécilia Ostertag, Muriel Visani, Thierry Urruty, Marie Beurton-Aimar | International Journal of Computer Assisted Radiology and Surgery | 2023 | Real | Public | ADNI (for pre-training) | https://adni.loni.usc.edu/data-samples/access-data/ | 381 | Declining: 180 Stable: 201 | 0.50 | balanced | \ | \ | bimodal | imaging tabular | \ | m1: MRI m2: clinical | No | m1: raw m2: raw | m1: CNN m2: FC m3: FC | Heterogeneous | m1: 826200 m2: 8 m3: 3 | m1: 256 m2: 8 m3: 3 | Not equal | m1: 0.0003 m2: 1.0000 m3: 1.0000 | Multiple | Gradual | 4 | 1 | Async | 2 | 0 | 2 | 0 | 0 | 2 | FC | Unimodal > Multimodal | Supervised | Classification | Prognosis | Mental health | 1 | No Yes | m1 m2 multimodal | supervised | same | related | entire net | No | \ | Yes | https://github.com/CeciliaOstertag/MultiNet | ||||||||||||||||||
44 | Public | PPMI (for transfer learning) | https://www.ppmi-info.org/ | 134 | Declining: 87 Stable: 47 | 0.46 | imbalanced | No | \ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
45 | MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images | Nasir Hayat | Machine Learning for Healthcare | 2022 | Real | Public | MIMIC-IV | https://physionet.org/content/mimiciv/1.0/ | 377.095 | bimodal | imaging time series | \ | m1: X-Ray (Chest X-Ray) m2: clinical time series | Yes | m1: raw m2: raw | m1: CNN m2: RNN | Heterogeneous | m1: 150000 m2: 76 | m1: 512 m2: 256 | Not equal | m1: 0.0034 m2: 3.3684 | Single | Sudden | 1 | 1 | Async | 0 | 0 | 1 | 0 | 0 | 1 | RNN-FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis Prognosis | Other (phenotype classification, all cause mortality) | 1 | No | m1 m2 | supervised | same | related | entire net | Yes | \ | Yes | Yes | Yes | Yes | \ | \ | \ | Yes | Yes | \ | \ | \ | Yes | https://github.com/nyuad-cai/MedFuse | ||||||||||||
46 | Public | MIMIC-CXR | https://physionet.org/content/mimic-cxr/2.0.0/ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
47 | MIFTP: A Multimodal Multi-Level Independent Fusion Framework with Improved Twin Pyramid for Multilabel Chest X-Ray Image Classification | 10.1109/ICTAI56018.2022.00170 | Jingni Zeng | International Conference on Tools with Artificial Intelligence (ICTAI) | 2022 | Real | Public | MIMIC-CXR | https://physionet.org/content/mimic-cxr/ | 61300 | Unhealthy: 48% Healthy: 52% | 0.50 | balanced | \ | \ | bimodal | imaging text | \ | m1: X-Ray (Chest X-Ray) m2: Free-Text Reports | No | m1: raw m2: raw | m1: CNN m2: RNN | Heterogeneous | Single | Sudden | 1 | 1 | Async | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Pneumology | 1 | No | m1 | supervised | different | different | entire net | No | \ | m1:GradCAM | Yes | Yes | \ | ||||||||||||||||||||
48 | MMHFNet: Multi-modal and multi-layer hybrid fusion network for voice pathology detection | https://doi.org/10.1016/j.eswa.2023.119790 | Hussein M.A. Mohammed | Expert Systems With Applications | 2023 | Real | Public | Saarbruecken Voice Database | https://stimmdatenbank.coli.uni-saarland.de/help_en.php4 | 1374 | Pathological: 687 Healthy: 687 | 0.50 | balanced | \ | \ | bimodal | audio time series | imaging imaging | m1: speech m2: electrical (EGG) | No | m1: handcrafted m2: handcrafted | m1: CNN m2: CNN | Homogeneous | Multiple | Multiflow | 5 | 4 | Sync | 5 | 0 | 0 | 0 | 0 | 1 | RNN-FC | Unimodal = Multimodal | Supervised | Classification | Diagnosis | Other (voice pathology) | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | Yes | \ | |||||||||||||||||||||
49 | Modeling uncertainty in multi-modal fusion for lung cancer survival analysis | https://doi.org/10.1109/ISBI48211.2021.9433823 | H Wang | IEEE 18th International Symposium on Biomedical Imaging (ISBI) | 2021 | Real | Public | NSCLC-Radiogenomics | https://wiki.cancerimagingarchive.net/display/Public/NSCLC+Radiogenomics | 107 | bimodal | imaging tabular | tabular tabular | m1: CT m2: genomics (RNA sequence) | No | m1: handcrafted m2: raw | m1: FC m2: FC | Homogeneous | m1: 107 m2: 5268 | m1: 50 m2: 50 | Equal | m1: 0.4673 m2: 0.0095 | Single | Sudden | 1 | 1 | Async | 0 | 0 | 0 | 0 | 1 | 1 | No-Architecture | Unimodal > Multimodal | Supervised | Classification | Prognosis | Oncology | 1 | Yes | \ | \ | \ | \ | \ | No | uncertainty model of individual modalities | Yes | Yes | Yes | Yes | \ | \ | \ | Yes | |||||||||||||||||
50 | MS2-GNN: Exploring GNN-Based Multimodal Fusion Network for Depression Detection | https://doi.org/10.1109/TCYB.2022.3197127 | Tao Chen | IEEE TRANSACTIONS ON CYBERNETICS | 2022 | Real | Public | DAIC-WOZ | https://dcapswoz.ict.usc.edu/daic-woz-database-download/ | 37 | Major depressive disorder: 16 Normal Control: 21 | 0.49 | balanced | \ | \ | bimodal | time series audio | graph graph | m1: electrical (EEG) m2: speech | No | m1: learned m2: learned | m1: GNN m2: GNN | Homogeneous | Multiple | Sudden | 3 | 1 | Sync | 1 | 0 | 1 | 0 | 1 | 3 | FC | Unimodal > Multimodal | Supervised | Classification | Detection | Mental health | 1 | Yes | \ | \ | \ | \ | \ | No | similarity loss; orthogonalization loss; | Yes | Yes | Yes | \ | ||||||||||||||||||||
51 | Public | MODMA | https://modma.lzu.edu.cn/data/index/ | 189 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
52 | MSMFN: An Ultrasound Based Multi-Step Modality Fusion Network for Identifying the Histologic Subtypes of Metastatic Cervical Lymphadenopathy | 10.1109/TMI.2022.3222541 | Zheling Meng | IEEE Transactions on Medical Imaging | 2022 | Real | Private | \ | \ | 301 | Squamous cell carcinoma: 121 Adenocarcinoma: 180 | 0.48 | imbalanced | No | \ | high-modal | imaging imaging imaging video tabular | \ | m1: Ultrasound (Color Doppler Flow Imaging) m2: Ultrasound (B-mode Ultrasound) m3: Ultrasound (Ultrasound Elastography) m4: DCE-US m5: clinical | No | m1: raw m2: raw m3: raw m4: raw m5: raw | m1: CNN m2: CNN m3: CNN m4: CNN m5: raw | Heterogeneous | m4: 163840 | m4: 256 | m4: 0.0016 | Multiple | Gradual | 5 | 1 | Async | 2 | 0 | 2 | 0 | 1 | 3 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Oncology | 1 | Yes | \ | \ | \ | \ | \ | No | self-supervised orthogonalization loss | m1:LayerCAM m2:LayerCAM m3:LayerCAM m4:LayerCAM | Yes | Yes | Yes | Yes | \ | Yes | \ | \ | https://github.com/RichardSunnyMeng/MSMFN | |||||||||||
53 | Multi-modal deep learning of functional and structural neuroimaging and genomic data to predict mental illness | 10.1109/EMBC46164.2021.9630693 | Md Abdur Rahaman | Engineering in Medicine & Biology Society | 2021 | Real | Public | fBIRN | https://www.nitrc.org/projects/fbirn/ | 437 | Schizophrenia: 162 Healthy Control: 275 | 0.47 | imbalanced | No | \ | trimodal | imaging imaging tabular | tabular tabular tabular | m1: MRI (fMRI) m2: MRI (sMRI) m3: genomics | No | m1: handcrafted m2: handcrafted m3: raw | m1: FC m2: FC m3: RNN | Heterogeneous | Single | Sudden | 1 | 1 | Async | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Mental health | 1 | Yes | \ | \ | \ | \ | \ | No | multimodal regularization | m1:saliency maps m2:saliency maps m3:saliency maps | modality-wise attention correlation of interaction | Yes | Yes | Yes | Yes | \ | Yes | \ | \ | ||||||||||||||
54 | Public | COBRE | https://fcon_1000.projects.nitrc.org/indi/retro/cobre.html | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
55 | Public | MRPC | https://onlinelibrary.wiley.com/doi/full/10.1002/hbm.24723?casa_token=lZwao4xN4usAAAAA%3AjHtnhy56Dtc0nVR70A6x6B12W9bUAniIB_gXsFGlBpW5txcZtmhAtyaRMgL31iDMybQdZbQu0h0 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
56 | Multi-modal fusion model for predicting adverse cardiovascular outcome post percutaneous coronary intervention | https://doi.org/10.1088/1361-6579/ac9e8a | A Bhattacharya | Physiological Measurement | 2022 | Real | Private | \ | \ | 17356 | bimodal | time series tabular | \ | m1: electrical (ECG) m2: clinical | No (imputed) | m1: raw m2: raw | m1: CNN m2: FC | Heterogeneous | m1: 11766 m2: 157 | m1: 16 m2: 128 | Not equal | m1: 0.0014 m2: 0.8153 | Single | Sudden | 1 | 1 | Async | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Prognosis | Other (heart failure) | 1 | Yes | \ | \ | \ | \ | \ | No | \ | m1:data ablation m2:LIME | Yes | Yes | Yes | \ | \ | \ | Yes | |||||||||||||||||
57 | Multi-objective optimization determines when, which and how to fuse deep networks: An application to predict COVID-19 outcomes | https://doi.org/10.1016/j.compbiomed.2023.106625 | Valerio Guarrasi | Computers in Biology and Medicine | 2023 | Real | Public | AIforCOVID | https://aiforcovid.radiomica.it/ | 820 | Sever: 436 Mild: 384 | 0.50 | balanced | \ | \ | bimodal | imaging tabular | \ | m1: X-Ray (Chest X-Ray) m2: clinical | No | m1: raw m2: raw | m1: CNN m2: FC | Heterogeneous | m1: 50000 m2: 34 | m1: 2 m2: 2 | Equal | m1: 0.00004 m2: 0.0588 | Single | Sudden | 1 | 1 | Async | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Prognosis | Pneumology | 1 | No | m1 | supervised | different | different | entire net | No | \ | m1:GradCAM m2:Integrated Gradient | Yes | Yes | Yes | Yes | Yes | Yes | \ | \ | Yes | \ | Yes | \ | \ | \ | Yes | ||||
58 | Multi-view Deep Neural Networks for multiclass skin lesion diagnosis | https://doi.org/10.1109/COINS54846.2022.9854997 | E Perez | IEEE International Conference on Omni-layer Intelligent Systems (COINS) | 2022 | Real | Public | ISIC2019 | https://www.isic-archive.com | 74347 | Class 1: 10630 Class 2: 9194 Class 3: 8624 Class 4: 9563 Class 5: 9434 Class 6: 8251 Class 7: 9085 Class 8: 9566 | 0.50 | balanced | \ | \ | bimodal | imaging tabular | \ | m1: skin lesion m2: clinical | No | m1: raw m2: raw | m1: CNN m2: FC | Heterogeneous | Single | Sudden | 1 | 1 | Async | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Oncology | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | Yes | Yes | Yes | \ | |||||||||||||||||||
59 | Multimodal deep learning to predict prognosis in adult and pediatric brain tumors | https://doi.org/10.1038/s43856-023-00276-y | S Steyaert | Communications Medicine | 2023 | Real | Public | Adult cohort from TCGA | https://www.cancer.gov/ccg/research/genome-sequencing/tcga | 880 | Gliobastoma: 454 Low-grade glioma: 426 | 0.50 | balanced | \ | \ | bimodal | imaging tabular | \ | m1: histopathology m2: genomics | No | m1: raw m2: raw | m1: CNN m2: FC | Heterogeneous | m1: 50176 m2: 12778 | m1: 2048 m2: 2048 | Equal | m1: 0.0408 m2: 0.1603 | Single | Sudden | 1 | 1 | Async | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Regression | Prognosis | Oncology | 1 | No | m1 | supervised | different | different | multimodal | No | different learning rates to balance learning of different modalities | m1: saliency maps m2:SHAP | Yes | Yes | Yes | Yes | Yes | Yes | \ | \ | \ | Yes | \ | Yes | \ | \ | Yes | https://github.com/gevaertlab/MultiModalBrainSurvival | |||
60 | Public | CPTAC-GBM (adult) | https://www.cancerimagingarchive.net/datascope/cptac/home/ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
61 | Public | PBTA (pedriatic) | https://kidsfirstdrc.org, | 305 | High-grade glioma: 107 Low-grade glioma: 198 | 0.46 | imbalanced | No | \ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
62 | Multimodal Dynamics: Dynamical Fusion for Trustworthy Multimodal Classification | 10.1109/CVPR52688.2022.02005 | Zongbo Han | IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) | 2022 | Real | Public | BRCA | https://portal.gdc.cancer.gov/projects/TCGA-BRCA | 875 | trimodal | tabular tabular tabular | \ | m1: genomics (mRNA) m2: genomics (DNA methylation) m3: genomics (miRNA) | No | m1: raw m2: raw m3: raw | m1: CNN m2: CNN m3: CNN | Homogeneous | Single | Sudden | 1 | 1 | Sync | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Oncology | 1 | Yes | \ | \ | \ | \ | \ | No | sparsity regularization; True class probability | m1:feature importance m2:feature importance m3:feature importance | modality importance | Yes | yes | Yes | \ | \ | github.com/TencentAILabHealthcare/mmdynamics. | |||||||||||||||||||||
63 | Public | LGG | https://portal.gdc.cancer.gov/projects/TCGA-LGG | 510 | Oncology | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
64 | Public | ROSMAP | https://www.niagads.org/datasets/ng00029 | 351 | Mental Health | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
65 | Public | KIPAN | https://www.cancer.gov/ccg/research/genome-sequencing/tcga | 658 | Oncology | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
66 | Multimodal fusion models for pulmonary embolism mortality prediction | 10.1038/s41598-023-34303-8 | Noa Cahan | Scientific Reports | 2023 | Real | Private | \ | \ | 358 | Died: 38 Alive: 320 | 0.19 | highly imbalanced | Yes | Weighted version of Binary Cross-Entropy loss | bimodal | imaging tabular | \ | m1: CT (CTPA volume) m2: clinical | No | m1: raw m2: raw | m1: CNN m2: Transformer | Heterogeneous | m1: 2097152 m2: 961 | m1: 64 m2: 64 | Equal | m1: 0.00003 m2: 0.0666 | Single | Sudden | 1 | 1 | Async | 1 | 0 | 0 | 0 | 0 | 1 | Attention-FC | Unimodal > Multimodal | Supervised | Classification | Prognosis | Pneumology | 1 | Yes | \ | \ | \ | \ | \ | \ | \ | m1:GradCAM ,t-SNE m2:feature importance, t-SNE | t-SNE | Yes | Yes | Yes | \ | Yes | \ | ||||||||||||
67 | Multimodal fusion of imaging and genomics for lung cancer recurrence prediction | 10.1109/ISBI45749.2020.9098545 | V Subramanian | International Symposium on Biomedical Imaging (ISBI) | 2020 | Real | Public | NSCLC-Radiogenomics | https://wiki.cancerimagingarchive.net/display/Public/NSCLC+Radiogenomics | 130 | bimodal | tabular imaging | tabular tabular | m1: genomics m2: CT | No | m1: raw m2: learned | m1: FC m2: FC | Homogeneous | m1: 500 m2: 1024 | m1: 64 m2: 64 | Equal | m1: 0.1280 m2: 0.0625 | Single | Sudden | 1 | 1 | Sync | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Regression | Prognosis | Oncology | 1 | yes | \ | \ | \ | \ | \ | No | \ | Yes | Yes | \ | ||||||||||||||||||||||
68 | Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection | https://doi.org/10.1038/s41598-020-78888-w | SC Huang | Scientific Reports | 2020 | Real | Private | \ | \ | 1837 | Positive: 726 Negative: 1111 | 0.48 | imbalanced | No | \ | high-modal | imaging tabular tabular tabular tabular tabular tabular | tabular tabular tabular tabular tabular tabular tabular | m1: CT m2: clinical (ICD9 codes) m3: clinical (vitals) m4: clinical (lab tests) m5: clinical (demographics) m6: clinical (inpatient medications) m7: clinical (outpatient medications) | No | m1: learned m2: raw m3: raw m4: raw m5: raw m6: raw m7: raw | m1: FC m2: FC m3: FC m4: FC m5: FC m6: FC m7: FC | Homogeneous | Single | Sudden | 1 | 1 | Sync | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal = Multimodal | Supervised | Classification | Diagnosis | Pneumology | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | Yes | Yes | Yes | \ | \ | \ | Yes | \ | \ | Yes | \ | Yes | https://github.com/marshuang80/pe_fusion | ||||||||||
69 | Multimodal Hierarchical CNN Feature Fusion for Stress Detection | 10.1109/ACCESS.2023.3237545 | RADHIKA KUTTALA | IEEE Access | 2023 | Real | Public | ASCERTAIN | https://ascertain-dataset.github.io | 58 | imbalanced declared | Yes | Synthetic Minority Oversampling Technique | bimodal | time series time series | tabular tabular | m1: electrical (ECG) m2: electrical (EDA) | No | m1: handcrafted m2: handcrafted | m1: CNN m2: CNN | Homogeneous | Multiple | Sudden | 1 | 1 | Sync | 0 | 0 | 3 | 1 | 0 | 2 | FC | Unimodal > Multimodal | Supervised | Classification | Detection | Mental health | 1 | Yes | \ | \ | \ | \ | \ | No | \ | m1:t-SNE | |||||||||||||||||||||||||
70 | Public | CLAS | https://dx.doi.org/10.21227/ybsw-yr53 | 59 | imbalanced declared | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
71 | Public | WAUC | https://musaelab.ca/resources/ | 22 | imbalanced declared | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
72 | Public | MAUS | https://ieee-dataport.org/open-access/maus-dataset-mental-workload-assessment-n-back-task-using-wearable-sensor | 45 | imbalanced declared | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
73 | Multimodal Information Fusion for Glaucoma and Diabetic Retinopathy Classification | https://doi.org/10.1007/978-3-031-16525-2_6 | Y Li | Ophthalmic Medical Image Analysis: 9th International Workshop, OMIA 2022 | 2022 | Real | Public | GAMMA | https://gamma.grand-challenge.org/ | 200 | Trainign Set: Moderate or advanced glauoma: 24 Early glauoma patients: 26 No glauoma patients: 50 | 0.46 | imbalanced | No | \ | bimodal | imaging imaging | \ | m1: 2D fundus images m2: 3D OCT scans | No | m1: raw m2: raw | m1: CNN m2: CNN | Homogeneous | m1: 602112 m2: 9404416 | m1: 512 m2: 512 | Equal | m1: 0.0009 m2: 0.00005 | Multiple | Sudden | 1 | 1 | Sync | 1 | 0 | 4 | 0 | 0 | 2 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Other (glaucoma, diabetic retinopathy) | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | Yes | Yes | Yes | \ | Yes | \ | \ | ||||||||||||
74 | Multimodal medical tensor fusion network-based DL framework for abnormality prediction from the radiology CXRs and clinical text reports | https://doi.org/10.1007/s11042-023-14940-x | S Shetty | Multimedia Tools and Applications | 2022 | Real, Simulated | Public | Indiana University dataset | https://openi.nlm.nih.gov/faq#collection | 6229 (3638 before synthetic data generation) | Abnormal: 62% Normal: 38% | 0.47 | imbalanced | No | \ | bimodal | imaging text | imaging tabular | m1: X-Ray (Chest X-Ray) m2: Free-Text Reports | No | m1: raw m2: learned | m1: CNN m2: CNN | Homogeneous | m1: 67500 m2: 26000 | m1: 1024 m2: 1024 | Equal | m1: 0.0152 m2: 0.0394 | Single | Sudden | 1 | 1 | Async | 0 | 0 | 1 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Prognosis | Pneumology | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | Yes | Yes | \ | ||||||||||||||||
75 | Private | \ | \ | 1498 (501 before synthetic data generation) | Abnormal: 48% Normal: 52% | 0.50 | balanced | \ | \ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
76 | Predicting Brain Degeneration with a Multimodal Siamese Neural Network | 10.1109/IPTA50016.2020.9286657 | Cécilia Ostertag, Marie Beurton-Aimar, Muriel Visani, Thierry Urruty, Karell Bertet | 2020 Tenth International Conference on Image Processing Theory, Tools and Applications (IPTA) | 2020 | Real | Public | ADNI | https://adni.loni.usc.edu/data-samples/access-data/ | 377 | Cognitive decline: 186 Stable: 191 | 0.50 | balanced | \ | \ | bimodal | imaging tabular | \ | m1: MRI m2: clinical | Yes | m1: raw m2: raw | m1: CNN m2: FC | Heterogeneous | m1: 826200 m2: 8 m3: 3 | m1: 256 m2: 8 m3: 3 | Not equal | m1: 0.0003 m2: 1.0000 m3: 1.0000 | Multiple | Gradual | 4 | 1 | Async | 2 | 0 | 2 | 0 | 0 | 2 | FC | Unimodal > Multimodal | Supervised | Classification | Prognosis | Mental health | 1 | Yes | \ | \ | \ | \ | \ | Yes | \ | Yes | Yes | Yes | \ | ||||||||||||||||
77 | Predicting heart failure in‐hospital mortality by integrating longitudinal and category data in electronic health records | 10.1007/s11517-023-02816-z | Meikun Ma | Medical & Biological Engineering & Computing | 2023 | Real | Public | MIMIC III | https://physionet.org/content/mimiciii/1.4/ | 7696 | Positive: 1190 Negative: 6505 | 0.26 | highly imbalanced | No | \ | bimodal | time series tabular | \ | m1: clinical time series m2: clinical | Yes | m1: raw m2: raw | m1: RNN m2: FC | Heterogeneous | Single | Sudden | 1 | 1 | Async | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal = Multimodal | Supervised | Classification | Prognosis | Other (heart failure mortality prediction) | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | Yes | \ | \ | Yes | \ | ||||||||||||||||||
78 | Predicting Successes and Failures of Clinical Trials With Outer Product–Based Convolutional Neural Network | 10.3389/fphar.2021.670670 | Sangwoo Seo | Frontiers in Pharmacology | 2021 | Real | Public | AACT | https://aact.ctti-clinicaltrials.org/download | 828 | Approved: 757 Failed: 71 | 0.16 | highly imbalanced | Yes | Synthetic Minority Oversamplig Technique + cost-sensitive learning method | bimodal | tabular tabular | \ | m1: chemical features m2: genomics | No (imputed) | m1: raw m2: raw | m1: FC m2: FC | Homogeneous | Single | Sudden | 1 | 1 | Sync | 0 | 0 | 1 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Drug Discovery | Oncology | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | Yes | Yes | Yes | \ | \ | https://github.com/sawoo9410/ Clinical-Trials-with-OPCNN | |||||||||||||||||
79 | Radiopaths: Deep Multimodal Analysis on Chest Radiographs | 10.1109/BigData55660.2022.10020356 | Kohankhaki M. | 2022 IEEE International Conference on Big Data (Big Data) | 2022 | Real | Public | MIMIC-CXR | https://physionet.org/content/mimic-cxr/ | 377110 | imbalanced declared | Yes | Focal loss | trimodal | imaging tabular text | \ | m1: X-Ray (Chest X-Ray) m2: clinical m3: Free-Text Reports | Yes | m1: raw m2: raw m3: raw | m1: CNN m2: FC m3: Transformer | Heterogeneous | m1: 50176 m2: 15 m3: 393216 | m1: 512 m2: 512 m3: 512 | Equal | m1: 0.0102 m2: 34.133 m3: 0.0013 | Single | Sudden | 1 | 1 | Sync | 1 | 0 | 0 | 0 | 0 | 1 | Attention-FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Pneumology | 1 | No | m1 m3 | supervised | same | related | multimodal | Yes | \ | m1:GradCAM++ m2:Integrated gradient m3:Integrated gradient | Yes | Yes | \ | \ | https://github.com/a-ayad/radiopaths | ||||||||||||||||
80 | Public | MIMIC-IV | https://physionet.org/content/mimiciv/1.0/ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
81 | Single Modality vs. Multimodality: What Works Best for Lung Cancer Screening? | 10.3390/s23125597 | Joana Vale Sousa | Sensors | 2023 | Real | Public | NLST | https://cdas.cancer.gov/nlst/ | 1079 | Malignant: 424 Benign: 655 | 0.48 | imbalanced | No | \ | bimodal | imaging tabular | \ | m1: CT m2: clinical | No | m1: raw m2: raw | m1: CNN m2: raw | Heterogeneous | m1: 50000 | m1: 512 | Single | Sudden | 1 | 1 | Async | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Oncology | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | Yes | \ | |||||||||||||||||||
82 | Stress Detection using CNN Fusion | 10.1109/TENCON54134.2021.9707438 | Radhika K, V Ramana Murthy Oruganti | 2021 IEEE Region 10 Conference (TENCON) | 2021 | Real | Public | ASCERTAIN | https://ascertain-dataset.github.io/ | 58 | imbalanced declared | Yes | Synthetic Minority Oversampling Technique | bimodal | time series time series | tabular tabular | m1: electrical (ECG) m2: electrical (EDA) | No | m1: handcrfated m2: handcrafted | m1: CNN m2: CNN | Homogeneous | Multiple | Sudden | 4 | 1 | Sync | 1 | 0 | 2 | 1 | 0 | 3 | FC | Unimodal > Multimodal | Supervised | Classification | Detection | Mental health | 1 | Yes | \ | \ | \ | \ | \ | No | \ | m1:t-SNE m2:t-SNE | t-SNE | ||||||||||||||||||||||||
83 | Public | CLAS | https://www.wwwsensornetworkslab.com/clas | 59 | imbalanced declared | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
84 | TinyM2Net-V2: A Compact Low Power Sotware Hardware Architecture for Multimodal Deep Neural Networks | https://dl.acm.org/doi/10.1145/3595633 | HASIB-AL RASHID | ACM Transactions on Embedded Computing Systems | 2023 | Real | Public | DiCOVA challenge | http://dicovachallenge.github.io/ | 929 | Positive: 172 Negative: 757 | 0.30 | highly imbalanced | Yes | Undersampling | trimodal | audio audio audio | \ | m1: cough audio m2: breathing audio m3: speech | No | m1: raw m2: raw m3: raw | m1: CNN m2: CNN m3: CNN | Homogeneous | m1: 4240 m2: 4980 m3: 2639 | m1: 32 m2: 32 m3: 32 | Equal | m1: 0.0075 m2: 0.0064 m3: 0.0121 | Single | Sudden | 1 | 1 | Sync | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Pneumology | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | Yes | \ | |||||||||||||||||
85 | Toward attention-based learning to predict the risk of brain degeneration with multimodal medical data | 10.3389/fnins.2022.1043626 | Xiaofei Sun | Frontiers in Neuroscience | 2023 | Real | Public | ADNI | https://adni.loni.usc.edu/data-samples/access-data/#access_data | 969 | Alzheimer disease: 288 Mild cognitive impairment: 365 Cognitive normal: 316 | 0.50 | balanced | \ | \ | bimodal | imaging tabular | \ | m1: MRI m2: clinical | No | m1: raw m2: raw | m1: CNN m2: FC | Heterogeneous | Multiple | Multiflow | 3 | 2 | Async | 1 | 2 | 0 | 0 | 0 | 2 | FC | Unimodal > Multimodal | Supervised | Classification | Diagnosis | Mental health | 1 | Yes | \ | \ | \ | \ | \ | No | \ | m1:attention maps | Yes | Yes | Yes | Yes | Yes | \ | \ | \ | Yes | Yes | \ | \ | \ | Yes | |||||||||
86 | Private | \ | \ | 396 | Normal Control: 99 T1 MRI diabetes mellitus: 135 T2 MRI diabetes mellitus: 162 | 0.49 | balanced | \ | \ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
87 | Transformer-based Self-supervised Multimodal Representation Learning for Wearable Emotion Recognition | 10.1109/TAFFC.2023.3263907 | Yujin Wu, Mohamed Daoudi, IEEE Senior, Ali Amad | IEEE Transactions on Affective Computing | 2023 | Real | Public | WESAD | https://dl.acm.org/doi/abs/10.1145/3242969.3242985?casa_token=rHxnaWaHXiUAAAAA:MuPOXSQTm_Q9H2-fYzKzleSxvBKFfUeqvDzJCAwpwQm2ryCuBzMXerm_pnwNzNsSvUUvKj0SS5jZ | 12185 | Stress: 36279 Non-stress: 85574 | 0.42 | highly imbalanced | No | \ | trimodal | time series time series time series | \ | m1: electrical (EDA) m2: clinical time series (blood volume pressure) m3: clinical time series (skin temperature) | No | m1: raw m2: raw m3: raw | m1: CNN m2: CNN m3: CNN | Homogeneous | Single | Sudden | 1 | 1 | Sync | 1 | 0 | 0 | 0 | 0 | 1 | Attention-FC | Unimodal > Multimodal | Supervised | Classification | Detection | Mental health | 1 | No | m1 m2 m3 multimodal | unsupervised | different | related | multimodal entire net | \ | modality specific classification | Yes | Yes | Yes | Yes | Yes | Yes | \ | Yes | \ | \ | ||||||||||||||
88 | Public | CASE | https://www.nature.com/articles/s41597-019-0209-0 | 95130 | Negative: 32017 Positive: 63113 | 0.45 | imbalanced | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
89 | Public | K-EmoCon | https://www.nature.com/articles/s41597-020-00630-y | 5217 | Negative: 4050 Positive: 1167 | 0.35 | highly imbalanced | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
90 | Trustworthy Deep Neural Network for Inferring Anticancer Synergistic Combinations | 10.1109/JBHI.2021.3126339 | M Alsherbiny | IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS | 2023 | Real | Public | NCI-ALMANAC | https://drive.google.com/drive/folders/1TmC5PjSCa0-oj551w758kZF2WluP6LK1 | 212979 | high-modal | tabular tabular tabular tabular tabular tabular tabular | \ | m1A: chemical features (drug A fingerprint) m1B: chemical features (drug B fingerprint) m2A: chemical features (drug A physicochemical properties) m2B: chemical features (drug B physicochemical properties) m3A: chemical features (drug A toxicophore) m3B: chemical features (drug B toxicophore) m4: genomics | No | m1A: handcrafted m1B: handcrafted m2A: handcrafted m2B: handcrafted m3A: handcrafted m3B: handcrafted m4: raw | m1: FC m2: FC m3: FC m4: FC m5: FC m6: FC m7: FC | Homogeneous | m1: 475 m2: 2235 m3: 3679 m4: 479 m5: 2209 m6: 3710 m7: 18046 | m: 8192 m2: 8192 m3: 8192 m4: 8192 m5: 8192 m6: 8192 m7: 8192 | Equal | m1: 17.246 m2: 3.6653 m3: 2.2267 m4: 17.102 m5: 3.7085 m6: 2.2081 m7: 0.4540 | Single | Sudden | 1 | 1 | Sync | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal > Multimodal | Supervised | Regression | Drug Discovery | Oncology | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | Yes | Yes | Yes | \ | \ | Yes | \ | |||||||||||||||||
91 | Public | ONEIL | https://drive.google.com/drive/folders/1TmC5PjSCa0-oj551w758kZF2WluP6LK1 | 68244 | No | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
92 | TWO-DIMENSIONAL ATTENTIVE FUSION FOR MULTI-MODAL LEARNING OF NEUROIMAGING AND GENOMICS DATA | 10.1109/MLSP55214.2022.9943519 | Md Abdur Rahaman | 2022 IEEE INTERNATIONAL WORKSHOP ON MACHINE LEARNING FOR SIGNAL PROCESSING | 2022 | Real | Public | COBRE | http://fcon_1000.projects.nitrc.org/indi/retro/cobre.html | 437 | Schizophrenia: 162 Healthy Control: 275 | 0.47 | imbalanced | No | \ | trimodal | imaging imaging tabular | imaging tabular tabular | m1: MRI (fMRI) m2: MRI (sMRI) m3: genomics (SNP) | No | m1: raw m2: handcrafted m3: raw | m1: AE m2: FC m3: RNN | Heterogeneous | m1: 2890 m2: 30 m3: 1280 | m1: 100 m2: 100 m3: 100 | Equal | m1: 0.0356 m2: 3.3333 m3: 0.0781 | Single | Sudden | 1 | 1 | Async | 1 | 0 | 0 | 0 | 0 | 1 | FC | Unimodal = Multimodal | Supervised | Classification | Diagnosis | Mental health | 1 | Yes | \ | \ | \ | \ | \ | No | \ | Yes | Yes | Yes | Yes | \ | \ | ||||||||||||||
93 | Public | fBIRN | https://www.nitrc.org/projects/fbirn/ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
94 | Public | MRPC | https://onlinelibrary.wiley.com/doi/pdf/10.1002/hbm.24723?casa_token=IxhDpbBZ2B8AAAAA:_eMvAnDorXVqFDrQd-oBFb6pZVBYR_PJ_X6LtoIhhh-FgbeO2vZA7GrB537Q0awZQUa0Pou2obeCoQ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
95 | Weakly supervised multimodal 30-day all-cause mortality prediction for pulmonary embolism patients | https://doi.org/10.1109/ISBI52829.2022.9761700 | N Cahan | IEEE 19th International Symposium on Biomedical Imaging (ISBI) | 2022 | Real | Private | \ | \ | 363 | Died: 38 Alive: 325 | 0.19 | highly imbalanced | Yes | Focal loss | bimodal | imaging tabular | \ | m1: CT (CTPA) m2: clinical | No | m1: raw m2: raw | m1: CNN m2: Transformer | Heterogeneous | Single | Sudden | 1 | 1 | Async | 1 | 0 | 0 | 0 | 0 | 1 | Attention-FC | Unimodal > Multimodal | Supervised | Classification | Prognosis | Pneumology | 1 | Yes | \ | \ | \ | \ | \ | \ | \ | Yes | Yes | Yes | Yes | \ | Yes | \ | \ | Yes | \ | \ | Yes | \ | |||||||||||
96 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
97 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
98 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
99 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
100 |