1 of 10

Omics-Enhanced Multimodal Representations for Cancer Prognosis and Staging

Sara De Luca, Federico D'Asaro, Paola Berchialla, Giuseppe Rizzo

LINKS Foundation LINKS Foundation Università di Torino LINKS Foundation

UCBM Politecnico di Torino Politecnico di Torino

2 of 10

Research question and state of the art

2

Research question Does adding omics-derived representations in a shared latent space with 3D imaging and clinical text yield more prognostically meaningful patient representations than keeping modalities separate, fusing them post-hoc, or omitting omics?

State of the art

Multimodal AI for precision medicine

Broad literature fusing heterogeneous patient data beyond unimodal models

Omics + imaging

Few models; mostly 2D histopathology slides, not radiology

Pathomic Fusion;

MCAT;

3D volumes + text

Medical MLLMs on volumetric CT and reports, without omics

M3D-LaMed

Merlin

Our contribution: OmicsLaMed

Omics + 3D CT + clinical text in a single MLLM token space

3 of 10

TCGA Dataset

3

7,534

patients with RNA-seq,

methylation and clinical data

805

patients with CT

(10.7%)

The Cancer Genome Atlas Project

Omics

RNA-seq (STAR, 60,660 genes) + DNA methylation (450K/EPIC, 486,426 probes); top 3,000 features per layer, selected per fold

CT scans

Volumetric DICOM series from TCIA; 3D volumes tiled into 32×256×256 patches (M3D-LaMed preprocessing)

Clinical text

Up to 41 GDC fields (demographics, tumour, lifestyle, treatment) serialised to text; target fields removed

- Three data layers per patient:

- Omics: RNA-seq (STAR, 60,660 genes) and DNA methylation (450K/EPIC, 486,426 probes). Per layer, keep the top 3,000 features by |Pearson r| with log survival time on deceased patients, computed on the training fold only -> 6,000 features, no test leakage.

- CT: volumetric DICOM series from TCIA, preprocessed as in M3D-LaMed (32x256x256 patches).

- Clinical text: up to 41 harmonised GDC fields serialised to text; fields encoding the targets (vital status, days to death, AJCC stage, cause of death, recurrence) are removed.

- TCGA, 18 cancer types: 7,534 patients with all three non-imaging layers.

- CT is the bottleneck: only 805 patients (10.7%), concentrated in KIRC, BRCA, BLCA, LIHC; CESC, OV, PRAD, SARC, UCEC have none.

- Splits: 5-fold CV stratified by cohort and CT availability, 80/10/10.

- ~60 s.

4 of 10

OmicsLaMed

4

Each stage is retrained independently within each of the 5 cross-validation folds.

(A) Omics encoder pretraining

(B) MLLM fine-tuning

DNA methylation

RNA-seq

CoxAE

Encoder

Decoder

z

4096-dim

❄ Freeze omics encoder

Connector

Omics token

Clinical records

comorbidities, treatment

CT volumetric scans

M3D-LaMed

+ LoRA fine-tuning

Prompt

Given the patient's clinical profile, omics data, and imaging:

Q1. Vital status? A) Alive B) Deceased

Q2. Estimated survival? A) <6 mo ... E) >5 y

Q3. Cancer stage? A) I ... D) IV

Answer:

Training target: "B, D, C"

Answer

"B, D, C"

Last hidden state

Embedding

clustering

5 of 10

Omics encoder

5

Encoder: 3-layer MLP in → H → H/2 → z (LayerNorm, ReLU); symmetric decoder

Bottleneck d = 4096, matching the LLaMA-2 hidden size

Three linear heads on z so that no single endpoint dominates the latent space

Multi-task objective

‖x − x̂‖²

reconstruction (MSE)

+ L_Cox

Breslow partial likelihood (survival)

+ L_BCE

vital status

+ L_CE

AJCC stage I–IV

6 of 10

Omics connector

6

Padding

z zero-padded to H, 1 token, 0 parameters. With d = 4096 it is an identity.

Q-Former

BLIP-2 style, 2 learnable queries, cross-attention, ~1.1M parameters

MLP

2-layer projector to 8 tokens, ~16M parameters

7 of 10

Evaluation paradigms

7

A

Generative

The fine-tuned model generates the answer string; each letter is compared with the ground-truth class.

Multi-task multiple-choice prompt

Q1 Vital status: Alive / Deceased

Q2 Survival: <6 mo · 6–12 mo · 1–2 y · 2–5 y · >5 y

Q3 AJCC stage: I · II · III · IV

Target "B, D, C"

B

Embedding clustering

Same fine-tuned model, answer withheld; each cluster takes its majority class as predicted label.

Prompt prefix

(no answer)

Mean-pooled last

hidden state

PCA-50 +

HDBSCAN

Patient embedding → unsupervised clusters

Cluster evaluated with majority label and compared with ground truth

Evaluation: balanced accuracy per task (vital status, survival bin, stage) and mean across tasks; mean ± 95% CI (t-distribution) over 5 folds

- Same fine-tuned model, two complementary evaluation paradigms.

- Left, generative: all three tasks are framed as one multiple-choice prompt; the model outputs one letter per task, e.g. "B, D, C" = deceased, 2-5 years, stage III. Accuracy of the generated letters against ground truth.

- Right, embedding clustering: the answer is withheld; the mean-pooled last hidden state of the prompt prefix is the patient embedding, reduced to 50 PCA components and clustered with HDBSCAN. Each cluster is labelled with its majority class. This tests whether the unsupervised structure of the representation aligns with clinical outcomes.

- Metric: balanced accuracy, because survival and stage distributions in TCGA are skewed; reported per task and as the mean of the three, with mean and 95% CI (t-distribution) over the 5 folds.

- ~60 s.

8 of 10

OmicsLaMed outperforms all baselines

8

Method	Modalities	Paradigm	Vital status	Survival	Stage	Mean BAcc
M3D-LaMed zero-shot	CT + Text	Clustering	70.9 ± 3.2	33.0 ± 4.1	40.5 ± 3.8	48.1 ± 2.6
Unimodal (BioGPT)	Text	Clustering	77.3 ± 1.6	41.2 ± 2.7	24.9 ± 2.5	47.8 ± 3.2
Unimodal (MedVAE)	CT	Clustering	78.6 ± 5.3	43.8 ± 6.2	19.0 ± 7.5	47.1 ± 2.6
M3D-LaMed zero-shot	CT + Text	Generative	0.0 ± 0.0	0.0 ± 0.0	23.9 ± 3.4	7.9 ± 3.4
M3D-LaMed fine-tuned	CT + Text	Generative	50.0 ± 4.4	19.8 ± 3.3	26.5 ± 3.6	32.1 ± 2.7
M3D-LaMed fine-tuned	CT + Text	Clustering	74.1 ± 1.2	35.0 ± 2.1	37.5 ± 3.8	48.8 ± 2.6
Late fusion (BioGPT, CoxAE)	Text + Omics	Clustering	76.1 ± 6.3	36.9 ± 2.4	52.9 ± 12.0	55.3 ± 6.9
Unimodal (CoxAE)	Omics	Clustering	80.7 ± 11.7	43.6 ± 8.9	51.0 ± 7.9	58.4 ± 3.1
Naive fusion (MedVAE, BioGPT, CoxAE)	CT + Text + Omics	Clustering	82.1 ± 2.7	50.0 ± 3.7	43.1 ± 3.9	58.4 ± 2.4
OmicsLaMed	CT + Text + Omics	Clustering	83.2 ± 2.6	45.8 ± 3.9	56.6 ± 3.3	61.9 ± 2.3
OmicsLaMed	CT + Text + Omics	Generative	85.9 ± 2.4	68.6 ± 3.4	36.3 ± 4.2	63.6 ± 2.2

Balanced accuracy (%), mean ± 95% CI over 5 folds. Bold: best per column.

9 of 10

Ablation: every modality contributes

9

Modalities	Paradigm	Vital status	Survival bin	Stage	Mean BAcc
CT + Text + Omics	Generative	59.3 ± 11.2	32.1 ± 5.9	44.2 ± 9.9	45.2 ± 2.8
CT + Text + Omics	Clustering	50.0 ± 0.0	33.4 ± 6.2	39.8 ± 6.2	41.1 ± 2.8
Omics	Generative	56.6 ± 21.9	12.6 ± 9.4	14.6 ± 19.9	27.9 ± 7.7
Omics	Clustering	53.3 ± 9.2	25.8 ± 6.2	37.8 ± 4.0	39.0 ± 2.7
Text	Generative	49.5 ± 1.0	21.0 ± 1.2	26.2 ± 3.3	32.2 ± 0.6
Text	Clustering	57.7 ± 3.2	25.5 ± 4.5	37.6 ± 5.8	40.3 ± 3.2
CT	Generative	50.8 ± 2.3	19.8 ± 1.4	25.1 ± 2.2	31.9 ± 1.1
CT	Clustering	50.0 ± 0.0	26.8 ± 7.6	37.0 ± 4.5	38.0 ± 3.9

Balanced accuracy (%), mean ± 95% CI over 5 folds, 805 CT-available patients.

Same 805 patients for all variants
Full model: best mean BAcc in both paradigms

10 of 10

Conclusions

Omics carry the dominant prognostic signal (CoxAE 58.4% vs CT 47.1%, text 47.8%)
Injecting a frozen omics embedding into M3D-LaMed via a parameter-free connector beats post-hoc fusion in both paradigms
Adding omics improves over both single-modality variants (ablation) and the CT + text fine-tuned M3D-LaMed (63.6% vs 32.1% generative mean BAcc)
Limitations: scarce CT (10.7%), ablation on 805 patients, connector tied to LLaMA-2 hidden size
Next: cross-cohort generalisation, more omics layers (CNV, mutations, proteomics)