1 of 38

Enhancing Medical Image Classification with Multi-Modal Inputs

Alain Fidahoussen

Danil Garmaev

2 of 38

2

Problem Statement and Approach

1

Results

2

Challenges and Next steps

3

Medical imaging plays a critical role in diagnosis, but traditional methods often don’t make full use of all the available information.

First of all, single-view X-ray models miss complementary information. For example, a frontal view might show an enlarged heart, but the lateral view confirms it by revealing the heart’s depth. Similarly, pleural effusion (fluid in the lungs) is often better visualized in the lateral view. For most of diseases it’s better to have both views to make a confident decision.

Second of all, textual data provides context. Radiology reports often describe findings like “enlarged heart shadow” or “fluid in the lungs,” which can help the model correlate visual patterns with clinical descriptions. By combining both views and text, our approach captures more comprehensive information, leading to better and more accurate diagnoses.

Our goal is to improve diagnostic accuracy in medical imaging by combining multi-modal inputs, frontal and lateral X-rays alongside textual data, using Visual Language Models.

3 of 38

Problem Statement

3

Motivation :

Single-view X-ray models miss complementary information.
Textual data (e.g., radiology reports) can provide context for better diagnosis.

Goal : Improve diagnostic accuracy in medical imaging by combining multi-modal inputs (frontal + lateral X-rays + text) using Visual Language Models.

Frontal and lateral X-rays of a patient showing cardiomegaly (enlarged heart size)

Frontal and lateral X-rays of a patient showing pleural effusion (fluid is present in the space around the lung)

Medical imaging plays a critical role in diagnosis, but traditional methods often don’t make full use of all the available information.

First of all, single-view X-ray models miss complementary information. For example, a frontal view might show an enlarged heart, but the lateral view confirms it by revealing the heart’s depth. Similarly, pleural effusion (fluid in the lungs) is often better visualized in the lateral view. For most of diseases it’s better to have both views to make a confident decision.

Second of all, textual data provides context. Radiology reports often describe findings like “enlarged heart shadow” or “fluid in the lungs,” which can help the model correlate visual patterns with clinical descriptions. By combining both views and text, our approach captures more comprehensive information, leading to better and more accurate diagnoses.

Our goal is to improve diagnostic accuracy in medical imaging by combining multi-modal inputs, frontal and lateral X-rays alongside textual data, using Visual Language Models.

4 of 38

Problem Statement

4

Cardiomegaly : heart looks larger than normal

Edema : extra fluid in the lungs

Consolidation : region of the lung filled with fluid instead of air

Atelectasis : part of the lung appears collapsed

Pleural Effusion : fluid is present in the space around the lung.

5 of 38

Related work

5

Model : CNN (DenseNet-121)

Inputs : frontal and lateral views

Outputs : 14 conditions (multi-label)

Dataset : MIMIC-CXR

~340K frontal images

~116K lateral images

The architecture of DualNet CNN architecture accepts a pair of frontal (PA or AP) and lateral input images.

6 of 38

Related work

6

Improvement for 10 of the 14 classes.

Metric : Area Under the Curve

7 of 38

Related work

7

Model : CLIP (Image encoder : ViT-B/16 - Text encoder : PubMedBERT)

Dataset : PMC-15M

~15M high-quality images-text pairs from 4.4M scientific papers

2 orders of magnitude larger than MIMIC-CXR

Another key piece of related work is BioMedCLIP, a vision-language model designed for medical imaging. BioMedCLIP uses a Vision Transformer (ViT-B/16) for images and PubMedBERT for text, and it is trained on PubMed Central Dataset, which contains ~15M image-text pairs from biological scientific papers.

BioMedCLIP excels at multi-modal understanding, correlating medical images with textual descriptions, and it even supports zero-shot classification for rare or unseen conditions.

This is highly relevant to our project because:

Textual Context: Unlike DualNet, which only uses images, BioMedCLIP can incorporate radiology reports to provide additional context for diagnosis.
Scalability: Trained on a much larger dataset, BioMedCLIP has learned richer representations of medical images and text.
Flexibility: We can fine-tune it for multi-label classification or use it in zero-shot mode for exploratory analysis.

8 of 38

Related work

8

Model : LLaVa (Mistral-7B-Instruct)

Dataset :

PMC-15M (Stage 1)

GPT-4 Instructions (Stage 2)

Finally, we have LLaVA-Med, a multi-modal instruction-tuned model designed for medical imaging and text. It builds on the LLaVA framework and uses Mistral-7B-Instruct as its backbone, making it highly capable of understanding and generating medical insights.

LLaVA-Med is also pre-trained on PMC-15M (15M image-text pairs) and fine-tuned on GPT-4-generated instructions for medical generation tasks.

This gives it two key strengths:

Instruction-Tuning: It can follow complex medical instructions, making it ideal for tasks like generating radiology reports or answering clinical questions.
Multi-Modal Understanding: It excels at combining visual and textual data, similar to BioMedCLIP, but with enhanced reasoning capabilities.

PubMed Central (PMC) is a publicly available, digital archive of full text biomedical and life sciences journal literature.

This is relevant to our project because:

Advanced Reasoning: LLaVA-Med can perform complex reasoning on medical images and text, improving diagnostic accuracy.
Generative Capabilities: Unlike BioMedCLIP, it can generate detailed descriptions of X-rays, providing additional context for clinicians.
Flexibility: It can be used for both classification and generative tasks, making it a versatile addition to our pipeline.

9 of 38

Approach

9

Hypothesis : Combining frontal and lateral X-rays with textual data improve diagnostic accuracy compared to using single-view X-rays.

Dataset : CheXpert dataset with frontal and lateral X-rays on 14 different conditions.

Baseline : Dual CNN trained on CheXpert.

Models :

BioMedCLIP
LLaVa-Med

Methods :

Zero-shot classification
Fine-tuning
Open-ended questions

Our approach aims to verify the benefits of combining frontal and lateral X-rays with textual data using the model.

We use publicly available dataset CheXpert with frontal and lateral X-rays on 14 different conditions to evaluate the performance of the models.

Baseline: We compare our approach to DualNet, a convolutional neural network that uses frontal and lateral X-rays but does not deal with textual data.

Methods:

Zero-shot Classification: Test the models’ ability to classify conditions without fine-tuning.
Fine-Tuning: Adapt BioMedCLIP and LLaVA-Med for multi-label classification.
Open-Ended Questions: Use LLaVA-Med to generate detailed descriptions of X-rays.

We fuse the image representations (frontal + lateral) and text representations using techniques like concatenation or summation.
This multi-modal fusion allows the model to leverage complementary information for better diagnosis.

10 of 38

Approach

10

BioMedCLIP

Freeze the CLIP parameters
Fine-tuning the image/text encoder
Add a multi-label classification head.

LLaVA-Med

Adapt the model to accept dual-view images
Fine-tune the model on CheXpert dataset.
And at first we evaluate it on zero-shot classification using closed questions. Further on, we will attempt to expand the model to answer open-ended questions.

Evaluation process : for each model train and evaluate

On frontal X-rays
On both frontal and lateral X-rays

For BioMedCLIP, we try different combinations: either we freeze the CLIP parameters, or we fine-tuning the image/text encoder and add a multi-label classification head. And we compare the performance for each combination.

Freeze CLIP Parameters: Add a multi-label classification head on top of the frozen CLIP encoder.
Fine-Tune Encoder: Adapt the image and text encoders to our dataset.
Compare Performance: Evaluate which strategy works best for our task.

For LLaVA-Med, we modify the architecture of the model to accept dual-view images. First, we evaluate its zero-shot classification capabilities using closed questions. Later, we will fine-tune it on CheXpert and explore its generative capabilities, we would need to generate a Question - Answer dataset to train LLaVA-Med and make it answer open-ended questions.

For evaluation, we first train and evaluate on frontal X-rays only to establish a baseline. Then, we will proceed with both frontal and lateral X-rays to measure the impact of multi-view inputs. This step-by-step approach helps us understand how each component contributes to the model’s performance.

11 of 38

11

Problem Statement and Approach

1

Results

2

Challenges and Next steps

3

Medical imaging plays a critical role in diagnosis, but traditional methods often don’t make full use of all the available information.

First of all, single-view X-ray models miss complementary information. For example, a frontal view might show an enlarged heart, but the lateral view confirms it by revealing the heart’s depth. Similarly, pleural effusion (fluid in the lungs) is often better visualized in the lateral view. For most of diseases it’s better to have both views to make a confident decision.

Second of all, textual data provides context. Radiology reports often describe findings like “enlarged heart shadow” or “fluid in the lungs,” which can help the model correlate visual patterns with clinical descriptions. By combining both views and text, our approach captures more comprehensive information, leading to better and more accurate diagnoses.

Our goal is to improve diagnostic accuracy in medical imaging by combining multi-modal inputs, frontal and lateral X-rays alongside textual data, using Visual Language Models.

12 of 38

12

Problem Statement and Approach

1

Results

2

Challenges and Next steps

3

Medical imaging plays a critical role in diagnosis, but traditional methods often don’t make full use of all the available information.

First of all, single-view X-ray models miss complementary information. For example, a frontal view might show an enlarged heart, but the lateral view confirms it by revealing the heart’s depth. Similarly, pleural effusion (fluid in the lungs) is often better visualized in the lateral view. For most of diseases it’s better to have both views to make a confident decision.

Second of all, textual data provides context. Radiology reports often describe findings like “enlarged heart shadow” or “fluid in the lungs,” which can help the model correlate visual patterns with clinical descriptions. By combining both views and text, our approach captures more comprehensive information, leading to better and more accurate diagnoses.

Our goal is to improve diagnostic accuracy in medical imaging by combining multi-modal inputs, frontal and lateral X-rays alongside textual data, using Visual Language Models.

13 of 38

Challenges

13

LLaVa-Med

Poor zero-shot results

Difficulty to find a good prompt to generate the Q/A from the report

Difficulty to find a good model to generate the Q/A from the report

LLAMA 3.2 ⇒ Local, fast, but not always accurate
Deepseek R1 ⇒ Local, but slow inference
GPT ⇒ not free

Fine-tuning

Computationally expensive (~30h / epoch - LORA)
Unbalanced dataset

14 of 38

Next steps

14

BiomedCLIP ✅

Frontal and lateral > Frontal
CLIP-based > CNN

LLaVa-Med ❌

Generate better Q/A
Weighted-loss
Under/Over sampling methods
Text-based augmentation

15 of 38

Experimental setup

15

Dataset

CheXpert

Focus on the 5 most important clinical conditions

16 of 38

Experimental setup

16

Dataset

CheXpert

Focus on the 5 most important clinical conditions

Filter patients with both frontal and lateral views

Training : 25188 patients
Validation : 6297 patients
Test : 143 patients

17 of 38

Experimental setup

17

Metric

Area Under the Curve

18 of 38

Experimental setup

18

Metric

Area Under the Curve

Not sensible to moderate imbalance

19 of 38

Experimental setup

19

Metric

Area Under the Curve

Not sensible to moderate imbalance

Random classifier (0.5) : TP = TN = FP = FN

Perfect classifier (1.0) : FP = FN = 0

20 of 38

Preliminary Results

20

BioMedCLIP

Zero-shot performance

21 of 38

Preliminary Results

21

BioMedCLIP

Zero-shot performance

For each condition, discriminate between having (+) and not having (-) the condition

(+) ⇒ "This chest X-ray suggests an overall enlargement of the cardiac silhouette, consistent with cardiomegaly.",

(-) ⇒ "This chest X-ray shows a normal-sized cardiac silhouette, with no indication of cardiomegaly."

22 of 38

Preliminary Results

22

BioMedCLIP

Zero-shot performance

For each condition, discriminate between having (+) and not having (-) the condition

(+) ⇒ "This chest X-ray suggests an overall enlargement of the cardiac silhouette, consistent with cardiomegaly.",

(-) ⇒ "This chest X-ray shows a normal-sized cardiac silhouette, with no indication of cardiomegaly."

Compute two cosine-similarity scores between the visual features and each of the (+) and (-) textual features, and choose the maximum one.

23 of 38

Preliminary Results

23

BioMedCLIP

Zero-shot performance

For each condition, discriminate between having (+) and not having (-) the condition

(+) ⇒ "This chest X-ray suggests an overall enlargement of the cardiac silhouette, consistent with cardiomegaly.",

(-) ⇒ "This chest X-ray shows a normal-sized cardiac silhouette, with no indication of cardiomegaly."

Compute two cosine-similarity scores between the visual features and each of the (+) and (-) textual features, and choose the maximum one

Frontal and Lateral visual features are averaging

24 of 38

Preliminary Results

24

BioMedCLIP

Zero-shot performance (AUC)

Not great (0.5 = random classifier)

Slight improvement using both Frontal and Lateral views

25 of 38

Preliminary Results

25

BioMedCLIP

Fine-tune a multi-label classification head

26 of 38

Preliminary Results

26

BioMedCLIP

Fine-tune a multi-label classification head

CLIP frozen

27 of 38

Preliminary Results

27

BioMedCLIP

Fine-tune a multi-label classification head

CLIP frozen

Frontal and Lateral visual features are averaging

28 of 38

Preliminary Results

28

BioMedCLIP

Fine-tune a multi-label classification head

CLIP frozen

Frontal and Lateral visual features are averaging

Use concatenation between visual and textual features

29 of 38

Preliminary Results

29

BioMedCLIP

Fine-tune a multi-label classification head

CLIP frozen

Frontal and Lateral visual features are averaging

Use concatenation between visual and textual features

For each (+) condition, predict a score between 0 and 1 (sigmoid)

30 of 38

Preliminary Results

30

BioMedCLIP

Fine-tune a multi-label classification head (AUC)

Significant improvement from Zero-Shot (~20%)

31 of 38

Preliminary Results

31

BioMedCLIP

Fine-tune a multi-label classification head (AUC)

Significant improvement from Zero-Shot (~20%)

Slight improvement using both Frontal and Lateral views

32 of 38

Preliminary Results

32

BioMedCLIP

Fine-tune CLIP and a multi-label classification head

33 of 38

Preliminary Results

33

BioMedCLIP

Fine-tune CLIP and a multi-label classification head

CLIP is fine-tuned for a few epochs

34 of 38

Preliminary Results

34

BioMedCLIP

Fine-tune CLIP and the multi-label classification head

Significant improvement from frozen CLIP (~10%)

35 of 38

Preliminary Results

35

BioMedCLIP

Fine-tune CLIP and the multi-label classification head

Significant improvement from frozen CLIP (~10%)

Slight improvement using both Frontal and Lateral views

36 of 38

Timeline

36

Week 1	Feb. 17 - 23	Literature review
Week 2	Feb. 24 - March 2	Data preparation Model selection
Week 3	March 3 - 9	Baselines BioMedCLIP
Week 4	March 10 -16	Baselines BioMedCLIP
Week 5	March 17 - 23	LLaVa-Med
Week 6	March 24 - 30	LLaVa-Med
Week 7	March 31 - April 6	Results Analysis
Week 8	April 7 - 15	Finalization and poster presentation

37 of 38

Q & A

37

Adapt LLM: Visual Instruction Synthesizer which takes image as the input, so we would generate a instruction and 2 responses: precise and informative.

Why LLM-based over non-LLM based methods?

Combining Multi-View X-Rays and Text: BioMedCLIP and LLaVA-Med excel at integrating multi-modal inputs, making them ideal for combining frontal and lateral X-rays with textual data.
Improved Diagnostic Accuracy: Their ability to correlate visual patterns with clinical descriptions leads to more accurate and confident diagnoses.
Generative and Reasoning Capabilities: LLaVA-Med can generate detailed descriptions of X-rays and answer clinical questions, adding value to your project.
Flexibility and Scalability: These models can generalize to new tasks and conditions, making them suitable for real-world clinical applications.

38 of 38

MIMIC-CXR

38