1 of 12

GPT-4V cannot generate radiology reports yet

Yuyang Jiang¹, Chacha Chen¹, Dang Nyugen¹�Benjamin Mervak², Chenhao Tan¹

¹University of Chicago, ²University of Michigan

2 of 12

3 of 12

❓

4 of 12

RQ1: Direct Report Generation

5 of 12

Why Failure?

Image task

Textual task

Very naturally, this pushes us to think more deeply: why does the most advanced vision-language model in the world still fail at direct report generation? What are the underlying reasons behind this failure? And most importantly, what should we be mindful of when integrating vision-language models into real-world radiology workflows?

To answer these questions, we proposed a decomposed evaluation framework. Given chest X-rays as input and generated reports as output, we break down the model’s generation process into a two-stage, radiologist-like workflow: image reasoning followed by report synthesis.

In the first stage, the model interprets the chest X-ray and extracts key clinical observations. We formulate this as an image task – in other words, we want to understand whether the model can accurately identify clinical conditions based on visual input.

In the second stage, after gathering these image observations, the model will organize and synthesize them into a coherent report. This becomes a textual generation task, where we evaluate the model’s ability to construct a clinically usable narrative based on its prior visual understanding.

6 of 12

Why Failure?

7 of 12

RQ2: GPT-4V cannot interpret medical images meaningfully

8 of 12

9 of 12

RQ3: Report Synthesis Given Groundtruth Conditions

Significant improvements

Gap to ground-truth reports

10 of 12

Additional human evaluation by two radiologists

Human written report are 100% usable, whereas even with groundtruth labels, model generated reports are still not perfect.
Human written reports contains richer and more nuanced information.
Model generated reports have the potential to have better clarity/readability.

Gap to human written reports

11 of 12

Report generation =

Image reasoning

Chest X-rays

<LABEL>

(Cardiomegaly, 0),

(Lung Lesion, 1),

(Lung Opacity, 1),

……

FINDINGS: Hyperinflated with diffuse bilateral opacities. No pleural effusion or pneumothorax. No visible fractures or lytic lesions.

IMPRESSION: Suspected COPD with superimposed infection. No acute disease.

report synthesis

GPT-4V struggles significantly with interpreting chest X-rays meaningfully, which directly impacts its ability to generate reports.

Even when we bypass the bottleneck of image reasoning by providing groundtruth conditions, GPT-4V still underperforms a finetuned LLaMA-2 baseline and fails to match the preferences of radiologists.

Failed terribly

12 of 12

Ongoing: building a radiology foundation model with better data

Key idea: High quality medical data curation for post-training

Find Us at the Poster Session!

Or yuyang2001@uchicago.edu