1 of 16

STRAGMED: Standardizing Retrieval-Augmented Generation Pipelines in Medical Domains

Leonardo Sanna, Esin Ezgi Yildiz and Mauro Dragoni

2 of 16

RAG today

3 of 16

STRAGMED

Finding a direction

4 of 16

4

Research Questions

RQ1

RQ2

RQ3

What is the optimal number of reranking steps in a RAG pipeline?

How should the RRF score be calculated in medical RAG systems?

Can applying the same pipeline lead to consistent and significant improvements across datasets?

5 of 16

5

Our directions: engineering solutions

​

​

6 of 16

6

Engineering solutions

Query Augmentation

Using an LLM to revise the base query and generate a new version of it

7 of 16

7

Engineering solutions

Query Augmentation

Hybrid Retrieval

Combining vector search with a lexical component

8 of 16

8

Engineering solutions

Query Augmentation

Hybrid Retrieval

Rerank-fuse

Use a re-reanker at each step of the pipeline and fuse different retrievers together

9 of 16

9

STRAGMED pipeline

10 of 16

Experiments

11 of 16

11

Datasets

PubMed

BioASQ

ChatFAQ

1,000 expert-annotated questions derived from PubMed article titles and abstracts

618 researcher-posed biomedical Yes/No questions with gold-standard PubMed articles

1,000 Italian questions and 1,512 verified documents on maternity and newborn care

12 of 16

RQ1: Best reranking at each step

Finding a direction

13 of 16

RQ2: Fusion strategy

Finding a direction

14 of 16

RQ2: K adds noise to retrieval!

15 of 16

15

RQ3: STRAGMED results (Hit@3)

16 of 16

Thank you