1 of 22

MetaScreener

Between efficiency, recall, and transparency

An evaluation of guided literature screening using LLMs

MetaScreener

Jonas Weinert · Anja Sautmann · Adisiri Swain · Emanuel Herrera | Development Research Group, World Bank

Measuring Development Conference 2026

Database search

→

Title & abstract screening

→

Full-text review

→

Meta-analysis

2 of 22

Every systematic review follows this pipeline

Screening is where two risks collide

2

IDENTIFICATION

Database search + dedup

▼

SCREENING

Title & abstract screening + record exclusion

▼

ELIGIBILITY

Full-text retrieval & assessment

▼

INCLUDED

Final studies in review

← The bottleneck

RECALL FAILURE

Missing a relevant study invalidates the meta-analysis.

PRECISION FAILURE

Including irrelevant studies wastes full-text review time and incurs high labor cost.

+ fatigue, subjectivity, inconsistency under volume

The traditional fix:

Dual-reviewer screening

+ narrow search scope

But narrow scope sacrifices recall.

You can’t miss what you never saw.

How would you know

if you missed something?

Moher D, Liberati A, Tetzlaff J, et al. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med. 2009 Jul 21;6(7): e1000097.

Everyone in this room who has done a literature review, knows this pipeline. What I want to highlight is that the screening stage is where two risks collide. If you miss a relevant study, a recall failure, that can change the conclusions of your entire meta-analysis. If you include too many irrelevant studies, you waste weeks of full-text review time. And the real problem is that human reviewers making thousands of these binary decisions are subject to fatigue, subjective drift, and inconsistency. How do we currently deal with that?��The standard approach is to write precise search terms, keep the volume manageable, and use two independent reviewers to catch errors. That is good practice. But it has a blind spot: you are relying on the search strategy to deliver the right papers, and you have no way to verify whether it did. If the search was too narrow, dual screening just means two people agreeing to miss the same studies. And if you broaden the search to be safe, the volume becomes unmanageable. That is the trap: you can have manageable scope or high recall, but manual screening forces you to choose.

3 of 22

Our project: AI for frontline health workers in LMICs

A real systematic review, not a synthetic benchmark

5

TOPIC	AI-based digital health interventions for frontline healthcare workers in LMICs
SEARCH	~150,000 initial records across multiple databases
SCREENING	LLM-assisted (MetaScreener) + 3 trained RAs + PI · partial dual review + reconciliation
FINAL INCLUDED	18 studies (all criteria met + LMIC setting)

INCLUSION CRITERIA

IC1

Is is about an AI-based healthcare intervention?

IC2

Is it an AI tool for frontline health care providers?

IC3

Is it empirically tested with real providers/patients?

IC4

Does the study measure patient outcomes?

We are conducting a systematic review of AI tools designed to support frontline health workers, community health workers, nurses, primary care staff, in low- and middle-income countries. Think diagnostic decision support, triage algorithms, screening tools deployed in real clinical settings. We started with roughly 150,000 records from a broad database search. That is not a typo, and it is not because the search was sloppy. It is because the topic sits at the intersection of AI, digital health, primary care, and LMIC contexts, and the relevant literature is scattered across many fields and vocabularies. Manual-only screening at that scale would have taken months. So we used LLM-assisted screening and then evaluated whether we could trust it. Every number I show you today comes from this project.

4 of 22

We evaluated three questions, using three exercises

TL;DR: This research project

3

Evaluation dimension	Exercise	Question answered	Main caution
Recall before screening	Search Recall Exercise	Would a narrower PICOS search have found the final included studies?	One domain; retrospective comparison
Human oversight	Human Ground-Truthing Exercise	Does LLM disagreement help correct human screening errors?	Corrected truth, not independent truth
Replicability / consistency	Bootstrap Consistency Exercise	Where does the model change its mind under repeated runs?	Consistency is not correctness
Efficiency + transparency	Costed Review Protocol	How should teams actually run an AI-assisted review?	Needs local cost and staffing assumptions

Takeaway: the evidence is encouraging only when these exercises are read together.

5 of 22

Summary of findings

Encouraging, but not conclusive

4

> 60%

of final included studies

missed by narrower search

8 / 13 · Retrospective search comparison

100%

post-moderation LLM-Human agreement

across all 4 criteria

309-paper moderated batch

5.7%

criterion-level bootstrap

overturn rate

100 papers × 300 iterations

3.4%

of iterations changed

full-text inclusion decision

30,000 iteration-decisions

Moderation is not perfect truth. Consistency is not validity.

6 of 22

Search Recall Exercise: a narrower search missed > 60% of our final included studies

Recall can fail before screening starts

6

Path A: LLM screening approach

Allows broad search → ~150,000 records

→ LLM-assisted screening → full-text review

→ 13 final included studies

13 / 13 found

Path B: Narrower PICOS-aligned search

Scopus	3,823	WoS	3,906	PubMed	2,064

After dedup: 4,291 records, typical size for (human-screened) literature reviews

5 found

8 MISSING

> 60%

of final included studies

were NOT in the narrower search

8 / 13

At scale, manual screening is infeasible

~8,500 abstracts → 33 days, $3,068 (Chai et al., 2021)

Median review time > 67 weeks (Borah et al., 2017)

Our project: ~150,000 records

After completing our review, we ran a retrospective test. We asked: what if we had used a tighter, PICOS-aligned search instead? That narrower search returned about 4,300 unique records -> very manageable. But when we checked which of our 13 final included studies appeared in that set, only five were there. Eight were missing. More than 60% of the studies we ultimately needed for the meta-analysis would never have been screened, because they were never retrieved. This is the scope-recall trap in action. The narrower search looks efficient, but it silently destroys recall. And you would never know, because you cannot miss what you never saw.��These benchmarks are from the literature, not from our project. But they illustrate the scale of the problem. Chai et al. estimated that screening about 8,500 abstracts takes 33 working days and costs over $3,000, and that is a mid-size review. Our search returned 150,000 records. Even with aggressive deduplication, a single-pass manual screen would take a reviewer roughly 300 working days. Dual screening doubles that. And Wang et al. showed that human reviewers under this kind of load make measurable errors. The bottleneck is real, and it is not just about time -- it is about the reliability of the decisions made under that time pressure.

7 of 22

Human Groundtruthing Exercise: After moderation, recall reaches 100% across all criteria

The LLM caught what humans missed | N = 309 papers

7

Human only

After moderation

IC1

0.900

→

1.000

IC2

0.435

→

1.000

IC3

0.182

→

1.000

IC4

0.167

→

1.000

	Precision	F1
IC1	0.938	0.968
IC2	0.966	0.982
IC3	0.875	0.933
IC4	0.923	0.960

All 5 papers ultimately included in the review were correctly identified by the LLM.

8 of 22

The "gold standard" is noisy -> changes how we validate

Many apparent LLM false positives were human misses (2 human reviewers) discovered during moderation

8

Paper type	Human said	LLM said	After adjudication
AI diagnostic tool tested on real patients	Exclude (IC3)	Include	Include - humans had missed the field trial
mHealth intervention with provider outcomes	Exclude (IC2)	Include	Include - humans had misread the target user
Retrospective algorithm validation	Include (IC3)	Exclude	Exclude - LLM correctly flagged no prospective test

Error rates of human reviewers during abstract screening in systematic reviews

Wang et al. (2020):

"We analyzed a total of 139,467 citations that underwent 329,332 inclusion and exclusion decisions […]

After abstract screening, the total error rate (false inclusion and

false exclusion) was 10.76% (95% CI: 7.43% to 14.09%)."

9 of 22

Bootstrap Consistency Exercise: most instability is boundary instability

If human review is not a good benchmark, LLM review many times as the best (flawed) benchmark?

9

5.7%

criterion-level overturn rate

15.6%

paper-class overturn rate

3.4%

full-text inclusion flip rate

When full-text inclusion changes, how many criteria flipped?

1 criterion: 966 (95.9%)

2 criteria: 28 (2.8%) | 3 criteria: 13 (1.3%)

Operational meaning

Human review should focus on boundary papers and LLM disagreements, not all papers equally.

100 papers × 300 iterations; modal decision as pseudo-truth

10 of 22

Conclusion: LLM screening is useful when it improves the review process

The value is not automation alone; it is auditable, high-recall evidence synthesis

10

Privacy

Local-first workflow keeps bibliographies under researcher control. BUT model API calls leave the browser.

Useful for sensitive search strategies and unpublished review corpora.

Reproducibility / Transparency

Prompts, criteria, model settings, outputs, and adjudication decisions become part of the audit trail.

AI decisions can be inspected, challenged, and reported.

Cost

Batch screening changes the economics of broad retrieval and repeated checks.

Reviewer time can move from bulk filtering to boundary-case moderations.

Quality

Groundtruthing suggests strong recall; bootstrap shows uncertainty is concentrated near the boundary.

The best workflow is guided review, not black-box replacement.

Bottom line: use LLMs to make screening broader, more inspectable, and more targeted for human judgment.

11 of 22

Recommended protocol: calibrate before screening the full corpus

Run broad, bootstrap the criteria, revise if needed, then target human review

11

The bootstrap converts uncertainty into a review rule, a human-time budget, and a recall-risk estimate.

1. Broad search + dedup

Keep retrieval recall high.

→

2. Bootstrap sample

1% × 100 or 0.5% × 200 ≈ one full-run equivalent.

→

3. Diagnose ICs

Inspect instability and correlated criterion overturns.

→

4. Stable enough?

Can the ICs support a full run?

No: revise ICs

clarify wording, rerun bootstrap

↓

5. Decide review rule

Disagreement, boundary cases, or conservative ≤2 no's.

→

6. Full LLM run(s)

Run full deduplicated set 1–3 times.

→

7. Human re-review

Review selected boundary/disagreement papers.

→

8. Estimate recall risk

Extrapolate from bootstrap and export audit trail.

Yes: Proceed

12 of 22

Scenario planning: human review depends on ambiguity

Illustrative estimates for 20k papers, 4 criteria, $11 per 1k LLM paper screens

12

• Assume one LLM paper screen = all 4 ICs for one paper.

• One full run over 20k papers = 20 × $11 = $220.

• Bootstrap calibration ≈ one full-run equivalent = $220.

• Human re-review time shown at 5 min / paper.

• Traditional comparison: 2 humans screen all papers at 2 min / paper + 10% moderation at 5 min / paper ≈ 1,500 hours.

Ambiguity case	LLM plan	Human re-review rule	LLM price	Human hrs	Est. recall
Low ambiguity stable ICs	Bootstrap + 2 full runs ≈ 60k screens	Review LLM disagreements + low-confidence boundary no's (~5% = 1,000 papers)	$660	≈83h	99.5–99.8%
Moderate ambiguity some boundary drift	Bootstrap + 3 full runs ≈ 80k screens	Review all disagreements + 1_no / 2_no overturns (~10% = 2,000 papers)	$880	≈167h	99.0–99.5%
High ambiguity correlated IC overturns	Bootstrap + 3 full runs ≈ 80k screens	Review all ≤2 no papers where any criterion overturned (~20% = 4,000 papers)	$880	≈333h	98.5–99.2%
Conservative / policy review near-zero miss tolerance	Bootstrap + 3 full runs ≈ 80k screens	Review all papers with ≤2 no criteria or any LLM disagreement (~35% = 7,000 papers)	$880	≈583h	99.5%+
Traditional dual human comparison	No LLM	2 humans screen all 20k + 1 human moderates 10% disagreements	$0 LLM	≈1,500h	Unknown; depends on human miss rate

Recall ranges are planning estimates: replace with your bootstrap-derived overturn rates and chosen review rule.

13 of 22

Guided screening = criteria + auditable prompts + moderation

MetaScreener is the instrument, not the object of study

13

CSV upload

(title + abstract)

→

Criterion prompts + free text field extractions

→

Model decision

per criterion

(yes / no / maybe)

→

Moderation queue

human reviews

disagreements

→

Adjudicated

dataset +

audit trail

https://sautmann.github.io/MetaScreener/

14 of 22

What we do not yet know

• Search sensitivity can dominate screening performance -> needs replication.

• Domain-specific: AI-based digital health interventions only.

• Ground truth is corrected, not independent.

• Bootstrap = consistency, not correctness.

• IC1–IC3 only; full 4-criterion stability untested.

• Prompt sensitivity untested.

We need your data, your criteria, and your disagreements

1. TEST IT

Run MetaScreener on your own review dataset.

Compare against your human labels.

2. ADJUDICATE WITH US

Share disagreement cases.

Help build a cross-domain library of boundary decisions.

3. IMPROVE THE TOOL

MetaScreener is open source.

Contribute prompt modules, metrics, or reporting templates.

14

MetaScreener

15 of 22

Appendix A: Full Metric Tables

Human-only baseline vs. Post-moderation (N = 309)

Baseline: Human-Only Screening

Criteria	TP	TN	FP	FN	Accuracy	Precision	Recall	F1
IC4	4 (1.3%)	271 (89.1%)	9 (3.0%)	20 (6.6%)	0.905	0.308	0.167	0.216
IC3	4 (1.3%)	278 (91.4%)	4 (1.3%)	18 (5.9%)	0.928	0.500	0.182	0.267
IC2	10 (3.3%)	262 (86.2%)	19 (6.3%)	13 (4.3%)	0.895	0.345	0.435	0.385
IC1	27 (8.9%)	236 (77.6%)	38 (12.5%)	3 (1.0%)	0.865	0.415	0.900	0.568

Post-Moderation LLM-Assisted Screening

Criteria	TP	TN	FP	FN	Accuracy	Precision	Recall	F1
IC4	12 (3.9%)	291 (95.7%)	1 (0.3%)	0 (0.0%)	0.997	0.923	1.000	0.960
IC3	7 (2.3%)	296 (97.4%)	1 (0.3%)	0 (0.0%)	0.997	0.875	1.000	0.933
IC2	28 (9.2%)	275 (90.5%)	1 (0.3%)	0 (0.0%)	0.997	0.966	1.000	0.982
IC1	61 (20.1%)	239 (78.6%)	4 (1.3%)	0 (0.0%)	0.987	0.938	1.000	0.968

16 of 22

Appendix B: Bootstrap Technical Details

Methodology for the consistency analysis

• Sample: 100 papers drawn from the screening corpus.

• Iterations: Each paper was screened 300 times using the same prompt, model (GPT-5), and parameters.

• Total decisions: 30,000 iteration-decisions analyzed.

• Criteria evaluated: IC1, IC2, and IC3.

• Binary rule: The model outputs 'yes', 'no', or 'maybe'. For this analysis, 'yes' and 'maybe' were collapsed into 'yes' (inclusion).

• Pseudo-truth baseline: For each paper-criterion combination, the modal decision across the 300 iterations was defined as the baseline.

• Modal paper classes: Based on the modal criterion decisions, papers were classified by how many criteria they failed: 3_yes (31 papers), 1_no (32 papers), 2_no (29 papers), 3_no (8 papers).

17 of 22

Appendix C: Human-only screening looks accurate - until you check criterion by criterion

Overall accuracy = 89.8% driven by true negatives in a class-imbalanced dataset

9

IC1 AI intervention?

0.900

IC2 Health providers?

0.435

IC3 Empirically tested?

0.182

IC4 Patient outcomes?

0.167

Once humans decide a paper is irrelevant, they stop evaluating subsequent criteria.

IC3 and IC4 recall below 0.2 = reviewers missed > 80% of eligible studies on those criteria.

18 of 22

Appendix D: If you ask the model 300 times, how often does it change its mind?

Accuracy is necessary but not sufficient. We also need consistency

12

	IC1	IC2	IC3	→	Decision
Run 1	yes	yes	yes	→	include
Run 2	yes	no	yes	→	exclude
Run 3	yes	yes	yes	→	include
...				→
Run 300	yes	yes	no	→	exclude

Modal decision: IC1=yes + IC2=yes + IC3=yes → include

The Bootstrap Test

100 papers

× 300 iterations each

= 30,000 decisions

The mode across 300 runs

becomes the pseudo-truth

baseline.

We ask: how often do

individual runs disagree?

This measures consistency, not correctness.

19 of 22

"Does the model classify well?" is an incomplete question

We need to evaluate across four dimensions

5

Efficiency

Does it reduce cost

and time?

Recall

Does it avoid

missing eligible studies?

Transparency

Can every decision be

inspected and reproduced?

Human Oversight

Where should humans

still review and adjudicate?

20 of 22

What these results tell us — and what they do not

15

ADJUDICATION (Exercise 1)

Question:

Does the model find what it should find?

Answer:

Yes: recall = 100% after moderation,

across all four criteria

Caveat:

Ground truth is corrected, not independent

BOOTSTRAP (Exercise 2)

Question:

Does the model give the same answer

when repeated?

Answer:

Mostly: 96.6% of runs preserve

the inclusion decision; nearly all flips

are single-criterion boundary cases

Caveat:

Consistency ≠ correctness

Combined: Guided LLM screening can support a high-recall, reasonably stable workflow IF paired with structured adjudication and boundary-focused review. Encouraging, but not conclusive.

21 of 22

Two exercises from the same project

7

SLR/MA on AI for frontline health workers in LMICs · ~150,000 records · 13 final included

Exercise 1: Adjudicated Ground Truth

309 papers from the screening corpus
4 criteria (IC1–IC4)
3 RAs + PI dual review + reconciliation
GPT-5 via MetaScreener
Human–LLM disagreements adjudicated
Measures: recall, precision, F1 per criterion

Does it find what it should find?

Exercise 2: Bootstrap Stability Test

100 papers from the screening corpus
300 iterations per paper
IC1–IC3 (yes + maybe → yes)
Modal decision = pseudo-truth baseline
Measures: overturn rates, inclusion flips

Does it give the same answer when repeated?

22 of 22

94% of criterion decisions are stable, but some criteria are harder

Instability clusters in conceptually ambiguous criteria

13

mean = 0.917

mean = 0.976

mean = 0.935

13 papers < 0.8

IC1

AI intervention?

IC2

Health providers?

20 papers < 0.8

IC3

Empirically tested?

4 papers < 0.8

94.3% of the time, an individual run agrees with the modal baseline. Overturn rate = 5.7%