1 of 143

Evaluating Search for οΏ½Systematic Review Creation

Allan Hanbury

​

Work with

Wojciech Kusa

1

2 of 143

Agenda

  • About me
  • Introduction to systematic reviews and citation screening
  • Evaluation metrics
  • Outcome-based evaluation
  • Q&A

​

2

3 of 143

About me

3

4 of 143

4

5 of 143

Medical Projects

2010-2014

​

​

2015-2017

​

​

5

6 of 143

7 of 143

8 of 143

Medical Projects

2010-2014

​

​

2015-2017

​

​

8

9 of 143

9

10 of 143

Medical Projects

2010-2014

​

​

2015-2017

​

​

10

​

2012-2015

11 of 143

12 of 143

12

13 of 143

Maier-Hain et al., Why rankings of biomedical image analysis competitions should be interpreted with care, Nature Communications, volume 9, Article number: 5217 (2018)

Annotator 1

Annotator 2

Rank1

Rank2

Rank3

Rank4

Rank5

Rank5

Rank7

Rank8

Rank9

Rank10

A6

A4

A1

A5

A10

A7

A9

A2

A8

A3

A8

A7

A2

A1

A6

A5

A4

A9

A10

A3

A6

A4

A1

A5

A10

A7

A9

A2

A8

A3

14 of 143

Maier-Hain et al., Why rankings of biomedical image analysis competitions should be interpreted with care, Nature Communications, volume 9, Article number: 5217 (2018)

A1

A2

A3

A4

A5

A6

A7

A8

A9

A10

A1

A2

A3

A4

A5

A6

A7

A8

A9

A10

A11

A12

A13

Ranking with HD

Ranking with HD95

Aggregation with mean

Aggregation with median

A5

A2

A3

A4

A1

A8

A7

A6

A9

A10

A11

A12

A13

A1

A2

A3

A4

A5

A6

A7

A8

A9

A10

A11

A12

A13

A10

A1

A2

A3

A5

A7

A4

A9

A6

A8

A1

A2

A3

A4

A5

A6

A7

A8

A9

A10

15 of 143

15

2019-2024

16 of 143

Professional Search

16

Professional search focuses on the work of paid professionals who are undertaking a work task that is predominately search-related and performed under a number of constraints such as budget and time

β€œ

”

Tony Russell-Rose, 2018

17 of 143

Systematic reviews

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

18 of 143

Evidence-based medicine

18

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

19 of 143

Evidence-based medicine

  • Evidence based medicine is the use of current best evidence in making decisions about the care of individual patients

​

  • Integrating individual clinical expertise with the best available external clinical evidence from systematic research

19

source: midway.edu

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

20 of 143

Randomised Controlled Trial (RCT)

https://teaching.blog.gov.uk/2016/04/12/designing-randomised-controlled-trials/

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

21 of 143

Evidence-based medicine

  • Evidence based medicine is the use of current best evidence in making decisions about the care of individual patients

​

  • Integrating individual clinical expertise with the best available external clinical evidence from systematic research

21

source: midway.edu

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

22 of 143

Systematic Reviews

22

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

23 of 143

Systematic Reviews

  • A systematic review attempts to collect and analyse all evidence that answers a specific question in transparent, systematic, and reproducible way
  • Recall oriented study
  • Consist of multiple stages

23

source: cochrane.org

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

24 of 143

Systematic Reviews

24

  • Very resource-intensive process – relies mostly on human labour
  • Full Systematic Review takes on average 67 weeks
  • Subject to human error

source: cochrane.org

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

25 of 143

Systematic Reviews

25

  • Very resource-intensive process – relies mostly on human labour
  • Full Systematic Review takes on average 67 weeks
  • Subject to human error

​

  • Exponentially growing number of publications

source: cochrane.org

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

26 of 143

Systematic Reviews

26

  • Very resource-intensive process – relies mostly on human labour
  • Full Systematic Review takes on average 67 weeks
  • Subject to human error

​

  • Exponentially growing number of publications
  • Top results are not sufficient, it is necessary to find all relevant items

source: cochrane.org

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

27 of 143

Systematic Reviews

27

  • Very resource-intensive process – relies mostly on human labour
  • Full Systematic Review takes on average 67 weeks
  • Subject to human error

​

  • Exponentially growing number of publications
  • Top results are not sufficient, it is necessary to find all relevant items
  • Sometimes decision needs to be made quickly (e.g. COVID-19)

source: cochrane.org

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

28 of 143

Systematic Reviews

28

  • Very resource-intensive process – relies mostly on human labour
  • Full Systematic Review takes on average 67 weeks
  • Subject to human error

​

  • Exponentially growing number of publications
  • Top results are not sufficient, it is necessary to find all relevant items
  • Sometimes decision needs to be made quickly (e.g. COVID-19)
  • Automation can help to overcome these challenges and improve the efficiency and accuracy

source: cochrane.org

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

29 of 143

Systematic Review example

29

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

30 of 143

Systematic Review example

30

Based on a review by Cochrane β€” a leading organisation focusing on SRs in medicine

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

31 of 143

Systematic Review example

Duehmke, R. M., Derry, S., Wiffen, P. J., Bell, R. F., Aldington, D., & Moore, R. A. (2017). Tramadol for neuropathic pain in adults.Β  Cochrane Database of Systematic Reviews, (6). https://doi.org/10.1002/14651858.CD003726.pub4

31

  • SR Protocol

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

32 of 143

Systematic Review example

Duehmke, R. M., Derry, S., Wiffen, P. J., Bell, R. F., Aldington, D., & Moore, R. A. (2017). Tramadol for neuropathic pain in adults.Β  Cochrane Database of Systematic Reviews, (6). https://doi.org/10.1002/14651858.CD003726.pub4

32

  • SR Protocol
  • PICO terms

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

33 of 143

Systematic Review example

Duehmke, R. M., Derry, S., Wiffen, P. J., Bell, R. F., Aldington, D., & Moore, R. A. (2017). Tramadol for neuropathic pain in adults.Β  Cochrane Database of Systematic Reviews, (6). https://doi.org/10.1002/14651858.CD003726.pub4

33

  • SR Protocol
  • PICO terms
  • Search query

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

34 of 143

Systematic Review example

Duehmke, R. M., Derry, S., Wiffen, P. J., Bell, R. F., Aldington, D., & Moore, R. A. (2017). Tramadol for neuropathic pain in adults.Β  Cochrane Database of Systematic Reviews, (6). https://doi.org/10.1002/14651858.CD003726.pub4

34

  • SR Protocol
  • PICO terms
  • Search query
  • Screening pipeline

Initial search:

152 + 388 + 737 = 1,277 records

Finally included in meta-analysis

6 records (0.47%)

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

35 of 143

Systematic Review example

Duehmke, R. M., Derry, S., Wiffen, P. J., Bell, R. F., Aldington, D., & Moore, R. A. (2017). Tramadol for neuropathic pain in adults.Β  Cochrane Database of Systematic Reviews, (6). https://doi.org/10.1002/14651858.CD003726.pub4

35

  • SR Protocol
  • PICO terms
  • Search query
  • Screening pipeline
  • Meta-analysis

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

36 of 143

Systematic Review example

Duehmke, R. M., Derry, S., Wiffen, P. J., Bell, R. F., Aldington, D., & Moore, R. A. (2017). Tramadol for neuropathic pain in adults.Β  Cochrane Database of Systematic Reviews, (6). https://doi.org/10.1002/14651858.CD003726.pub4

36

  • SR Protocol
  • PICO terms
  • Search query
  • Screening pipeline
  • Meta-analysis

SR outcomes and forest plot

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

37 of 143

Systematic Review example

Duehmke, R. M., Derry, S., Wiffen, P. J., Bell, R. F., Aldington, D., & Moore, R. A. (2017). Tramadol for neuropathic pain in adults.Β  Cochrane Database of Systematic Reviews, (6). https://doi.org/10.1002/14651858.CD003726.pub4

37

  • SR Protocol
  • PICO terms
  • Search query
  • Screening pipeline
  • Meta-analysis

Risk of bias assessment

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

38 of 143

Systematic Review example

Duehmke, R. M., Derry, S., Wiffen, P. J., Bell, R. F., Aldington, D., & Moore, R. A. (2017). Tramadol for neuropathic pain in adults.Β  Cochrane Database of Systematic Reviews, (6). https://doi.org/10.1002/14651858.CD003726.pub4

38

  • SR Protocol
  • PICO terms
  • Search query
  • Screening pipeline
  • Meta-analysis
  • Final review πŸŽ‰

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

39 of 143

Systematic Review example

Duehmke, R. M., Derry, S., Wiffen, P. J., Bell, R. F., Aldington, D., & Moore, R. A. (2017). Tramadol for neuropathic pain in adults.Β  Cochrane Database of Systematic Reviews, (6). https://doi.org/10.1002/14651858.CD003726.pub4

39

  • SR Protocol
  • PICO terms
  • Search query
  • Screening pipeline
  • Meta-analysis
  • Final review? πŸ€”πŸ€”

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

40 of 143

Systematic Review example

Duehmke, R. M., Derry, S., Wiffen, P. J., Bell, R. F., Aldington, D., & Moore, R. A. (2017). Tramadol for neuropathic pain in adults.Β  Cochrane Database of Systematic Reviews, (6). https://doi.org/10.1002/14651858.CD003726.pub4

40

  • SR Protocol
  • PICO terms
  • Search query
  • Screening pipeline
  • Meta-analysis
  • Final review? πŸ€”πŸ€”

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

41 of 143

Systematic Review example

Duehmke, R. M., Derry, S., Wiffen, P. J., Bell, R. F., Aldington, D., & Moore, R. A. (2017). Tramadol for neuropathic pain in adults.Β  Cochrane Database of Systematic Reviews, (6). https://doi.org/10.1002/14651858.CD003726.pub4

41

  • SR Protocol
  • PICO terms
  • Search query
  • Screening pipeline
  • Meta-analysis
  • Final review
  • Review updates

2004:

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

42 of 143

Systematic Review example

Duehmke, R. M., Derry, S., Wiffen, P. J., Bell, R. F., Aldington, D., & Moore, R. A. (2017). Tramadol for neuropathic pain in adults.Β  Cochrane Database of Systematic Reviews, (6). https://doi.org/10.1002/14651858.CD003726.pub4

42

  • SR Protocol
  • PICO terms
  • Search query
  • Screening pipeline
  • Meta-analysis
  • Final review
  • Review updates

2004:

2006:

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

43 of 143

Systematic Review example

Duehmke, R. M., Derry, S., Wiffen, P. J., Bell, R. F., Aldington, D., & Moore, R. A. (2017). Tramadol for neuropathic pain in adults.Β  Cochrane Database of Systematic Reviews, (6). https://doi.org/10.1002/14651858.CD003726.pub4

43

  • SR Protocol
  • PICO terms
  • Search query
  • Screening pipeline
  • Meta-analysis
  • Final review
  • Review updates

2004:

2017:

2006:

…

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

44 of 143

Systematic Review example

Duehmke, R. M., Derry, S., Wiffen, P. J., Bell, R. F., Aldington, D., & Moore, R. A. (2017). Tramadol for neuropathic pain in adults.Β  Cochrane Database of Systematic Reviews, (6). https://doi.org/10.1002/14651858.CD003726.pub4

44

  • SR Protocol
  • PICO terms
  • Search query
  • Screening pipeline
  • Meta-analysis
  • Final review
  • Review updates

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

45 of 143

Citation Screening

45

source: cochrane.org

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

46 of 143

Citation Screening

46

source: cochrane.org

One of the most time-consuming steps of Systematic Review

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

47 of 143

Citation Screening

47

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

48 of 143

Citation Screening

48

  1. Boolean query to identify as many articles as possible

Helfand, M. et al. (2007). Drug class review on beta adrenergic blockers. Oregon Health & Science University.

Documents

search

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

49 of 143

Citation Screening

49

  1. Boolean query to identify as many articles as possible

Helfand, M. et al. (2007). Drug class review on beta adrenergic blockers. Oregon Health & Science University.

Documents

search

~30 million studies

PubMed + EMBASE + …

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

50 of 143

Citation Screening

50

Helfand, M. et al. (2007). Drug class review on beta adrenergic blockers. Oregon Health & Science University.

  1. Boolean query to identify as many articles as possible

2072

papers

Documents

search

~30 million studies

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

51 of 143

Citation Screening

51

Helfand, M. et al. (2007). Drug class review on beta adrenergic blockers. Oregon Health & Science University.

  1. Boolean query to identify as many articles as possible

2072

papers

Documents

search

~30 million studies

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

52 of 143

Citation Screening

52

Helfand, M. et al. (2007). Drug class review on beta adrenergic blockers. Oregon Health & Science University.

  1. Boolean query to identify as many articles as possible
  2. Title and abstract screening

2072

papers

Title and abstract

screening

Documents

search

~30 million studies

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

53 of 143

Citation Screening

53

Helfand, M. et al. (2007). Drug class review on beta adrenergic blockers. Oregon Health & Science University.

  1. Boolean query to identify as many articles as possible
  2. Title and abstract screening

2072

papers

302 papers

(14%)

Title and abstract

screening

Documents

search

~30 million studies

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

54 of 143

Citation Screening

54

Helfand, M. et al. (2007). Drug class review on beta adrenergic blockers. Oregon Health & Science University.

  1. Boolean query to identify as many articles as possible
  2. Title and abstract screening
  3. Full text screening

2072

papers

302 papers

(14%)

Title and abstract

screening

Full text

screening

Documents

search

~30 million studies

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

55 of 143

Citation Screening

55

Helfand, M. et al. (2007). Drug class review on beta adrenergic blockers. Oregon Health & Science University.

  1. Boolean query to identify as many articles as possible
  2. Title and abstract screening
  3. Full text screening

2072

papers

302 papers

(14%)

42 papers

(2%)

Title and abstract

screening

Full text

screening

Documents

search

~30 million studies

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

56 of 143

Citation Screening

56

Helfand, M. et al. (2007). Drug class review on beta adrenergic blockers. Oregon Health & Science University.

  1. Boolean query to identify as many articles as possible
  2. Title and abstract screening
  3. Full text screening

2072

papers

302 papers

(14%)

42 papers

(2%)

Title and abstract

screening

Full text

screening

Documents

search

~30 million studies

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

57 of 143

Citation Screening

57

Helfand, M. et al. (2007). Drug class review on beta adrenergic blockers. Oregon Health & Science University.

  1. Boolean query to identify as many articles as possible
  2. Title and abstract screening
  3. Full text screening

2072

papers

302 papers

(14%)

42 papers

(2%)

Title and abstract

screening

Full text

screening

Documents

search

Data extraction and meta-analysis

~30 million studies

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

58 of 143

Citation Screening

58

Helfand, M. et al. (2007). Drug class review on beta adrenergic blockers. Oregon Health & Science University.

  1. Boolean query to identify as many articles as possible
  2. Title and abstract screening
  3. Full text screening

2072

papers

302 papers

(14%)

42 papers

(2%)

Title and abstract

screening

Full text

screening

Documents

search

~30 million studies

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

59 of 143

Citation Screening

59

Helfand, M. et al. (2007). Drug class review on beta adrenergic blockers. Oregon Health & Science University.

  1. Boolean query to identify as many articles as possible
  2. Title and abstract screening
  3. Full text screening

2072

papers

302 papers

(14%)

42 papers

(2%)

Title and abstract

screening

Full text

screening

Documents

search

~30 million studies

Publication of Systematic Review

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

60 of 143

Automated Citation Screening

60

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

61 of 143

Automated Citation Screening

  • First automation approaches in 2006 [1]
  • Methods:
    • binary classification
    • ranking, prioritisation
    • retrieval

61

[1] Cohen, Hersh, Peterson, Yen, (2006). Reducing workload in systematic review preparation using automated citation classification.Β JAMIA

[2] Bannach-Brown, PrzybyΕ‚a, Thomas, Rice, Ananiadou, Liao, Macleod, (2019). Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error.Β Systematic reviews.

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

62 of 143

Automated Citation Screening

  • First automation approaches in 2006 [1]
  • Methods:
    • binary classification
    • ranking, prioritisation
    • retrieval

62

[1] Cohen, Hersh, Peterson, Yen, (2006). Reducing workload in systematic review preparation using automated citation classification.Β JAMIA

[2] Bannach-Brown, PrzybyΕ‚a, Thomas, Rice, Ananiadou, Liao, Macleod, (2019). Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error.Β Systematic reviews.

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

63 of 143

Automated Citation Screening

  • First automation approaches in 2006 [1]
  • Methods:
    • binary classification
    • ranking, prioritisation
    • retrieval

63

[1] Cohen, Hersh, Peterson, Yen, (2006). Reducing workload in systematic review preparation using automated citation classification.Β JAMIA

[2] Bannach-Brown, PrzybyΕ‚a, Thomas, Rice, Ananiadou, Liao, Macleod, (2019). Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error.Β Systematic reviews.

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

64 of 143

Automated Citation Screening

  • First automation approaches in 2006 [1]
  • Methods:
    • binary classification
    • ranking, prioritisation
    • retrieval

64

[1] Cohen, Hersh, Peterson, Yen, (2006). Reducing workload in systematic review preparation using automated citation classification.Β JAMIA

[2] Bannach-Brown, PrzybyΕ‚a, Thomas, Rice, Ananiadou, Liao, Macleod, (2019). Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error.Β Systematic reviews.

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

65 of 143

Screening models

  • Retrieval: lexical [1,2,3] + dense models [4]
  • Classification and prioritisation: random forest [5], CLM [6], SVMs [7], neural networks [8,9], prompt-based learning [10] , LLMs [11]

65

[1] Kanoulas, Evangelos, et al. 2017. ”CLEF 2017 technology assisted reviews in empirical medicine overview.” CLEF

[2] Kanoulas, Evangelos, et al. 2018. ”CLEF 2018 technology assisted reviews in empirical medicine overview.” CLEF

[3] Kanoulas, Evangelos, et al. 2019. ”CLEF 2019 technology assisted reviews in empirical medicine overview.” CLEF

[4] Wang, Shuai, et al. 2022. β€œNeural Rankers for Effective Screening Prioritisation in Medical Systematic Review Literature Search.” ACDS

[5] Khabsa, Madian, et al. 2016. "Learning to identify relevant studies for systematic reviews using random forest and external information."Β Machine LearningΒ 

[6] Scells, Harrisen, et al. 2020. "You can teach an old dog new tricks: Rank fusion applied to coordination level matching for ranking in systematic reviews."Β ECIR

[7] Cohen, Aaron M., et al. 2006. "Reducing workload in systematic review preparation using automated citation classification."Β JAMIA

[8] Kontonatsios, Georgios, et al. 2020 "Using a neural network-based feature extraction method to facilitate citation screening for systematic reviews."Β Expert Systems with Applications: XΒ 

[9] Kusa, Wojciech, et al. 2022 "Automation of citation screening for systematic literature reviews using neural networks: A replicability study."Β ECIR

[10] Romagnoli, Alice, et al. 2023. β€œAutomatic Citation Screening Using Pattern-Exploiting Training and Paraphrasing”. ALTARS

[11] Wang, Shuai, et al. 2024 "Zero-shot Generative Large Language Models for Systematic Review Screening Automation."Β ECIR findings

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

66 of 143

Screening models

  • Retrieval: lexical [1,2,3] + dense models [4]
  • Classification and prioritisation: random forest [5], CLM [6], SVMs [7], neural networks [8,9], prompt-based learning [10] , LLMs [11]

66

Technologically- Assisted Reviews (TAR)

[1] Kanoulas, Evangelos, et al. 2017. ”CLEF 2017 technology assisted reviews in empirical medicine overview.” CLEF

[2] Kanoulas, Evangelos, et al. 2018. ”CLEF 2018 technology assisted reviews in empirical medicine overview.” CLEF

[3] Kanoulas, Evangelos, et al. 2019. ”CLEF 2019 technology assisted reviews in empirical medicine overview.” CLEF

[4] Wang, Shuai, et al. 2022. β€œNeural Rankers for Effective Screening Prioritisation in Medical Systematic Review Literature Search.” ACDS

[5] Khabsa, Madian, et al. 2016. "Learning to identify relevant studies for systematic reviews using random forest and external information."Β Machine LearningΒ 

[6] Scells, Harrisen, et al. 2020. "You can teach an old dog new tricks: Rank fusion applied to coordination level matching for ranking in systematic reviews."Β ECIR

[7] Cohen, Aaron M., et al. 2006. "Reducing workload in systematic review preparation using automated citation classification."Β JAMIA

[8] Kontonatsios, Georgios, et al. 2020 "Using a neural network-based feature extraction method to facilitate citation screening for systematic reviews."Β Expert Systems with Applications: XΒ 

[9] Kusa, Wojciech, et al. 2022 "Automation of citation screening for systematic literature reviews using neural networks: A replicability study."Β ECIR

[10] Romagnoli, Alice, et al. 2023. β€œAutomatic Citation Screening Using Pattern-Exploiting Training and Paraphrasing”. ALTARS

[11] Wang, Shuai, et al. 2024 "Zero-shot Generative Large Language Models for Systematic Review Screening Automation."Β ECIR findings

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

67 of 143

Information Retrieval Evaluation

Dataset / Documents

Queries

Relevance Judgements

Measures

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

68 of 143

Information Retrieval Evaluation

Dataset / Documents

Queries

Relevance Judgements

Measures

Search Engine

Result Lists

Compare

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

69 of 143

Evaluation measures

[1] Kusa, Lipani, Knoth, Hanbury. 2023. An analysis of work saved over sampling in the evaluation of automated citation screening in systematic literature reviews.Β ISwA

​

[2] Kusa, Lipani, Knoth, Hanbury. 2023. VoMBaT: A Tool for Visualising Evaluation Measure Behaviour in High-Recall Search Tasks. SIGIR

​

[3] Kusa, Peikos, Lipani, Hanbury. 2024. Normalised Precision at Fixed Recall for Evaluating TAR. ICTIR

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

70 of 143

Evaluation Measures

The most popular measure used in SR literature is Work Saved Over Sampling οΏ½at π‘Ÿ% recall (WSS@r%) [1]:

70

[1] Cohen, Hersh, Peterson, Yen. 2006. Reducing workload in systematic review preparation using automated citation classification. JAMIA

    • measures amount of work saved when using machine learning models to screen irrelevant publications
    • typically used at recall = 95% (WSS@95%)

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

71 of 143

Evaluation of Screening Prioritisation

71

N = 1,000

Documents

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

72 of 143

Evaluation of Screening Prioritisation

72

N = 1,000

Documents

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

73 of 143

Evaluation of Screening Prioritisation

73

N = 1,000

Documents

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

74 of 143

Evaluation of Screening Prioritisation

74

N = 1,000

Documents

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

75 of 143

Evaluation of Screening Prioritisation

75

N = 1,000

Documents

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

76 of 143

Evaluation of Screening Prioritisation

76

N = 1,000

Documents

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

77 of 143

Evaluation of Screening Prioritisation

77

N = 1,000

Documents

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

78 of 143

Evaluation of Screening Prioritisation

78

N = 1,000

Documents

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

79 of 143

Evaluation of Screening Prioritisation

79

N = 1,000

Documents

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

80 of 143

Evaluation of Screening Prioritisation

80

N = 1,000

Documents

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

81 of 143

Evaluation of Screening Prioritisation

81

N = 1,000

Documents

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

82 of 143

Evaluation of Screening Prioritisation

82

N = 1,000

Documents

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

83 of 143

Evaluation of Screening Prioritisation

83

N = 1,000

Documents

Recall = 95% for a document at rank 656

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

84 of 143

Evaluation of Screening Prioritisation

84

N = 1,000

Documents

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

85 of 143

Work Saved over Sampling

85

N = 1,000

Documents

WSS@r% is the percentage of papers that meet the original search criteria that the reviewers do not have to read (because they have been screened out by the classifier)

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

86 of 143

Work Saved over Sampling

86

Higher WSS

Lower WSS

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

87 of 143

WSS - same Recall, different datasets

87

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

88 of 143

WSS - different Recall, same dataset

88

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

89 of 143

WSS properties

  • (1 βˆ’ π‘Ÿ) term is a constant for a specific recall r% for all datasets and models
  • FN term for a specific r% recall is equal to βŒŠπΌβ‹…(1 βˆ’ π‘Ÿ)βŒ‹ where 𝐼 is the number of includes

​

​

  • The numerator is a sum of:
    • TN (a factor that should be maximised) and
    • FN (a factor which should instead be minimised)

89

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

90 of 143

Min and max WSS

  • minWSS β€” all excludes should be ranked before more than 𝐼⋅(1 βˆ’ π‘Ÿ) includes
  • maxWSS β€” at least r% of includes should be ranked before any exclude

90

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

91 of 143

Normalised Work Saved over Sampling

91

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

92 of 143

Normalised Work Saved over Sampling

92

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

93 of 143

Normalised Work Saved over Sampling

93

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

94 of 143

Normalised Work Saved over Sampling

  • Normalised WSS is equal to the True Negative Rate
  • nWSS@r% == specificity (TNR) at a recall rate of r%

94

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

95 of 143

Why this is important?

95

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

96 of 143

Work Saved over Sampling

96

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

97 of 143

Normalised Work Saved over Sampling

97

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

98 of 143

Open problems

  • All measure require calculating 95% of recall first
    • They cannot be used prospectively, i.e., until all documents are evaluated
  • nP and TNR are strongly influenced by random effects
    • Tend to have a large variance
    • Although comparable to MAP
  • They do not account for the document length and complexity
  • Is Recall that important?

98

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

99 of 143

[1] Kusa, Zuccon, Knoth, Hanbury. 2023. Outcome-based Evaluation of Systematic Review Automation.Β ICTIR

Outcome-based Evaluation

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

100 of 143

Systematic Review Outcomes

100

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

101 of 143

Systematic Review Outcomes

  • For all relevant publications, reviewers extract outcome information

101

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

102 of 143

Systematic Review Outcomes

  • For all relevant publications, reviewers extract outcome information
  • Outcome data is compared between two intervention groups (β€˜effect measures’)

102

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

103 of 143

Systematic Review Outcomes

  • For all relevant publications, reviewers extract outcome information
  • Outcome data is compared between two intervention groups (β€˜effect measures’)

​

  • Examples of outcomes:
    • Clinical outcomes (e.g. mortality, morbidity)
    • Patient-reported outcomes (e.g. quality of life, satisfaction)
    • Economic outcomes (e.g. cost-effectiveness)

103

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

104 of 143

Review outcome example

104

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

105 of 143

Review outcome example

105

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

106 of 143

Review outcome example

106

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

107 of 143

Review outcome example

107

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

108 of 143

Review outcome example

108

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

109 of 143

Review outcome example

109

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

110 of 143

Motivation

110

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

111 of 143

Motivation

Not every study has the same β€œimpact” on the final outcome of the review

111

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

112 of 143

Motivation

Not every study has the same β€œimpact” on the final outcome of the review

112

Recall = 100%

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

113 of 143

Motivation

Not every study has the same β€œimpact” on the final outcome of the review

113

Recall = 100%

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

114 of 143

Motivation

Not every study has the same β€œimpact” on the final outcome of the review

114

Recall = 100%

Recall = 20%

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

115 of 143

Motivation

Not every study has the same β€œimpact” on the final outcome of the review

115

Recall = 100%

Recall = 20%

Recall = 80%

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

116 of 143

Motivation

Not every study has the same β€œimpact” on the final outcome of the review

116

Recall = 100%

Recall = 20%

Recall = 20%

Recall = 80%

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

117 of 143

EVALUATION FRAMEWORK

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

118 of 143

Evaluation framework steps

118

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

119 of 143

Evaluation framework steps

  1. Data extraction β€” find and extract meta-analysis section

119

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

120 of 143

Evaluation framework steps

  1. Data extraction β€” find and extract meta-analysis section
  2. Model evaluation β€” re-calculate the meta-analysis for a ranking/classification

120

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

121 of 143

Evaluation framework steps

  1. Data extraction β€” find and extract meta-analysis section
  2. Model evaluation β€” re-calculate the meta-analysis for a ranking/classification
  3. Result analysis β€” examine the outcomes generated by the run and compare them with the outcomes obtained by the original review

121

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

122 of 143

Evaluation framework steps

  1. Data extraction β€” find and extract meta-analysis section
  2. Model evaluation β€” re-calculate the meta-analysis for a ranking/classification
  3. Result analysis β€” examine the outcomes generated by the run and compare them with the outcomes obtained by the original review
  4. Measure publication Influence β€” measure the influence of each publication on review outcomes

122

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

123 of 143

Results analysis

123

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

124 of 143

Results analysis

We distinguish five aspects of analysis for review outcomes against the original review:

​

​

​

​

​

​

​

​

​

​

​

​

124

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

125 of 143

Results analysis

We distinguish five aspects of analysis for review outcomes against the original review:

​

  1. Magnitude of Difference β€” By how much are the outcomes different in their effect size? Measured by calculating the absolute difference in effect size between the original outcome π‘‚π‘œ and predicted outcome 𝑂𝑝: MoD = |π‘‚π‘œ βˆ’ 𝑂𝑝|/|π‘‚π‘œ|.

​

​

​

​

​

​

​

​

125

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

126 of 143

Results analysis

We distinguish five aspects of analysis for review outcomes against the original review:

​

  1. Magnitude of Difference β€” By how much are the outcomes different in their effect size? Measured by calculating the absolute difference in effect size between the original outcome π‘‚π‘œ and predicted outcome 𝑂𝑝: MoD = |π‘‚π‘œ βˆ’ 𝑂𝑝|/|π‘‚π‘œ|.
  2. Distance from CI β€” Is the new outcome within the Confidence Interval (CI) of the original outcome? The answer is a distance between the predicted outcome 𝑂𝑝 and the closest of the pair (πΆπΌπ‘™π‘œπ‘€π‘’π‘Ÿ,πΆπΌπ‘’π‘π‘π‘’π‘Ÿ).

​

​

​

​

​

126

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

127 of 143

Results analysis

We distinguish five aspects of analysis for review outcomes against the original review:

​

  1. Magnitude of Difference β€” By how much are the outcomes different in their effect size? Measured by calculating the absolute difference in effect size between the original outcome π‘‚π‘œ and predicted outcome 𝑂𝑝: MoD = |π‘‚π‘œ βˆ’ 𝑂𝑝|/|π‘‚π‘œ|.
  2. Distance from CI β€” Is the new outcome within the Confidence Interval (CI) of the original outcome? The answer is a distance between the predicted outcome 𝑂𝑝 and the closest of the pair (πΆπΌπ‘™π‘œπ‘€π‘’π‘Ÿ,πΆπΌπ‘’π‘π‘π‘’π‘Ÿ).
  3. Overestimation/underestimation β€” Is the generated outcome overestimated or underestimated compared to the original one?

​

​

​

127

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

128 of 143

Results analysis

We distinguish five aspects of analysis for review outcomes against the original review:

​

  1. Magnitude of Difference β€” By how much are the outcomes different in their effect size? Measured by calculating the absolute difference in effect size between the original outcome π‘‚π‘œ and predicted outcome 𝑂𝑝: MoD = |π‘‚π‘œ βˆ’ 𝑂𝑝|/|π‘‚π‘œ|.
  2. Distance from CI β€” Is the new outcome within the Confidence Interval (CI) of the original outcome? The answer is a distance between the predicted outcome 𝑂𝑝 and the closest of the pair (πΆπΌπ‘™π‘œπ‘€π‘’π‘Ÿ,πΆπΌπ‘’π‘π‘π‘’π‘Ÿ).
  3. Overestimation/underestimation β€” Is the generated outcome overestimated or underestimated compared to the original one?
  4. Sign β€” Does the outcome have the same sign as the original one? In other words, are the new conclusions opposite to the original ones?

​

128

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

129 of 143

Results analysis

We distinguish five aspects of analysis for review outcomes against the original review:

​

  1. Magnitude of Difference β€” By how much are the outcomes different in their effect size? Measured by calculating the absolute difference in effect size between the original outcome π‘‚π‘œ and predicted outcome 𝑂𝑝: MoD = |π‘‚π‘œ βˆ’ 𝑂𝑝|/|π‘‚π‘œ|.
  2. Distance from CI β€” Is the new outcome within the Confidence Interval (CI) of the original outcome? The answer is a distance between the predicted outcome 𝑂𝑝 and the closest of the pair (πΆπΌπ‘™π‘œπ‘€π‘’π‘Ÿ,πΆπΌπ‘’π‘π‘π‘’π‘Ÿ)
  3. Overestimation/underestimation β€” Is the generated outcome overestimated or underestimated compared to the original one?
  4. Sign β€” Does the outcome have the same sign as the original one? In other words, are the new conclusions opposite to the original ones?
  5. Estimability β€” Is it possible to calculate the outcome? An outcome cannot be calculated if there are no included studies concerning it.

129

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

130 of 143

Results analysis

We distinguish five aspects of analysis for review outcomes against the original review:

​

  1. Magnitude of Difference β€” By how much are the outcomes different in their effect size? Measured by calculating the absolute difference in effect size between the original outcome π‘‚π‘œ and predicted outcome 𝑂𝑝: MoD = |π‘‚π‘œ βˆ’ 𝑂𝑝|/|π‘‚π‘œ|.
  2. Distance from CI β€” Is the new outcome within the Confidence Interval (CI) of the original outcome? The answer is a distance between the predicted outcome 𝑂𝑝 and the closest of the pair (πΆπΌπ‘™π‘œπ‘€π‘’π‘Ÿ,πΆπΌπ‘’π‘π‘π‘’π‘Ÿ)
  3. Overestimation/underestimation β€” Is the generated outcome overestimated or underestimated compared to the original one?
  4. Sign β€” Does the outcome have the same sign as the original one? In other words, are the new conclusions opposite to the original ones?
  5. Estimability β€” Is it possible to calculate the outcome? An outcome cannot be calculated if there are no included studies concerning it.

130

categorical

real-valued

real-valued

categorical

categorical

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

131 of 143

Publication Influence

  • For each publication, we can measure its influence as a difference between the gold Outcome Oπ‘œ and the β€˜new’ predicted outcome when this publication is missing 𝑂𝑝:οΏ½ MoD = |π‘‚π‘œ βˆ’ 𝑂𝑝|

​

  • The more β€˜influential’ is the publication on the review outcomes, the bigger the change in the outcomes when that publication is missing
  • If the publication would be β€˜useless’ for a given SR, the outcomes would not change at all and the Influence would be zero

131

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

132 of 143

Dataset

132

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

133 of 143

Dataset

  • We use 32 systematic reviews of interventions from CLEF TAR 2019 shared task

133

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

134 of 143

Dataset

  • We use 32 systematic reviews of interventions from CLEF TAR 2019 shared task

134

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

135 of 143

Dataset

  • We use 32 systematic reviews of interventions from CLEF TAR 2019 shared task
  • We re-evaluate 74 different runs using the full text screening relevance judgements (qrels)

135

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

136 of 143

Evaluation on CLEF TAR

136

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

137 of 143

Evaluation on CLEF TAR

137

  • We measure the mean relative difference of all outcomes calculated at 30% cut-off of dataset size (MoD@30%)

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

138 of 143

Evaluation on CLEF TAR

138

  • We measure the mean relative difference of all outcomes calculated at 30% cut-off of dataset size (MoD@30%)
  • Runs sorted by their MAP score

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

139 of 143

Evaluation on CLEF TAR

139

  • We measure the mean relative difference of all outcomes calculated at 30% cut-off of dataset size (MoD@30%)
  • Runs sorted by their MAP score

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

140 of 143

Evaluation on CLEF TAR

140

  • We measure the mean relative difference of all outcomes calculated at 30% cut-off of dataset size (MoD@30%)
  • Runs sorted by their MAP score

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

141 of 143

CONCLUSION

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

142 of 143

Conclusion

  • Traditional evaluation of citation screening automation relies on binary relevance
  • Based on the assumption that not every publication is equally important to the review outcome we introduce outcome-based evaluation
            • Our analysis on CLEF TAR 2019 data shows different ordering of runs compared to the traditional evaluation measures
            • Adapting the retrieval measurements to the domain requires understanding of the domain

​

            • And there is a dataset…

​

142

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721

143 of 143

Dataset

37th Conference on Neural Information Processing Systems Track on Datasets and Benchmarks

​

pdf: https://arxiv.org/pdf/2311.12474.pdf

code: https://github.com/WojciechKusa/systematic-review-datasets

MSCA ITN/ETN No. 860721

MSCA ITN/ETN No. 860721