1 of 29

Exploring different review methodologies

2 of 29

Publishing of evidence synthesis has grown considerably

2

Systematic reviews and meta-analysis

n=410,787 as of March 2024

Data and image from Web of Science

!!

3 of 29

Publishing of evidence synthesis has grown considerably

3

Diverse ES methods not including SRs

n=37,252 as of March 2024

Data and image from Web of Science

4 of 29

4

5 of 29

5

Littell, J.H. (2018), Conceptual and practical classification of research reviews and other evidence synthesis products. Campbell Systematic Reviews, 14: 1-21. https://doi.org/10.4073/cmdp.2018.1

6 of 29

Open the example review that you chose

In the chat, tell us what type of review you chose (e.g. systematic review, scoping review, something else?)

We’ll use it again in an exercise at the end of this section!

6

7 of 29

Evidence Synthesis Methodologies

7

Narrative Review

Provides an overview or description of the literature review without using systematic methods.

Scoping Review

Systematic Map/Evidence Gap Map

Looks at themes, trends, and gaps in research. "Where the research is".

Uses rigorous methods for search and selection. Often answers questions of 'what works' or 'what the research says'.

Systematic Review

Rapid reviews

Living reviews

Umbrella Reviews

Other Methodologies

8 of 29

Show of hands:

How many of you have had a student come to you for help with a “systematic review” when in fact it’s a literature review for their dissertation, or a short paper for an undergraduate assignment?

8

9 of 29

Show of hands: How many of you have had a student come to you for help with a “systematic review” when in fact it’s a literature review for their dissertation, or a short paper for an undergraduate assignment?

9

More on this in the breakout room on Day 4!

10 of 29

Maybe a systematized literature review instead?

Systematized literature reviews use systematic approaches in different steps in the review process but without adhering to established systematic review or other standards.

10

11 of 29

11

12 of 29

More methodology selection tools on OSF

12

https://osf.io/zn3md/

13 of 29

13

https://whatreviewisrightforyou.knowledgetranslation.net/

14 of 29

Visit our public Zotero library for methodology resources

14

https://www.zotero.org/groups/2418750/evidence_synthesis_articles/library

15 of 29

Systematic Review

Often answers a question of whether or not an intervention is effective.

“Does this intervention work in this population? Or is there insufficient evidence?”

Sometimes limited to certain types of study design (RCTs, experimental research, etc.)
Should always include risk of bias and quality assessment
Sometimes includes a quantitative analysis of findings (meta-analysis)

15

16 of 29

Example format of a typical systematic review question

What is the effect of intervention X on population Y in achieving Z outcomes?

16

17 of 29

Systematic Review

While questions of effectiveness are most common for a systematic reviews, they can be used to answer many types of well-formulated research questions:

How a construct is defined and measured
Questions about rates, trends and prevalence
Whether things are correlated
Cause and effect questions

17

18 of 29

A look at a systematic review research question

Murray, J., Wickramasekera, N., Elings, M., Bragg, R., Brennan, C., Richardson, Z., Wright, J., Llorente, M. G., Cade, J., Shickle, D., Tubeuf, S., & Elsey, H. (2019). The impact of care farms on quality of life, depression and anxiety among different population groups: A systematic review. Campbell Systematic Reviews, 15(4). https://doi.org/10.1002/cl2.1061

18

19 of 29

“What is the impact of care farming (intervention) on quality of life, depression and anxiety (outcomes), on a range of service user groups (population)?”

19

20 of 29

Scoping Reviews &

Systematic Maps

Describes the current state of knowledge and/or shows research themes, trends and gaps
Research question can be broader than a systematic review but criteria still well defined
Often includes all types of studies (though maps may be limited to high impact study designs)
Thematic analyses, tables, maps but no synthesis per se
Typically does not include risk of bias or quality assessment

20

21 of 29

Example formats for scoping review or systematic map questions

What interventions are used to address outcomes X in population Y?

What are the characteristics of the literature on interventions X in setting Y?

21

22 of 29

Evidence & Gap Map examples: https://www.3ieimpact.org/

22

Saran, A., & White, H. (2018). Evidence and gap maps: A comparison of different approaches. Campbell Systematic Reviews, 14(1), 1–38. https://doi.org/10.4073/cmdp.2018.2

“Evidence maps are a relatively new approach to systematically identify and report the range of research activity in broad topic areas or policy domains. Although they go under various names, they have the common feature of adopting a systematic approach to mapping begins with a broad question, theme or issue which defines the scope of the evidence map” (Saran & White, 2018).

Question setting for Evidence Map:

“Broad scope of interventions across a sector or sub-sector, with full range of outcomes across causal chain. A PICOS is specified to guide study inclusion criteria…broader in scope than systematic reviews.” (Saran & White, 2018).

In a nutshell evidence and gap map or systematic maps can be defined as:

‘a systematic [visual] presentation of the availability of relevant evidence [of effects] for a particular policy domain. The evidence is identified by a search following a pre-specified, published search protocol. [The map may be accompanied by a descriptive report to summarize the evidence for stakeholders such as researchers, research commissioners, policy makers, and practitioners]

[Evidence maps summarize what evidence there is, not what the evidence says]’.

23 of 29

A look at a scoping review research question

Piñeiro, V., Arias, J., Dürr, J., Elverdin, P., Ibáñez, A. M., Kinengyere, A., Opazo, C. M., Owoo, N., Page, J. R., Prager, S. D., & Torero, M. (2020). A scoping review on incentives for adoption of sustainable agricultural practices and their outcomes. Nature Sustainability, 3(10), 809–820. https://doi.org/10.1038/s41893-020-00617-y

Methods section is towards the end of the article

23

24 of 29

“What are the market, non-market, regulatory and compliance incentives or compulsory/voluntary programmes (interventions) for farmers (population) to adopt environmentally sustainable practices? (outcomes)”.

24

25 of 29

Additional methodologies

Living systematic reviews

A systematic review that is continually updated, incorporating relevant new evidence as it becomes available

Elliott, J. H., Synnot, A., Turner, T., Simmonds, M., Akl, E. A., McDonald, S., Salanti, G., Meerpohl, J., MacLehose, H., Hilton, J., Tovey, D., Shemilt, I., Thomas, J., Agoritsas, T., Hilton, J., Perron, C., Akl, E., Hodder, R., Pestridge, C., … Pearson, L. (2017). Living systematic review: 1. Introduction—the why, what, when, and how. Journal of Clinical Epidemiology, 91, 23–30. https://doi.org/10.1016/j.jclinepi.2017.08.010

Rapid reviews

Rapid reviews are a form of knowledge synthesis in which components of the systematic review process are simplified or omitted to produce information in a timely manner.

Tricco, A. C., Antony, J., Zarin, W., Strifler, L., Ghassemi, M., Ivory, J., Perrier, L., Hutton, B., Moher, D., & Straus, S. E. (2015). A scoping review of rapid review methods. BMC Medicine, 13(1), 224. https://doi.org/10.1186/s12916-015-0465-6

25

And many many more!

26 of 29

Poll: Scoping review or systematic review

👪

#1 ‘What machine learning methods have been used for mental health clinical practice and research?'

https://doi.org/10.1017/S0033291719000151

🌏

#2 ‘How effective are different methods for improving student achievement scores for algebra?’

https://www.jstor.org/stable/40927286

26

27 of 29

Poll: Scoping review or systematic review

👪Scoping review

#1 ‘What machine learning methods have been used for mental health clinical practice and research?'’

🌏Systematic review

#2 ‘How effective are different methods for improving student achievement scores for algebra?’

27

28 of 29

Open the example review that you chose

Aside from the title, how can you tell what type of evidence synthesis methodology was used? Look for clues in the methods section.

Do you think the authors chose an appropriate methodology? Or is the methodological term incorrectly applied?

Share your answer in the chat.

28

29 of 29

Questions?

29