Automating the evidence synthesis process

1

Prompt Questions

What does automation in the evidence synthesis context mean to you?

What aspects of evidence synthesis do you think would most benefit from automation?

What are some of the downsides of automation that you can think of?

What skill sets might be needed to apply automation techniques?

​

2

What does automation mean in

evidence synthesis?

Automation can be as simple as using a tool to auto-generate a Boolean search from a list of keywords and as complex as applying machine learning algorithms to auto-exclude records in the screening process.

3

Why consider automation?

The evidence synthesis process can be lengthy and time-consuming. Automation can help speed up parts of the process to arrive at answers more quickly and save on resources.

But it is important to exercise caution!

4

Automated vs. computer-assisted

Automation tools and techniques can be used to assist otherwise human-led processes, such that we are never relying solely on machines to make decisions for us.

As in all other aspects of evidence synthesis, transparency in how these methods are applied is key.

5

Many tools exist for automating ES

Automation tools can range from very user-friendly, point-and-click tools to more complex tools that require knowledge of a programming language like R or Python.

There are many automation tools to screening and analysis, but let's focus on the steps librarians are typically involved in.

6

7

A few notes about automation in screening

Many screening platforms include machine learning tools to aid in the screening process. For example:

• EPPI-Reviewer: eppi.ioe.ac.uk/CMS/Default.aspx
• sysrev: sysrev.com
• Abstrackr: abstrackr.cebm.brown.edu/account/login

Most do this by training an algorithm as the team makes eligibility decisions. Some tools re-sort records based on the algorithm to speed up screening (computer-assisted). Others provide prediction metrics to aid in bulk exclusion or inclusion.

There is not much consensus on how to apply these tools, particularly when it comes to bulk exclusion.

​

R packages and Shiny apps

• There are increasingly more R packages designed for evidence synthesis steps (like revtools, litsearchr, Automated Systematic Search Deduplicator (ASySD), and synthesisr)
• More and more these tools come with graphical user interfaces called Shiny apps, that make them accessible and usable to non-coding end user
• For example, litsearchr: https://elizagrames.shinyapps.io/litsearchr/

​

Resources for learning R for evidence synthesis

• litsearchr: "an R package to quasi-automate search strategy development for systematic reviews": https://elizagrames.github.io/litsearchr/
• Library Carpentry Lesson: https://carpentries-incubator.github.io/lc-litsearchr/
• swirl: https://swirlstats.com/
• R for Data Science: https://r4ds.had.co.nz/

​

Resources and readings for automation in ES

SR Toolbox: http://systematicreviewtools.com/

SR Accelerator: https://sr-accelerator.com/#/

UNC SR Automation Guide: https://guides.lib.unc.edu/automation

Evidence Synthesis Hackathon: https://www.eshackathon.org/

Haddaway, N. R. (2015). The use of web-scraping software in searching for grey literature. Grey J, 11(3), 186-90. https://www.researchgate.net/publication/282658358_The_Use_of_Web-scraping_Software_in_Searching_for_Grey_Literature

Haddaway, N. R., Callaghan, M. W., Collins, A. M., Lamb, W. F., Minx, J. C., Thomas, J., & John, D. (2020). On the use of computer‐assistance to facilitate systematic mapping. Campbell Systematic Reviews, 16(4). https://doi.org/10.1002/cl2.1129

Marshall, I. J., & Wallace, B. C. (2019). Toward systematic review automation: A practical guide to using machine learning tools in research synthesis. Systematic Reviews, 8(1), 163. https://doi.org/10.1186/s13643-019-1074-9

​

​

​

​

11

Let's demo a tool or two

12

Questions?

13