Oompa-Loompas (AI) for Systematic Literature Review

Marcel Tkacik

PhD student @ Prague University of Economics and Business

Data Science Manager, GAI Lead @ PwC Austria

�

Workshop website: https://www.marceltkacik.me/teaching/ml-for-literature-review

Motivation

• Doing a systematic literature review (LR) can be a headache. Especially with a large number of papers.
• Fortunately we have a framework that can guide us through the LR process (PRISMA)
• We also now have a software that can speed up one of the parts of the process (ASReview), it gives you the most relevant abstracts to the front of your workload
• We will also see how to extract knowledge from papers into table
• These steps are not relevant only for writing LR paper but also only when you are scanning for literature and getting domain knowledge

How to do systematic LR

We have to be systematic!

​

4 stages

1. Identification
2. Screening
3. Eligibility assesment, notes on the reason for rejection
4. Final inclusion, extracting knowledge into table (when possible)

Pic source

PRISMA - history

• Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) is a guideline on what to do and how to report items from papers to systematic review
• The original PRISMA statement, published in 2009, was designed to help systematic reviewers transparently report why the review was done, what the authors did, and what they found.
• It has been originally developed and is mostly used for (health) interventions
• Has been replaced by PRISMA 2020 statement – checklist, explanations and elaboration,and flow diagram
• Everything is here: The PRISMA 2020 statement: an updated guideline for reporting systematic reviews�

PRISMA tells you how to write your paper!

• Your LR paper should closely follow this checklist
• The checklist guides you what to do and what to report
• http://www.prisma-statement.org/documents/PRISMA_2020_checklist.pdf
• The search and retrieval process can be best described by a workflow diagram
• Both abstract and main paper checklist can be filled-in online here https://prisma.shinyapps.io/checklist/

PRISMA workflow - abstracts

You should fill this diagram for any LR you do

See the note on automation tools

Screening part is very time-consuming

• Usually only dozens of papers are relevant for our LR/our research
• But because we don’t know which ones, we have to screen a large number of abstracts to identify papers that might be relevant/worth reading
• What could help?
• In the past we would try to make clever queries to search for relevant literature
• Unfortunately this never really worked too well (terms often tend to diverge even in one field, easy to miss some papers)
• Fortunately now we have machine learning

Go away long queries… (in the past we tried to limit the number of abstracts to be screened to be as low as possible, which sometimes resulted in very complicated queries)

• Appendix 1�Search code�1)Treatment (abstract/title/keywords):
• “Venture capital” OR VC investment*” OR “VC investor” OR “VC fund*” OR “VC financ*” OR “VC*backed” OR “seed funding” OR “initial VC” OR “funding round*” OR IVC OR GVC “government* venture capital” OR IVC OR “independent venture capital” OR CVC OR “corporate venture capital” OR “syndicated investments”
• 2)EU MEMBER STATES (28 countries)
• AND (anywhere)
• Austria* OR Italy* OR Belgium* OR Latvia* OR Bulgaria* OR Lithuania* OR Croatia* OR Luxembourg* OR Cyprus* OR Malta* OR Czech* OR Czech Republic* OR Netherlands* OR Holland* OR Denmark* OR Poland* OR Estonia* OR Portugal* OR Finland* OR Romania* OR France* OR Slovakia* OR Germany* OR Slovenia* OR Greece* OR Spain* OR Hungary* OR Sweden* OR Ireland* OR United Kingdom* OR “European Union” OR “EU” OR “Europe” OR “European”
• 3) FIRM CHARACTERISTICS
• AND (anywhere)�“start up” OR “start-up” OR “startup” OR “young firm” OR “new firm” OR “innovative firm” OR “high tech” OR “high-tech” OR “IT” OR “ICT” OR “tech” OR “new technology-based firm” “NTBF” OR “knowledge intensive” OR “innovation”
• 4) OUTCOME VARIABLES: AND (anywhere)
• “firm growth” OR “firm performance” OR “economic performance” OR revenue* OR turnover OR sales OR “value added” OR “value-added” OR employment OR employee* OR productivity OR “scale-up” OR “scaleup” OR “scaling up” OR assets
• 5) METHODS
• AND (anywhere)�“counterfactual evaluation” OR “treatment effect” OR “causal effect” OR “propensity score”
• OR “matching” OR “regression discontinuity” OR “dif-in-dif” OR “difference-in-differences” OR “difference in differences” OR “instrumental variable*” OR “identification strategy” OR “GMM” OR “Generalized Method of Moments”

Today we will show how to overcome this

Machine Learning

• How can machine learning help us in this process?
• It’s not gonna extract the knowledge from us to the tables, neither it’s going to write the paper for us
• But it’s going to help us in the screening phase
• What is machine learning actually?
• These are methods that can automatically “learn” (find, explore, estimate) rules between target and feature when only these two are provided
• In other words it can find patterns in the data

Machine Learning

• What would be the example of a pattern in LR?
• We could employ it on my Scopus
• It would look at the history of articles I read and could recommend me new articles based on what I read the most in the past
• E.g. when it “learns” on my past behavior, it would see that I like (spend most time) on papers that are talking about machine learning, natural language processing, human behavior, gender associations, top managers and upper echelon theory
• At the same time it would learn that I never read about material science, aerospace enginneering or knitting
• The machine would then read (process) new articles that I haven’t read yet and could recommend me (in terms of probability) which articles I might like the most
• This is e.g. now employed on Google Scholar – let’s see

Pitfalls

• This already sounds great!
• Where is the potential problem?
• When we are doing literature review, we very often don’t know what we are looking for… We are exploring new research
• If we knew exactly what we want, we wouldn’t have to do all these steps
• So when we don’t know what we are looking for or we are exploring new research/domain, classic ML might not be that helpful
• We need somethig that is more reactive and learns more adaptively
• Fortunately we have it and it’s called Active Learning

Active Learning = human interacts with ML algorithm

• Active Learning is a special case of machine learning where the algorithm interacts with the user to get / query him to label new data
• ASReview is a prime example of active learning
• Every time you rate an abstract, it learn (updates) it’s knowledge about what you might be interested in and reshuffles your workload
• Paper about ASReview has been published in Nature Human Intelligence this year
• It really learns after every abstract decision, we will see it in action soon!

• ASReview has been experimentally shown to decrease the reviewing time by over 90% on most applications
• Note two things:
• When we start, you will only see title and abstract – why?
• You decide when to stop reviewing, we usually stop after we have 20-30 irrelevant abstracts

Biases and publication bias

• The reason why we only see the the title and abstracts is a bias
• We might be primed to rate abstracts more relevant when there are famous authors, famous journal, English-sounding names and so on
• But only the text of the abstracts itself should matter for our decision!
• I have to admit I am sure I was also under influences of many biases when I was screening abstracts manually before ASReview
• Special type of bias concerning the papers is publication bias – top journal in virtually all fields are much more likely to publish studies that report significant results (usually along with high magnitude of effects)
• If you would only search in top journals, you might seriously overestimate the effects in LR. In lower tier journals the effects reported are usually lower

How to stop screening?

• Time-based strategy: If you choose a time-based strategy, you decide to stop after an x amount of time. This strategy can be useful when you have a limited amount of time to screen.
• Data-driven strategy: When using a data-driven strategy, you e.g. decide to stop after an x amount of consecutive irrelevant papers (this number can be found in the statistics panel). Whether you choose 50, 100, 250, 500, etc. is dependent on the size of the dataset and the goal of the user. You can ask yourself: how important is it to find all the relevant papers?
• Mixed strategy: Another option is to stop after an x amount of time unless you exceed the predetermined threshold of consecutive irrelevant papers before that time.
• Live thread on this topic here

Case study: Experimental evidence on gender beliefs

• We want to run an experiment on how gender beliefs (especially beliefs that men have about women) affect decision in organizational/work settings
• But we have no domain knowledge of gender experiments! I only know about 4 or 5!
• This also means I am quite sure we don’t know what we are looking for exactly..
• What to do?
• Let’s search Scopus and use ASReview and Rayyan to help us find what we are interested in!

First – get your articles by searching in Scopus

• You can access Scopus advanced search via VSE on this link https://library.vse.cz/resources/scopus/
• Then export your articles in RIS or Bib format, you might have to join them later in an app if it’s too many of them

Next steps

• Step 2: Screening – use ASReview Lab (techie people usually like this more) or Rayyan (easier)
• Step 3: Assess papers for eligibility – define inclusion/rejection criteria beforehand
• Step 4: Extract knowledge, you can use chatGPT to assist you

Another case study: Pantea & Tkacik (2020):Venture Capital and Growth of Young High-Tech Firms in the EU: A Systematic Review of the Empirical Evidence �

• Before ASReview: this is just the search code (query)
• Appendix 1�Search code�1)Treatment (abstract/title/keywords):
• “Venture capital” OR VC investment*” OR “VC investor” OR “VC fund*” OR “VC financ*” OR “VC*backed” OR “seed funding” OR “initial VC” OR “funding round*” OR IVC OR GVC “government* venture capital” OR IVC OR “independent venture capital” OR CVC OR “corporate venture capital” OR “syndicated investments”
• 2)EU MEMBER STATES (28 countries)
• AND (anywhere)
• Austria* OR Italy* OR Belgium* OR Latvia* OR Bulgaria* OR Lithuania* OR Croatia* OR Luxembourg* OR Cyprus* OR Malta* OR Czech* OR Czech Republic* OR Netherlands* OR Holland* OR Denmark* OR Poland* OR Estonia* OR Portugal* OR Finland* OR Romania* OR France* OR Slovakia* OR Germany* OR Slovenia* OR Greece* OR Spain* OR Hungary* OR Sweden* OR Ireland* OR United Kingdom* OR “European Union” OR “EU” OR “Europe” OR “European”
• 3) FIRM CHARACTERISTICS
• AND (anywhere)�“start up” OR “start-up” OR “startup” OR “young firm” OR “new firm” OR “innovative firm” OR “high tech” OR “high-tech” OR “IT” OR “ICT” OR “tech” OR “new technology-based firm” “NTBF” OR “knowledge intensive” OR “innovation”
• 4) OUTCOME VARIABLES: AND (anywhere)
• “firm growth” OR “firm performance” OR “economic performance” OR revenue* OR turnover OR sales OR “value added” OR “value-added” OR employment OR employee* OR productivity OR “scale-up” OR “scaleup” OR “scaling up” OR assets
• 5) METHODS
• AND (anywhere)�“counterfactual evaluation” OR “treatment effect” OR “causal effect” OR “propensity score”
• OR “matching” OR “regression discontinuity” OR “dif-in-dif” OR “difference-in-differences” OR “difference in differences” OR “instrumental variable*” OR “identification strategy” OR “GMM” OR “Generalized Method of Moments”

​

Thank you for your attention!

• All materials are on the website:

​