1 of 75

Presentation for EA Anywhere, 5 Nov. 2023

Youtube video: The Unjournal: Bridging the gap between EA and academia

David Reinstein - Founder and Co-Director

Unjournal.org: Our full explanation and progress

These slides are linked at bit.ly/ujpresents; see speakers’ notes for more details

Also see

Intro - The Unjournal Presentation for March 25 Event

2 of 75

What is The Unjournal?

The Unjournal is not a journal.

We coordinate and fund the public evaluation of research projects in any format.

We’re building an open, sustainable system

for evaluation, feedback, ratings, and assessment.

Our initial focus is quantitative work that informs global priorities,

especially in economics, policy, and other social sciences.

Links: Unjournal.org, An Introduction to The Unjournal

Output: unjournal.pubpub.org

3 of 75

“Academic peer review” (background)

In Economics:

~A ‘working paper’ is publicly released

The ‘publication’ (review) process takes 6 months to 10 years

At the end, we only know “which journal it ‘landed’ in”

4 of 75

Main ingredients

Research Submission/Identification and Selection

Paid Evaluators (AKA 'reviewers')

Eliciting Quantifiable and Comparable Metrics

Public Evaluation

Linking, Not Publishing

Financial Prizes

Transparency

5 of 75

Our Theory of Change

6 of 75

To believe The Unjournal has value one must believe...

Research matters

Rigorous prioritization research can positively influence funding, decision-making, and/or policy.

Rigor and expertise add value.

(Peer) review & evaluation can add value to research (and research-use)

The status quo peer review system is suboptimal

Academic publishing has substantial room for improvement and/or
Global-priorities-relevant research world would benefit from more scrutiny.

The UJ’s approach can succeed

7 of 75

EA Research

High focus on impact
High flexibility/agency/agility
High coordination on innovations
Limited subject matter expertise
Limited external credibility
Limited formal feedback, evaluation, or quality-control processes

Academic Research

High general resources
High subject matter expertise
High prestige and credibility
Limited incentive/funding for impact
Limited flexibility/agency
Inefficient, rent-seeking publishing ecosystem
Limited ability to coordinate innovation

Commission Direct Public Evaluation

The Unjournal Collaboration

Focus on impact
Flexibility, agency, agility, innovation
Funding to change incentives
Connection to resources
Subject matter expertise
Prestige and credibility

8 of 75

Our Approach: leveraging problem synergy

EA needs research, but EA research has problems.

Esp.: rigorously answering the question “what is most impactful?”.
EA research can struggle with rigor and external credibility.

Academia could be highly valuable to EA.

It conducts “impactful-adjacent” research.
It has immense resources (~2.5% of US GDP), expertise, and cachet.

“Peer review” is broken in several ways.

Solution: Direct public evaluation of (global-priorities-relevant) research

Addresses many limitations of the outdated “journal system”
Leverage this to shift attention and resources towards impact

→ The Unjournal is funding, organizing, and scaling this up. See our full ToC here.

We can consider other EA research questions/goals later.
We need to be able to trust our research and know which research/orgs to fund.
We want the research (into priority causes/areas) to be credible outside EA, to have influence.

[Maybe hard to convince you?]
“Academic journals”

In different ways in different fields (‘all unhappy families’). In economics, it is slow, inefficient, noisy, opaque, and uninformative to research users/policymakers. It takes rents and puts up paywalls. It encourages ‘journal shopping’ and game-playing. It discourages open science and innovative formats.
“Just commission research to be evaluated, rated and discussed” is a fairly obvious and simple solution.
But academia is stuck in a bad equilibrium

More on all this later...

focusing this on impactful research, nudging academia towards impact
nudging EA research towards rigor and credibility,
and fostering EA/academia connections

We’ve made progress already

(growing our team, building tools and system, communicating this and increasing our exposure)
We want your feedback on pivotal choices
We want to engage with researchers and research-users

9 of 75

10 of 75

Impactful Research and Evaluation:

What is it and why does it matter?

11 of 75

How do we ‘have a positive impact’?

Define a moral framework. What matters?

(E.g.) human and animal well-being over the long run

Work towards success according to that framework.

How do we make progress on the things that matter?
Which activities are most likely to make things better?

12 of 75

How can research support impact?

Research needs to produce true, useful information which enables better decision-making, driving choices and behavior that lead to better outcomes.

This can occur through:

Influencing resource allocation, i.e., via funding decisions
Affecting the nature of policies or other interventions
Providing models and logical arguments to improve decision-makers’ thinking

→ We focus on global-priorities-relevant research.

13 of 75

Brief… some of Unjournal’s paths to impact

==

14 of 75

Prioritizing research: existing frameworks

EA “Cause Areas”, INT framework. Ex.: 80k problem profiles, OP focus areas.

But cause prioritization ≠ research prioritization!

Crux/pivotal issues: uncertainty and importance
Value of Information: how much would a decisionmaker pay for the information gain (generated by the research) prior to making a decision?
Indirect evidence: researcher credibility and track-record

15 of 75

CF: GPI framework

Foundations

Moral theory, decision theory, epistemology

“Theory”-building to support intervention design

E.g., Behavioral research on altruism,

game theory & peace building

Empirical measurement of interventions & outcomes

E.g., development economics, monitoring & evaluation, cost-benefit and predictive modeling

Increasingly relevant to UJ focus

16 of 75

What global-priorities-relevant research does Unjournal (currently) cover?

Fields: “Human behavior and its consequences”
Economics and quantitative social science (psychology, political science, etc.)
Business/Policy, forecasting, cost/benefit, etc.

Not: Philosophy, computer science (AI interpretability), animal behavior, pure math

Approaches
Empirical measurement (evidence)
~Theory/modeling, methodology with direct applications to policy/prioritization

Not: Pure theory, research inputs, shallow reviews, informal discussion

Sources and formats
Academic-aimed work

Working papers/preprints; ideal: notebooks, dynamic formats etc.
Journal-published ‘under-evaluated’ work

Our ‘rigorous policy/non-academic papers stream’ (10-20%)

4. Causes/outcomes including… →

17 of 75

Our “Field Specialist groups” �Updated 3-Nov-23, see “our team” for more

GH&D

Dev Econ

Economics/welfare

Psychology and attitudes

Innovation, meta-science

Catastrophic risks & AI gov.

Environmental Economics

Building:

Animal Welfare,

Social impact of tech., Macro/growth/finance,

LT trends and demographics

18 of 75

Why should research be evaluated?

19 of 75

The value of journals

Journals are not really publishers.

Journals are evaluators.

Journals offer quality control, credibility, prestige.

We have arXiv (and RePEc etc.) for that.

20 of 75

The value of evaluation

Rigor & Quality Control. Researchers should receive feedback and be held accountable to high standards of logic and evidence.

21 of 75

The value of evaluation

Credibility, domain, usefulness. Research-users want to know how much to trust research, update their beliefs, & adjust their decisions in different contexts.

(Without validating it all themselves)

22 of 75

The value of evaluation

Prioritizing research. We need a way to choose which researchers and organizations should receive more funding.

23 of 75

The state of the art in EA research evaluation

Ad-hoc internal discussion.

Ad-hoc external feedback.

EA Forum posts (academics rarely engage, technical posts get fewer comments)
Blogs
Widely shared Google docs

Some private red-teaming takes place

Occasional formal academic publications*

24 of 75

Weak underbelly of “EA/GPI/adjacent research”?

HLI on Deworming: Discrepancy between GiveWell’s data and model

25 of 75

26 of 75

From authors’ response to Alex Bates’ evaluation

Bates:

… A cost of $86m to mitigate approximately 40% of the impact of a full-scale nuclear war between the US and a peer country seems prima facie absurd, and the level of exploration of such an important parameter is simply not in line with best practice in a cost-effectiveness analysis (especially since this is the parameter on which we might expect the authors to be least expert). … these issues could potentially reverse the authors’ conclusions, and should have been substantially defended in the text.

Authors:

We agree that this estimate from the published work is likely low and have since updated our view on cost upwards. The nuclear war probability utilized does not include other sources of nuclear risk such as accidental detonation of nuclear weapons leading to escalation, intentional attack, or dyads involving China.

27 of 75

Other ‘wins, rethinks, and potential’

CBT therapy: little long-term effect? See UJ evaluations of “The Comparative Impact of Cash Transfers and a Psychotherapy Program…”

Evaluations of “Artificial Intelligence and Economic Growth”: Proofs checked and correction → quality standards and legibility

Animal welfare: attitudes, interventions, markets – almost no rigorous work in economics+

Major grant to water quality largely based on un-reviewed paper?

28 of 75

Apr. 2022 Evidence Action’s Dispensers for Safe Water program “… a remarkable new investment of up to $64.7 million.” “recommended by GiveWell… and funded by Open Philanthropy”

“Underpinned by rigorous research by Nobel Laureate Michael Kremer and colleagues…”

a recent meta-analysis by Michael Kremer … shows that water treatment reduces the odds of mortality of children under five, from all causes, by around 25%.

Release cites:

“Social Engineering: Evidence from a Suite of Take-up Experiments in Kenya”; 2011 working paper without evident peer review

“Water Treatment and Child Mortality: A Meta-analysis and Cost-effectiveness Analysis”; WP updated 2023, no evident peer review

29 of 75

Why not just use academic publishing?

30 of 75

31 of 75

Problems with academic publishing:

It rewards ‘flexing’ and research ties, not realism and impact

Rents & barriers to research access.

Static, limited formats: “the PDF prison”.

Inefficient, convoluted systems diverts researcher effort.

Private evaluation keeps users in the dark, slows down feedback loops.

As well as concerns linked to priorities (and gripes) of academic editors and referees
Expensive, rent-seeking publications: either paywalls or pay-to-publish.
A barrier to transparency and robustness: hard to publish dynamic documents/notebooks; or unfolding explanations. Cannot update the publication with new evidence, corrections or links. “PDF prison” was coined by Willem Sleegers, I believe.
Researchers have to focus on gaming the system and journal-shopping, as well as engaging in tedious arcane processes to format their work for different journal styles
For research users too: all they learn is ‘which tier journal it was published in’ … not why, nor what it’s strengths and weaknesses were. E.g., if it was seen as highly credible in the real-world claim and evidence, but not seen as innovative/clever or didn’t help build research methodology → It may be exactly what policymakers want to rely on

�All of this frustrates promising researchers, driving them out of academia, and discourages non-academic researchers from engaging in any rigorous evaluation

32 of 75

Why are we still doing this?

33 of 75

How do we solve these problems?

34 of 75

Towards a New Equilibrium: The Unjournal

Commission evaluations which are:

Modular
Public
Paid
Credible
Quantifiable
Impact-Oriented

35 of 75