Building a Theory of Change for Your Research

AKA: one tool for maximally kicking global problems in the butt

While we wait to start, please start filling in the worksheet

Note: I might share a recording of this session with some people or on YouTube.

What are some ways research projects can go wrong?

A lot of research is (at least!) one of the following...

• Simply inaccurate.
• Unclearly written.
• Accurate. Clearly written. But irrelevant to any important topics.
• Accurate. Clearly written. But irrelevant to any important decisions.
• Accurate. Clearly written. But hard for key decision-makers to use.
• Accurate. Clearly written. But never seen by key decision-makers.

Other common “failure modes”

• Good research & impact, but you could’ve done even better research
• Good research & impact, but you’d have more impact via something other than research
• Good research & impact, but not building you up to excellence

Learning objectives

Basically: Don’t do those things!

Let this ship be almost all past researchers, not you

Learning objectives

This workshop should make you able to…

1. Understand the concepts “theory of change” (ToC), “paths to impact”, and “backchaining”
2. Emotionally appreciate the value of those things & of not settling for “filling a gap in the literature”
3. Understand & generate multiple types of path to impact research can have
4. Understand why & how to think about decision-makers when planning research
5. Suggestion for later in Sep: Sketch a project plan for a research project?

There are many things we won’t cover, such as how to do high-quality, efficient research or how to write/communicate well. Resources relevant to such topics can be found here.

Theory of change

• “defines long-term goals and then maps backward to identify necessary preconditions [...] Theory of Change explains the process of change by outlining causal linkages in an initiative, i.e., its shorter-term, intermediate, and longer-term outcomes.” (Wikipedia)

Alternative framing:

• What ultimate goal(s) should you prioritise? What path(s) for getting there should you prioritise? What are the key steps in those paths? How can you maximise the chance you successfully take all those steps?
• Note: Doesn’t have to be about direct impact of this project itself!

Some broad paths to impact

• Impact via the work itself (or associated presentations, conversations, etc.)
• See Dafoe’s “product model of research”
• Testing fit
• For future research roles
• For other roles
• Building career capital
• See Dafoe’s “field building model of research”

Bad examples (don’t do this!)

“I’ll investigate how this worm’s behavior changes when we increase its exposure to sunlight.”

• Probably not a high-priority topic!

“...This is important because prior research has considered only other worms / considered only changes in temperature / raised two hypotheses that this could distinguish between.”

• But what important decisions will be improved due to your research?!
• Aim much higher than just “fill a gap in the research”!

Meh examples (probably don’t do this!)

“I want to reduce x-risk. Nuclear weapons create x-risk. I live in France. I’ll look into French disarmament movements.”

• France’s arsenal isn’t one of the biggest factors!

“I want to reduce x-risk. Nuclear weapons create x-risk. I’ll look into how valuable reaching zero nuclear weapons globally would be.”

• That goal is probably too intractable anyway!
• Have you checked if there’s anyone who could do something big about this and whose mind you could change?

Meh examples (probably don’t do this!)

“I’ll look into what interventions could best reduce the chance of huge increases in arsenal size. Then I’ll just publish all the thoughts I have… somewhere…”

• Good topic
• But you should:
• Figure out what decision-makers are relevant
• figure out key implications for those people
• highlight those implications clearly & up-front
• publish where they’ll see
• share with them actively

Good example (but simplified, & not the only type of good example)

“Want to reduce x-risk. Best-suited to work on politics, policy, natsec, or similar. Should try each of nuclear risk, AI governance, and biosecurity policy. I’ll start with nuclear risk.

Who are key decision-makers on that issue who might talk to me? Some EA-aligned funders, policy people, and researchers. Ask what they think are key risk pathways, decisions, & uncertainties.

…Ok, they say a key uncertainty is how to best reduce the chance of huge increases in nuclear arsenal size.

I’ll research that & figure out key implications for what these decision-makers should do. Then write that up in a way they can see the usefulness of right away. Then publish on EA Forum & share directly with the decision-makers.

They can then fund/do advocacy, diplomacy, or further research to cause these interventions to happen.”

Good example (but simplified, & not the only type of good example)

Path to impact:

1. Increase clarity on what actions would best reduce the chance of nuclear arsenal increases
2. Share with EA-aligned funders, policy people, and researchers
3. More advocacy, diplomacy, or research tailored to reducing the chance of nuclear arsenal increases
4. Lower chance of nuclear arsenal increases
5. Lower chance of extremely large nuclear war
6. Reduced x-risk

What are some ways the same project could also help with testing fit & building career capital?

Good example (but simplified, & not the only type of good example)

“Backchaining” version of that path to impact via the work itself:

• Want to reduce x-risk
• Maybe focus on reducing nuclear risk? (tbd)
• Maybe focus on reducing the chance of extremely large nuclear war
• Maybe focus on reducing the chance of huge nuclear arsenal increases
• Maybe focus on helping EA-aligned funders achieve that goal
• Maybe increase their clarity on what actions would best achieve that goal

Y’all with me so far?

(1-min Q&A break)

Now 3 minutes to do Q1-4 of the worksheet

Then 2 minutes to discuss 1:1 in breakout rooms

Please let me know if you have questions / confusions!

Developing a ToC could help you make better decisions about...

• which research questions/directions to pursue
• which sub-questions to focus on
• how long to spend on a given line/piece of research
• how to frame & write any output(s)
• whether & how to disseminate your findings
• whether, when, & how to assess progress & impact

...basically, it can help you avoid the failure modes mentioned earlier

Relevant (but over-simplistic) quote: “Plans are worthless, but planning is essential.”

Some broad paths to impact via the work itself

What kind of decisions/activities can you influence?

• Other research
• Funding
• Policies
• Careers
• Entrepreneurship & org strategy
• Technology R&D & deployment (e.g., AI, bioengineering, clean meat)
• Other projects/programs

(Via influencing “final” decision-makers/actors or people who can influence them.)

Can consider a similar breakdown of stakeholders. Also EA vs non-EA.

Considering stakeholders

To illustrate one way to approach this, here’s an excerpt from a database I made (originally to help guide various research project decisions):

Now 3 minutes to do Q5-7 of the worksheet

Then 2 minutes to discuss 1:1 in breakout rooms

Please let me know if you have questions / confusions!

Dimensions along which theories of change (or ways of developing them) differ

• Backward chaining vs forward chaining
• Explicit/foreseeable paths to impact vs speculative/curiosity-driven
• Other-directed vs self-directed (or reactive vs proactive)
• Applied vs fundamental/basic

(The 2nd dimension is probably somewhat correlated with the others.)

Which is best?

• Varies between people and projects
• Often best to do a mix / multiple
• Probably often best to move from the left to the right over time

Some takeaways from me

1. Making a massive positive expected impact on the world is hard
2. Important cause areas are complicated
3. Almost all research has way less impact than the impact many of you could probably have
4. “This project idea seems relevant to [important cause area]” is a great start, but not enough
5. Explicitly thinking about ToCs & decision-makers could supercharge your impact
6. There are many types of ToC and ways of developing them
7. Often you shouldn’t optimise for direct impact; often the ToC will flow mostly/entirely through improving your own later work

Recap of learning objectives

This workshop hopefully made you super mega able to…

• Understand the concepts “theory of change” (ToC), “paths to impact”, and “backchaining”
• Emotionally appreciate the value of those things & of not settling for “filling a gap in the literature”
• Understand & generate multiple types of path to impact research can have
• Understand why & how to think about decision-makers when planning research
• Suggestion for later in Sep: Sketch a project plan for a research project?

Now 3 minutes to do Q8-10 of the worksheet

Then 2 minutes to fill in the feedback form

Then 2 minutes to discuss 1:1 in breakout rooms

​

Please let me know if you have questions / confusions!

Questions?

Appendix: Additional slides from previous versions of this workshop

Some broad paths to impact via the work itself

Elaboration here

What kind of things can you change about actors?

• Advance the frontiers of knowledge
• Advance the frontiers of knowledge within a given community
• Bring more people to those frontiers of knowledge
• Change things other than knowledge (e.g., inclination, attitudes)

ToC example: Happier Lives Institute (in 2019)

~ToC example: Rethink Priorities

(This is just one high-level summary. RP has more detailed ToCs for specific areas of work, and especially for specific projects.)

ToC example: Convergence Analysis (in 2020; excerpt)