1 of 21

Economies of Scale for Research,�Economies of Scope for Policy

Dean Karlan

Professor of Economics and Finance, Northwestern University

Founder, Innovations for Poverty Action

2022 Africa Evidence Summit

Kigali, Rwanda

2 of 21

Two Themes

Research could be more efficient

Eg: Same data for multiple studies

For policy, all research is incremental

Analog: No one study should change the world (radically)

3 of 21

Four Research “Infrastructure” Initiatives

Ghana Panel Survey
Multi-site impact evaluations
Structured Ethics Appendix (Asiedu et al 2020)
Northwestern Feedback Project

4 of 21

Ghana Socioeconomic Panel Survey

Nationally-representative panel survey; n = ~5,000

https://egc.yale.edu/data/egc-isser-northwestern-ghana-panel-survey

Cross-university collaboration

Institute of Statistical, Social and Economic Research (University of Ghana)
Global Poverty Research Lab (Northwestern University, USA)
Economic Growth Center (Yale University, USA)

Goals

Long-term, representative panel data
Promote inductive exploration of connections between diverse areas of life and inferential analysis spanning a wide range of topics
Facilitate policy-relevant research through overlay evaluations

Now I’ll talk a bit about the Ghana Socioeconomic Panel Survey, a nationally-representative longitudinal survey with a sample of roughly 5,000 households that we’ve been fielding every 4 years since 2009. The survey is a collaboration between the University of Ghana, Northwestern, and Yale.

The survey fills an important niche since it’s the only recent, nationally-representative panel survey in Ghana. So it is uniquely equipped to descriptively chart social change in Ghana. Also, the questionnaire covers an extremely wide range of topics from education and livelihood to health and intra-household decision-making, to housing and land tenure. As a result, it’s useful for exploratory analyses of patterns across the sectors of life that would be missed by more narrowly-focused surveys. For some research questions, the Ghana Panel by itself can provide substantial leverage for identification, whereas for other questions it’s best used in combination with other data sources.

Beginning in Wave 3, we’ve been integrating field experiments into the Ghana Panel. For example, there was a randomized evaluation where parents of infants were shown a video on the importance of “infant-directed speech”. And the upcoming wave of the panel will serve as a midline in a large-scale agricultural development evaluation testing the impacts of a tropical weather forecasting service.

5 of 21

Ghana Panel design

2-stage, representative sampling strategy

Stratified selection of 334 clusters (census enumeration areas) for representativeness within the 10 (erstwhile) regions
Simple random sampling of 15 households per cluster

Followed moved and split-off households

Data collection every ~4 years

Wave 1, 2009-10, n = 5,009
Wave 2, 2013-14, n = 4,774
Wave 3, 2017-18, n = 5,669
Wave 4, 2022-23, n = TBD

To construct a representative sample, we used a 2-stage strategy. First came selection of census clusters, stratified so that the sample would also be representative of each of the country’s former 10 regions (as of 2020, these have been split into 16). For the second stage, we used simple random sampling to select 15 households from each of the census clusters.

Since geographic mobility is an important aspect of social change in Ghana, we try to track all sample households for each wave, even if they’ve moved across the country. The only exception is for households who leave the country entirely. Additionally, anytime household members split off and form new households or join others within the same community, we add them to the sample as new households. And we do the same for a randomly-selected half of splitoff households that move to a separate community from the root household.

Three waves have been completed, and the data is available online through the Yale Economic Growth Center webiste. At present, we’re gearing up for Wave 4, which has a planned launch in Fall 2022.

6 of 21

Survey modules

Household

Background
Testing
Employment
Education
Migration
Assets
Agriculture
Household enterprise
Health
Psychology
Consumption
Housing

Community

School, health, and financial institutions
Infrastructure
Land
Agricultural rental, livestock, and crop prices
Unit conversion
Extension services
Agricultural sales outlets
Shocks and conflicts
Employment and business structure
Elections
Social and political groups

7 of 21

Example Outputs #1: � Psychological distress

Found high levels of depression in rural areas

Sipsma, Ofori-Atta, Canavan, Osei-Akoto, Udry and Bradley. BMC Public Health, 2013

Motivated testing Cognitive Behavioral Therapy

CBT effective, even for those not “depressed”
Barker, Bryan, Karlan and Ofori-Atta. American Economics Review – Insights, 2022

Income shocks for youth (cocoa) 🡪 mental health impact as adult

Adhvaryu, Fenske and Nyshadham. Journal of Political Economy, 2019

An example of the type of insight that can be gleaned from the Ghana Panel is the close connection between poverty and mental health. Early explorations of Ghana Panel data showed strong correlations between poverty on one hand and psychological distress—as measured by the Kessler-10 question battery—on the other. This finding helped to motivate the inclusion of a group cognitive behavioral therapy component in an evaluation of a large-scale livelihood program.

Relatedly, the above maps are taken from a Journal of Political Economy article by Adhvaryu and colleagues that uses Ghana Panel data to explore the link between early childhood circumstances and mental health, with an identification strategy that exploits variation in global cocoa prices. Higher Kessler scores—indicated by the darker regions—mean more psychological distress which means greater likelihood of depression and anxiety. Not surprisingly, distress is generally higher in the country’s northern section, which is poorer. But the data also reveals more fine-grained variation within the two main geographic divisions (i.e., North and South), the type of prompt that can highlight areas for future studies.

8 of 21

Example Output #2: � COVID-19 Cash Transfer Evaluation

Sampling

Poorest ~1,500 households from Ghana Panel sample

Intervention

Weekly transfers of 90GHS (US$42 PPP)

Of the evaluations that have been integrated with the Ghana Panel so far, the most important in terms of intervention intensity and policy relevance has been a randomized study on the effects of mobile money cash transfers among relatively poorer households. As the pandemic began to spread through the developing world, policymakers increasingly came up against the potential tension between the dire need of many households to engage in in-person livelihood activities on the one hand and social distancing protocols on the other. Since mobile money is a central tool in Ghana’s social protection programs, we designed an intervention where recipient households would receive biweekly transfers of 90 GHS, (i.e., US$42 adjusted for PPP). This amount represents 65% of weekly household food expenditure as reported at baseline, and is significantly larger than LEAP, a flagship social protection program run by the Ghanaian government.

To test the intervention, we invited the poorest half of Ghana Panel respondents to join the study (as measured by per capita food consumption adjusted for adult equivalence) and ended up with a sample of around 1,500. All households received a baseline survey, immediately followed by the first transfer. Treatment households were then informed that they would receive 7 additional transfers, and control households that they would not receive any more transfers. We then carried out four followup phone surveys, each separated by a few weeks (while transfers were still being received), and a fifth around 8 months after the final transfer.

9 of 21

Cash transfer evaluation results

Positive impacts

~40% of transfer spent on food
Food expenditure up 8%
Income up 20%

But… social distancing up 0.08 standard deviations

Income effect persists at 8 months after last transfer

The first followup survey took place when treatment and control households had each received a single transfer. Since treatment households had been promised 7 additional transfers while control groups had been informed that they would not receive additional transfers, this survey wave allowed us to test for anticipation effects. Differences between the experimental groups during this survey were generally minimal and not statistically significant, so anticipation effects do not seem to have been influential here.

While the first transfer was carried out without a hitch, several of the later transfers were delayed, ultimately averaging 25 rather than 7 days apart from one another. These delays may mean that the impacts we calculated are underestimates, although it’s also possible to keep in mind that logistical difficulties could also lead to delays if a government agency or NGO were to scale up the program.

Despite the challenges, the overall story is positive. Food expenditure and income both increased substantially in the treatment group relative to control, with the data suggesting that almost half of the transfer value was pent on food. The impact on social distancing was smaller in magnitude, but this is likely because nearly the whole sample reported engaging in some social distancing behaviors like mask wearing regardless of the treatment. The social distancing impact seems to have been driven by respondents reporting that they had stayed home all day. Encouragingly, the economically significant positive coefficient on income was visible even eight months after the last survey with little reduction in magnitude, although too imprecisely measured to register as statistically significant.

10 of 21

Multi-Site Studies

Context matters! But not entirely.
Replicate & Coordinate

Don’t put too much weight on any one study
Pooling allows for more nuanced tests (heterogeneity, e.g.)
Coordination allows for common measurement, perhaps common design

11 of 21

The basic premise

Cause of poverty is complicated

Which is why singular-approaches often have mixed results

Work on many problems at once,

but probably not all problems

12 of 21

The Multi-Faceted Approach�(“Graduation” or “Productive Inclusion” or “Livelihood Plus” or “Cash plus”)

food aid

health services

livelihoods

social development

financial services

Pre-existing cash-transfer

13 of 21

Eight Sites of Original Graduation Tests

Honduras

Peru

Ghana

Ethiopia

Yemen*

Pakistan

India

Bangladesh*

14 of 21

Impacts on Many Dimensions

15 of 21

Niger: Project Overview

First of four-country study (also Burkina Faso, Senegal & Mauritania)
Built on top of existing cash transfer programs
Implemented by government
Cost-effective after 18 months
Psycho-social component

Film, individual AND social aspect
EG, respecting elder = continuing their work. What about respecting their adaptative skills?

Published in Nature (2022)

Bossuroy, Goldstein, Karimou, Karlan, Kazianga, Parienté, Premand, Thomas, Udry, Vaillant, Wright

16 of 21

Niger: Program built on Cash Transfer program�4-country Sahel Adaptive Social Protection Program (ASP)

		CONTROL	CAPITAL Package	PSYCHOSOCIAL Package	FULL Package
	Regular cash transfer program	•	•	•	•

Core components	Group formation and coaching		•	•	•
	Savings groups		•	•	•
	Micro-entrepreneurship training		•	•	•
	Market access facilitation		•	•	•

Psychosocial components	Community sensitization on aspirations and social norms			•	•
Psychosocial components	Life-skills training			•	•

Cash grant component	Lump-sum cash grant		•		•

Number of villages (322)		81	80	78	83
Number of sample households (4712)		1206	1191	1112	1203

17 of 21

Niger Results (18 months): �Consumption and Business Revenue

18 of 21

Structured Ethics Appendix

“A Call for Structured Ethics Appendices in Social Science Papers” by Edward Asiedu, Dean Karlan, Monica Lambon-Quayefio and Chris Udry, PNAS 2021

Too much reliance on IRB (IRBs are not on the ground!)

Majority of “ethics” issues essentially come down to staff training and organizational culture

Discussion could

Improve management of ethics issues
Improve adherence to standards
Avoid misunderstandings, incorrect assumptions

19 of 21

Structured Ethics Appendices

Policy Equipoise & Scarcity
Researcher Roles with Respect to Implementation
Potential harms to participants or nonparticipants from the interventions or policies
Potential Harms from Data Collection or Research Protocols
Financial and reputational conflicts of interest
Intellectual freedom
Feedback to participants or communities
Foreseeable misuse of research results
Any other ethical issues

20 of 21

Northwestern Feedback Project

Conferences great.
Project collaboration great.
Journal feedback when rejected, often not so great.

Northwestern Research Feedback Project for Scholars at LMIC Universities

Submit a paper
We try to find a “feedback partner”
Instead of a distant, anonymous referee report adjudicating yea/nay, we schedule a zoom meeting to get feedback, discuss the paper, etc.
Journal of Development Economics helping to recruit

21 of 21

Thank you!

Dean Karlan

dean.karlan@gmail.com