STAMP MEMORIAL LECTURE

Designing & Evaluating Digital Interventions for Social Impact

​

​

SUSAN ATHEY

Economics of Technology Professor, Graduate School of Business

Faculty Director of the Golub Capital Social Impact Lab

Faculty Affiliate, Institute for Computational and Mathematical Engineering

Senior Fellow at the Stanford Institute for Economic Policy Research

We partner with social sector organizations to develop and apply technology solutions for social impact.

CO-CREATION �WITH IMPLEMENTING ORGANIZATION

Clear path to impact

CO-CREATION

WITH IMPLEMENTING ORGANIZATION

Clear path to impact

Interdisciplinary collaboration to solve problems

CO-CREATION

WITH IMPLEMENTING ORGANIZATION

Clear path to impact

Interdisciplinary collaboration to solve problems

​

Both tactical and foundational research

CO-CREATION

WITH IMPLEMENTING ORGANIZATION

WHY DIGITAL TECHNOLOGY?

Low cost, scalable

WHY DIGITAL TECHNOLOGY?

Low cost, scalable

Academic lab can prototype

WHY DIGITAL TECHNOLOGY?

Low cost, scalable

Academic lab can prototype

General purpose technology

WHY DIGITAL TECHNOLOGY?

Low cost, scalable

Academic lab can prototype

General purpose technology

Personalization & access

WHY DIGITAL TECHNOLOGY?

Targeted Digital Interventions

With attention to equity-efficiency tradeoffs

SIX RECENT

IMPLEMENTATIONS

TARGETED CALL TIMES FOR

FARMERS

IMPLEMENTATION I:

Collaborated with agricultural advisory service in India

​

Free agricultural advice on crop cultivation, livestock and fisheries management.

​

Calls 1 million farmers on a weekly basis

​

​

# of Calls Allocated

Very Likely to Pick up

Unlikely to Pick up

Use farmer characteristics to predict how likely to pick up each hour

Who to call at what hour under capacity constraints?

Only 1 spot left – who gets called at their best predicted call time?

Max. Calls per Hour

Impact of Personalization in Call Times

Value of Targeting

• Personalizing call time increases prob. of picking up 8%.
• Impact: potential to reach 26k-33k additional farmers with ed. content.

Equity-Efficiency Tradeoff

• Capacity constraints: not everyone gets their “right time.”
• Women farmers lower average engagement.
• Can improve engagement from women by 9% if we reduce men’s engagement by 1.7%.

​

Shocks/external validity

• Targeted policy underperforms in practice.
• A farmer's “right time” shifts from week to week through season.
• Weight more recent data for better performance.

8% difference

Athey, Cole, Nath, and Zhu (2023 WP)

More men get “right time”

More women get “right time”

IMPLEMENTATION II - III:

Built recommendation system to rank children’s stories in English-reading application to help kids learn to read in India.

​

Created a contest with a leaderboard that helped children develop habits.

​

How can personalization and gamification improve children’s engagement with educational apps?

HELPING CHILDREN IN INDIA READ

Which stories to show first?

Historical Data

Personalized story ranking

1

2

3

User 1

User 2

User 3

Ranking

1

Editor’s Pick

Same story ranking for everyone

2

3

User 1

User 2

User 3

Ranking

Baseline

Results

Compared to editorial policy, personalized recommendations increase:

​

• Engagement, usage
• Story diversity (niche content is shown more)

​

​

Heterogeneous treatment effects (of algorithm) show more improvements for:

​

• Niche users (consuming more non-popular content)
• Heavy users (long history or more activity)

Agrawal, Athey, Kanodia, and Palikot (2023a WP)

TREATMENT

CONTROL

New users randomized into:

​

Treated users enter contest to win a set of books.

​

• 79% (SE 14%) more completed stories
• 48% (SE 14%) higher Engagement
• HTE: Greater impact for younger students, poorer schools

After contest, treated users continue higher engagement – habit formation.

​

• 35% (SE 17%) more completed stories
• 34% (SE 17%) higher Engagement
• 8% (SE 1%) higher retention

​

• 34% (SE 15%) increase in difficult stories

During Contest

After Contest

100-day contest created lasting habits and learning gains.

Agrawal, Athey, Kanodia, and Palikot (2023b WP)

DIGITAL ASSISTANT FOR NURSES

IMPLEMENTATION IV:

Developed a digital tablet application that augmented nurses’ counseling of patients in a hospital in Cameroon.

​

How can getting patients the right info support informed decision-making?

​

How can subsidies be targeted to maximize impact per dollar?

​

​

Support App for Contraceptive Methods Discussion

Discussion Structure

Shared decision-making

App-Ranked: app prioritizes one method at a time based on interview responses

vs.

Individual decision-making Patient-Ranked: quick overview in random order, then patient prioritizes

Sample:

Age 15-49  

Want to wait > 1 year before next pregnancy

LARCs

SARCs

Athey, Bergstrom, Hadad, Jamison, Ozler, Parisoto and Sama (2023)

Treatments

$7.00 (4000 CFA)

App-ranked

Patient Ranked

IUD, Implant Price

Recommendation Method

$3.50 (2000 CFA)

$1.75 (1000 CFA)

Free

App-ranking leads to more discussion.

• Consistent with rational search models with costly search (Athey & Ellison, 2011).

• Discounts are effective for increasing LARC adoption.
• At full price, app-ranked triples LARC uptake.
• App-ranked also makes clients less price-sensitive, possibly by increasing willingness to pay above full price.

LARC Adoption Within 100 Days

Next: How can we use personalization to

1. Improve LARC uptake further?
2. Balance $$ cost with LARC uptake?

App-ranking best for all subgroups

Free LARCs for more vulnerable clients (young or recently pregnant).

Half price for

everyone else

(cost-effectiveness).

Data-Driven Personalization of Discounts: �An Adaptive Experiment

Methods: Policy estimation - Athey & Wager (2021); Zhou, Athey, & Wager (2023); Software: policyTree

Contextual bandits – Dimakopoulou, Zhou, Athey & Imbens (2019), Krishnamurthy, Zhan, Athey & Brunskill (2023), Carranza, Krishnamurthy, & Athey (2023), Offer-Westort, Athey, & Rosenzweig (2023), Zhan, Hadad, Hirschberg, & Athey (2021), …

Learn who to show which price to optimize objective:

• Weighted average of financial cost of discounts and LARC uptake.

​

Adaptive experiment:

• Learn for which groups discounts have biggest bang-for-buck.
• Whether any groups benefit from patient-ranking.

IMPLEMENTATION V:

NUDGING WORKERS

TO SHARE CREDENTIALS

Can non-standard credentials help disadvantaged workers, like those from developing countries or without a college degree, get better jobs?

​

Developed a feature to simplify sharing credentials from Coursera on LinkedIn, and nudge workers to use it.

Learner

Not now

Share now

Do you want to boost your career? Only [XYZ]% of learners manage to complete [course name] on Coursera and get a certificate. Let everyone know you did it! Add the certificate to your LinkedIn profile in just 2 clicks.

Not now

Share now

Looking to boost your career?�LinkedIn profiles with credentials receive 6x more views! Don’t waste your hard-earned certificate!�Add the certificate to your LinkedIn profile in just 2 clicks.

PS. This is your last reminder.

Nudge 1

All treated users get Nudge 1, which offers a simple, 2-click process.

​

Treated users who didn’t click get Nudge 2, with more behavioral cues.

Nudge 2

Average treatment effects estimates

N= 37K

N= 37K

N= 37K

N= 31K

All new jobs

(>1 mo)

​

All new jobs

(>4 mo)�

New jobs in

tech or managerial

(>1 mo)

New jobs in

tech or managerial with no exp.

(>1 mo)

Share reporting

a new job

Results

In-app notifications + nudges improve prob. of finding a job.

• Impact persistent
• Treatment effect larger for the outcome of transitioning into jobs directly related to the credential, especially for workers with no related experience.

Treatment effect is driven by those with low baseline predicted employability.

Athey and Palikot (2024 WP)

Predicted Baseline Employability

IMPLEMENTATION VI:

How can we create scalable solutions to transition more women into tech jobs?

​

Created a low-cost, online program that helped women develop portfolios to demonstrate capability.

HELPING WOMEN GET INTO TECH

2 free programs for women aiming to transition to tech job

Mentoring Program

• Mentees (tech skills, but no tech job) matched to mentors (mid-career women in tech).
• 1:1 meetings for 3 months.
• Difficult to scale

Challenges Program

​

• Research team developed in collaboration with employers.
• Demonstrate skills/talent in a way that employers will recognizes.

​

• Submit 6 assignments that add up to a portfolio item, which signals relevant skills.

Two parallel experiments

Mentoring

Challenges

~ 1000 applicants after screening

Each mentor shortlists 2 applicants

160 pairs

treatment

treatment

control

control

~ 600 applicants

First 100 applicants

Admitted to the program and removed from the analysis.

Mentoring Control

Mentoring Treated

Challenges Control

Challenges Treated

Effect on having a tech job

…in the 8th month after application

• Mentoring: 13pp (SE 5)
• Challenges: 9pp (SE 4)

…across 16 months after application (Cox)

• Mentoring: 13pp (SE 2.5)
• Challenges: 17pp (SE 2)

Note: different populations for two experiments.

Conditional Average Treatment Effect – Tech Job in 12 Months

Mentoring benefits those with experience.

Challenges benefits those entering the market.

Athey and Palikot (2023 WP)

Personalization & customization work

Digital assistants improve counseling

Digital credentials enable transitions

Low cost and scalable

DIGITAL INTERVENTIONS SHOW GREAT PROMISE

THANK YOU