SICSS-Rutgers 2024

Day 5: Digital Experiments

June 17th - June 28th, 2024 | Rutgers University - New Brunswick

We thank the core sponsors of the Summer Institutes in Computational Social Science and the Department of Political Science at Rutgers University for their support for this event.

Today’s Goals

• Discuss digital surveys and sampling
• Gain practice designing and launching an online survey
• Discuss key elements of experimental design
• Practice designing an experiment and assessing its strengths & weaknesses

​

​

​

Experiments

Experiments in a digital age

Goal of explanation and causality

​

• Conceptual shift between prediction and explanation/causality
• “Social media mentions of ice cream are a good predictor of swimsuit sales” (Text as Data pg. 244)

Do social media mentions of ice cream cause swimsuit sales?

​

​

​

​

​

​

• In prediction, we focus on accuracy of predicting the outcome (swimsuit sales)

-> Ice cream may still be a valuable independent variable (predictor)

• In explanation, we focus on the (causal) effect of the independent variable on the outcome

-> Ice cream less useful.

​

Experiments in a digital age

Why conduct experiments: Correlation and causation

Unobserved common cause

Experiments in a digital age

What does it mean to make a causal claim about this relationship?

• We think that manipulating some treatment
• Seeing vs. Not seeing social media mentions of ice cream

​

• Causes a change in some outcome
• Buying a swimsuit

​

​

​

Experiments in a digital age

What does it mean to make a causal claim about this relationship?

• Ideal Comparison
• Simultaneously observe the swimsuit buying behavior (Y) of Billy when Billy has seen Y(1) vs. has not seen Y(0) social media mentions of ice cream

​

• Problem: For any single person, we can only observe their outcome under one condition

“Fundamental problem of causal inference”

​

• Randomized experiments as a way to
• Approximate ideal comparisons
• Account for confounders (e.g., unobserved common causes, selection)

​

Ingredients of an Experiment

Randomized controlled trial

• Participant recruitment
• Treatment randomization
• Treatment delivery
• Outcome measurement

Experimental design

Within subjects experiments:

Repeated measures of subjects �(e.g. pre- and post- treatment)

Controls for individual differences.

Between subjects experiments:

Compare different subject groups (e.g. treatment & control)

Controls for environment change.

Lab and field experiments

Lab experiments

• Controlled settings
• No extraneous variables
• Standardized, reproducible

Field experiments

• Natural settings
• No experimenter effects
• Generalizable to real life

Online field experiments with zero variable cost

Bit by Bit

Combining Natural Events + Digital Data Collection

Combining Natural Events + Digital Data Collection

Digital Field Experiments + Enriched Data Collection

Pre-Debate Survey + Experimental Assignment

Post-Debate Survey

Treatment Exposure

Adaptive Experimentation in Digital Settings

“This adaptive design allowed us to continue to learn which treatments were best, while reducing the probability that users were assigned to ineffective or harmful interventions.”

Adaptive experimentation tutorial

​

Experiments in the Digital Age

Today’s Activity: Speedy Experimental Research Design

• Develop a research question and/or hypotheses about how a treatment will affect an outcome.

​

• We will focus on randomized controlled experiments
• Recruiting participants and platform for the study
• Randomized assignment into conditions
• Delivery of treatment and control
• Measuring outcomes

​

​

​