What Persuades Voters?

2020 Metastudy Findings

Results

Research supported by:

Executive summary

Campaigns are presented with competing claims about what messaging strategies best win votes for Democrats. Median voter strategies have been given heightened publicity this cycle: these emphasize polling to identify issues that appeal to persuadable voters, defined in terms of specific demographics and moderate partisanship. However, despite the appeal to empiricism that can accompany these claims, polling data does not offer good evidence about who is persuadable, or what persuades them, leaving the status of the claims unsettled.

Measuring persuasion reliably requires carefully controlled RCT (randomized controlled trial) experiments. Major Democratic and progressive campaigns have recognized this since at least 2018, and used the Swayable platform to conduct such experiments on their ads at an accelerating rate. As a result, Swayable — a public benefit corporation dedicated to effective truth telling — now has the largest dataset in existence of academic-quality RCT experiment data on American political opinion. Using this dataset, the team looked for evidence that might have bearing on messaging strategy. Based on the most recent cycle, the team found:

• Political ads do persuade voters. In our captive experiments, the average ad generates about 1 percentage point lift in support for Democrats, but there is a wide range, with the best ones getting above 5 points. This highlights why systematic RCT pre-testing is a must-have.
• Voters of all backgrounds were comparably persuadable, across all geographies. There was no evidence than any specific demographic is significantly more persuadable than others.
• Issue choice was not a reliable predictor of what ads persuade voters. Some issues were more successful in 2020, but these were not reliably predictable by median voter theories. Some issues that theory proponents have cautioned against (e.g. racism, abortion) turn out to be no less effective than ones they recommend.
• Moderates mostly looked like everyone else. There was no evidence that political moderates were persuaded by significantly different ads than other voters.
• Identity of narrators didn’t generally impact persuasion much. Americans of all backgrounds were comparably persuaded by people of other backgrounds. There may always be other important reasons to pay attention to representation of course, so we see this as saying that storytellers have many good options.
• There is emerging, suggestive evidence that specific narrative structures may be more successful than others, for example, clear, sharp critiques of the right. We see this suggesting that storytellers may have more success focusing experimentation on clearly defined narratives—and that there is no reason they should feel constrained from campaigning on important issues.

3

LARGE VIRTUAL PANEL

Randomly allocated

CONTROL CONTENT

(e.g. PSA or other �neutral content)

CONTROL �GROUP

SURVEY

BASELINE

RESPONSE

STATISTICS

Swayable Impact Metric

TEST

CONTENT

TEST

GROUP

SURVEY

CONTENT-

INFLUENCED

RESPONSES

5

Custom segments

Contrast ad 02

Candidate bio 01

Likely Voters

Moderates

Very Liberal

Very Conservative

Suburban

Women

Likely Voters

Candidate bio 01

Support

The right’s strategy:

Stoke fear�of the “other”

6

One strategy: Appeal to the middle

Use polling to select issues�and appeal to “median voters”

–Slow Boring (Substack)

The median voter is a 50-something white person who didn’t go to college

7

An alternative approach

• Lead on important issues��
• Call out the politics of division��
• Prioritize a clear narrative

Not just follow polling��

Not just change the topic��

Not just issues and groups

8

Responses

Ad treatments

Experiments

1.1 million

1,146

255

Agreement -->

Age →

Each dot in the background is one complete response

We conducted a meta-analysis of 255 independent RCT survey experiments on over 1,000 ads to see what worked in 2020

31 people

$11 million invested

Team

Yale ad features team

Jesse Bryant

Katie Michels

Anna Reside

Ellie Singer

Emma Mueller

Mitchell Mares

Cloe Dickson

Elizabeth Himschoot

Ben Feshbach

Charlie Mueller

Cameron Danesh-Pajou

North Bennett

Karen Blakelock

Thomas Harris

Prerna Bhat

Swayable metastudy team

James Slezak

Valerie Coffman

Mike Baumer

Tanya Marton

Kwaku Ofori-Atta

Kathy Gerlach

Hang Su

Josh Dean

Mohammad Hamidian

Gavan Driscoll

Academic paper team

Luke Hewitt

David Broockman

Alexander Coppock

Ben Tappin

Josh Kalla

Nate Lubin

Mohammad Hamidian

Valerie Coffman

James Slezak

​

Input and consultation

Christina Coloroso

Saul Cunow

Michael Podhorzer

Jen Fernandez Ancona

Johannes Fischer

Elizabeth Spiers

Anat Shenker

Dan Ancona

Financial support

Way To Win

New Media Ventures

Higher Ground Labs

Other Swayable investors

Hewitt et al 2023 (forthcoming)

Important note: In this presentation, Swayable is presenting its own conclusions and not speaking for the academic group. Please see P50 for more details on collaboration structure

11

There was a wide spread in how persuasive ads were

Notes: A minority of treatments are not ‘ads’ but may include, for example, clips of candidates�60% of results are positive, including 13% that reach 95% confidence. Experiment description is simplified.

Distribution of persuasion (treatment effect), all ads

Persuasion (percentage points)

Average: 1 percentage point

Sample:

1,103,386

Treatments:

1,146

How likely are you to vote for Joe Biden?

← Unlikely Likely →

1000 people

1000 people

Ad

Placebo

Persuasive ad

Backlashing ad

No measurable impact at 95% confidence level

-5

+5

0

Persuasion = how much an�ad causes a change in answers

Standard experiment

Issues

President

Non-president candidate

0.89 points

0.41–0.91 points

1.57 points

Treatment Effect (percentage points) →

Average

Note: “President” range minimum figure includes all content for which impact on presidential support was measured, even if primary purpose was not persuasion (e.g. GOTV and issue ads). Maximum includes only ads for Candidate Biden. All subsequent analysis uses the broader definition.

+5

-5

0

Distribution of persuasion caused by each ad

Ads were most persuasive in non-presidential races

Persuasive ad

Backlashing ad

No measurable impact at 95% confidence level

Sample:

1,087,934

Treatments:

1,120

Mobilization

0.78 points

Experts did not predict what worked

Measured persuasion vs prediction

+5

-5

Predicted effectiveness →

Actual persuasion (pp)

President

Other candidates

Issues

Mobilization

r=0.0

r=0.1

r=0.0

r=0.0

Predictions by Yale students trained by political scientists. Prediction is normalized. Y-axis clipped at ±5

Persuasive ad

Backlashing ad

No measurable impact at 95% confidence level

Sample:

1,087,934

Treatments:

1,120

14

Rapid RCT pre-testing is critical for campaigns

To maximize votes, campaigns should reallocate 10–15% of media dollars to rapid pre-testing

Very favorable $/vote

Large expected vote gain

From: Hewitt, et al. 2022 (forthcoming)

15

Persuasion was most effective early

Persuasion (p.p.)

Highest persuasion early

Minor rebound�in final months

Number of ads

Jan�2020

Feb

Mar

Apr

May

Jun

Jul

Aug

Sep

Oct

Average persuasion by month, in lead-up to Nov 2020 general election

16

Every group was persuadable.

Age and politics made a difference to what persuaded, but race, gender, and education had much less impact.

Demographics

17

All parts of the country were persuadable

Average persuasion (pp), by partisanship of the area

Average persuasion (pp)

Partisanship by zip code area (x-axis) calculated based on 2020 Presidential vote. Error bars are 95% confidence intervals.

Democratic�zipcodes

Republican�zipcodes

All areas

Higher persuadability in conservative and moderate areas for non-Presidential races

Sample:

1,087,934

Treatments:

1,120

President

Issue

Non-President Candidate

Mobilization

←

→

All demographics were persuadable

Average persuasion (pp), by demographics

Average persuasion (pp)

President

Issue

Non-President Candidate

Mobilization

Republicans were almost as persuadable as Democrats in non-Presidential races

Error bars are one standard deviation (= 95% confidence intervals)

Sample:

1,087,934

Treatments:

1,120

High correlation (r=0.73)

High correlation (r=0.72)

High correlation (r=0.86)

Persuasion of each ad by population segment

Gender, race, and education did not have a major impact on what persuaded people

Gender

Race

Education

Persuasive ad

Backlashing ad

No measurable impact at 95% confidence level

Sample:

723,158

Treatments:

696

Results from Presidential support metrics

The identity of a messenger had no significant effects

Persuasion (pp) of ads having messengers with various qualities/identities

Sample:

1,045,250

Treatments:

814

Male

Female

Young adult

Middle-aged

Elderly

White

Black

Latinx

Asian

“Standard US” accent

Afr.-Am. Vernacular English

Latinx accent

Perceived as “diverse”

Average ad

Results from Presidential support metrics. Error bars are 68% confidence intervals. Gray region is a guide to the eye, representing the approximate range of statistical equivalence at 95% confidence

0.50 ±0.06

0.50 ±0.07

0.47 ±0.08

0.53 ±0.06

0.33 ±0.09

0.51 ±0.06

0.38 ±0.08

0.55 ±0.10

0.37 ±0.14

0.46 ±0.05

0.50 ±0.14

0.45 ±0.17

0.46 ±0.06

0.44 ±0.05

No detectable significant differences�at 95% confidence (p<0.05)

Speaker

Average persuasion

0.18

0.24

0.73

0.06

0.18

0.17

0.36

0.25

0.59

0.38

0.71

0.99

0.67

-

p-value (of difference�relative to other ads)

21

Matching narrator identity to target audience�didn’t help much (1/2)

Relative persuasion among like audiences

​

Male

Female

​

Young

Old

White

Black

Latinx

Asian

​

AAVE voice

Latinx voice

​

High diversity

​

Male

Female

​

Young

Old

​

White

Black

Latinx

Asian

​

Black

Latinx

​

People of color

Overperformance

Underperformance

Consistent with average performance

Sample:

1,045,250

Treatments:

814

Speaker

Audience

​

Relative persuasion (pp)

Results from Presidential support metrics. Error bars are 95% confidence intervals.

22

Matching narrator identity to target audience�didn’t help much (2/2)

​

Relative persuasion among contrasting audiences

​

Male

Female

​

Young

Old

White

White

Black

Latinx

AAVE voice

Latinx voice

​

High diversity

​

Female

Male

​

Old

Young

Black

POC

White

White

White

White

​

White

Relative persuasion (pp)

Results from Presidential support metrics. Error bars are 95% confidence intervals.

Sample:

1,045,250

Treatments:

814

Overperformance

Underperformance

Consistent with average performance

Speaker

Audience

​

23

Persuasion of targeted ads (pp), by ethnicity

Black

Ads that targeted one ethnicity were similarly effective with others

Ads targeting Black voters vs others

Ads targeting Latinx voters vs others

Results from presidential support metrics. ATEs plotted with 68% confidence intervals; differences are reported at 95% confidence (p < 0.05). The apparent targeted group for each ad was assessed by our academic content tagging team based on watching each ad

Sample

722,145

Treatments

151+135

Targeted

All others

-0.07 ±0.26

-0.17 ±0.24

-0.10 ±0.28

-0.11 ±0.22

-0.05 ±0.28

Black (n=65,873)

Latinx

White

People of Color

Asian

Latinx (n=45,234)

White

People of Color

Asian

+0.09 ±0.28

-0.04 ±0.28

+0.05 ±0.30

+0.01 ±0.23

+0.09 ±0.30

Difference

Difference

24

Low correlation (r=0.13)

Correlation in persuasiveness between age groups

Older voters were different

Age

Persuasive ad

Backlashing ad

No measurable impact at 95% confidence level

Sample:

723,158

Treatments:

696

Results from presidential support metrics

• Plenty of ads persuaded both young and old
• But a similar amount worked only with one group

55+

18–24

Selected responses to ads with backlash from older respondents�but persuasive for younger ones

Format appeared to be part of the explanation

“The video is silly and based on a very slanted ideological view rather than reality.”

–56 year-old woman, Lexington, KY

“Very inspiring and emotional video. And it’s very touching.”

–20 year-old woman, Tampa, FL

“It was very confusing. It was all over the place. I did not understand the point the video was ultimately trying to make.”

–72 year-old man, Philadelphia, PA

Sample:

336,582

Treatments:

696

“Too much going on. Should be simplified.”

–65 year-old woman, Greensboro, NC

Results from presidential support metrics

26

Results from presidential support metrics

High correlation: r=0.67

High correlation: r=0.86

High correlation: r=0.87

High correlation: r=0.68

Moderates were mostly persuaded by the same ads�as everyone else

Persuasive ad

Backlashing ad

No measurable impact at 95% confidence level

Sample:

713,237

Treatments:

684

Persuasion of moderates →

Persuasion of non-moderate segments →

Very liberal ->

Liberal ->

Conservative ->

​

Very conservative ->

​

Ads that persuaded the most liberal 20% of voters had no relation to ads that persuaded most conservative 20%.

Importantly, that does not mean that the liberal ads were bad for conservatives (that would be a negative correlation, rather than zero). In fact, some ads were highly persuasive with voters at both ends of the spectrum.

Politics mattered when it was most different

← Liberal

Conservative ->

← Conservative

Liberal →

Persuasive ad

Backlashing ad

No measurable impact at 95% confidence level

Sample:

713,237

Treatments:

684

Results from presidential support metrics

There were only limited patterns in what issues drove support, and no significant evidence that Democrats should avoid issues some see as divisive.

The economy and corruption/character were issues where ads were consistently more persuasive.

Issues

29

“Median voter” issues worked no better than others

Presidential support metrics. *Denotes significant difference (95% confidence) between ads that mention an issue and ads that don’t. Persuasion (i.e. ATE) is reported at 68% confidence.

Persuasion associated with selected issues

“Median voter” issues

More contested issues

Bipartisanship

Healthcare

Law & Order

Jobs

Veterans

Abortion

Foreign policy

Criminal justice reform

Immigration

Average ad

Average ad

Racism

The Economy

Persuasion, pp

Character/decency

+0.52 ±0.17

+0.54 ±0.07

+0.55 ±0.21

+0.35 ±0.13

+0.63 ±0.17

+0.68* ±0.10

+0.48 ±0.14

+0.51 ±0.22

+0.54* ±0.07

+0.40 ±0.08

+0.59 ±0.17

+0.64 ±0.18

+0.44 ±0.05

+0.44 ±0.05

Persuasion, pp

30

Moderates looked similar to everyone else

Presidential support metrics. ATEs reported at 68% confidence. * indicates statistically significant difference (95% confidence; p < 0.05) from ads not mentioning each issue.�All issues with more than 35 tested ads are included. Gray region is a guide to the eye, representing the approximate range of statistical equivalence at 95% confidence

Persuasion by issue, pp

Corruption*

Veterans

The Economy*

Foreign Policy

Campaign Finance

Big Business

Character/decency*

Law & Order Policing

Healthcare

Education

Coronavirus

Racism

Labor

Workers

Jobs

Climate Change

The Environment

Economic Inequality

Natural Disasters

Black Lives Matter

All ads (average)

Everyone

Moderates

Lobbying

Criminal Justice Reform

Bipartisanship

Pre-existing Health Conditions

Abortion

Immigration

Poverty/Homelessness

Corruption

Veterans

The Economy*

Foreign Policy

Campaign Finance

Big Business

Character/decency*

Law & Order Policing

Healthcare

Education

Coronavirus

Racism

Labor

Workers

Jobs

Climate Change

The Environment

Economic Inequality

Natural Disasters

Black Lives Matter

All ads (average)

Lobbying

Criminal Justice Reform

Bipartisanship

Pre-existing Health Conditions

Abortion

Immigration

Poverty/Homelessness

+0.77 ±0.15

+0.63 ±0.17

+0.68 ±0.10

+0.59 ±0.17

+0.64 ±0.21

+0.66 ±0.17

+0.54 ±0.07

+0.55 ±0.21

+0.54 ±0.07

+0.74 ±0.16

+0.48 ±0.08

+0.38 ±0.12

+0.37 ±0.15

+0.35 ±0.13

+0.33 ±0.15

+0.43 ±0.10

+0.41 ±0.09

+0.37 ±0.14

+0.33 ±0.20

+0.24 ±0.19

+0.44 ±0.05

+0.64 ±0.18

+0.52 ±0.17

+0.68 ±0.19

+0.51 ±0.22

+0.45 ±0.11

+0.31 ±0.12

+0.48 ±0.14

+0.75 ±0.15

+0.59 ±0.17

+0.69 ±0.18

+0.57 ±0.07

+0.61 ±0.16

+0.68 ±0.17

+0.56 ±0.20

+0.57 ±0.20

+0.56 ±0.07

+0.71 ±0.17

+0.52 ±0.08

+0.48 ±0.12

+0.43 ±0.09

+0.42 ±0.12

+0.39 ±0.15

+0.48 ±0.23

+0.48 ±0.14

+0.43 ±0.11

+0.39 ±0.19

+0.31 ±0.19

+0.49 ±0.05

+0.64 ±0.20

+0.56 ±0.15

+0.69 ±0.10

+0.55 ±0.13

+0.49 ±0.14

+0.36 ±0.12

+0.49 ±0.09

The independently tagged academic study by political scientists found no persistent pattern to what worked best

A small set (3/43) of issues had significantly higher persuasiveness

Presidential support metrics. Errors are 95% confidence intervals. Arrows represent relative persuasiveness. * indicates significance at 95% confidence (p < 0.05).

Corruption

Education

The economy

Big business

+0.39* ±0.32

Relative persuasiveness of issue

Persuasion (pp) relative to ads without the issue

+0.32 ±0.34

+0.29* ±0.22

+0.25 ±0.4

Persuasiveness of each ad

Persuasion of each ad featuring the issue

Pre-existing health conditions

+0.23 ±0.36

Campaign finance / money in politics

Criminal justice reform

+0.21 ±0.44

+0.21 ±0.39

Character / decency

+0.20* ±0.19

Significant at 95%

0

+3

-3

33

Ads discussing racism were no less effective than others

Persuasion (pp)

Sample:

388,891

Treatments:

169

Black Lives Matter

Criminal Justice Reform

Racism/Discrimination

Confronting Past Injustices

Equity

Diversity

All ads (average)

+0.40 ±0.08

+0.64 ±0.18

+0.45 ±0.12

+0.45 ±0.07

+0.44 ±0.08

+0.31 ±0.12

+0.44 ±0.05

Presidential support metrics. ATEs reported with 68% confidence intervals. Grey box is a guide to the eye representing the approximate range of insignificant difference from the average ad result

Persuasion associated with selected issues/features

P-value�Of difference�relative to other ads

0.50

0.27

0.94

0.96

0.96

0.23

–

Statistically equivalent (approx.)

34

Swayable Impact Metric: Biden Support

+2.1 pp

Overall

+2.5 pp

Deep red ZIP codes in the South

+2.4 pp

Non-college white men

35

This was true for voters of all political backgrounds

Presidential support metrics. Error bars are 68% confidence intervals.

Persuasion of ads mentioning racism vs all others, percentage points

Very Liberal

Very Conservative

Ads mentioning racism

Other ads

0.60

1.00

0.91

0.89

0.79

P-values:

Liberal

Moderate

Conservative

.65

±.07

.59�±.11

.56

±.06

.56

±.10

.48

±.05

.49

±.09

.34

±.07

.32

±.10

.15

±.10

.10

±.14

No significant differences for any segment

36

Ads mentioning racism were not significantly less persuasive for white people than Black people

Presidential support metrics. Error bars are 68% confidence intervals; differences reported at 95% confidence.

Persuasion of ads mentioning racism vs others, percentage points

Everyone

Black

Asian

People of Color

White

Latinx

-0.04 ±0.21

+0.13 ±0.27

-0.05 ±0.29

+0.04 ±0.23

-0.11 ±0.28

-0.13 ±0.23

Difference

Significance of difference is below 72%�p-value = 0.28

Ads mentioning racism

Other ads

Persuasion of group

0

0.5

37

This was also true for other relevant segments

Presidential support metrics. Error bars are 68% confidence intervals. Low income defined as <$25K/y

Persuasion of ads mentioning racism vs all others, percentage points

Non-college x white x female

Non-college x white x male

White x suburban x female

Rural

South

South x deep red zips

Suburban

Low income

2022 battleground districts

Ads mentioning racism

Other ads

Non-college x white

p=0.24

p=0.25

p=0.23

p=0.49

p=0.75

p=0.46

0

0.5

0

0.5

P-value of difference

38

Emphasizing bipartisanship was no more nor less effective than other topics

Presidential support metrics. Error bars are 68% confidence intervals.

Persuasion of ads mentioning bipartisanship/shared values, by personal politics

Ads with the feature

All other ads

Sample:

579,320

Treatments:

37+394

Everyone

<- Liberal

Conservative->

Bipartisanship

No significant differences

Everyone

<- Liberal

Conservative->

Shared values

39

Abortion: There was no penalty in discussing it

Presidential support metrics. Same trend observed in area partisanship breakdown. Error bars are 68% confidence intervals.

Persuasion of ads mentioning abortion vs others, percentage points

Ads mentioning abortion

Other ads

Persuasion, pp

<- Liberal

Conservative ->

Moderates and conservatives no less persuaded than liberals

Sample:

38,959

Treatments:

37

Everyone

40

Ads mentioning corruption were significantly more persuasive

Presidential support. Error bars are 68% confidence intervals.

Persuasion of ads mentioning corruption vs others, by politics

“Trump is hiding something”

–64 year-old white female republican (KY)

Mentions corruption

All other

Sample:

189.579

Treatments:

109

Persuasion, pp

All

<- Liberal

Conservative->

p-values

.02

*

.59

.33

.05

.007

**

.002

**

41

Character of public officials was an effective theme across the board

Error bars are 68% confidence intervals. */**/*** denotes significant difference at 95%/99%/99.9% confidence

Persuasion of ads mentioning character and decency vs others, by politics

Persuasion, pp

Mentions character/decency

All other

Sample:

454,768

Treatments:

377

All

Conservative->

← Liberal

All

Conservative->

← Liberal

Other candidate support

Presidential support

p-values

.03

*

.06

.04

*

.04

*

.06

.21

.0001

***

.0002

***

.0001

***

.0002

***

.0005

***

.01

*

42

Economic issues also overperformed, especially among low and middle-income respondents

Presidential support metrics. */** denotes significant difference at 95%/99% confidence between ads having/lacking the feature. Error bars are 68% confidence intervals; differences are reported at 95% confidence.

Relative persuasion of ads mentioning economic issues, overall and by income level

Sample:

152,187

Treatments:

121

Ads mentioning economic issues

All other ads

Everyone

$25k - $49k

$100k or more

Less than $25k

$50k - $100k

+0.29* ±0.22

+0.30* ±0.23

+0.21 ±0.24

+0.28* ±0.23

+0.36** ±0.23

Difference

0

0.5

43

Narrative approach

We saw significant impact from broader narrative choices.

This suggests storytellers could have success focusing experimentation on clearly defined narratives, rather than only on narrower elements like issues.

44

Clarity was associated with stronger persuasiveness

Persuasion of ads, president

Sample:

694,994

Treatments:

655+463+254

+0.36* ±0.34

Clear, non- confusing story

Clearly indicated source/funder

Call to action: Support/vote for candidate/party

Relative persuasiveness

Persuasion (pp) relative to ads without the feature

Persuasiveness of each ad

Persuasion of each ad featuring the issue

+0.24* ±0.20

+0.28** ±0.19

*/** denotes significant difference at 95%/99% confidence between ads having/lacking the feature. Errors reported at 95% confidence. Arrows represent relative persuasion.

45

Betrayal was an effective concept on all three electoral metrics

​

​

​

​

Persuasion of ads, president

+0.31** ±0.32

Presidential

​

Other candidates

​

Mobilization

Relative persuasiveness

Persuasion (pp) relative to ads without the feature

Persuasiveness of each ad

Persuasion of each ad featuring the issue

+1.15*** ±0.32

+0.25 ±0.29

Sample:

549,821

Treatments:

546

*/**/*** denotes significant difference at 95%/99%/99.9% confidence between ads having/lacking the feature. Errors are reported at 95% confidence. Arrows represent relative persuasion.

46

Sharp critiques of the right performed well in non-presidential races

Error bars + ATE errors are reported at 68% confidence. */**/*** denotes significant difference at 95%/99%/99.9% confidence between ads having/lacking the feature. Gray region is a guide to the eye, representing the approximate range of statistical equivalence at 95% confidence.

Betrayal

Deception

Extremism

Anger

Sadness

Degradation

Harm

Concerning

Average ad

Persuasion, percentage points

+2.02*** ±0.28

+2.09* ±0.33

+2.94*** ±0.59

+1.90*** ±0.27

+2.13*** ±0.30

+2.32* ±0.53

+2.03*** ±0.27

+1.84*** ±0.24

+0.87 ±0.20

Average persuasion of ads featuring each notion, non-president candidate results

Highly significant results: most above 99.9% level

47

Swayable Impact Metric: Biden Support

+2.7 pp

Overall

+3.9 pp

Deep red ZIP codes in the South

+3.2 pp

Non-college white men

48

Conclusions

Poll-selected issues and narrow demographic targeting do not appear to be the key to winning votes with ads.

This work suggests storytellers can have more success focusing experimentation on clearly defined narratives—and should not feel constrained from campaigning on important issues.

It also underlines that systematic RCT pre-testing is a must-have for campaigns.

Notes on study structure

• This metastudy was conducted in parallel with a team of academic social scientists referenced on the credits slide. The Swayable team initiated this collaboration in order to maximize the insights that emerge from this unique dataset, and expose the data and experiment structure to as much useful expert scrutiny as possible while remaining compliant with confidentiality obligations to campaign partners.
• Swayable does not speak here for the academic team. The academic team will report their findings in a forthcoming peer-reviewed paper, which will be available once the peer-review process is complete. Results presented here represent the conclusions of the Swayable team, whose agreement with the universities was that each subgroup could draw conclusions independently should they wish to. However, we remained in close contact at all times, and, as the experimentalists conducting the RCTs, Swayable team members will be authors on the paper.
• The academic team pre-registered its publication hypotheses as per current standards in academic political science, so findings are robust against any counter-claims relating to statistical significance in the context of multiple hypotheses. Swayable’s independent analysis did not do that, for several reasons:
• We wanted maximum flexibility to explore the widest range of ad features for which we might find evidence of greater persuasive impact
• Although each test is a rigorously controlled RCT, the broader metastudy is effectively an observational study on top of these RCTs, since we do not control the content of the ads. Studies of this type can establish associations between persuasion and ad features, and these associations can disprove causal relationships but by themselves cannot prove them. The causal significance of any choice of feature is the central concern of any campaign (“if I do X can I cause more persuasion?”), and our purposes as Swayable are to inform these practitioners, so we chose not to compromise on the breadth of hypotheses in an attempt to resolve multiple-hypothesis concerns that don’t affect this more fundamental causal significance question anyway.
• Our intent is to work with campaigns and academic institutions to establish direct evidence of causality by creating follow-up experiments that look at the effect of varying specific features that emerge as associated with stronger persuasion in the metastudy.

​

51

Narratives include a broad set of elements

Narrative

Elements include:

• Notions, concepts, moral foundations, frames
• Emotions/feelings
• Topics, issues
• Protagonists
• Their actions, behaviors
• Their intentions, goals
• Argument, structure (e.g. attack, praise, contrast)

Beliefs

• By asking people questions in a controlled experimental context, survey experiments give us evidence of beliefs and how they can change

• Narrative is communicated via stories (e.g. ads, media or person to person)
• People interpret it
• This can influence their beliefs

Storytelling

What about my issue / election / district / idea?

The full metastudy dataset can be explored live on the Swayable platform - contact us for details:

53