2 of 8

Study question and design

Does watching "Lemons" an episode of Black-ish decrease toxic polarization?

Design: Pre-post experimental design, looking at change in PQ, with control subjects watching otherwise equivalent sitcom episode (season, ratings, viewership).

PQ Pre

Affective polarization

Empathy

Humanization

Pluralist Norms

Intellectual Humility

Respect/Understanding

Identity

Anger

Morality

Threat

"Super Index"

Blackish

Equivalent sitcom

Mom

Modern Family

Life in Pieces

Goldbergs

Speechless

American Housewife

PQ Post

Compare the difference in Pre and Post PQ

3 of 8

Subjects

Successfully completed study (w/ 4+/5 attention check questions):

90 Dems, 74 Reps (34 Dem, 19 Rep in Murk. 56 Dem, 55 Rep Prolific)

Recruitment and Quality of data

Prolific

Prequalification asking which streaming services use. (350 Dems and 350 Reps) About 60% have HULU (220 Dems and 192 Reps)
Posted 120 surveys to ~400 eligible participants
111 successfully completed (93%)

Mturk round 1

Disaster: 18/40 successfully completed, with over 30% attriting

Mturk round 2

1st Prequalification asking which streaming services use and partisan identity (2000 Americans) About 30% have HULU and partisan identity (396 Dems and 215 Reps)
2nd Prequalification doing short video attention check task (100 Dems, 100 Reps - but only 64 Reps completed) 71/100 Dems successful, 48/64 Reps successful
Posted 40 surveys to all above
35 successfully completed (88%)

4 of 8

Results: indexing all outcomes into one measure

Upshot: Statistically significant (p:0.03) and doubles effect, but only 2 pct. pt movement (i.e. two points out of 100)

5 of 8

Results: Separating out outcomes/constructs

Upshot: except for Respect/Understanding (which has odd result in control), no statistically significant effect on any individual construct.

Note: Viewing any sitcom appears to have a positive effect, yet that may be an artefact of the study design (discussed later).

Results with * are statistically significant.

6 of 8

Results: Dems v Reps

Upshot: effect seems to be concentrated among Dems, particularly in Empathy & Perceived Threat.

Results with * are statistically significant.

7 of 8

Additional testing

Further testing on effect of Lemons (recommended): Although there is some indication that Lemons episode decreases toxic polarization (among Ds), results could be artefact of demand effect or of reactance. To test that is not the case:

Rerun as panel study (i.e. run Pre-PQ in separate survey) to see if get similar results. $
Rerun as between subjects (i.e. not Pre-PQ) study to see if get similar results. $$$
Decrease PQ items and add filler items to obscure study question?
Rerun study randomizing order of survey questions?

Further testing on effect of Sitcoms in general? We observed that control subjects also decreased toxic polarization, suggesting viewing any sitcom may have an effect - but, again, it could be an artefact of the design (eg. being asked same set of question twice in a row). To test "sitcom effect", could:

Rerun with non-sitcom control video
Rerun with non video control

8 of 8

Scoping capacity to do future studies

Which pool to use:

Prolific is clear winner: inexpensive, high quality data, ease of use (bonusing, creating own panel, contacting participants), also transparent and ethical
Only limitation: Under 8,000 Republicans in pool. Assuming 60% have streaming services and a substantial number are not active, I ballpark we may top out at 1,000 Reps in a potential pool

Cost

$0.35 to recruit potential panelist (Hulu, Amazon Prime. More for HBO, less for Netflix) -> $700 to recruit 2000 panelists
$8.5/participant to PQ 22 minute show
100 participants / study to see main effect. 200 participants / study to see effect on Dems or Reps. -> $850 to $1,700

Frequency / capacity

Limit to one study / month per participant in pool
Assuming can get pool of 2,000 and 50% participation / month, can run 5 studies / month