1 of 32

YouTube, The Great Radicalizer?�

Auditing and Mitigating Ideological Biases

in YouTube Recommendations

​

​

​

Muhammad Haroon, Anshuman Chhabra,

Xin Liu, Prasant Mohapatra, Zubair Shafiq, Magdalena Wojcieszak

​

2 of 32

3 of 32

The “Problem” with Recommendations

How often have you noticed that you interact with something once and are then bombarded with similar recommendations?

4 of 32

Loop Effect

5 of 32

The “Problem” with Recommendations

  • Exacerbated when the content is political
    • Partisanship, hate-crimes, misinformation, etc.
    • Radicalization

​

  • Why YouTube?
    • Used by 81% of the American population
    • Over 70% of views are driven by recommendations

​

​

6 of 32

Prior Work

  • Several prior audits have been conducted
    • Hosseinmardi et al., Papadamou et al., Ribeiro et al., Ledwich et al. ...
  • Little consensus in findings
  • Methodological differences
    • Fake vs. real users
    • Impact of personalization

7 of 32

What we did

  • Automated audit using a 100,000 sock puppets
    • Left, Center-left, Center, Center-right, Right ideologies
  • Extensible ideological slant estimation approach at video-level granularity
  • Proposed mitigation using reinforcement learning

8 of 32

Background

9 of 32

YouTube Interface: Homepage Recommendations

10 of 32

YouTube Interface: Up-next Recommendations

11 of 32

Slant Estimation

Step 2

Collect following lists of the tweet authors

Step 1

Search for tweets using Twitter API

Step 3

Determine liberal and conservative landmarks

12 of 32

Slant Estimation

​

​

​

​

​

​

  • Scale of -1 to +1
    • -1 is liberal
    • +1 is conservative

​

13 of 32

The Audit

14 of 32

Overview of Audit

15 of 32

Data Collected (100,000 sock puppets)

16 of 32

Research Questions

​

RQ1: Are recommendations ideologically biased?

​

RQ2: Does following the recommendation trail increase exposure to ideologically biased content?

​

RQ3: Does following the recommendation trail lead to increasingly radical videos?

​

17 of 32

RQ1�Are recommendations ideologically biased?

  • Homepage recommendations

18 of 32

RQ1�Are recommendations ideologically biased?

  • Up-next recommendations

19 of 32

RQ1�Are recommendations ideologically biased?

  • Kolmogorov-Smirnov test
  • Compare Center sock puppet with others

​

​

  • Statistically significant results

​

20 of 32

RQ2�Does following the recommendation trail increase exposure to ideologically biased content?

  • Left sock puppets
  • Exposure = 1.32x

21 of 32

RQ2�Does following the recommendation trail increase exposure to ideologically biased content?

  • Right sock puppets
  • Exposure = 1.37x

22 of 32

RQ3: Does following the recommendation trail lead to increasingly radical videos? �

  • Slant of each up-next recommendation
  • Compare at different depths
  • Mean and stdev move towards extreme

23 of 32

Bias Mitigation

24 of 32

Mitigating Bias

  • An intervention tool for bias mitigation
    • “Bottom-up”
    • Control to users
  • Watching counter-videos to escape the loop-effect
  • But how to determine the videos to watch?
    • Answer: Reinforcement learning

​

​

25 of 32

Mitigating Bias

26 of 32

Mitigating Bias

  • The framework is adaptable to mitigate other biases as well

​

  • Users might be unwilling to adopt such tools
    • Those who need it most are the least likely to actually use it
    • Considerable recruitment efforts are required

​

  • YouTube’s response
    • Positive: YouTube is compelled to increase the transparency of their system
    • Negative: Undermine its effectiveness through restrictions

​

​

27 of 32

Contributions

28 of 32

Code Release

  • YouTube-Driver
    • Programmatically interact with YouTube's interface.

​

  • YouTube Sock Puppet
    • Sock puppet implementation for collecting YouTube data.

​

  • YouTube Slant Estimation
    • Estimate the ideological slant of a YouTube video using the Twitter API.

29 of 32

Data Release

  • Sock puppet data
    • Training videos and recommendations
    • Estimated slants

​

  • Video metadata
    • Collected metadata for over ~400k unique videos

​

  • Tweets
    • Tweets collected for over ~400k unique videos

30 of 32

Visualization Tools

  • Daily top recommendations
    • See what YouTube's recommendations look like every day for left, center, and right political ideology users.

​

  • Sock puppet explorer
    • Explore sock puppet data

​

31 of 32

Daily Top Recommendations

32 of 32

YouTube, The Great Radicalizer?

Muhammad Haroon

mharoon@ucdavis.edu

www.muhammadharoon.xyz

www.youtubeaudit.com