1 of 9

Tor + Google/DuckDuckGo

Andrew Min

2 of 9

The “Problem”

  • Search results can be (and often are) biased → “search neutrality”
  • Search engines use info about us to provide relevant content
  • → “information bubbles” + effects on ideologies

3 of 9

The “Solution”

  • Google/DuckDuckGo searches via Tor Browser → manually scraping results
  • Automating this process via Stem, Socks, and Selenium → scraping with BeautifulSoup (with some caveats…)

4 of 9

Results: “LeBron James GOAT” - Manual (San Angelo, TX)

Google

- Is LeBron James the GOAT? — The Undefeated

- LeBron James says there's one big reason he's the GOAT.

- Why Lebron James is the GOAT - Odyssey

- Why LeBron James is the G.O.A.T., not Michael Jordan

- The comparison between LeBron James and Michael Jordan, in their ...

- Doug Robinson: LeBron James' boast that he's the GOAT tells you all ...

- Celtics legend calls LeBron James' GOAT comments "disrespectful" to ...

- Yes, LeBron James Is the GOAT - National Review

- LeBron James or Michael Jordan: Who Is the GOAT? - GQ

DuckDuckGo

- LeBron James: The Greatest of All Time | National Review

- Jordan vs. LeBron 'GOAT' debate: Most Americans choose...

- LeBron James Collection | GOAT

- LeBron James's GOAT talk got Internet to dig up Michael ...

- LeBron James calling himself the GOAT has rankled NBA world ...

- LeBron James passes Michael Jordan: Who is the GOAT ...

- LeBron James: Why [MJ] always will be the GOAT

- Kareem Abdul-Jabbar: LeBron James Is Bigger Than the 'GOAT ...

- King James: 7 Reasons LeBron Is The GOAT And 8 Reasons Why He ...

- Why Lebron James is the GOAT - The Odyssey Online

5 of 9

Results: “LeBron James GOAT” - Auto (Providence, UT)

Google

- LeBron James says there's one big reason he's the GOAT.

- Is LeBron James the GOAT? — The Undefeated

- LeBron James says win over Warriors made him the GOAT

- Celtics legend calls LeBron James' GOAT comments "disrespectful" to ...

- Why Lebron James is the GOAT - Odyssey

- Why LeBron James is the G.O.A.T., not Michael Jordan

- LeBron James or Michael Jordan: Who Is the GOAT? - GQ

- Doug Robinson: LeBron James' boast that he's the GOAT tells you all ...

- Skip Bayless calls LeBron James 'delusional' over GOAT talk

DuckDuckGo

- LeBron James Collection | GOAT

- LeBron James calling himself the GOAT has rankled NBA world ...

- LeBron James's GOAT talk got Internet to dig up Michael ...

- Jordan vs. LeBron 'GOAT' debate: Most Americans choose...

- Kareem Abdul-Jabbar: LeBron James Is Bigger Than the 'GOAT ...

- LeBron James passes Michael Jordan: Who is the GOAT ...

- LeBron James: Why [MJ] always will be the GOAT

- LeBron James declares six rings as threshold for GOAT status

- Kevin McHale: LeBron James' GOAT chatter 'disrespectful..

- King James: 7 Reasons LeBron Is The GOAT And 8 Reasons Why He ...

6 of 9

Challenges

  • Tor/Socks APIs are not very robust
    • Troubleshooting is a pain point
    • Tor process failures → manually kill
  • Limiting exit nodes
    • Can control country (with some APIs) but nothing more granular than that

7 of 9

Challenges

  • Automating Google searches is annoying
    • Avoiding bot detection requires setting cookies/headers/browser information, which are difficult to configure and not privacy-preserving
    • Even manually Googling via Tor Browser can lead to bot detection

8 of 9

Takeaways

  • Numerous factors [demonstrably] affect search results
    • Location/ISP
    • Browser/cookies/fingerprints
    • Date + time
  • Privacy <> trade-offs

9 of 9

Next Steps...

  • Sentiment analysis for a more quantitative description of what’s going on
  • Notion of similarity scores for search results between Google and DuckDuckGo
  • Similarity scores for search results within Google and DuckDuckGo (variance depending on exit-route location)
  • Remove confounding variables (cookies)