Tor + Google/DuckDuckGo

Andrew Min

The “Problem”

• Search results can be (and often are) biased → “search neutrality”
• Search engines use info about us to provide relevant content
• → “information bubbles” + effects on ideologies

The “Solution”

• Google/DuckDuckGo searches via Tor Browser → manually scraping results
• Automating this process via Stem, Socks, and Selenium → scraping with BeautifulSoup (with some caveats…)

Results: “LeBron James GOAT” - Manual (San Angelo, TX)

Google

- Is LeBron James the GOAT? — The Undefeated

- LeBron James says there's one big reason he's the GOAT.

- Why Lebron James is the GOAT - Odyssey

- Why LeBron James is the G.O.A.T., not Michael Jordan

- The comparison between LeBron James and Michael Jordan, in their ...

- Doug Robinson: LeBron James' boast that he's the GOAT tells you all ...

- Celtics legend calls LeBron James' GOAT comments "disrespectful" to ...

- Yes, LeBron James Is the GOAT - National Review

- LeBron James or Michael Jordan: Who Is the GOAT? - GQ

​

DuckDuckGo

- LeBron James: The Greatest of All Time | National Review

- Jordan vs. LeBron 'GOAT' debate: Most Americans choose...

- LeBron James Collection | GOAT

- LeBron James's GOAT talk got Internet to dig up Michael ...

- LeBron James calling himself the GOAT has rankled NBA world ...

- LeBron James passes Michael Jordan: Who is the GOAT ...

- LeBron James: Why [MJ] always will be the GOAT

- Kareem Abdul-Jabbar: LeBron James Is Bigger Than the 'GOAT ...

- King James: 7 Reasons LeBron Is The GOAT And 8 Reasons Why He ...

- Why Lebron James is the GOAT - The Odyssey Online

​

Results: “LeBron James GOAT” - Auto (Providence, UT)

Google

- LeBron James says there's one big reason he's the GOAT.

- Is LeBron James the GOAT? — The Undefeated

- LeBron James says win over Warriors made him the GOAT

- Celtics legend calls LeBron James' GOAT comments "disrespectful" to ...

- Why Lebron James is the GOAT - Odyssey

- Why LeBron James is the G.O.A.T., not Michael Jordan

- LeBron James or Michael Jordan: Who Is the GOAT? - GQ

- Doug Robinson: LeBron James' boast that he's the GOAT tells you all ...

- Skip Bayless calls LeBron James 'delusional' over GOAT talk

​

DuckDuckGo

- LeBron James Collection | GOAT

- LeBron James calling himself the GOAT has rankled NBA world ...

- LeBron James's GOAT talk got Internet to dig up Michael ...

- Jordan vs. LeBron 'GOAT' debate: Most Americans choose...

- Kareem Abdul-Jabbar: LeBron James Is Bigger Than the 'GOAT ...

- LeBron James passes Michael Jordan: Who is the GOAT ...

- LeBron James: Why [MJ] always will be the GOAT

- LeBron James declares six rings as threshold for GOAT status

- Kevin McHale: LeBron James' GOAT chatter 'disrespectful..

- King James: 7 Reasons LeBron Is The GOAT And 8 Reasons Why He ...

​

Challenges

• Tor/Socks APIs are not very robust
• Troubleshooting is a pain point
• Tor process failures → manually kill
• Limiting exit nodes
• Can control country (with some APIs) but nothing more granular than that

​

Challenges

​

​

• Automating Google searches is annoying
• Avoiding bot detection requires setting cookies/headers/browser information, which are difficult to configure and not privacy-preserving
• Even manually Googling via Tor Browser can lead to bot detection

Takeaways

• Numerous factors [demonstrably] affect search results
• Location/ISP
• Browser/cookies/fingerprints
• Date + time
• Privacy <> trade-offs

Next Steps...

• Sentiment analysis for a more quantitative description of what’s going on
• Notion of similarity scores for search results between Google and DuckDuckGo
• Similarity scores for search results within Google and DuckDuckGo (variance depending on exit-route location)
• Remove confounding variables (cookies)