SICSS Atlanta 2022

​

​

Announcements

• Sign up for lightning talks (link pinned in the Slack & in your email)
• Daily logistics

The Plan for Today

• An example SICSS project
• Introduction to digital trace data
• Omar Asensio and Olga Churkina
• Lunch & research Talk
• Group work

SICSS Sample Project

Day 2

Trying to find a new apartment

Trying to find a new apartment

Where is the Data From?

• Crimereports.com
• Mines data from over 1000 participating agencies, however they report on their website that “Each agency controls their data flow to CrimeReports, including how often they send data, which incidents are included.

​

• Lucido (2011): Trulia’s data is not weighted by population and gives limited crime type information
• Coronado Realty Group (2016): crime rates are combined by nearby areas that are actually separated by large bodies of water

​

Trying to find a new apartment

Information Asymmetries Housing Markets

• Who wins?
• Real Estate Agents v. Individuals
• People with resources or people without

​

​

• Crime affects housing prices (Kat)
• Consumer knowledge of foreclosure (Maria)

​

• What if we could quantify the cost of the data distortion – thereby generating the literal price of misinformation?

counts

counts

rates

Total Sample

• Purchasing
• 52.68% of participants would not purchase
• 47.32% would purchase
• Average cost
• $246,362

14

Counts

• Purchase price
• 251,298
• Buying
• No: 40%
• Yes: 60%

15

Counts

• Purchase price
• 251,298
• Buying
• No: 40%
• Yes: 60%

Rates

• Purchase Price
• 241,276
• Buying
• No: 66%
• Yes: 34%

16

Results

• Everything else about the house was held constant

​

• The crime data is all true
• It’s just projected differently

​

• Change in projection changed willingness to purchase substantially and changed the purchase price by over 10 thousand dollars

17

What happened after SICSS

• We applied for funding from SICSS
• We ran a pretest
• We then spent the next full year running some more pretests
• We launched the real thing
• What started as a comment, became a mock-up, then became a study

​

• More importantly, it became a collaboration and a friendship between two people interested in the same problem who never would have worked together

Digital Trace Data

Day 2

Readymades vs. Custommades

• Taking something that was created for one purpose and turning it into something different (creative re-purposing)
• Digital Trace data
• Data Scientists

​

• Data created intentionally
• Social science

Cleo the Clownfish from the Shedd Aquarium

Readymades vs. Custommades

• Readymade: Twitter data

​

• Custommade: survey data

Cleo the Clownfish from the Shedd Aquarium

What is digital trace data

• Large, digital datasets that describe human behavior
• (e.g. social media sites, internet search data, blogs, administrative records, historical archives, audio-visual data, or geospatial data)

Strengths of digital trace data

• Big
• Always on
• Non-reactive
• Captures social relationships

Weaknesses of digital trace data

• Inaccessible
• Non-representative

Weaknesses of digital trace data

• Inaccessible
• Non-representative
• Drifting
• Algorithmically confounded
• Unstructured
• Sensitive
• Incomplete
• Elite/publication bias

Application Programming Interfaces

• With the advent of web 2.0 the internet became more inter-dependent
• Over 22,000 APIs today (but a lot of them are not public)
• Links to public APIs
• https://www.programmableweb.com/apis/directory  
• https://any-api.com/

Application Programming Interfaces

• When we make an API, in theory we are stitching together a long url that is telling the web browser exactly what information we want

Testing some easy APIs where you control the call

• https://v2.jokeapi.dev/joke/Any?safe-mode

​

Testing some easy APIs where you control the call

• https://api.agify.io/?name=kat

Testing some easy APIs where you control the call

• https://house-stock-watcher-data.s3-us-west-2.amazonaws.com/data/all_transactions.json

R Packages for APIs

• Example: Reddit.com

​

​

​

​

​

​

​

​

• Documentation https://rpubs.com/mswofford/redditAPI�

​

Throttling & Rate Limiting

Rate limiting: client-side response the maximum capacity of a channel

Throttling: server-side response providing feedback to the caller that there are too many requests coming in from that client or that the server is overloaded

Screen scraping basics in R

Screen-Scraping

• Process of automatically extracting data from web pages

​

• Usually lists of sites/pages that can’t be extracted by hand

​

• 4 Basic steps
• Loads the name of a web-page to be scraped
• Downloads the website in html or xml
• Finds pieces of information on that website
• Puts the information in a convenient format

Adapted from SICSS Day 2

Is it Illegal?

• In the early days of the internet – a free for all

​

• More recently, websites have tried to stop screen-scraping via Terms of Service
• Often their screen-scraping policy is stored in a doc called ‘robots.txt’ in their terms of service

​

• Difference between personal or research use and commercial use

​

• Verdict: Use your best judgement and/or consult a lawyer

Let’s Try

• https://en.wikipedia.org/wiki/World_Health_Organization_ranking_of_health_systems_in_2000

Setting up the R Environment

• Install packages that we need
• Make sure those packages are downloaded into our virtual environment

​

​

install.packages("rvest")

install.packages("selectr")

Tell R we want to use those packages now

Find what we want to scrape

• Return to the webpage of interest (Wikipedia) and figure out where on the page the information we want actually is
• But – we can’t look as people; we need to look as if we are the computer

​

• Source code
• In Google Chrome you can use the drop-down menu at the top of your screen and select View -> Developer -> View Source

Read this source code into R

• Using the handy function read_html()

The Website is now in R…�but we need to parse the html file

Right click on the part of the webpage you want to scrape and choose ‘inspect’

Right click inside the developer window and select copy then copy Xpath

Now we can use this information to point R in the right direction

Put the information back into Table form

Complications

Parsing with a CSS Selector

• Some websites are more complicated, and we need a more advanced way of finding the xpath
• Here we will use an interactive tool (a Chrome Plug-in Extension) called “Selector Gadget”
• You can find this at selectorgadget.com

​

Scraping Duke’s Mainpage (duke.edu)

Try clicking around the site with Selector Gadget to identify the xpath

Feed the information to R

Other Complications

• Scraping many many pages within pages
• Simply nest your code into a ‘for loop’ that will iterate through all available webpages

​

• Needing to interact with a browser
• Implement Selenium

​

• Evading Captcha’s

​