1 of 20

EV Car Data Scraping Updates 11/8

Alex Liu, Sally Yin, Rob Carleski, Sabina Tomkins

2 of 20

Table of contents

Summary of work

Engineering challenges- getting all listings

Sample datasets- EV and All cars

Next steps

3 of 20

Summary of work

Pipeline development to get all car listings

Improved metric reporting

Dataset preparation, summary statistics

4 of 20

Engineering challenges

scheduled daily at

8:00 am

send

HTTP request

static website

dynamic website

car sales web server

get web content on eligible car sales listings

parse content into individual listing urls

car sales web server

car sales web server

car sales web server

car sales web server

parse listing information from HTML into JSON

get web content on individual listing

send

HTTP request

send

HTTP request

send

HTTP request

send

HTTP request

5 of 20

Engineering challenges

​

scheduled daily at

8:00 am

send

HTTP request

static website

dynamic website

car sales web server

get web content on eligible car sales listings

parse content into individual listing urls

car sales web server

car sales web server

car sales web server

car sales web server

parse listing information from HTML into JSON

get web content on individual listing

send

HTTP request

send

HTTP request

send

HTTP request

send

HTTP request

EV's: 10's of requests

All cars: 1,000's

6 of 20

Engineering challenges

​

scheduled daily at

8:00 am

send

HTTP request

static website

dynamic website

car sales web server

get web content on eligible car sales listings

parse content into individual listing urls

car sales web server

car sales web server

car sales web server

car sales web server

parse listing information from HTML into JSON

get web content on individual listing

send

HTTP request

send

HTTP request

send

HTTP request

send

HTTP request

EV's: 10's of requests

All cars: 1,000's

7 of 20

Engineering challenges

​

scheduled daily at

8:00 am

send

HTTP request

static website

dynamic website

car sales web server

get web content on eligible car sales listings

parse content into individual listing urls

car sales web server

car sales web server

car sales web server

car sales web server

parse listing information from HTML into JSON

get web content on individual listing

send

HTTP request

send

HTTP request

send

HTTP request

send

HTTP request

EV's: 10's of requests

All cars: 1,000's

EV's: 1,000's of listings; All cars: 100,000's

8 of 20

Engineering challenges

​

scheduled daily at

8:00 am

send

HTTP request

static website

dynamic website

car sales web server

get web content on eligible car sales listings

parse content into individual listing urls

car sales web server

car sales web server

car sales web server

car sales web server

parse listing information from HTML into JSON

get web content on individual listing

send

HTTP request

send

HTTP request

send

HTTP request

send

HTTP request

EV's: 10's of requests

All cars: 1,000's

EV's: 1,000's of listings; All cars: 100,000's

BLOCKED

SLOW

9 of 20

Engineering challenges- getting all listings

  • General approach- improve infrastructure to handle 100x throughput
    • Avoid Platform 1 blockage
    • Make listing scraping faster
  • Current approach
    • Not guaranteed to always work
    • Collected all listings for today's dataset
  • Future approach
    • More robust; Will not fail due to blockage
    • Working on it…

10 of 20

Sample datasets- Configuration

Vehicle type

EV

All cars

Dates

10/27-11/2

11/6-11/7

11 of 20

Sample datasets- Configuration

Vehicle type

EV

All cars

Dates

10/27-11/2

11/6-11/7

Search ranges

17 cities, 50mi

17 cities, 50mi

Platform

Platform 1

Platform 1

12 of 20

Sample datasets- Web scraping success

Dataset

EV

All cars

Listings identified

11,795

174,132

Scraped listings

10,761

169,810

Unique listings

2,774

156,538

13 of 20

Sample datasets- Data fields success

Dataset

EV

All cars

Unique listings w/ Essential columns

100%

100%

Ess. + Important

83%

78%

Ess. + Imp. + Nice

45%

47%

14 of 20

Sample datasets- Basic statistics (price)

Vehicle type

EV

All cars

Mean price

$30,260

$17,733

Max price

$448,888

$999,999 (max)

15 of 20

Sample datasets- Basic statistics (price)

16 of 20

Sample datasets- Basic statistics (price)

EV's

​

All cars

17 of 20

Sample datasets- Basic statistics (proportion of pop.)

EV's

​

All cars

18 of 20

Sample datasets- Basic statistics (EV:Car ratio)

19 of 20

Sample datasets- Basic statistics (fuel type)

20 of 20

Next steps

Share these sample datasets

Build out more robust pipeline for collecting all cars

Investigate data loss for systematic bias

Questions

  • How should all cars be reported?

​