Week 4: Exploratory Data Analysis

Introduction to Data Visualization

W4995.003 Spring 2025

​

​

00 Quiz

01 Sparks Presentation

02 Revisit: C ⊂ O ⊂ Q

03 Monthly Budget Redesigns

04 What is EDA?

05 EDA Examples: Air Pollution, Stop & Frisk

06 EDA Assignment: Group critique

​

Quiz

​

5 min

Closed book

​

Slide via Jeff Heer

01

Sparks Presentation

​

Catalogtree

​

02

Review: Categorical, Ordinal, & Quantitative

​

Data Types: C ⊂ O ⊂ Q

​

C: Categorical

Operations: =, ≠

Categories are of equal importance, or “equidistant”

O: Ordered

Operations: =, ≠, <, >

Items of equal importance, or “equidistant”

Q-Interval (location of zero arbitrary)

Operations: =, ≠, <, >, -

Can measure distances or spans, only delta (i.e. intervals) may be compared

Q-Ratio (zero fixed)

Operations: =, ≠, <, >, -, %

Can measure ratios or proportions e.g. Length, Mass, Temp, counts and amounts

​

Slide via Jeff Heer

William Playfair (1786)

​

Image via Wikipedia

Inventor of line charts, bar charts, and pie charts.

British pounds

X-axis: year (Q)

Y-axis: currency (Q)

Color: imports/exports (C)

From the Pudding

​

Rectangle Area: laugh share (Q)

Rectangle Position: category (C)

Color Hue: category (double-encoded) (C)

​

​

Map of the Market (Wattenberg 2000)

​

Rectangle Area: market cap (Q)

Rectangle Position: market sector (C)

Color Hue: loss vs. gain (C)

Color Value: magnitude of loss or gain (Q)

​

Exercise: data type of zip code?

Ben Fry, Zipdecode (1999)

Zip code trivia (and why it is categorical)

​

• Zip codes that begin with '0': most of New England + New Jersey, Puerto Rico, the U.S. Virgin Islands and APO/FPO military addresses
• 095* for mail to the Navy
• ​

• 00501, a ZIP Code exclusively for the U.S. Internal Revenue Service

Via wikipedia

Note: it’s difficult for a single O/Q value to span 2D

​

​

Via wikipedia

One option:

The Hilbert Curve

Compare & Contrast

Left: John Hopkins dashboard, right: Bloomberg

Compare & Contrast

NYTimes

Compare & Contrast

NYTimes

XKCD/NYTimes

03

Monthly Budgets

​

Stream graphs

A variation on the stacked area chart

03

What is Exploratory Data Analysis?

​

Design

Computer Graphics

HCI

Psychology

Statistics

Cartography

dataviz

EDA is...

​

A style of data analysis that employs graphical & statistical techniques to:

• uncover underlying structure
• detect outliers and anomalies
• test assumptions

​

EDA always precedes formal (confirmatory) data analysis.

​

​

“The greatest value of a picture is when it forces us to notice what we never expected to see.”

— John Tukey

Anscombe’s Quartet

Anscombe’s Quartet (1973)

Mean & Variance

uX = 9.0, σX = 11

uY = 7.5, σY = 4.125

​

Linear Regression

Y = 3 + 0.5X

R^2 = 0.67

Tukey 1977

​

Based on insights developed at Bell Labs in ’60s

​

Introduced new techniques for visualizing and summarizing data:

• 5-number summary
• Box plots (visual 5-number)
• Stem and leaf diagrams

Boxplot 5-Number Summary

Boxplot 5-Number Summary

Note: Max/Min in Boxplots

https://www.leansigmacorporation.com/box-plot-with-minitab/

Counterintuitive: less space means more density

https://www.leansigmacorporation.com/box-plot-with-minitab/

EDA differs from classical analysis

​

Exploratory data analysis is sometimes compared

to detective work: it is the process of gathering evidence.

​

Confirmatory data analysis is comparable to a

court trial: it is the process of evaluating evidence.

​

We use graphics in data analysis to...

​

• understand data properties (1D)
• find patterns in the data (2D+)
• suggest modeling strategies
• "debug" analyses
• communicate results

​

Use visual pattern detection to guide analysis

Identifying trends

Constant

Periodic

Linear

Exponential

Identifying trends

Identifying trends

Simple distributions

Uniform

Normal

Exponential

Graphics via Petra Isenberg

Simple distributions

Log-Normal

Skewed right (tail on right)

Skewed left

Graphics via Petra Isenberg

Use EDA to help you guess appropriate models

https://blog.cloudera.com/blog/2015/12/common-probability-distributions-the-data-scientists-crib-sheet/

What we do when we do EDA...

​

• First step: formulate hypotheses

​

Next steps, iterative:

• Test hypotheses to later confirm them
• Refute them (or just discard them)
• Generate new hypotheses from results

“The goal of data exploration is to generate many promising leads that you can later explore in more depth.”

R for Data Science, Wickham

Characteristics of exploratory graphs

​

• make quickly
• large number are made (keep saving copies)
• goal is for personal understanding
• axes/legends are generally cleaned up later
• color/size are primarily used for information
• (cf. in presentation you might think more carefully about your encodings)

​

04

EDA Example: EPA Air Pollution

​

Via Roger Peng, John Hopkins Biostatistics

Dataset: Air Pollution in the US

​

• EPA says: fine particle pollution (PM2.5), the “annual mean, averaged over 3 years” cannot exceed 12 μg/m³

​

• Question: Are there any counties in the U.S. that exceed that national standard for fine particle pollution?

​

Overview of Process

​

• import/clean
• single variable exploration (1D)
• pairwise exploration (2D)
• multivariate analysis (3+)

Dataset: Air Pollution in the US

​

pm25 fips region long lat

9.771 01003 east -87.75 30.59

9.994 01027 east -85.84 33.27

10.689 01033 east -87.73 34.73

11.337 01049 east -85.80 34.46

12.120 01055 east -86.03 34.02

10.828 01069 east -85.35 31.19

​

Via Roger Peng, EPA 2008-2010 https://aqs.epa.gov/aqsweb/documents/data_mart_welcome.html

Boxplot: PM2.5

​

Boxplot with overlaid feature

​

Histogram: PM2.5

Histogram with different bin size

Second variable: Categorical

Small multiples: PM2.5 x Region

Small multiples: PM2.5 x Region

the levels in eastern counties are on average higher than the levels in western counties.

Multiple histograms: PM2.5 x Region

Scatterplot: PM2.5 x Latitude

South

North

Scatterplot: PM2.5 x Latitude

highest levels of PM2.5 tend to be in the middle region of the country.

~South

~North

Scatterplot: PM2.5 x Latitude x Region

East: black

West: red

Easier if separate: PM2.5 x Latitude x Region

04

EDA Example: NYPD Stop & Frisk

​

“Stop, Question, and Frisk” by Bloomberg’s NYPD

​

• 2003–2013: over 100,000 stops were made per year
• 2002: 97,296 people stopped
• 2011: 685,724

​

• In 2010: 85% of people stopped by the police were Black or Latino, although those groups make up only slightly more than half of the city’s population.

​

​

“Stop, Question, and Frisk” by Bloomberg’s NYPD

​

• Opponents of the program have criticized that it is racist and failed to reduce robbery, burglary, or other crime.

​

• Proponents argue “stops save lives in communities disproportionately affected by crime”
• citing: young men of color represented 90% of murder victims and 96% of shooting victims in NYC, 2010

“Stop, Question, and Frisk” by Bloomberg’s NYPD

​

• In Floyd v. City of New York, Aug. 12, 2013: US District Court Judge Shira Scheindlin ruled the practice unconstitutional and ordered NYPD to adopt a written policy to specify where such stops are authorized. (“probable cause”)

​

• Oct. 31, 2013: Court of Appeals ruled against appeal and allowed city to proceed with overhaul.

Hypotheses & questions

​

1. The number of people stopped, questioned, and frisked in New York City declined after Floyd v. City of New York on Oct. 31, 2013.

​

• After adopting the “probable cause” standard, are police making fewer and/or more effective stops?

​

• Is the racial disparity narrowing?

Data Source

​

• NYPD (via NYC Open Data)
• 2013–2015
• data format is uniform so using this as preliminary exploration
• Fields
• Date
• Frisked?
• Arrested?
• Race

Hypothesis 1

​

The number of people stopped, questioned, and frisked in New York City declined after Floyd v. City of New York on Oct. 31, 2013.

Overall, looks like significant decline in number of stops

Plotting by day, we see there might be misformatted or missing data.

Missing data?

Zooming in we see:

​

Number of stops seems to fluctuate significantly but regularly.

​

This could be due to weekend/week-

day or reporting patterns.

Overall, stop and frisks declined after the Floyd decision.

​

However, there was already a downward trend from early 2012 in the number of stops.

​

Further research is needed to understand how the court orders were implemented.

Lawsuit

Decision

Hypothesis 2

​

After adopting the stricter “probable cause” standard, are police making a) fewer and b) more effective stops?

Most stops don’t result in an arrest.

​

In fact only about 1 out of 10 results in an arrest.

Over 2013-2015, with decline in number of stops, number of arrests have also declined.

By 2015, percentage of stops that resulted in arrests were more than double that of 2013.

Hypothesis 3

​

With the change in procedure, is racial disparity also narrowing?

There’s a lot of messy data in the field.

Using valid values from data spec sheet, we can filter out invalid values.

Already we see disproportionate representation.

When splitting into into small multiples by year, overall decrease in number of stops masks any changes we can see. What we’re interested in comparing is part-of-whole.

Abela, Advanced Presentations by Design, 2013 redrawn by Berinato in Good Charts

There may be a very slight trend towards less racial disparity.

A slightly clearer view of the same, since very slight differences in slope are hard to perceive.

Summary: Exploratory Data Analysis

​

• Exploratory plots are "quick and dirty"
• Let you summarize the data (usually graphically) and highlight any broad features
• Explore basic questions and hypotheses (and perhaps rule them out)
• Suggest modeling strategies for the "next step"

Summary: Exploratory Data Analysis

​

• Use histograms to look at distributions + basic statistics
• Use scatterplots and aligned bar charts to see potential correlations between pairs of variables
• Use line charts to find patterns in time series

Additional EDA Resources

​

• Handbook of Statistical Methods (good summary)

https://itl.nist.gov/div898/handbook/eda/eda.htm

​

• Allen Downey Think Stats (examples in python)

http://greenteapress.com/thinkstats2/html/thinkstats2002.html

​

• Spotify Team @ Tableau Conference (real-world)

https://www.youtube.com/watch?v=pgSLSYLNEq0

​

06

EDA: Brainstorming session

​

Form pairs with someone not in your group

​

3 min each

Describe the topics you’re examining and the data set you’ve found, then share your initial charts

​

5 min

Work independently to come up with 3 hypotheses for the other group’s topic. Suggest at least one additional variable that could be interesting.

Form pairs with someone not in your group

​

5 min

Compare your hypotheses with your partner’s and choose which seem most interesting/fruitful for further investigation

06

Tableau Concepts

​

Tableau/Polaris Contribution

​

• One declarative language (“formalism”) can simultaneously specify both database queries and visualization
• Choose data, then visualization, not vice versa
• Use smart defaults for visual encodings

Taxonomy of vis types

Stolte, et. al. Polaris, ACM 2008.

Tableau UI chart types

Valid chart types for the selected dimensions & measures

Tableau Fields: “Scale” of Fields

​

​

​

Nominal

Ordinal

​

Q-Ratio

Q-Interval

Discrete

Continuous

Continuous

Discrete

Table Algebra to specify configurations

​

Operands are the database fields

Each operand interpreted as a set {...}

Continuous and Discrete fields treated differently

​

Three operators

concatenation (+)

cross product (x)

nest (/)

​

Via Jeff Heer.

x

/

+

/

+

x

Via Jeff Heer.

/

+

x

Via Jeff Heer.

GROUP BY category, region, segment

Relational Data Model

​

• Represent data as a table (or relation)
• Each row (or tuple) represents a record
• Each record is a fixed-length tuple
• Each column (or field) represents a variable
• Each field has a name and a data type
• A table’s schema is the set of names and types A database is a collection of tables (relations)

Example: U.S. Census Data

​

People Count: # of people in group

Year: 1850–2000 (every decade)

Age: 0–90+

Sex: Male, Female

Martial Status: married,

never married,

widowed,

divorced

​

Example via Jeffrey Heer.

“Roll-Up” and “Drill-Down”

​

Examine population by year and age?

Roll-up the data along the desired dimensions

​

​

SELECT year, age, SUM(people)

FROM census

GROUP BY year, age

​

Via Jeff Heer.

Dimensions

Measure

Drill-Down

​

see the breakdown by marital status?

Drill-down into additional dimensions

​

SELECT year, age, marst, SUM(people) FROM census

GROUP BY year, age, marst

Via Jeff Heer.

Data Cube Concept

Via Jeff Heer.

Via Jeff Heer.

Via Jeff Heer.

​

Questions?

​

Next Week…

Topics Next Week

​

• Marks & Channels

​

• Expressiveness & Effectiveness

​

• Grammar of Graphics

​

• (A2 feedback next week)

​

Checklist For Next Week

​

• Assignment 3
• Submit PDF or slides, video or link to your Tableau Dashboard
• Follow template structure
• Yes it is EDA, but remember to label everything for graders

​

• Readings
• Norman Power of Representation
• Tufte Visual Display of Quantitative Information
• McCloud Understanding Comics: Lines, Six Steps, Color
• Optional: Choropleth maps (read this if you want to make a map)
• Optional: Original D3 paper