Data Exploration

CMSC 320: Introduction to Data Science

2026

�

Lecture - 09

– FARDINA FATHMIUL ALAM

           (fardina@umd.edu)

�

Topics to Cover

Chapter 9 in Textbook

Exploratory data analysis (EDA) Checklist

• Asking the right questions
• Variations, Co-variations, Key Variable Of Interest, Dependencies/Relationships between variables.
• Understand Your Data Structure
• Summarizing Variables: Categorical and Continuous
• Some Basic Data Manipulation
• Few Common Syntax in Pandas

​

Why have a lecture on data exploration?

When we start our process, all we have is a CSV, with no idea what we have.

• There’s generally no label and no guide.
• It’s useful to have a process for exploring it.

Exploratory Data Analysis refers to the critical process of performing initial investigations on data so as to

• discover patterns and gain insights
• spot anomalies
• test hypothesis and
• to check assumptions with the help of summary statistics and graphical representations.

Exploratory data analysis (EDA)

Before making inferences from data it is essential to examine all your variables.

Why?

To listen to the data:

- to catch mistakes

- to see patterns in the data

- to find violations of statistical assumptions

- to generate hypotheses

…and because if you don’t, you will have trouble later

It is a good practice to understand the data first and try to gather as many insights from it.

EDA is all about making sense of data in hand, before getting them dirty with it.

Aim of Exploratory data analysis (EDA)

EDA provides a structured process for approaching this scenario. Involves a series of steps and techniques to systematically explore the data to

• Maximize insight into a dataset.
• Uncover underlying structure.
• Extract important variables.
• Detect outliers and anomalies.
• Test underlying assumptions.
• Develop valid models.

This process ensures that you don't miss critical information and can make informed decisions about subsequent analyses or modeling.

​

EDA also emphasizes the importance of data vis.

Use suitable visualizations and statistical tests to support findings.

​

3 steps to understand the Cycle of EDA

Rinse and repeat (until you publish a paper or your boss approved it!)

HISTORY

• Created by statistician John Tukey
• Seminal book is Exploratory Data Analysis by Tukey
• A nice online introduction can be found in Chapter 1 of the NIST Engineering Statistics Handbook

​

Depending on the number of columns we are analyzing we can divide EDA into three types.

Types of Exploratory Data Analysis (EDA)

Example

use “Titanic Dataset” for EDA

• real-world relevance and diverse features

Example Code: https://colab.research.google.com/drive/11JjW055RQ-poNCXTlzPiUxgs9hMi5Xns?usp=sharing

Dataset Link

Asking the Right Questions

What type of questions we should ask that gives us a clear picture.

​

Two types of questions will always be useful for making discoveries with your data

​

1. WHAT TYPE OF VARIATION OCCURS WITHIN MY VARIABLES?

Ex. Understand the distribution and range of key metrics like revenue and engagement.

​

2. WHAT TYPE OF CO-VARIATION OCCURS BETWEEN MY VARIABLES?

Ex. Explore how different variables interact. For example, does increased engagement lead to higher revenue?

​

More: Asking the Right Questions

What type of questions we should ask that gives us a clear picture.

​

Start your analysis with a list of questions, but don’t be afraid to change them!

IDENTIFYING THE KEY VARIABLE OF INTEREST

Generally, there will be one column you care about the most: Revenue, engagement, survival, etc. Focus on how the other variables relate to that.

​

UNDERSTANDING RELATIONSHIPS, CONSIDERING DEPENDENCIES

You’ll also need your common sense! Think about what columns might be related, what might contribute to what, what might dependent or independent.

​

• Demographic Breakdown: What is the demographic profile of passengers?
• Overall Survival Rate: What was the overall survival rate on the Titanic?
• Age Impact: Did age affect survival rates?
• Class Survival: Were higher-class passengers more likely to survive?
• Fare Distribution: What was the distribution of fares paid?
• Family Survival: Were families more likely to survive together?
• Embarkation Point: Did the point of embarkation influence survival?
• Gender Influence: How did gender affect survival rates?
• Name Patterns: Were there interesting patterns in passenger names or titles?
• Feature Correlation: What is the correlation between features like age, fare, and class?
• Class Survival by Gender: What was the survival rate by class, broken down by gender?

Example: Asking the Right Questions

Titanic Dataset

Start your analysis with a list of questions based on your initial hypotheses and domain knowledge.

• Do first-class passengers have a higher survival rate compared to third-class passengers?
• Is there a significant difference in survival rates between males and females?"
• How does age impact the likelihood of survival?
• Are passengers who embarked from different ports more likely to survive?

Refine your questions to gain a deeper understanding of the factors, if needed.

​

Titanic Dataset

​

​

Focus on: you're not only describing the data but also seeking meaningful explanations for observed patterns

​

EDA Checklist

• Define Objective: Clearly state the question you want to answer. A well-defined goal keeps the analysis focused and purposeful.�
• Understand Data Types: Identify what type of data you have (e.g., numerical, categorical) and treat each appropriately.�
• Handle Missing Data: Identify missing values and decide how to address them (e.g., imputation or removal). [Next class topic]
• Did passengers with missing values for certain features have different survival rates?
• Detect Outliers: Identify unusual values and evaluate their causes and impact on results. [Later class topic]
• ​
• Were there any outliers in the dataset, especially regarding fares or ages?�

(NOT PART OF EDA, LATER STAGE): Feature Engineering: Create, modify, or remove features to improve model performance or analysis quality.

If a checklist is essential for pilots on every flight, it’s equally important for data scientists to use with every dataset.

Before Doing EDA… Understand Your Data Structure

EDA is not just plotting. First ask:

• What does one row represent?
• What does each column represent?
• Are measurements repeated over time?
• Which variables are fixed vs changing?

Because analysis depends on data structure.

Example Dataset: Cancer Registry

New cancer cases in the U.S. based on a cancer registry.

• Rows = individuals
• Columns = attributes

Some variables describe the person once. Some variables change over time.This is longitudinal data.

What questions can we ask from this data?

• Identifier (who?)
• Baseline attributes (start)?
• Time-varying attributes (changes)?

Variable Roles in Longitudinal Data

​

Identifiers

​

• Patient ID, visit number, measurement date
• Link records for the same individual over time

​

• Age at Enrollment date
• Demographics (BMI, etc.)
• Treatment or intervention group

Time-Varying Attributes (Follow-up)

​

• Health metrics: blood pressure, cholesterol, weight
• Symptoms: pain level, fatigue

Baseline Attributes (Study Start)

Research Question:

• How do symptoms change across patient groups or over time?
• Do treatment groups differ?
• Which baseline factors predict outcomes?

Typical Analysis

Example

distributions, group comparison

trends, change over time

Tracking, grouping

Next: Understanding Variable Types

Before summarizing or visualizing data, identify what type of variable you have.

Categorical variables

Frequency tables - how many observations in each category?

Relative frequency table - percent in each category.

Bar chart and other plots.

Summarizing Variables: Categorical

The goal for both categorical and continuous data is data reduction while preserving/extracting key information about the process under investigation.

Frequency Table: Categories with counts

Relative Frequency Table: Percentage in each category

Graphing a Frequency Table - Bar Chart

Plot the number of observations in each category

Create Frequency Table :

Continuous variables

• Grouping Data into Categories: Bin the observations (create categories .e.g., (0-10), (11-20), etc.) then, treat as ordered categorical.
• Plots specific to Continuous variables.

Summarizing Variables: Continuous

Plot Continuous Data: Age histogram of 10 adult leukemia patients

A histogram is a bar chart showing the frequency of binned data, highlighting how many observations fall into each bin

Create Frequency Table for this new categorical age variable:

Back to Previous Example “Titanic”

We will now start EDA using Pandas!

A Starting Point for Basic Data Manipulation

​

Once your DataFrame is loaded, first inspect the structure of your data. Some Examples:

• View column names: df.columns
• Check data types of each column: df.dtypes
• Transform data into usable formats: Data is often not analysis-ready. You can create or modify columns to make them easier to work with.
• Convert dates to numeric values
• df["date"] = pd.to_datetime(df["date"])
• df["date_in_days_since_zero"] = (df["date"] - df["date"].min()).dt.days
• Encode ordered survey responses

scale = {

"strong agree": 2,

"agree": 1,

"neutral": 0,

"disagree": -1,

"strong disagree": -2

}

df["survey_score"] = df["survey_response"].map(scale)

• Change measurement units
• df["seconds"] = df["days"] * 24 * 60 * 60
• df["grams"] = df["pounds"] * 453.592
• Apply custom transformations Use .apply() when a transformation requires a function.
• df["new_col"] = df["old_col"].apply(lambda x: x * 2)

Sometimes a data dictionary explains what each column means ; but often it doesn’t. If unclear, infer meaning from the column name, units, or ask the data provider.

Key idea: Always reshape, recode, or convert data into forms that make analysis clearer and more meaningful.

Learning About Individual Columns (EDA)

​

1. Summary Statistics: Use .describe() to quickly understand a numeric column.

df["column"].describe() Provides:

• count → number of values
• mean → average
• std → spread (variability)
• min / max → range
• 25%, 50%, 75% → quartiles (distribution shape)

2. Visualize Distribution (Box Plot) Box plots show spread, median, and outliers.

import matplotlib.pyplot as plt

​

fig, ax = plt.subplots()

ax.boxplot(df["column"].values)

plt.show()

​

​

count 69.000000 �mean 112.696254 �std 75.825020 �min 23.590017 �25% 69.131239 �50% 95.814021 �75% 143.869780 �max 532.230000 �Name: damage, dtype: float64

Q1 (25th Percentiles)

Q3 (75th Percentiles)

Understanding Distribution of Categorical Variables

To examine how a categorical variable is distributed, use:

df["column"].value_counts()

Example output:

​

​

​

​

​

If you want proportions instead of counts:

df["column"].value_counts(normalize=True)

​

shows:

• Frequency of each category
• Which values are most common
• Whether the distribution is balanced or skewed�

New Orleans 9 �Seattle 5 �Boston 4 �San Francisco 3 �Long Beach 3 �Memphis 3 �Virginia Beach 3 �Detroit 3 �Dallas 2 �Orlando 2

Understanding Distribution of Continuous Variables

For numeric (continuous) variables, use histograms to visualize how values are distributed.

Ex: df[“column”].hist() will usually work:

​

​

​

​

Each histogram can tell a different story!�

df["Age"].hist()

df["column"].hist(bins=20)

plt.xlabel("Value")

plt.ylabel("Frequency")

plt.title("Distribution of Column")

plt.show()

Variable Relationships

​

Visualize how variables are related using plots such as scatter plots or heatmaps; helps reveal patterns, trends, and associations between variables.

Correlation Matrix: To quickly examine relationships between numeric variables: df.corr()

• Computes pairwise correlations
• Returns a square matrix
• Each value shows the linear relationship between two variables

Pearson Correlation (r): Measures strength and direction of a linear relationship.

Range: −1 to 1

• +1 → perfect positive relationship
• −1 → perfect negative relationship
• 0 → no linear relationship

Key Idea: Correlation summarizes how strongly two variables move together.

Example: Correlation

df["X"].corr(df["Y"])

Examples: Variable Relationships

​

​

​

​

​

​

​

Example: You gather a sample of 5,000 college graduates and survey them on their high school SAT scores and college GPAs. You visualize the data to check for a linear pattern:

Scatter Plot = Visualize the Relationship

A scatter plot shows how two numerical variables relate by plotting each observation as a point (shows the relationship pattern between two variables).

df.plot.scatter("col1", "col2")

plt.title("Scatter Plot: Variable Relationship")

plt.show()

Remember: Scatter plot visualizes the relationship (correlation and covariance quantify it.)

Correlation vs Covariance

Covariance → How two variables move together�

​

​

Measures joint deviation from the mean.

Example: df["X"].cov(df["Y"])

Interpretation

• Positive → move together
• Negative → move opposite
• Magnitude depends on units

Advanced: Event Frequency Over a Continuous Variable (e.g., Time)

How often specific events occur across a continuous variable (most commonly time).

Purpose:

• Identify patterns or trends
• Detect peaks or clusters of activity
• Observe changes over time�

Example: Count events by date

df["date"] = pd.to_datetime(df["date"])

df["date"].value_counts().sort_index().plot()

plt.xlabel("Time")

plt.ylabel("Event Count")

plt.title("Events Over Time")

plt.show()

​

Key Idea: Visualizing event frequency over time helps reveal when activity increases, decreases, or follows recurring patterns.

Advanced: Event Timeline Using a Scatter Plot

How? Create a scatter plot where each dot represents one event along a continuous

variable (e.g., time). Use alpha to make overlapping points transparent so dense areas are easier to see.

df["zeroes"] = 0

df.plot.scatter("days", "zeroes", alpha=0.5)

Purpose:

• Creates a constant column (zeroes = 0)
• Plots time (days) on the x-axis
• Places every event at y = 0
• Each dot = one event occurrence

​

This is a simple way to visualize when events happen along a continuous scale, especially time.

Sometimes you want to show when events happened, not how many. A simple trick: place all events along a single horizontal line.

Advanced: Checking Seasonal Patterns in Temporal Data

For time-based data, always check for seasonality (yearly, monthly, weekly patterns).

One simple approach is to map time into a repeating cycle and visualize it.

​

​

# Calculate day of year (0-364)

df["day_of_year"] = df["days_since_start] % 365

​

# Create color gradient (red intensity based on day of year)

df["color"] = df["day_of_year"].apply(lambda d: (d/365, 0, 0))

​

# Plot with colors

df.plot.scatter("day_of_year", "x", c=df["color"])

In this graph, each shade of red corresponds to one specific year.

Idea: If patterns repeat at similar times in the cycle, you likely have seasonality.

Statistical Significance

Identifying Errors

If you have two datasets with different averages, you aren’t allowed to assume they come from different distributions! Oftentimes you’ll need to run a ttest:

stats.ttest_ind(df[x], df[y])

Look for:

• Duplicated values. Compare the relative length of len(df) and len(df.drop_duplicates())
• len(df) =the total number of rows in the original DataFrame
• len(df.drop_duplicates()) = the number of rows after removing duplicates.
• compare
• Check for missing values: sum(df[“column”].isna()) will tell you how many missing values are in a series
• Check for outliers: These should show up in your box and whisker plot.
• Make sure you understand why your outliers are outliers.

Finishing

Once we are done, we should be an expert on our data.

• Understand all data ranges
• helps to identify outliers, anomalies, and extreme values that may require further investigation or treatment.
• Know what variables relate to each other
• identify correlations and dependencies between variables
• Know where your errors are
• handling missing values, correcting data entry mistakes, and addressing outliers.
• Errors in the data can lead to incorrect analysis and conclusions,
• Have an idea of what other data you may need

Some Commonly used Pandas syntax patterns for dataframe “df”

​

Explore by yourself!

https://colab.research.google.com/drive/1OGNLwvqDhLyxTtyoRUidXdHPu-gG2Fl2?usp=sharing

​

Applying Functions

Applying Functions Directly: You can apply functions directly to a DataFrame column:

df[column_name].function()

​

Applying Custom Functions to a Dataframe Column: df['column_name'].apply(custom_function)

Example: df['frequency'].sum()

# Define a custom function to double a value

Def double_value(value):

return value * 2

​

# Apply the custom function to the 'values' column and create a new column 'doubled_values'

df['doubled_values'] = df['values'].apply(double_value)

​

Or use lambda:

df['doubled_values'] = df['values'].apply(lambda x: x**2))

Create a new column in the DataFrame: df['new_col_name']= df['other_column_name'] .apply(custom_function)

In-Place Operations

​

Perform in-place operations on a column.

# In-place operation

df['column_name'].function(value, inplace=True)

​

# Non-in-place operation (default behavior)

df_copy = df['column_name'].function (value, inplace=False)

​

Commonly used syntax patterns for df

​

Filtering Rows Based on a Condition You can filter rows in a DataFrame based on a condition and then apply statistical functions to a specific column:

​

df[df['column_name'] > value]

boolean_condition

df['Quantity_Kg'] > 5

Q: What about multiple condition?

Grouping and Aggregating Data

You can group data by a condition and calculate statistics within each group:

df.groupby('condition')[column_name].statistics_function()

​

​

df.groupby('Category')['Price_USD'].mean()