DS161 Introduction to Data Science & Artificial Intelligence

Self-Study 6

Exploring and Understanding Real-World Data

Krishnendu Ghosh

What is Real-World Data?

Real-world data is rarely clean, complete, or perfectly structured.

​

It may contain:

• Missing values
• Duplicate records
• Incorrect data types
• Inconsistent categories
• Outliers
• Noisy or erroneous values
• Irrelevant features

​

Goal: understand and prepare the data.

Example: Titanic Dataset

Load the Dataset

import pandas as pd

​

df = pd.read_csv("train.csv")

​

print(df.head())

​

​

​

• What does our data actually look like?

Exploring the Dataset

df.shape

df.columns

df.info()

df.describe()

​

• How many observations?
• How many variables?
• Which variables are numerical?
• Which variables are categorical?
• Are there missing values?

Looking at the Raw Data

​

​

​

​

​

​

​

​

​

​

​

​

df.head()

​

Observation: Different columns contain different types of information.

Real Data Is Not Always Ready

Imagine that the Titanic data was collected from multiple sources.

​

During data collection:

• Some ages were not recorded.
• Some passenger categories were entered differently.
• Some fares were incorrectly recorded.
• Some records were duplicated.
• Some values were entered as text instead of numbers.

​

• Can we detect and fix these problems using Python?

Create a Modified Dataset

import pandas as pd

import numpy as np

​

data = df.copy()

​

data.loc[10:20, "Age"] = np.nan

​

data.loc[30:35, "Sex"] = " Male "

​

data.loc[40:45, "Embarked"] = "southampton"

​

data.loc[50:55, "Fare"] = "unknown"

make a copy

introduce some problems

Introduce Duplicate Records

duplicates = data.iloc[100:105].copy()

​

data = pd.concat(

[data, duplicates],

ignore_index=True

)

​

Detect Missing Values

data.isnull().sum()

​

import matplotlib.pyplot as plt

​

data.isnull().sum().plot(kind="bar")

​

plt.title("Missing Values by Feature")

plt.ylabel("Number of Missing Values")

plt.show()

Detect Duplicate Records

data.duplicated().sum()

​

data[data.duplicated()]

​

data = data.drop_duplicates()

​

• Is this actually a duplicate observation, or could it represent a legitimate repeated event?

Data Types Matter

data.dtypes

​

pd.to_numeric(data["Fare"], errors="coerce")

​

data["Fare"] = pd.to_numeric(

data["Fare"],

errors="coerce"

)

Inconsistent Categories

data["Sex"].unique()

​

data["Sex"] = (

data["Sex"]

.str.strip()

.str.lower()

)

​

data["Sex"].unique()

Cleaning Categorical Data

data["Embarked"] = (

data["Embarked"]

.str.strip()

.str.upper()

)

​

data["Embarked"].value_counts()

​

The computer treats "S", "s" and " S " as different values.

What to do With Missing Data?

Option 1

Remove rows.

data.dropna(subset=["Age"])

Option 2

Replace with mean.

data["Age"].fillna(

data["Age"].mean() Which method should we choose?

)

Option 3

Replace with median.

data["Age"].fillna(

data["Age"].median()

)

Mean vs Median

Consider:

​

18, 20, 21, 22, 23, 24, 25, 90

​

Mean is strongly affected by the extreme value 90.

​

Median is more robust.

​

• The appropriate treatment depends on the distribution and meaning of the variable.

Handling Missing Age

data["Age"] = data["Age"].fillna(

data["Age"].median()

)

​

data["Age"].isnull().sum()

Numerical Summary

data.describe()

​

This provides:

• Count
• Mean
• Standard deviation
• Minimum
• Quartiles
• Maximum

​

data["Age"].describe()

Understanding Distributions

data["Age"].hist(bins=20)

​

plt.xlabel("Age")

plt.ylabel("Number of Passengers")

plt.title("Age Distribution")

plt.show()

​

• What does the distribution tell us?

Categorical Data

data["Sex"].value_counts()

​

data["Sex"].value_counts().plot(

kind="bar"

)

​

plt.title("Passenger Distribution by Sex")

plt.show()

​

data["Pclass"].value_counts()

data["Embarked"].value_counts()

Does Passenger Class Matter?

data.groupby("Pclass")["Survived"].mean()

​

data.groupby("Pclass")["Survived"].mean().plot(

kind="bar"

)

​

plt.ylabel("Survival Rate")

plt.show()

​

• What pattern do you observe?

Does Gender Matter?

data.groupby("Sex")["Survived"].mean()

​

data.groupby("Sex")["Survived"].mean().plot(

kind="bar"

)

​

plt.ylabel("Survival Rate")

plt.show()

Creating New Information

Sometimes useful information is hidden inside existing columns.

For example:

data["FamilySize"] = (

data["SibSp"] +

data["Parch"] +

1

)

​

Now we have:

SibSp + Parch + 1 → FamilySize

This is called feature engineering.

Creating Age Groups

data["AgeGroup"] = pd.cut(

data["Age"],

bins=[0, 12, 18, 35, 60, 100],

labels=[

"Child",

"Teen",

"Young Adult",

"Adult",

"Senior"

]

)

Data Quality After Cleaning

Raw Data

↓

Data Profiling

↓

Missing Value Handling

↓

Duplicate Removal

↓

Type Correction

↓

Category Standardization

↓

Feature Engineering

↓

Exploratory Analysis

↓

AI/ML-Ready Data

Complete Python Workflow

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

​

# 1. Load

df = pd.read_csv("train.csv")

​

# 2. Inspect

print(df.shape)

print(df.info())

print(df.describe())

​

# 3. Check missing values

print(df.isnull().sum())

​

# 4. Remove duplicates

df = df.drop_duplicates()

​

# 5. Clean categories

df["Sex"] = df["Sex"].str.strip().str.lower()

df["Embarked"] = df["Embarked"].str.strip().str.upper()

# 6. Fix numeric data

df["Fare"] = pd.to_numeric(

df["Fare"],

errors="coerce"

)

​

# 7. Handle missing values

df["Age"] = df["Age"].fillna(

df["Age"].median()

)

​

# 8. Feature engineering

df["FamilySize"] = (

df["SibSp"] + df["Parch"] + 1

)

​

# 9. Explore

print(df.groupby("Pclass")["Survived"].mean())