Week 2

Practical Data Transformation

Agenda

1. Data Reading & Data Cleaning

Break

• Exploratory Data Analysis

Grading

70% - Final Project

20% - Assignments

10% - Live Quizzes

+10 Points for every invited person.

​

What is Google Collab?

What data are we gonna work today?

Common Data Science Formats

• Comma-separated values (csv)
• XLSX
• ZIP
• Plain Text (txt)
• JSON
• HTML
• SQL

​

• Images
• Hierarchical Data Format
• PDF
• DOCX
• MP3
• MP4

​

​

Spreadsheet Format

In spreadsheet format, data is stored in cells. Each cell is organized in rows and columns. A column in the spreadsheet file can have different types.

Common File Formats for Spreadsheets

• CSV
• XLSX
• JSON
• HTML

How to read all this data formats?

How to read all this data formats?

How to read all this data formats?

CSV file format

​

​

Each line in CSV file represents an observation or commonly called a record. Each record may contain one or more fields which are separated by a comma.

​

CSV file format

​

Sometimes you may come across files where fields are not separated by using a comma but with other delimiter. For example: ‘ ; ’, ‘ \t ’, ‘ # ’ etc.

​

​

How to read CSV file?

XLSX file format

XLSX is a Microsoft Excel Open XML file format. It is an XML-based file format created by Microsoft Excel.

How to read XLSX?

JSON file format

JavaScript Object Notation(JSON) is a text-based open standard designed for exchanging the data over web.

How to read JSON?

HTML file format

Hypertext Markup Language (HTML) is the standard markup language for documents designed to be displayed in a web browser.

How to read HTML table?

Pandas IO tools

Ex: Load Dataset

df.shape quiz

Linear Regression - House Price/Number Rooms

Linear Regression - House Price/Number Rooms

Major Data Quality Issues

1. Duplicates
2. Missing Data
3. Incorrect Data
4. Inconsistent Formats
5. Insecure Data

​

Duplicates

​

​

How to deal with duplicates?

Ex. shape of the array after removing dub

Missing Data

Type of Missing Values

• Missing Completely at Random (MCAR)
• Missing at Random (MAR)
• Missing not at Random (MNAR)

Missing Completely at Random (MCAR)

​

​

• Probability for a data point to be missing is completely random.
• There’s no relationship between whether a data point is missing and any values in the data set, missing or observed.
• The missing data are just a random subset of the data.

Missing at Random (MAR)

​

​

• Probability for a data point to be missing is not related to the missing data, but it is related to some of the observed data.
• Example: ‘Whether or not someone answered #13 on your survey has nothing to do with the missing values, but it does have to do with the values of some other variable.’

Missing not at Random (MNAR)

• There is a relationship between the probability of a value to be missing and its values
• Example: ‘Survey with regard to drug usage. Individuals being surveyed could potentially leave fields blank if they used drugs that are currently illegal out of fear of being prosecuted.’

How to count Missing Data?

​

How to remove Missing Data?

SUM of missing values in column “X”

Incorrect Data

Inconsistent Formats

Insecure Data

Data security & privacy laws are being put into place giving business extra financial incentive to follow these newly placed laws.

With steep fines for non-compliance, insecure data is quickly becoming one of the most dangerous types of dirty data.

​

Insecure Data

How to deal with Insecure Data?

DROP NAME

Most common data types?

Exploratory Data Analysis

What is EDA?

Cleaning your data

Summarizing your data

Finding patterns

​

​

​

What is EDA?

Cleaning your data

Summarizing your data

Finding patterns

Telling a story with you data.

​

​

​

​

​

Why EDA?

Understand the quality of your data.

Gain some quick insights.

Find potential patterns.

​

​

​

​

​

​

Descriptive Statistics

​

Descriptive Statistics

​

The art of describing the basic features of your data.

The basis of all quantitative analysis of data.

Univariate Analysis

MAX: 9.000€

MEAN: 79.560€

MIN: 54.000€

Why is there a salary of -10.000€?

Technique #1 - Missing values?

Determine how many values are missing.

2 Missing values

Sometimes missing values, are not obvious

3 Missing values

Technique #2 - Distribution of the data

Describe your data with basic statistical functions

MAX: 96.000€

MEAN: 79.560€

MEDIAN: 79.560€

MIN: 54.000€

STD: 15.000€

​

​

​

What is the mean of this table?

???

What is the mean of this table?

Categorical data statistics are different.

​

Farmer Scientist?

???

What is the mean of this table?

Categorical data statistics are different.

​

Potato Scientist?

???

Categorical data statistics are different.

Unique Values:

Data Scientist

Potato Scientist

Data Engineer

Categorical data statistics are different.

Number of

Unique Values:

3

Categorical data statistics are different.

Value Counts:

Data Scientist: 3

Data Engineer: 3

Potato Scientist: 1

​

​

Technique #3 - Histogram Plot

Visualizing the shape tells you more about the data.

Technique #4 - Count Plot

Visualizing the shape tells you more about the data.

Technique #5 - Identifying Extreme Points

Some data points stick out.

The median salary is 74.000€?

500.000€ is way over the median.

We can use the previous techniques to identify these points as well.

We can use the previous techniques to identify these points.

Why is the salary so high?

It is hard to answer questions this question with one variable only.

Bivariate Analysis

Technique #6 - Compare Statistics

We can group previously analyzed statistics.

Technique #6 - Compare Statistics

We can group previously analyzed statistics.

Data Scientist

MAX: 74.000€

MEAN: 72.000€

​

CEO

MAX: 500.000€

MEAN: 500.000€

Technique #7 - Scatterplot

Great way to compare visually 2 continuous variables

Technique #7 - Scatterplot

Great way to compare visually 2 continuous variables

Technique #8 - Linear Correlation

Linear correlation refers to straight-line relationships between two variables.

​

Ok, so what.

What does this help me?

​

Ok, so what.

What does this help me?

​

​

​

Correlation measures the extent to which variables:

• depend on one another
• predict one another

Rainfall is positively correlated with amount of vegetation

​

Pollen count is positively correlated with bee activity.

​

​

Pollen count is positively correlated with bee activity.

​

Is bee activity the cause of the pollen count?