1 of 15

Exploratory Data Analysis�By Nidhi Ranjan�

1

2 of 15

What is Exploratory Data Analysis

2

EDA is an approach for data analysis using variety of techniques to gain insights about the data.

    • Cleaning and preprocessing
    • Statistical Analysis
    • Visualization for trend analysis, anomaly detection, outlier detection (and removal).

Basic steps in any exploratory data analysis:

3 of 15

Importance of EDA

3

Improve understanding of variables by extracting averages, mean, minimum, and maximum values, etc.

Discover errors, outliers, and missing values in the data.

Identify patterns by visualizing data in graphs such as bar graphs, scatter plots, heatmaps and histograms.

4 of 15

EDA using Pandas

4

Import data into workplace(Jupyter notebook, Google colab, Python IDE)

Descriptive statistics

Removal of nulls

Visualization

5 of 15

6 of 15

1. Packages and data import

  • Step 1 : Import pandas to the workplace.
        • “Import pandas”

​

  • Step 2 : Read data/dataset into Pandas dataframe. Different input formats include:
      • Excel : read_excel
      • CSV: read_csv
      • JSON: read_json
      • HTML and many more

​

6

7 of 15

2. Descriptive Stats (Pandas)

  • Used to make preliminary assessments about the population distribution of the variable.
  • Commonly used statistics:

1. Central tendency :

      • Mean – The average value of all the data points. : dataframe.mean()
      • Median – The middle value when all the data points are put in an ordered list: dataframe.median()
      • Mode – The data point which occurs the most in the dataset :dataframe.mode()

2. Spread : It is the measure of how far the datapoints are away from the mean or median

      • Variance - The variance is the mean of the squares of the individual deviations: dataframe.var()
      • Standard deviation - The standard deviation is the square root of the variance:dataframe.std()

3. Skewness: It is a measure of asymmetry: dataframe.skew()

7

8 of 15

Descriptive Stats (contd.)

Other methods to get a quick look on the data:

  • Describe() : Summarizes the central tendency, dispersion and shape of a dataset’s distribution, excluding NaN values.
      • Syntax: pandas.dataframe.describe()
  • Info() :Prints a concise summary of the dataframe. This method prints information about a dataframe including the index dtype and columns, non-null values and memory usage.
      • Syntax: pandas.dataframe.info()

8

9 of 15

3. Null values

9

Detecting

Detecting Null-values:

    • Isnull(): It is used as an alias for dataframe.isna(). This function returns the dataframe with boolean values indicating missing values.
    • Syntax : dataframe.isnull()

Handling

Handling null values:

    • Dropping the rows with null values: dropna() function is used to delete rows or columns with null values.
    • Replacing missing values: fillna() function can fill the missing values with a special value value like mean or median.

10 of 15

4. Visualization

  • Univariate: Looking at one variable/column at a time
    • Bar-graph
    • Histograms
    • Boxplot
  • Multivariate : Looking at relationship between two or more variables
    • Scatter plots
    • Pie plots
    • Heatmaps(seaborn)

​

10

11 of 15

Bar-Graph,Histogram and Boxplot

  • Bar graph: A bar plot is a plot that presents data with rectangular bars with lengths proportional to the values that they represent.
  • Boxplot : Depicts numerical data graphically through their quartiles. The box extends from the Q1 to Q3 quartile values of the data, with a line at the median (Q2).
  • Histogram: A histogram is a representation of the distribution of data.

​

11

12 of 15

Scatterplot, Pieplot

  • Scatterplot : Shows the data as a collection of points.
        • Syntax: dataframe.plot.scatter(x = 'x_column_name', y = 'y_columnn_name’)

​

  • Pie plot : Proportional representation of the numerical data in a column.
        • Syntax: dataframe.plot.pie(y=‘column_name’)

​

12

13 of 15

Outlier detection

  • An outlier is a point or set of data points that lie away from the rest of the data values of the dataset..
  • Outliers are easily identified by visualizing the data.
  • For e.g.
    • In a boxplot, the data points which lie outside the upper and lower bound can be considered as outliers
    • In a scatterplot, the data points which lie outside the groups of datapoints can be considered as outliers

​

​

13

14 of 15

Outlier removal

  • Calculate the IQR as follows:
      • Calculate the first and third quartile (Q1 and Q3)
      • Calculate the interquartile range, IQR = Q3-Q1
      • Find the lower bound which is Q1*1.5
      • Find the upper bound which is Q3*1.5
      • Replace the data points which lie outside this range.
      • They can be replaced by mean or median.

14

15 of 15

References

​

​

​

15