1 of 57

CSE & II SEM

PYTHON PROGRAMMING AND DATA SCIENCE

PPDS

III

Mr B.DILIP CHAKRAVARTHY

DEPT & SEM : ��SUBJECT NAME :��COURSE CODE :��UNIT :��PREPARED BY :

COURSE: PPDS UNIT: 3 Pg. 1

2 of 57

CONTENTS

Introduction to NumPy
Pandas
Matplotlib
Exploratory Data Analysis (EDA)
Data Science life cycle
Descriptive Statistics
Basic tools (plots, graphs and summary statistics) of EDA
Philosophy of EDA

COURSE: PPDS UNIT: 3 Pg. 2

3 of 57

CONTENTS

Data Visualization:

. Scatter plot

. bar chart

. histogram

. boxplot

. heat maps

COURSE: PPDS UNIT: 3 Pg. 3

4 of 57

Introduction to NumPy

NumPy

COURSE: PPDS UNIT: 3 Pg. 4

5 of 57

Introduction to NumPy

Data Science: is a branch of computer science where we study how to store, use and analyze data for deriving information from it.
What is NumPy? NumPy stands for Numerical Python.

NumPy is a Python library used for working with arrays.

It also has functions for working in domain of linear algebra, fourier transform,and matrices.

NumPy was created in 2005 by Travis Oliphant. It is an open source project and you can use it freely.

Why Use NumPy?
In Python we have lists that serve the purpose of arrays, but they are slow to process.
NumPy aims to provide an array object that is up to 50x faster than traditional Python lists.

COURSE: PPDS UNIT: 3 Pg. 5

6 of 57

Introduction to NumPy

The array object in NumPy is called ndarray, it provides a lot of supporting functions that make working with ndarray very easy.
Why is NumPy Faster Than Lists?
NumPy arrays are stored at one continuous place in memory unlike lists, so processes can access and manipulate them very efficiently. This behaviour is called locality of reference in computer science. This is the main reason why NumPy is faster than lists. Also it is optimized to work with latest CPU architectures.
Which Language is NumPy written in?
NumPy is a Python library and is written partially in Python, but most of the parts that require fast computation are written in C or C++.

COURSE: PPDS UNIT: 3 Pg. 6

7 of 57

Introduction to NumPy

Installation of NumPy:
If you have Python and PIP already installed on a system, then installation of NumPy is very easy.
Install it using this command
C:\Users\Your Name>pip install numpy
If this command fails, then use a python distribution that already has NumPy installed like, Anaconda, Spyder etc.
Import NumPy
Once NumPy is installed, import it in your applications by adding the import keyword:
import numpy
Now NumPy is imported and ready to use.

COURSE: PPDS UNIT: 3 Pg. 7

8 of 57

Introduction to NumPy

Example
import numpy
arr=numpy.array([1,2,3,4])
print(arr)
[1 2 3 4 ]

NumPy as np
NumPy is usually imported under the np alias.
alias: In Python alias are an alternate name for referring to the same thing.
Create an alias with the as keyword while importing:
import numpy as np
Now the NumPy package can be referred to as np instead of numpy.

COURSE: PPDS UNIT: 3 Pg. 8

9 of 57

Introduction to NumPy

Eg:
import numpy as np
arr=np.array([1,2,3,4])
print(arr)
[1 2 3 4 ]

Checking NumPy Version
The version string is stored under __version__ attribute.
Eg:
print(np.__version__) # 1.16.2

COURSE: PPDS UNIT: 3 Pg. 9

10 of 57

Introduction to NumPy

type():
This built-in Python function tells us the type of the object passed to it. Like in above code it shows that arr is numpy.ndarray type.
import numpy as np
arr=np.array([1,2,3,4])
print(arr)
print(type(arr))
o/p:
[1 2 3 4]
<class 'numpy.ndarray’>

COURSE: PPDS UNIT: 3 Pg. 10

11 of 57

Introduction to NumPy

NumPy Creating Arrays
To create an ndarray, we can pass a list, tuple or any array-like object into the array() method, and it will be converted into an ndarray:
Use a List to create a NumPy array:
import numpy as np
arr=np.array([1,2,3,4])
print(arr)
[1 2 3 4]
Creating arrays with different dimensions
A dimension in arrays is one level of array depth (nested arrays).
nested array: are arrays that have arrays as their elements.

COURSE: PPDS UNIT: 3 Pg. 11

12 of 57

Introduction to NumPy

0-D Arrays
0-D arrays, or Scalars, are the elements in an array. Each value in an array is a 0-D array.
arr=np.array(42)
print(arr)
o/p: 42
1-D Arrays
An array that has 0-D arrays as its elements is called uni-dimensional or 1-D array.
These are the most common and basic arrays.
arr=np.array([1,2,3,4])
print(arr) # [1 2 3 4]

COURSE: PPDS UNIT: 3 Pg. 12

13 of 57

Introduction to NumPy

2-D Arrays
An array that has 1-D arrays as its elements is called a 2-D array. These are often used to represent matrix or 2nd order tensors.

Create a 2-D array containing two arrays with the values 1,2,3 and 4,5,6:
arr=np.array([[1,2,3],[4,5,6]])
print(arr)
[[1 2 3]
[4 5 6]]

COURSE: PPDS UNIT: 3 Pg. 13

14 of 57

Introduction to NumPy

3-D arrays
An array that has 2-D arrays (matrices) as its elements is called 3-D array. These are often used to represent a 3rd order tensor.
Create a 3-D array with two 2-D arrays, both containing two arrays with the values 1,2,3 and 4,5,6:
arr=np.array([[[1,2,3],[4,5,6]],[[1,2,3],[4,5,6]]])
print(arr)
[[[1 2 3]
[4 5 6]]

[[1 2 3]
[4 5 6]]]

COURSE: PPDS UNIT: 3 Pg. 14

15 of 57

Introduction to NumPy

Array Properties or characteristics
1. size
2. ndim
3. shape
4. dtype
size()
The NumPy array object has a property called size that returns the total no of elements present in array
arr=np.array([1, 2, 3, 4])
print(np.size(a))

COURSE: PPDS UNIT: 3 Pg. 15

16 of 57

Introduction to NumPy

ndim()
NumPy Arrays provides the ndim attribute that returns an integer that tells us how many dimensions the array have.
Example
Check how many dimensions the arrays have:
a=np.array(42)
b=np.array([1,2,3])
c=np.array([[1,2,3],[4,5,6]])
print(a.ndim) #0
print(b.ndim) #1
print(c.ndim) #2

COURSE: PPDS UNIT: 3 Pg. 16

17 of 57

Introduction to NumPy

shape()
The shape of an array is the number of elements in each dimension.
import numpy as np
arr=np.array([1, 2, 3, 4])
print(arr.shape) (4,)
arr=np.array([[1, 2, 3, 4], [5, 6, 7, 8]])
print(arr.shape) (2, 4)
The example above returns (2, 4), which means that the array has 2 dimensions, and each dimension has 4 elements.

COURSE: PPDS UNIT: 3 Pg. 17

18 of 57

Introduction to NumPy

dtype()
The NumPy array object has a property called dtype that returns the data type of the elements used in array
Data Types in NumPy
NumPy has some extra data types, and refer to data types with one character, like i for integers, u for unsigned integers etc.
Below is a list of all data types in NumPy and the characters used to represent them.
• i - integer
• b - boolean
• u - unsigned integer
• f - float

COURSE: PPDS UNIT: 3 Pg. 18

19 of 57

Introduction to NumPy

• c - complex float
• m - timedelta
• M - datetime
• O - object
• S - string
• U - unicode string
• V - fixed chunk of memory for other type ( void )
arr=np.array([10,20,30])
print(arr.dtype) # int64

COURSE: PPDS UNIT: 3 Pg. 19

20 of 57

Introduction to NumPy

Creating Arrays With a Defined Data Type
dtype also allows us to define the expected data type of the array elements:
import numpy as np
arr=np.array([1, 2, 3, 4], dtype='f')
print(arr)
print(arr.dtype)
[1. 2. 3. 4.]
float32

COURSE: PPDS UNIT: 3 Pg. 20

21 of 57

Introduction to NumPy

Initialization of numpy arrays
1. Initialize arrays with zeros
2. Initialize arrays with full
3. Initialize arrays with random.rand
4. Initialize arrays with ones
5. Initialize arrays with eye

COURSE: PPDS UNIT: 3 Pg. 21

22 of 57

Introduction to NumPy

initialize arrays with zeros
a=np.zeros(3)
print(a)
[0. 0. 0.]

a=np.zeros([2,3])
print(a)
[[0. 0. 0.]
[0. 0. 0.]]

COURSE: PPDS UNIT: 3 Pg. 22

23 of 57

Introduction to NumPy

a=np.zeros([2,3,4])
print(a)
[[[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]

[[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]]]

COURSE: PPDS UNIT: 3 Pg. 23

24 of 57

Introduction to NumPy

initialize arrays with full
a=np.full([2,3],5) which fills all values with 5
print(a)
[[5 5 5]
[5 5 5]]
initialize arrays with random.rand
a=np.random.rand(10) which prints random values which are unpredictable
print(a)
[0.21490551 0.72400899 0.19993116 0.66810077 0.22899403 0.09913034
0.91488207 0.33019666 0.65544687 0.24655334]

COURSE: PPDS UNIT: 3 Pg. 24

25 of 57

Introduction to NumPy

initialize arrays with ones
a=np.ones([2,3]) all elements filled with 1
print(a)
[[1. 1. 1.]
[1. 1. 1.]]
initialize arrays with eye
a=np.eye(3) diagonal elements filled with 1
print(a)
[[1. 0. 0.]
[0. 1. 0.]
[0. 0. 1.]]

COURSE: PPDS UNIT: 3 Pg. 25

26 of 57

Introduction to NumPy

Numerical ranges
arange
This function returns an ndarray object containing evenly spaced values within a given range. The format of the function is as follows −
numpy.arange(start, stop, step, dtype)

start:The start of an interval. If omitted, defaults to 0
stop:The end of an interval (not including this number)
step:Spacing between values, default is 1
dtype: Data type of resulting ndarray. If not given, data type of input is used

COURSE: PPDS UNIT: 3 Pg. 26

27 of 57

Introduction to NumPy

Eg1:import numpy as np
x = np.arange(5)
print(x)
[0 1 2 3 4]
Eg2:import numpy as np
x = np.arange(5, dtype = float)
print(x) [0. 1. 2. 3. 4.]
Eg3:import numpy as np
x = np.arange(10,20,2)
print(x) [10 12 14 16 18]

COURSE: PPDS UNIT: 3 Pg. 27

28 of 57

Introduction to NumPy

linspace()
np.linspace(start,stop,num,endpoint,retstep,dtype)
np.linspace(10,100,5)
[ 10. , 32.5, 55. , 77.5, 100. ]
np.linspace(10,100,5,endpoint=False)
[10., 28., 46., 64., 82.]
np.linspace(10,100,5,endpoint=False,retstep=True)
([10., 28., 46., 64., 82.]), 18.0
np.linspace(10,100,5,endpoint=False,retstep=True,dtype=int)
([10, 28, 46, 64, 82]), 18.0

COURSE: PPDS UNIT: 3 Pg. 28

29 of 57

Introduction to NumPy

logspace() default base=10
np.logspace(start,stop,num,base,dtype)
np.logspace(1,10,5)
[1.00000000e+01, 1.77827941e+03, 3.16227766e+05, 5.62341325e+07,
1.00000000e+10]
np.logspace(1,10,5,base=2)
array([ 2. , 9.51365692, 45.254834 , 215.2694823 ,
1024. ])
np.logspace(1,10,5,base=2,dtype=int)
[ 2, 9, 45, 215, 1024]

COURSE: PPDS UNIT: 3 Pg. 29

30 of 57

Introduction to NumPy

copy()
import numpy as np
arr=np.array([1, 2, 3, 4, 5])
x=arr.copy()
arr[0]= 42
print(arr)
print(x)
[42 2 3 4 5]
[1 2 3 4 5]
He copy SHOULD NOT be affected by the changes made to the original array.

COURSE: PPDS UNIT: 3 Pg. 30

31 of 57

Introduction to NumPy

Make changes in the copy:
import numpy as np
arr=np.array([1, 2, 3, 4, 5])
x=arr.copy()
x[0]= 31
print(arr)
print(x)
[1 2 3 4 5]
[31 2 3 4 5]

COURSE: PPDS UNIT: 3 Pg. 31

32 of 57

Introduction to NumPy

view()
import numpy as np
arr=np.array([1, 2, 3, 4, 5])
x=arr.view()
arr[0]= 42
print(arr)
print(x)
[42 2 3 4 5]
[42 2 3 4 5]
The view SHOULD be affected by the changes made to the original array.

COURSE: PPDS UNIT: 3 Pg. 32

33 of 57

Introduction to NumPy

Make changes in the view:
import numpy as np
arr=np.array([1, 2, 3, 4, 5])
x=arr.view()
x[0]= 31
print(arr)
print(x)
[31 2 3 4 5]
[31 2 3 4 5]
The original array SHOULD be affected by the changes made to the view.

COURSE: PPDS UNIT: 3 Pg. 33

34 of 57

Introduction to NumPy

reshape()
Reshaping means changing the shape of an array.
import numpy as np
arr=np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])
newarr=arr.reshape(4, 3)
print(newarr)
[[ 1 2 3]
[ 4 5 6]

[ 7 8 9]
[10 11 12]

COURSE: PPDS UNIT: 3 Pg. 34

35 of 57

Introduction to NumPy

flatten()
to get a copy of array collapsed into one dimension
arr=np.array([1,2,3],[4,5,6]])
a=arr.flatten()
[1 2 3 4 5 6]

COURSE: PPDS UNIT: 3 Pg. 35

36 of 57

Introduction to NumPy

Accessing Array Elements:
You can access an array element by referring to its index number.
The indexes in NumPy arrays start with 0, meaning that the first element has index 0, and the second has index 1 etc.
arr=np.array([1,2,3])
print(arr[0]) 1
print(arr[1]) 2
print(arr[2]) 3

COURSE: PPDS UNIT: 3 Pg. 36

37 of 57

Introduction to NumPy

Access 2-D Arrays
To access elements from 2-D arrays we can use comma separated integers representing the dimension and the index of the element.
Example
Access the 2nd element on 1st dim:
import numpy as np
arr=np.array([[1,2,3,4,5],[6,7,8,9,10]])
print('2nd element on 1st dim: ', arr[0, 1])
2nd element on 1st dim: 2

COURSE: PPDS UNIT: 3 Pg. 37

38 of 57

Introduction to NumPy

Access 3-D Arrays
To access elements from 3-D arrays we can use comma separated integers representing the dimensions and the index of the element.
Example
Access the third element of the second array of the first array:
import numpy as np
arr=np.array([[[1, 2, 3],[4, 5, 6]],[[7, 8, 9],[10, 11, 12]]])
print(arr[0, 1, 2])
6

COURSE: PPDS UNIT: 3 Pg. 38

39 of 57

Introduction to NumPy

Negative Indexing
Use negative indexing to access an array from the end.
Example
Print the last element from the 2nd dim:
import numpy as np
arr=np.array([[1,2,3,4,5],[6,7,8,9,10]])
print('Last element from 2nd dim: ', arr[1, -1])
Last element from 2nd dim: 10

COURSE: PPDS UNIT: 3 Pg. 39

40 of 57

Introduction to NumPy

Array Slicing
Slicing in python means taking elements from one given index to another given index.
We pass slice instead of index like this: [start:end].
We can also define the step, like this: [start:end:step].
If we don't pass start its considered 0
If we don't pass end its considered length of array in that dimension
If we don't pass step its considered 1

COURSE: PPDS UNIT: 3 Pg. 40

41 of 57

Introduction to NumPy

Example
Slice elements from index 1 to index 5 from the following array:
import numpy as np
arr=np.array([1, 2, 3, 4, 5, 6, 7])
print(arr[1:5])
OUTPUT:
[2 3 4 5]
eg2: print(arr[4:]) [5 6 7]
eg3: print(arr[:4]) [1 2 3 4]

COURSE: PPDS UNIT: 3 Pg. 41

42 of 57

Introduction to NumPy

Slicing 2-D Arrays
Example
From the second element, slice elements from index 1 to index 4 (not included):
import numpy as np
arr=np.array([[1, 2, 3, 4, 5],[6, 7, 8, 9, 10]])
print(arr[1, 1:4])
[7 8 9]

COURSE: PPDS UNIT: 3 Pg. 42

43 of 57

Introduction to NumPy

Negative Slicing
Use the minus operator to refer to an index from the end:
Example
Slice from the index 3 from the end to index 1 from the end:
import numpy as np
arr=np.array([1, 2, 3, 4, 5, 6, 7])
print(arr[-3:-1])
[5 6]

COURSE: PPDS UNIT: 3 Pg. 43

44 of 57

Introduction to NumPy

Arithmetic Operations on single array using scalar
a=np.array([1,2,5,3])
a+1 [2 3 6 4]
a-3 [-2 -1 2 0]
a*10 [10 20 50 30]
a**2 [ 1 4 25 9]
a/2 [0.5 1. 2.5 1.5]

COURSE: PPDS UNIT: 3 Pg. 44

45 of 57

Introduction to NumPy

Transpose
a=np.array([[1,2,3],[4,5,6],[7,8,9]])
print(a)
[[1 2 3]
[4 5 6]
[7 8 9]]
print(a.T) #print(a.transpose())
[[1 4 7]
[2 5 8]
[3 6 9]]

COURSE: PPDS UNIT: 3 Pg. 45

46 of 57

Introduction to NumPy

Operations on two arrays
a=np.array([[1,2],[3,4]])
b=np.array([[4,5],[6,7]])
print(a+b) [[ 5 7]
[ 9 11]]
print(a-b) [[-3 -3]
[-3 -3]]
print(a*b) [[ 4 10]
[18 28]]
print(a@b) [[16 19]
[36 43]]

COURSE: PPDS UNIT: 3 Pg. 46

47 of 57

Introduction to NumPy

Statistical operations
Numpy provides functions to perform statistical operations on numpy
a=np.array([1,0,2,-3,6,8,4,7])
b=([[3,6],[4,2]])
a.max() 8
b.max(axis=1) [6,4]
a.min()
a.sum()
a.mean()
a.std()

COURSE: PPDS UNIT: 3 Pg. 47

48 of 57

Introduction to NumPy

Array functions(ufunc)
using one array
a=np.array([30,40,20,10])
sin()
np.sin(a) [-0.98803162 0.74511316 0.91294525 -0.54402111]
cos()
np.cos(a) [ 0.15425145 -0.66693806 0.40808206 -0.83907153]
exp()
np.exp(a) [1.06864746e+13 2.35385267e+17 4.85165195e+08 2.20264658e+04]

COURSE: PPDS UNIT: 3 Pg. 48

49 of 57

Introduction to NumPy

sqrt()
np.sqrt(a) [5.47722558 6.32455532 4.47213595 3.16227766]
insert()
np.insert(a,2,25) [30 40 25 20 10]
delete()
np.delete(a,2) [30 40 10]
Split()
a=np.array([1, 2, 3, 4, 5, 6])
np.array_split(a, 3)
[array([1, 2]), array([3, 4]), array([5, 6])]

COURSE: PPDS UNIT: 3 Pg. 49

50 of 57

Introduction to NumPy

Search()
np.where(a== 4) (array([3], dtype=int64),)
mean()
np.mean(a) 3.5
a=np.array([[10,30,20],[20,40,50]])
print(np.mean(a)) 28.333333333333332
print(np.mean(a,axis=1)) [20. 36.66666667]
print(np.mean(a,axis=0)) [15. 35. 35.]
median()
np.median(a) 25.0
print(np.median(a,axis=1)) [20. 40.]

COURSE: PPDS UNIT: 3 Pg. 50

51 of 57

Introduction to NumPy

var()
np.var(a) 180.55555555555557
print(np.var(a,axis=1)) [ 66.66666667 155.55555556]
std()
np.std(a) 13.43709624716425
print(np.std(a,axis=1)) [ 8.16496581 12.47219129]
amin()
np.amin(a) 10
print(np.amin(a,axis=1)) [10 20]
sort()
print(np.sort(a,axis=1)) [[10 20 30]
[20 40 50]]

COURSE: PPDS UNIT: 3 Pg. 51

52 of 57

Introduction to NumPy

amax()
np.amax(a) 50
print(np.amax(a,axis=1)) [30 50]
sum()
np.sum(a) 170
print(np.sum(a,axis=1)) [ 60 110]
reciprocal()
print(np.reciprocal(a,dtype=float))
[[0.1 0.03333333 0.05 ]
[0.05 0.025 0.02 ]]

COURSE: PPDS UNIT: 3 Pg. 52

53 of 57

Introduction to NumPy

using two arrays
a=np.array([10,20,30])
b=np.array([40,50,60])
np.append(a,b) [10 20 30 40 50 60]
np.add(a,b) [50 70 90]
np.subtract(a,b) [-30 -30 -30]
np.multiply(a,b) [ 400 1000 1800]
np.divide(a,b) [0.25 0.4 0.5 ]
np.mod(a,b) [10 20 30]

COURSE: PPDS UNIT: 3 Pg. 53

54 of 57

Introduction to NumPy

Advanced Indexing:
It is possible to make a selection from ndarray that is a non-tuple sequence, ndarray object of integer or Boolean data type. Advanced indexing always returns a copy of the data. As against this, the slicing only presents a view.
There are two types of advanced indexing − Integer and Boolean.
Integer Indexing:
Generally an index is a number.In integerindexing we can give index as a array or a list or a tuple of integers

COURSE: PPDS UNIT: 3 Pg. 54

55 of 57

Introduction to NumPy

Eg: a=np.arange(1,10)
print(a) array([1,2,3,4,5,6,7,8,9])
y=np.array([1,4,5])
a[y]
array((2,5,6])
Eg:
x = np.array([[1, 2], [3, 4], [5, 6]]) #suppose if I want 1 4 5,first we take row index of 1,4,5
y = x[[0,1,2], [0,1,0]] #the column index of 1,4,5
print y
[1 4 5]

COURSE: PPDS UNIT: 3 Pg. 55

56 of 57

Introduction to NumPy

Boolean Array Indexing
This type of advanced indexing is used when the resultant object is meant to be the result of Boolean operations, such as comparison operators.
a=np.array([1,-2,3],[4,-2,3])
print(a)
array([[1,-2,3],
[4,-2,3]])
a[a<0]
array([-2,-2])

COURSE: PPDS UNIT: 3 Pg. 56

57 of 57

Introduction to NumPy

import numpy as np
x = np.array([[ 0, 1, 2],[ 3, 4, 5],[ 6, 7, 8],[ 9, 10, 11]])
print x
print 'The items greater than 5 are:'
print x[x > 5]
[[ 0 1 2]
[ 3 4 5]
[ 6 7 8]
[ 9 10 11]]
The items greater than 5 are:
[ 6 7 8 9 10 11]

COURSE: PPDS UNIT: 3 Pg. 57