ניתוח נתונים ולמידת מכונה

אמית רננים – חלופה ליחידה 3

https://pandas.pydata.org/

מוטי בזר

mottibz@gmail.com

https://www.commridge.com

What is Pandas?

• Open-source data manipulation and analysis library for Python
• Essential for data science, machine learning, and data analysis tasks
• Provides high-performance, easy-to-use data structures and tools

2

Why use Pandas?

• Efficient handling of large datasets
• Intuitive data manipulation with DataFrames and Series
• Built-in data alignment and handling of missing data
• Powerful data merging and joining capabilities
• Time series functionality
• Integration with other Python libraries (NumPy, Matplotlib, Scikit-learn)

3

Before We Start – How To Load Datasets The Easy Way�Not needed when using Visual Studio Code

• Go to www.kaggle.com
• Use the search function to search for a dataset
• When the search results are displayed, click on the Datasets button on top
• Select the required dataset from the list and click on it
• Click on the Download button on top-right
• In the DOWNLOAD VIA selection box, select “Kaggle CLI”
• Copy the command that is displayed using the copy button (command box top-right corner)
• Paste the copied command to a Colab notebook’s cell
• Add the “!” character at the beginning of the line. It will look like “!kaggle datasets…”
• Run the cell. The dataset will be downloaded to Colab’s Files section
• Running this command again will not download the dataset again as it will notice that it was already downloaded
• Add a new cell below and write the command “!unzip –o <file name>”
• <file name> is the name of the downloaded file
• Run the cell. The dataset file(s) will be extracted from the downloaded zip file

4

How To Load Datasets With Visual Studio Code

• Go to www.kaggle.com
• Use the search function to search for a dataset
• When the search results are displayed, click on the Datasets button on top
• Select the required dataset from the list and click on it
• Click on the Download button on top-right
• Select the “Download dataset as zip”
• The dataset file will be downloaded to your PC
• Uncompress the zip file in order to get the dataset file
• Copy the dataset (.csv file) to the c:\Projects directory
• In your Python code, when using pd.read_csv(), provide the dataset full filename

5

Installing Pandas

• To install Pandas, use pip (not needed when using Colab):

pip install pandas

​

• To import Pandas in your Python code:

import pandas as pd

6

פקודת ייבוא�לספרייה

שם הספרייה

כינוי מקוצר

Main Data Structures in Pandas

• Series: one-dimensional labeled array
• DataFrame: two-dimensional labeled data structure with columns of potentially different types

​

7

# Series

s = pd.Series([1, 3, 5, np.nan, 6, 8])

print(s)

Output:

0 1.0

1 3.0

2 5.0

3 NaN

4 6.0

5 8.0

dtype: float64

# DataFrame

data = {'Name': ['John', 'Anna', 'Peter', 'Linda’],

'Age': [28, 34, 29, 32],

'City': ['New York', 'Paris', 'Berlin', 'London’]}

df = pd.DataFrame(data)

print(df)

Output:

Name Age City

0 John 28 New York

1. Anna 34 Paris
2. Peter 29 Berlin

3 Linda 32 London

NaN - Not a Number

Creating DataFrames

• There are multiple ways to create DataFrames
• From a CSV file (Comma-Separated Values) - simple text-based file format used to store tabular data
• Each line in a CSV file represents a row of text data, and the values within each row are separated by commas
• Pandas also supports reading�other file formats such as�Excel [read_excel()] and more

8

# From a dictionary of lists:

df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7, 8, 9]})

​

# From a list of dictionaries:

data = [{'a': 1, 'b': 2}, {'a': 5, 'b': 10, 'c': 20}]

df = pd.DataFrame(data)

​

# From a NumPy array:

arr = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

df = pd.DataFrame(arr, columns=['A', 'B', 'C’])

​

# From a CSV file:

df = pd.read_csv(‘filename.csv')

Basic DataFrame Operations (1/5)

Summary of key operations

• Viewing data:
• df.head(): View first 5 rows (parameter can define another number)
• df.tail(): View last 5 rows (parameter can define another number)
• df.info(): Get concise summary of the DataFrame
• df.describe(): Generate descriptive statistics
• df.columns: View the list of column names in the DataFrame
• Selecting columns:
• Single column: df[‘<column name>'] or df.<column name>
• Multiple columns: df[[‘<column1 name>', ‘<column2 name>’]]
• Selecting rows:
• By index: df.loc[index]
• By position: df.iloc[position]
• Slicing: df[start:end]
• Filtering data:
• Using boolean conditions: df[df[‘<column name>'] > value]
• Sorting by column
• Using the df.sort_values(by=‘<column name>’)

9

No sleeping in class ☺

Notice the

double brackets

Basic DataFrame Operations (2/5)

10

data = {'Name': ['John', 'Anna’, 'Peter’, 'Linda’],

'Age': [28, 34, 29, 32],

'City': ['New York', 'Paris’, 'Berlin', 'London’]}

df = pd.DataFrame(data)

​

# View first 3 rows

print(df.head(3))

​

# Select 'Name' and 'Age' columns

print(df[['Name', 'Age’]])

​

# Filter - inner comparison generates True/False array,

# outer part selects the rows which are marked as True

print(df[df['Age'] > 30])

print(df[df[‘City’] == ‘Paris’])

​

# Combine filters

print(<comparison 1> & <comparison 2>)

Basic DataFrame Operations (3/5)

11

Selecting one feature (column) as a Series

Selecting one feature (column) as a DataFrame

Selecting multiple features

Basic DataFrame Operations (4/5)

12

1. Select by index
2. Select by index and column names
3. Select using multiple comparisons
4. Sort values

Basic DataFrame Operations (5/5)

13

Create a new column (if it does not exist)�based on a calculation done on another column

Display statistics about the DataFrame

• Count – מספר הרשומות
• Mean – ממוצע
• Std – סטיית תקן
• Min / Max – ערך מינימלי ומקסימלי
• 25%/50%/75% - רבעון ראשון, חציון, רבעון שלישי

Data Manipulation (1/2)

• Adding a new column: df['New_Column'] = [value1, value2, value3]
• Dropping columns: df = df.drop('Column_Name', axis=1)
• Renaming columns: df = df.rename(columns={'OldName': 'NewName’})
• Sorting data: df_sorted = df.sort_values('ColName', ascending=True)
• Calculating mean(), median(), mode(), sum(), etc.
• Mean: Average value, Median: Mid point value, Mode: Most common value
• When specifying the column axis for mode(), it searches column-wise and returns the mode value for each row
• Handling missing data:
• Drop rows with missing values: df.dropna()
• Fill missing values: df.fillna(value)
• Value can be calculated using methods such as mean(), mode(), etc.

14

Data Manipulation (2/2)

15

df = pd.DataFrame({'A': [1, 2, np.nan, 4],

'B': [5, np.nan, np.nan, 8],

'C': [9, 10, 11, 12]})

print(df)

# Add new column

df['D'] = [13, 14, 15, 16]

print(df)

​

# Drop column 'B’

df = df.drop('B', axis=1)

print(df)

​

# Rename column 'A' to 'Alpha’

df = df.rename(columns={'A': 'Alpha’})

print(df)

# Sort by 'Alpha' column

df_sorted = df.sort_values('Alpha', ascending=True)

print(df_sorted)

​

# Drop rows with missing values

df_clean1 = df.dropna(axis=1)

df_clean2 = df.dropna(axis=0)

print(df_clean1)

print(df_clean2)

​

# Fill missing data with value

df1 = df[‘Alpha’].fillna(df[‘Alpha’].mean())

print(df1)

df2 = df.fillna(df.mean())

print(df2)

df3 = df.fillna(value=0.0)

print(df3)

Grouping and Aggregation (1/2)

Grouping allows you to split the data into groups based on some criteria, apply a function to each group independently, and then combine the results.

16

# Grouping:

grouped = df.groupby('Column_Name’)

​

# Aggregation:

result = grouped.agg({'Column1': 'mean', 'Column2': 'sum’})

​

# Common aggregation functions:

- count(): Count of non-null values

- sum(): Sum of values

- mean(): Mean of values

- median(): Median of values

- min(), max(): Minimum and maximum values

- std(), var(): Standard deviation and variance

Grouping and Aggregation (2/2)

17

data = {'Category': ['A', 'B', 'A', 'B', 'A', 'C’],

'Value1': [10, 20, 30, 40, 50, 60],

'Value2': [100, 200, 300, 400, 500, 600]}

df = pd.DataFrame(data)

​

# Group by 'Category' and calculate mean of 'Value1' and sum of 'Value2’

result = df.groupby('Category').agg({'Value1': 'mean', 'Value2': 'sum’})

print(result)

Exploring the “diamonds” dataset (1/3)

18

1. Load the diamonds dataset

2. Group By “color” and calculate mean for all numeric columns

3. Add soring by “price” in ascending order

4. Grouping by “cut ” and presenting max() of “depth”

Exploring the “diamonds” dataset (2/3)

19

5. Group on “cut ”, present max() of “depth” in sorted order

6. Group on “cut”, describe the “depth” column

7. List unique “color” values

8. Get number of unique “color” types

9. nunique() for all “object” type (string) fields

Exploring the “diamonds” dataset (3/3)

20

10. Number of items per “color”

11. Same results as above

12. List the dataset’s columns

Merging and Joining DataFrames (1/2)

• Pandas provides various methods to combine DataFrames

21

# Concatenation (stacking DataFrames):

result = pd.concat([df1, df2])

​

# Merging (combining based on a common column):

result = pd.merge(df1, df2, on='common_column’)

​

# Joining (combining based on index):

result = df1.join(df2)

Merging and Joining DataFrames (2/2)

22

# Create two DataFrames

df1 = pd.DataFrame({'A': ['A0', 'A1', 'A2’],

'B': ['B0', 'B1', 'B2’]},

index=['K0', 'K1', 'K2’])

df2 = pd.DataFrame({'C': ['C0', 'C1', 'C2’],

'D': ['D0', 'D1', 'D2’]},

index=['K0', 'K2', 'K3’])

​

# Concatenate DataFrames

result_concat = pd.concat([df1, df2], axis=1)

​

# Merge DataFrames

df3 = pd.DataFrame({'key': ['K0', 'K1', 'K2', 'K3’],

'A': ['A0', 'A1', 'A2', 'A3’],

'B': ['B0', 'B1', 'B2', 'B3’]})

df4 = pd.DataFrame({'key': ['K0', 'K1', 'K2’],

'C': ['C0', 'C1', 'C2’],

'D': ['D0', 'D1', 'D2’]})

result_merge = pd.merge(df3, df4, on='key’)

print("Concatenation result:")

print(result_concat)

print("\nMerge result:")

print(result_merge)

Working with Time Series Data (1/2)

• Pandas has powerful capabilities for working with time series data

23

# Creating date ranges:

dates = pd.date_range(start='2023-01-01', end='2023-12-31', freq='D’)

​

# Converting columns to datetime:

df['Date'] = pd.to_datetime(df['Date’])

​

# Setting date as index:

df.set_index('Date', inplace=True)

​

# Resampling time series data:

df_monthly = df.resample('M').mean()

Working with Time Series Data (2/2)

24

# Create a DataFrame with date index

dates = pd.date_range(start='2023-01-01', end='2023-12-31', freq='D’)

df = pd.DataFrame({'value': np.random.randn(len(dates))}, index=dates)

​

# Resample to monthly frequency

df_monthly = df.resample('M').mean()

​

# Shift dates

df_shifted = df.shift(periods=7)

​

print("Original DataFrame (first 5 rows):")

print(df.head())

print("\nMonthly resampled DataFrame:")

print(df_monthly)

print("\nShifted DataFrame (first 5 rows):")

print(df_shifted.head())

Data Visualization with Pandas (1/2)

• Pandas integrates well with Matplotlib for data visualization

25

# Line plot:

df.plot(kind='line’)

​

# Bar plot:

df.plot(kind='bar’)

​

# Histogram:

df['column'].hist()

​

# Scatter plot:

df.plot(kind='scatter', x='column1', y='column2')

Data Visualization with Pandas (2/2)

26

import pandas as pd

import matplotlib.pyplot as plt

​

# Create a sample DataFrame

data = {'Year': [2010, 2011, 2012, 2013, 2014],

'Sales': [100, 130, 120, 140, 150],

'Expenses': [90, 110, 100, 130, 120]}

df = pd.DataFrame(data)

​

# Create a line plot

df.plot(x='Year', y=['Sales', 'Expenses'], kind='line’)

plt.title('Sales and Expenses Over Time’)

plt.ylabel('Amount’)

plt.show()

​

# Create a bar plot

df.plot(x='Year', y=['Sales', 'Expenses'], kind='bar’)

plt.title('Sales and Expenses Comparison’)

plt.ylabel('Amount’)

plt.show()

Advanced Pandas Features (1/2)

27

# Pivot Tables

pivot_table = df.pivot_table(values='value_column', index='index_column’, columns='column_name', aggfunc='mean’)

​

# Multi-indexing:

df_multi = df.set_index(['column1', 'column2’])

​

# Categorical Data:

df['category'] = pd.Categorical(df['category’])

​

# Applying custom functions:

df['new_column'] = df['column'].apply(custom_function)

Advanced Pandas Features (2/2)

28

# Create a sample DataFrame

data = {'Date': ['2023-01-01', '2023-01-01', '2023-01-02', '2023-01-02’],

'Category': ['A', 'B', 'A', 'B’],

'Value': [100, 150, 120, 180]}

df = pd.DataFrame(data)

​

# Create a pivot table

pivot_table = df.pivot_table(values='Value', index='Date', columns='Category', aggfunc='sum’)

​

# Create a multi-index DataFrame

df_multi = df.set_index(['Date', 'Category’])

​

# Apply a custom function

def double_value(x):

return x * 2

df['Double_Value'] = df['Value'].apply(double_value)

print("Pivot Table:")

print(pivot_table)

print("\nMulti-index DataFrame:")

print(df_multi)

print("\nDataFrame with custom function applied:")

print(df)

Performance Optimization

29

# Use appropriate data types:

df['integer_column'] = df['integer_column'].astype('int32’)

df['float_column'] = df['float_column'].astype('float32’)

​

# Use vectorized operations instead of loops:

# Slow:

for i in range(len(df)):

df.loc[i, 'new_column'] = df.loc[i, 'column'] * 2

# Fast:

df['new_column'] = df['column'] * 2

​

# Use `inplace=True` for operations that modify the DataFrame:

df.drop('column_name', axis=1, inplace=True)

​

# Use `chunk` for loading large datasets:

chunk_size = 10000

for chunk in pd.read_csv('large_file.csv', chunksize=chunk_size):

process_chunk(chunk)

Best Practices and Tips

• Make a copy of the DataFrame before modifying it:

df_copy = df.copy()

• Use method chaining for cleaner code:

result = (df.groupby('column').agg({'value': 'mean’}).� sort_values('value', ascending=False).reset_index())

Not always true, sometimes it is confusing ☺

• Use `loc` and `iloc` for explicit indexing:

df.loc[df['column'] > 5, 'new_column'] = 1

• Regularly check your DataFrame's memory usage:

df.info(memory_usage='deep’)

• Use appropriate data types to reduce memory usage and improve performance
• Document your code and use meaningful variable names for better readability

30