PANDAS

HODP Spring 2025 BOOTCAMP

ANNOUNCEMENTS

• Last Week
• Ryan - Intro to Web Scraping [High Level]
• Michael - Intro to Python [Low Level]
• This Week - Pandas
• Upcoming Work - Think about Questions

End Goal

Your Article Here!

THE PLAN

• Review - Modeling
• Review - Python Basics
• Learn - Pandas
• Intro - EDA

Review - Modeling

WHAT IS MODELING

• Computers learn from data and improve over time by recognizing patterns and making decisions
• AI vs ML?
• AI is just the concept of machines doing “human intelligent tasks”
• ML is a subset of AI built on data
• Other concepts?
• Computer Vision?
• Natural Language Processing?
• Large Language Models?

WHAT IS MODELING

• Computers learn from data and improve over time by recognizing patterns and making decisions
• AI vs ML?
• AI is just the concept of machines doing “human intelligent tasks”
• ML is a subset of AI built on data
• Other concepts?
• Computer Vision?
• Natural Language Processing?
• Large Language Models?

Review - Data Structures

Lists

Data storage type that is ordered and can store different types of information

To declare a list, use brackets [ ] to enclose the list and commas , to separate each item.

my_list = [‘red’, ‘green’, ‘blue’, ‘yellow’, ‘white’, ‘black’]

Lists are indexed! Each value in the list is assigned an index

More methods

Add element to list

lst.append("z")

​

Remove element from list

lst.remove("z")

​

Insert an element into a specific index in the list

lst.insert(2, "zzz")

​

​

​

Reverse list

lst.reverse()

​

Sort list

lst.sort()

​

Extend a list (different than append)!

lst.extend(thing)

​

List methods are done “in place”, which means you do not need to do �lst = lst.append(“z”)! Typing lst.append(“z”) will add “z” to the list directly.

Dictionaries

• Dictionaries are like named lists, in that they are mutable and can hold values.
• Attach a key (as opposed to an index) to each value
• These key-value pairs make up the dictionary.
• Values can be any data-type (e.g. strings, ints, lists, dictionaries, etc)
• Keys must be unique and immutable (e.g. strings, ints, etc. but not lists).

grades = {

"freshman": 9,

"sophomore": 10,

}

�“Freshman” is a key, 9 is the value, which you can access with grades[“freshman”]

Adding Elements to the Dictionary

We can add elements (or replace elements) using hopefully familiar syntax!

grades[“junior”] = 11

grades[“senior”] = 12

​

grades = {

"freshman": 9,

"sophomore": 10,

“junior”: 11,

“senior”: 12,

}

​

To delete an element, you can do del grades[“junior”].

KeyErrors and Iterations

CAUTION: What happens if you try to access a key that is NOT in the dictionary?

• You get something called a “KeyError” and your program stops running

​

Bypassing:

• Use dict.get(key, defaultVal) ⇒ e.x. grades[‘freshman’] = grades.get(‘freshman’, 0) + 1

​

Checking:

• Use dict.keys(): can be in a loop heading too ⇒ e.x. for key in dict.keys():
• Use dict.values() for specific searching ⇒ e.x. for val in dict.values():
• Use dict.items() for key-val pairing ⇒ e.x. for key, val in dict.items():

Functions

• Oftentimes, we will want to perform the same action (with the same chunk of code) in multiple areas.
• Define functions to avoid copy/pasting
• Essentially customizing

def square(x):

return x**2

​

square(5)

>> 25

Pandas

FOLLOW ALONG ON DEEPNOTE

hodp.org/bootcamp

WHAT IS PANDAS

• A powerful Python library for data manipulation and analysis.
• Easy-to-use data structures: Series and DataFrame.
• Fast and efficient: commonly used dataset files like CSVs
• Good for cleaning, input for modeling

​

Conventions:

​

“import pandas as pd” // always like this by convention, do NOT change

df = name of dataframe

s = name of series

​

ALL PANDAS COMMANDS WILL HAVE THE PREFIX “pd.”

PANDAS SPECIFIC STRUCTURES

• Series: one-dimensional labeled array, similar to a list or array in Python.
• DataFrame:
• A two-dimensional table of data with labeled rows and columns.
• Multiple series glued together
• Same indexing
• “Spreadsheet analogous”
• Can be made by either:
• List of Dictionaries
• Dictionary of Lists
• CSV reading

DFs - List of Dictionaries

DFs - Dictionary of Lists

DFs - Reading a CSV

The command is df = pd.read_csv(path):

• Path should be a string
• Might look something like:
• Colab: "/content/drive/MyDrive/CoolFolder1/CoolFolder2/file.csv"
• Terminal: “CS_61/cs61-f24-psets-USERNAME/file.csv”
• Jupyter Notebook: “data/file.csv”
• Think of it as directing your computer to the file
• Pandas will do the rest!

Exercise 1

Go onto Deepnote and give Exercise 1 a shot! If you completed it successfully, you should be able to see your loaded in Data.

Basic Data Exploration

• Now we have our DataFrame, so what?
• Examination:
• df.head(x)
• Returns the first x rows of the dataset
• df.tail(x)
• Returns the last x rows of the dataset
• If you don’t input an integer value x, default is first/last 5 rows
• EDA:
• df.shape
• returns a tuple value of (# rows, # columns)
• df.columns
• Returns all the columns, can be cast to a list with list(df.columns)
• Note there are NO () after
• df.info, df.describe

EDA Continued

• Numeric Stats
• df.describe - returns a table of various numerical statistics
• Overall Picture
• df.info()
• Columns, non-null, dtype
• What are “non-null”?
• Seen in data as “NaN”
• Represents empty cells
• See next week…

Exercise 2

Go onto Deepnote and give Exercise 2 a shot! Note:

Every time you want to display something, you either need to create a new code block (preferred) or use the print function. We prefer the first, since the formatting looks a bit cleaner!

Indexing

• Accessing columns: use df[‘col_name’] which returns a Series
• Can switch to a list with df['col_name'].tolist() for a list
• How to access specific parts of the dataset?
• .loc: access by label, think rows and columns by their labels (names or indices).
• .iloc: access by index, think numerical (integer) positions, like standard Python indexing (0-based).
• Example LOC: ILOC:

result = df.loc['b', ['Name', 'Age']] print(result)

result = df.iloc[1, [0, 1]] print(result)

Multiple Entry Access

• Suppose we have this code: what does our dataframe look like?

​

data = {'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],

'Age': [25, 30, 35, 40, 45],

'City': ['New York', 'Los Angeles', 'Chicago', 'Houston', 'Phoenix']}

df = pd.DataFrame(data, index=['a', 'b', 'c', 'd', 'e'])

​

• Loc: both endpoints are included
• iloc: right endpoint is excluded

​

result = df.loc['b':'d']

print(result)

​

result = df.iloc[1:3]

print(result)

Exercise 3

Go onto Deepnote and give Exercise 3 a go!

Restructuring Dataframes - Sort

• Suppose our running example is all messed up i.e:
• We can fix this in many ways:
• sorted_df = df.sort_values(by='Age')
• This sorts by age ascending order
• Order is: A, B, C, D, E
• Can also do opposite order:
• new_sorted_df = df.sort_values(by=’Name’, ascending = False)
• What might the new order be?
• We can also do restructurings inplace
• Use df.sort_values(by='Age', inplace=True)
• Can also sort with a hierarchy
• sorted_df_multi = df.sort_values(by=['Age', 'Name'])
• This sorts by age first, then in case of ties does it by name

Restructuring Dataframes - Func

Start with:

// start

data = {'Name': ['Alice', 'Bob', 'Charlie'],

'Age': [25, 30, 35],

'Salary': [50000, 60000, 70000]}

df = pd.DataFrame(data)

Original (top) vs post-restructure (bottom)

// restructuring

df['Bonus'] = df['Salary'] * 0.10

df['Age'] = df['Age'].apply(add_ten)

​

// end

​

What might add_ten look like?

Restructuring Dataframes - Drop

Start with:

// start

data = {'Name': ['Alice', 'Bob', 'Charlie'],

'Age': [25, 30, 35],

'Salary': [50000, 60000, 70000],

‘Bonus’: [5000.0, 6000.0, 7000.0]}

df = pd.DataFrame(data) Original (top) vs post-restructure (bottom)

​

df.drop('Bonus', axis=1, inplace=True)

​

// axis = 1 means column

​

Row drop: df.drop(1, axis=0, inplace=True)

Filtering

Start with:

// start

data = {'Name': ['Alice', 'Bob', 'Charlie'],

'Age': [25, 30, 35],

'Salary': [50000, 60000, 70000],

‘Bonus’: [5000.0, 6000.0, 7000.0]}

df = pd.DataFrame(data) Original (top) vs post-restructure (bottom)

​

filtered_df = df[df['Age'] > 28]

Simple Plotting

Make sure you have import matplotlib.pyplot as plt

data = {'Name': ['Alice', 'Bob', 'Charlie'],

'Age': [25, 30, 35],

'Salary': [50000, 60000, 70000],

‘Bonus’: [5000.0, 6000.0, 7000.0]}

df = pd.DataFrame(data)

​

df.plot()

​

// might need this if not last line of notebook cell

​

// plt.show()

Common Dangers

• Dropping rows too early
• If you drop a row and then try accessing it later, even if it is earlier, you can’t
• Have to rerun previous cells to ‘restore’ the previous cell instance
• Notebook runs sequentially
• This means you might have to rerun the entire thing and restart runtime
• Use Shift + Enter to do this really quickly
• Reset_index function:
• If you do an operation like sort_values, indexing is preserved
• To account for this, can run df.reset_index(drop=True, inplace=True)
• Helps also for merging datasets
• Datasets merge by default via index so if one dataset has indices out of order, will put things in the wrong place

Exporting

• Saving DataFrames: Suppose you’ve done all your hard work and want to export some data. How to preserve the python notebook or send to people?

​

cleaned_df_path = ‘/path/to/your/file/desired_file_name.csv’

// optional - will only run if the file if the desired file doesn’t exist yet

if not os.path.exists(cleaned_df_path):

df.to_csv(cleaned_df_path, index=False)

​

Project Time

https://tinyurl.com/hodp-spring25-project

​

Use the link on the first page of the form to find people and their interests. As groups form, we will try to update that spreadsheet (which is also here)

​

https://docs.google.com/spreadsheets/d/1QpwyljIJ8NnM-AN6ggfeG4PFC4bRGS8kgp8HMtTc8yQ/edit?gid=0#gid=0

Attendance Code: pd