Python for Scouting and Data Analysis

​

Session 08

Advanced Data Structures

​

Issaquah Robotics Society

FRC 1318

Session 8 Topics

• List and Dictionary Comprehensions
• Compact (and sometimes faster) syntax for creating Python lists and dictionaries
• Numpy Arrays
• Python add-on package for creating and manipulating large numeric
• Other Python Data Structures
• Tuple, Sets, Ordered Dictionaries,
• Pandas Dataframes
• Python add-on package for manipulating tables of data, where different columns can be named and have different data types

Session 8 Overview

1. Review the presentation slides on HTTP
2. For all topics (comprehensions, Numpy, Tuples, Sets, Ordered Dictionaries, Pandas Dataframes):
1. Review presentation slides
2. Complete assigned reading on that topic
3. Complete the exercises for that topic
3. Review presentation material on nested and normalized data and complete the corresponding exercises
4. Answer the questions at the end of the presentation

Reading List

1. Python Tutorial (List and Dictionary Comprehensions)
1. https://docs.python.org/3/tutorial/index.html
2. Sections 5.1, and 5.5 for comprehensions. All other sections for tuples, sets and looping techniques
2. Numpy User Guide
• https://docs.scipy.org/doc/numpy/user/quickstart.html
• Read first section called The Basics (beginning through Indexing, Slicing, and Iterating)
3. ​
4. Pandas Dataframes
• http://pandas.pydata.org/pandas-docs/stable/10min.html
• Object Creation, Viewing Data, Selection, Missing Data

​

​

List Comprehensions

• Compact method for creating lists
• Get the districts.json file from the github repository
• Execute git commands below
• districts.json file will be in the 08_Adv_Data folder
• Try out the two list comprehensions below in a Jupyter notebook. Look at the source JSON data and the resulting lists.

# Open JSON file

with open("districts.json", "r") as file:

dist = json.load(file)

​

# Simple list comprehension

[x["abbreviation"] for x in dist]

​

# More complex list comprehension (get

[x["abbreviation"] for x in dist if len(x["abbreviation"])==2]

git checkout master

git pull origin master

git checkout <your_branch_name>

git rebase master

Dictionary Comprehensions

• Compact method for creating dictionaries
• district_rankings.json file will be in the 08_Adv_Data folder
• Load the district_rankings.json file in a Jupyter notebook and try out the dictionary comprehension below

# Dictionary Comprehension

{x["team_key"]: x["point_total"] for x in dist_rankings if x["rank"] <= 20}

Comprehension Exercises

1. Using the matches.json data, create a list of all matches with frc1318 on the blue alliance.
2. Using the matches.json data, create a list of all matches with the maximum number of ranking points (6)
1. Hint: check out how the in keyword is used here: https://docs.python.org/3/library/stdtypes.html#sequence-types-list-tuple-range
3. Create a list of match keys for matches that had FRC 1318 on the blue alliance.
4. Comprehensions can be nested. Use nested list and dictionary comprehensions (and the district_rankings.json file) to create this dictionary:
• Dictionary key = team key
• Dictionary value = list of event keys for events attended by the team
• Example: �{'frc1318': ['2018wamou', '2018wayak', '2018pncmp'], …}

Why Use List or Dictionary Comprehensions?

• List and dictionary comprehension syntax is compact but still easy to understand
• Often faster than creating lists or dictionaries in for loops

Numpy

• Numpy is a Python scientific computing package for handling very large numeric arrays
• Calculations on Numpy arrays are much faster than on Python built-in objects (e.g, lists or dictionaries)
• Numpy arrays have a fixed size
• With some exceptions, elements of numpy arrays must all be the same datatype
• Many Python mathematical and scientific packages depend on Numpy (so you need to be familiar with it if you are going to use Python for scientific or engineering work)
• numpy.nan is a special numpy data type that means “Not a Number”. It’s frequently used for missing data

Numpy Exercises

1. Use the numpy.arange() function to create a one-dimensional numpy array consisting of every number between 0 and 100 that is divisible by 5.
2. Use list comprehensions and numpy to create two numpy arrays (ndarray object).
• The first array will be a list of district ranking points earned by FRC teams at their first competition, in descending order by district rank.
• The second array will be identical except that it will contain points earned at the second district competition.
3. From the arrays in exercise #2, create a numpy array containing the total points earned at the first two competitions for each team.
• Don’t use a for loop or list comprehension.
• Hint: review the Basic Operations portion of the Numpy Quickstart Guide

Ordered Dictionaries

• Standard Python dictionaries have no defined order
• If you loop over all values in a dictionary, there is no way to predict the order of execution.

Exercises

1. Run the code above in a Python notebook.
2. Convert the unordered dictionary to an ordered dictionary. Test it in a Jupyter notebook

​

• OrderedDict datatype in collections package fixes this

​

unordered_dict = {"1": "a", "2": "b", "3": "c"}

​

for key, val in unordered_dict.items():

print(key, val)

​

# Will output be 1, 2, 3? 3, 1, 2? No way to know.

​

​

​

Look!� A tuple!

Pandas Dataframes

1. Convert the district JSON text to a Pandas DataFrame =>

Pandas converted each dictionary to a row and each dictionary key to a column

JSON TEXT (dist)

[{'abbreviation': 'chs',� 'display_name': 'FIRST Chesapeake',� 'key': '2017chs',� 'year': 2017},� {'abbreviation': 'fim',� 'display_name': 'FIRST In Michigan',� 'key': '2017fim',� 'year': 2017},� {'abbreviation': 'in',� 'display_name': 'Indiana FIRST',� 'key': '2017in',� 'year': 2017},

…

]�

​

​

​

import pandas as pd

​

# Open JSON file

with open("districts.json", "r") as file:

dist = json.load(file)

​

df = pd.DataFrame(dist)

​

​

​

Pandas DataFrame (as displayed in Jupyter)

Anatomy of a Dataframe

Column

Row Index

Column Index

# Select single column

df["display_name"]

​

# Select several rows

df[2:4]

​

# Select portion

df.loc[7:9, "abbreviation":"key"]

​

# Select portion with integers

df.iloc[0:3, 0:1]

Row

DataFrame Exercises�(Flat Data)

1. Create a Pandas dataframe from the districts JSON text.
2. Select the display_name column.
3. Use integer indexing (Hint: iloc[]) to select the first three display names.
4. Delete the year column (Hint: use the Dataframe.drop() function)

DataFrame Exercises (Nested Data)

• Create a Pandas dataframe from the district rankings JSON text.
• What's going on with all that text in the event_points column? Look at the source JSON text to figure out what's going on.

Nested and Normalized Data

[{'event_points':

[{'alliance_points': 16,� 'award_points': 5,� 'district_cmp': False,� 'elim_points': 30,� 'event_key': '2018orwil',� 'qual_points': 18,� 'total': 69},� {'alliance_points': 16,� 'award_points': 5,� 'district_cmp': False,� 'elim_points': 30,� 'event_key': '2018orlak',� 'qual_points': 22,� 'total': 73},� {'alliance_points': 48,� 'award_points': 15,� 'district_cmp': True,� 'elim_points': 90,� 'event_key': '2018pncmp',� 'qual_points': 54,� 'total': 207}],� 'point_total': 349,� 'rank': 1,� 'rookie_bonus': 0,� 'team_key': 'frc2471'},�

• The JSON data consisted of a list of dictionary objects.
• When we pass such a list to Pandas.DataFrame():
• Each dictionary becomes a row
• Each key (e.g., rank, event_points) becomes a column
• The problem occurs because the event_points key itself contains a list of dictionaries
• Pandas cannot nest rows inside of other rows, so it just inserts all of the JSON text into the event_points cell
• We can fix this with a Pandas function called json_normalize

Normalizing Dataframe

• Use the json_normalize() function to create a data frame from the district rankings JSON data
• What happened to every item in the event_data column? What happened to items in the meta columns?
• Use the query() function to identify teams that won an event (hint: they receive at least 30 elimination points)

• Pandas provides a function that will create a flat dataframe from nested JSON
• The record_path attribute contains the name of the column containing the nested data
• The meta attribute contains the names of other columns

from pandas.io.json import json_normalize

​

json_normalize(dist_rankings, record_path="event_data",

meta=["point_total", "rank",

"rookie_bonus", "team_key"])

Questions

1. Which data structures are mutable, and which are immutable? What does mutable and immutable mean?
2. In your own words, describe the difference between flat and nested data
3. Which data structures have both a key and a value? What are they?
4. Are any of the data structures unordered? In a practical sense, what is the impact of an unordered data structure?
5. What are some of the differences between a numpy array and a Pandas data frame?

Minimize the Cost, Maximize the Value!

Change Log