NumPy, Indexing, Materials-Based Research
1
Data 6 Summer 2022
LECTURE 05
What it means to work with data.
Developed by students and faculty at UC Berkeley and Tuskegee University
Icebreaker & Check-In
2
Week 2
Announcements!
3
Today’s Roadmap
Lecture 05, Data 6 Summer 2022
4
Warmup
Suppose we have two arrays containing the resident populations of several states in 2020 and 2021. Assume each array contains information about the same states in the same order.
How would you compute the percentage change in state populations from 2020 to 2021?
5
| 2019 pop (in millions) | 2020 pop (in millions) |
.Alabama | 5.025 | 5.040 |
.Alaska | 0.732 | 0.733 |
.Arizona | 7.178 | 7.276 |
.Arkansas | 3.012 | 3.026 |
.California | 39.500 | 39.238 |
pop_2020 = make_array(5.025, .732, 7.178, 3.012, 39.500)
pop_2021 = make_array(5.040, .733, 7.276, 3.026, 39.238)
In [ ]:
...
In [ ]:
Warmup
Suppose we have two arrays containing the resident populations of several states in 2020 and 2021. Assume each array contains information about the same states in the same order.
How would you compute the percentage change in state populations from 2020 to 2021?
6
pop_2020 = make_array(5.025, .732, 7.178, 3.012, 39.500)
pop_2021 = make_array(5.040, .733, 7.276, 3.026, 39.238)
In [ ]:
...
In [ ]:
100 * (pop_2021 - pop_2020) / pop_2020
In [3]:
array([ 0.29850746, 0.13661202, 1.36528281, 0.46480744, -0.66329114])
Out [3]:
(final - initial) / initial
| 2019 pop (in millions) | 2020 pop (in millions) |
.Alabama | 5.025 | 5.040 |
.Alaska | 0.732 | 0.733 |
.Arizona | 7.178 | 7.276 |
.Arkansas | 3.012 | 3.026 |
.California | 39.500 | 39.238 |
Array Functions
7
1. Array Functions
2. NumPy
3. Indexing
4. Materials-Based Research
➤
Standard Functions
8
Call expression format | Example(s) |
len(arr) | len(str_arr) # 5�len(empty_arr) # 0 |
max(arr) | min(int_arr) # -4 |
min(arr) | max(str_arr) # 'yd' |
sum(arr) | sum(int_arr) # 6 sum(str_arr) # TypeError |
Compare
Standard Functions
9
Call expression format | Example(s) |
len(arr) | len(str_arr) # 5�len(empty_arr) # 0 |
max(arr) | min(int_arr) # -4 |
min(arr) | max(str_arr) # 'yd' |
sum(arr) | sum(int_arr) # 6 sum(str_arr) # TypeError |
While the function names are identical to what we saw for int/float/strs, the call expressions evaluate differently with our new array data type.
Compare
Quick Check 1
Recall the definition of an average:
The average, or mean, of a collection of numbers is the sum of all the elements of the collection, divided by the number of elements in the collection.
How would you compute the average of an array arr?
10
arr = make_array(30, -40, -4.5, 0, 35)
avg = ...
avg
In [ ]:
Quick Check
NumPy
11
➤
1. Array Functions
2. NumPy
3. Indexing
4. Materials-Based Research
NumPy: A Convenient Function Library
Earlier, we computed averages using built-in Python functions:
Computing averages of array elements happens a lot in data science!
The NumPy package function np.average() is human-readable and convenient.
12
arr = make_array(30, -40, -4.5, 0, 35)
avg = sum(arr)/len(arr)
avg
In [2]:
4.1
Out [2]:
arr = make_array(30, -40, -4.5, 0, 35)
avg = np.average(arr)
avg
In [2]:
4.1
Out [2]:
The NumPy package
NumPy (pronounced “num pie”) is �a Python package* with convenient and�powerful functions for manipulating arrays.
13
*For our purposes, “library”, “package”, and “module” all mean similar things.
arr = make_array(30, -40, -4.5, 0, 35)
avg = np.average(arr)
avg
In [2]:
4.1
Out [2]:
import numpy as np
In [1]:
import numpy as np
Anytime we want to use NumPy, we run
We generally put this import statement at the top of our notebook,�then prepend np. to call a NumPy function.
Element-wise NumPy Functions
We’ll point you to NumPy functions as they come up; you don’t need to memorize them. The course website has a list of some of them.
14
NumPy functions
Many of these functions work on both arrays and individual numbers.
N-length array
N-length array
Demo
Common NumPy Functions
15
NumPy function | Return value |
np.average(arr) np.mean(arr) | The average (i.e., mean) value of arr |
np.sum(arr) | The sum of all elements in arr |
np.prod(arr) | The product of all elements in arr |
np.count_nonzero(arr) | The number of elements in arr that are not equal to 0 |
NumPy functions
N-length array
N-length array
Demo
Even More Functions
16
NumPy function | Return value |
np.average(arr) np.mean(arr) | The average (i.e., mean) value of arr |
np.sum(arr) | The sum of all elements in arr |
np.prod(arr) | The product of all elements in arr |
np.count_nonzero(arr) | The number of elements in arr that are not equal to 0 |
NumPy functions
N-length array
N-length array
Demo
Questions About Functions?
17
Data 6 Python Reference™
Side Note: The datascience Package
datascience package import statement:
18
Fun fact: The datascience package was written �by UC Berkeley specifically for data science education. It’s designed to support many Python packages like NumPy.
“Import everything”
Indexing
19
➤
1. Array Functions
2. NumPy
3. Indexing
4. Materials-Based Research
Array Methods
Methods are functions that we call with “dot” syntax. There are several array methods that make it easy to calculate values of interest.
Terminology note: Method calls are where the function operates directly on the array arr.
In these examples, method calls are equivalent to the NumPy package functions.
20
The most common array method is item(), which is used for array indexing.
An Element’s Index Is Its Position in an Array
When people stand in a line, each person has a position.
Similarly, each element (i.e., value) of an array has a position – called its index.
Python, like most programming languages, is 0-indexed. This means that in an array, the first element has index 0, not 1.
21
Person 1
Person 7
Index 0
Index 6
Indices
0 1 2 3 4
For a length-5 array:
Array Indexing
We can access an element in an array by using its index and the item() method:
arr.item(index)
22
Though int_arr has 5 elements, the largest valid index is 4.
Demo
Negative Indexing
We can also “count backwards” using negative indexes.
23
Demo
Questions?
24
Quick Check 2
What is the value of five after running this code?
25
Quick Check
Research Using
Existing Data
26
➤
1. Array Functions
2. NumPy
3. Indexing
4. Materials-Based Research
Gathering Data for Research and Analysis
All data are created/generated from human input:
However, the researcher and data scientist are often not the same person.�Data scientists therefore need to consider the context of how researchers generated data from existing materials.
27
Conduct surveys, experiments, interviews, ethnography,�measurements…
Data
Conceptualize variables and define domains
Pose research question
+
+
Data Scientist
Researcher
Gathering Data for Research and Analysis
Validity A quality of a measure concerning how accurate it is.
Generalizability The extent to which results or conclusions based on one population can be applied to others.
28
A researcher considers both validity and generalizability throughout the data gathering process.
Conduct surveys, experiments, interviews, ethnography,�measurements…
Data
Conceptualize variables and define domains
Pose research question
+
+
Data Scientist
Researcher
Using Existing Data from a Different Context
Validity A quality of a measure concerning how accurate it is.
Generalizability The extent to which results or conclusions based on one population can be applied to others.
29
If we did not create the original dataset, we will need to separately consider whether the data are valid and generalizable for our own research question.
Conduct surveys, experiments, interviews, ethnography,�measurements…
Data
Conceptualize variables and define domains
Pose research question
+
+
Data Scientist
Researcher
Using Existing Data
Pros
30
Using Existing Data
Cons
31
Units of Analysis
32
➤
1. Array Functions
2. NumPy
3. Indexing
4. Materials-Based Research
Units of Analysis
The types of entities we are analyzing:
33
Units of Analysis
The types of entities we are analyzing:
Income
individual�income��household income��average income
Race/Ethnicity
# People
individual�race/ethnicity��–���population demographics
–���# members���population size
34
The concept of unit of analysis is directly linked to the concept of “variables”.�Variables describe the unit of analysis.
UC Berkeley and Simpson’s Paradox
Graduate School Admissions at UC Berkeley, Fall 1973.
35
However, when disaggregated (or disentangled) by department,�Women had small but significantly higher admission rates in many departments.
| All | Men | Women | |||
| Applicants | Admitted | Applicants | Admitted | Applicants | Admitted |
Total | 12,763 | 41% | 8,442 | 44% | 4,321 | 35% |
Dept | All | Men | Women | |||
Applicants | Admitted | Applicants | Admitted | Applicants | Admitted | |
A | 933 | 64% | 825 | 62% | 108 | 82% |
B | 585 | 63% | 560 | 63% | 25 | 68% |
C | 918 | 35% | 325 | 37% | 593 | 34% |
D | 792 | 34% | 417 | 33% | 375 | 35% |
E | 584 | 25% | 191 | 28% | 393 | 24% |
F | 714 | 6% | 373 | 6% | 341 | 7% |
Total | 4526 | 39% | 2691 | 45% | 1835 | 30% |
Men are significantly more likely than women to be admitted to UC Berkeley graduate school.
Greater percentage of successful applicants�than other gender
Greater number of applicants than other gender
(note: UC changed gender counting policy to include non-binary as of 2016)
(and men to less competitive depts with higher rates of admission).
Furthermore, women tended to apply to more competitive depts with lower rates of admission
Back to the ACS
36
Household survey
Private dataset by household
Public dataset by geographic region
aggregation
Table 1
Table 2
(last lecture)
Individual Level (Private Dataset)
37
ACS: What Is Released to the Public
38
Household survey
Private dataset by household
Public dataset by geographic region
aggregation
Table 1
Table 2
(this lecture)
ACS’s Public Units of Analysis
39
The ACS only provides an aggregated view of data by region and specific disaggregations by income and sex.
Researchers will therefore find it difficult (if not impossible) to disentangle the impact of certain variables, e.g. by race/ethnicity.
Table 2
Public dataset by geographic region
Table 2
Questions?
40
In Conclusion…
41
Summary
42
Recap
Next Time
43
Week 1
Announcements!
44