EE 629 �Internet of Things �Lesson 8: Data Analysis
Kevin W. Lu
2022-10-24
Lesson 8: Data Analysis
2
Lab 8A — Examples
3
Lab 8B — Data Analysis
4
IoT Value Chain
5
Components, Sensors, Semiconductors | Things | Connectivity, Infrastructure, Gateway | Software, Platforms, Analytics | Services |
Complexity Levels of IoT Systems
6
Level | Node | Analysis | Storage | Example |
1 | Single | Local | Local | Home Automation |
2 | Single | Local | Cloud | Smart Irrigation |
3 | Single | Cloud | Cloud | Vibration Monitoring |
4 | Multiple | Local | Cloud | Noise Monitoring |
5 | Multiple + Coordinator | Cloud | Cloud | Forest Fire detection |
6 | Multiple + Centralized Controller | Cloud | Cloud | Weather Monitoring |
Characteristics of Big (Fast) Data
* correlation is not causation
7
Decimal and Binary Prefixes
8
Decimal (Hard Drive and Bit Rate) | Binary (RAM) | ||||
Value | SI Prefix | Value | IEC Prefix | ||
1000 | k | kilo | 1024 | Ki | kibi |
10002 | M | mega | 10242 | Mi | mebi |
10003 | G | giga | 10243 | Gi | gibi |
10004 | T | tera | 10244 | Ti | tebi |
10005 | P | peta | 10245 | Pi | pebi |
10006 | E | exa | 10246 | Ei | exbi |
10007 | Z | zetta | 10247 | Zi | zebi |
10008 | Y | yotta | 10248 | Yi | yobi |
SI: International System of Units IEC: International Electrotechnical Commission
Power of 10
9
Power of 10 | Short Scale | Long Scale | Power of 10 | Short Scale | Long Scale |
million | 106 | 106 | duodecillion | 1039 | 1072 |
billion | 109 | 1012 | tredecillion | 1042 | 1078 |
trillion | 1012 | 1018 | quattuordecillion | 1045 | 1084 |
quadrillion | 1015 | 1024 | quindecillion | 1048 | 1090 |
quintillion | 1018 | 1030 | sexdecillion | 1051 | 1096 |
sextillion | 1021 | 1036 | septendecillion | 1054 | 10102 |
septillion | 1024 | 1042 | ⋮ | ⋮ | ⋮ |
octillion | 1027 | 1048 | centillion | 10303 | 10600 |
nonillion | 1030 | 1054 | ⋮ | ⋮ | ⋮ |
decillion | 1033 | 1060 | 103003 | 106000 | |
undecillion | 1036 | 1066 | ⋮ | ⋮ | ⋮ |
In long scale, 109 is milliard, 1015 is billiard, 1021 is trilliard, etc.
Short and Long Scale Usage
10
Exponential vs. Factorial
11
Hyperoperation
12
Hyper0 | 1 + b | | |
Hyper1 | a + b | | |
Hyper2 | a × b | a ∙ b | |
Hyper3 | ab | a ↑ b | |
Hyper4 | ba | a ↑↑ b | |
Hyper5 | a ↑3 b | a ↑↑↑ b | |
Hyper6 | hexation | a ↑4 b | a ↑↑↑↑ b |
Other Large Numbers
6.02214076 x 1023
where 3↑↑↑↑3=3↑↑↑(3↑↑↑3) and 3↑↑↑3=7,625,597,484,987
13
Googol and Googolplex
1920: Prof. Edward Kasner 1878—1955 of Columbia University sought a name for 10100 and one of his nephews, nine-year-old Milton Sirotta 1911—1981 who might have read The Google Book (1913) and/or Barney Google (1919), suggested googol (perhaps, three o's resemble 10100) and googolplex "for writing zeros until one gets tired"
1940: Edward Kasner and James Newman 1909—1966 coauthored Mathematics and the Imagination that introduced googol for 10100 and googolplex for 10googol
1996: Larry Page and Sergey Brin called their search engine "BackRub" for its analysis of the web's backlinks
1997: Larry's officemate, Sean Anderson, suggested a new name Googolplex for BackRub; Larry shortened it to Googol; Sean misspelled Googol as Google in search of the domain name; Larry liked it, and registered Google.com for himself and Sergey on 1997-09-15
14
Google and Googleplex
1913: Vincent Cartwright Vickers 1879—1939 wrote and illustrated The Google Book, a children's book about a strange creature Google and imaginary birds living in Google Land
1919: Billy DeBeck 1890—1942 created a comic strip, Take Barney Google, F'rinstance
1923: Billy Rose 1899—1966 wrote the lyrics for "Barney Google (with the Goo-Goo-Googly Eyes)"
1934: The strip became Barney Google and Snuffy Smith with Snuffy as the main character
1997: Larry Page registered the domain name Google.com instead of Googol.com
2002: The first book that Google Books scanned was Vickers' The Google Book
2004: Google moved headquarters to the Googleplex
15
Baidu
东风夜放花千树，更吹落，星如雨。宝马雕车香满路。凤箫声动，玉壶光转，一夜鱼龙舞。
蛾儿雪柳黄金缕，笑语盈盈暗香去。众里寻他千百度，蓦然回首，那人却在，灯火阑珊处。
16
Regret Minimization Framework
Jeff Bezos left D. E. Shaw & Co, L.P. and founded Amazon.com, Inc. on 1994-07-05
17
Yes
When I am 80 years old, will I regret not doing this?
Do it
Let it go
No
Idea
Apache Hadoop
The name Hadoop is not an acronym; it's a made-up name. The project's creator, Doug Cutting, explains how the name came about: “The name my kid gave a stuffed yellow elephant. Short, relatively easy to spell and pronounce, meaningless, and not used elsewhere: those are my naming criteria.”
18
ACID Database Transaction Properties
Jim Gray, “The transaction Concept: Virtues and Limitations,” June 1981 http://research.microsoft.com/en-us/um/people/gray/papers/theTransactionConcept.pdf
Andreas Reuter and Theo Härder, “Principles of transaction-oriented database recovery,” December 1983
http://web.stanford.edu/class/cs340v/papers/recovery.pdf
19
Data Warehouse vs. Data Lake
20
| Data Warehouse | Data Lake or Data Reservoir |
Data | Structured, processed | Structured, semi-structured, unstructured |
Processing | Schema-on-write | Schema-on-read |
Storage | Expensive for large data volumes | Designed for low-cost storage |
Agility | Less agile, fixed configuration | Highly agile, configure and reconfigure as needed |
Security | Mature | Maturing |
Users | Business professionals | Data scientists |
Sharding
21
Edwin A. Stevens Hall Est. 1870
22
What does this symbol mean?
Pendentive Dome
23
D. Vaccari, “The Barbed Quatrefoil Is a 1500-Year-Old Symbol of Architectural Advance,” 2018. [Online]. Available.
Data Analysis
Data analysis is the process of
"Responsible Conduct in Data Management," Northern Illinois University and the Office of Research Integrity (ORI) of the U.S. Department of Health and Human Services (HHS)
24
Types of Data Analysis
25
Descriptive
Diagnostic
Predictive
Hindsight:
What has happened?
Insight:
Why did it happen?
Foresight:
What could happen?
Prognosis
Diagnosis
Prescriptive
Oversight:
What needs to happen?
"Knowledge is telling the past. Wisdom is predicting the future."
— W. Timothy Garvey, M.D.
Black Swan and Gray Rhino
26
Falsifiability vs. Verifiability
27
Left of Bang
28
Machine Learning and Deep Learning
29
Kaggle
30
COVID-19 Data
"Are Countries Flattening the Curve for the Coronavirus?" | GitHub repository
31
IEEE DataPort
32
IEEE DataPort Dataset Categories
33
Additional Data Sources
34
Data Visualization
Edward Tufte, The Visual Display of Quantitative Information
Excellence in statistical graphics consists of complex ideas communicated with clarity, precision and efficiency. Graphical displays should:
35
A Picture Is Worth 500 Billion Words
36
Popularity
37
Prometheus and Frankenstein
38
Data Visualization Examples
39
Related Coronavirus Genomes
40
SARS-CoV-2 Phylogenetic Network
41
Coronavirus Infectivility
42
Install Packages on Raspberry Pi
Install SciPy, Matplotlib, pandas, and dependencies on Raspberry Pi
pi@piot4:~ $ sudo apt update
pi@piot4:~ $ sudo apt install python3-scipy
pi@piot4:~ $ sudo apt install python3-matplotlib
pi@piot4:~ $ sudo apt install python3-pandas
pi@piot4:~ $ sudo apt install libopenblas-dev
pi@piot4:~ $ sudo apt install libatlas-base-dev
Install NumPy, scikit-learn, TensorFlow, and Keras on Raspberry Pi
pi@piot4:~ $ pip3 -V
pip 18.1 from /usr/lib/python3/dist-packages/pip (python 3.7)
pi@piot4:~ $ sudo pip3 install -U numpy
pi@piot4:~ $ sudo pip3 install --only-binary :all: -U scikit-learn
pi@piot4:~ $ sudo pip3 install -U tensorflow
pi@piot4:~ $ sudo pip3 install -U keras==2.3.1
43
Contribution by Scott Maslin, 2018 Fall
Install Packages on Laptop
All examples in Lesson 8 can run on a laptop; use conda on Windows, or pip3 on Linux and macOS to install or update Python3 data analysis packages
$ pip3 -V
pip 19.3.1 from /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip (python 3.7)
$ sudo pip3 install -U numpy scipy scikit-learn matplotlib pandas
$ sudo pip3 install -U tensorflow keras
$ sudo pip3 list
44
Numpy Array 0/4
pi@piot4:~ $ python3
>>> import numpy as np
>>> a = np.arange(6)
>>> a
array([0, 1, 2, 3, 4, 5])
>>> print(a)
[0 1 2 3 4 5]
>>>
>>> b = np.arange(12).reshape(4,3)
>>> print(b)
[[ 0 1 2]
[ 3 4 5]
[ 6 7 8]
[ 9 10 11]]
>>> c = np.arange(24).reshape(2,3,4)
>>> print(c)
[[[ 0 1 2 3]
[ 4 5 6 7]
[ 8 9 10 11]]
[[12 13 14 15]
[16 17 18 19]
[20 21 22 23]]]
45
Numpy Array 1/4
>>> b.shape
(4, 3)
>>> b.reshape(-1)
array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])
>>> b.reshape(-1, 1)
array([[ 0],
[ 1],
[ 2],
[ 3],
[ 4],
[ 5],
[ 6],
[ 7],
[ 8],
[ 9],
[10],
[11]])
>>> b.reshape(2, -1)
array([[ 0, 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10, 11]])
>>>
46
Numpy Array 2/4
>>> d = np.array([20, 30, 40, 50])
>>> e = np.arange(4)
>>> e
array([0, 1, 2, 3])
>>> f = d-e
>>> f
array([20, 29, 38, 47])
>>> e**2
array([0, 1, 4, 9], dtype=int32)
>>> A = np.array([[1, 1], [0, 1]])
>>> B = np.array([[2, 0], [3, 4]])
>>> A*B
array([[2, 0],
[0, 4]])
>>> A.dot(B)
array([[5, 4],
[3, 4]])
>>> np.dot(A, B)
array([[5, 4],
[3, 4]])
47
Numpy Array 3/4
>>> g = np.ones((2, 3), dtype=int)
>>> h = np.random.random((2, 3))
>>> g *= 3
>>> g
array([[3, 3, 3],
[3, 3, 3]])
>>> h += g
>>> h
array([[3.98175695, 3.86494818, 3.52676445],
[3.31238832, 3.97164736, 3.22392774]])
>>> k = np.random.random((2, 3))
>>> k
array([[0.56497847, 0.13008329, 0.41460729],
[0.56262637, 0.16092589, 0.63772776]])
>>> k.sum()
2.4709490742230726
>>> k.min()
0.1300832909029529
>>> k.max()
0.6377277578978647
>>>
48
Numpy Array 4/4
>>> m = np.arange(12).reshape(3, 4)
>>> m
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
>>> m.sum(axis=0)
array([12, 15, 18, 21])
>>> m.min(axis=1)
array([ 0, 4, 8])
>>> m.cumsum(axis=1)
array([[ 0, 1, 3, 6],
[ 4, 9, 15, 22],
[ 8, 17, 27, 38]], dtype=int32)
>>> n = np.arange(5)
>>> n
array([0, 1, 2, 3, 4])
>>> n[[1, 3, 4]] = 0
>>> n
array([0, 0, 2, 0, 0])
>>> exit()
pi@piot4:~ $
49
Matplotlib Examples
50
John D. Hunter 1968—2012
51
Bokeh and Seaborn
52
Learning How to Learn
53
X Window System
$ export DISPLAY=localhost:0
$ ssh -Y pi@xxx.xxx.xxx.xxx
$ ssh -Y pi@xxx.xxx.xxx.xxx
$ ssh -Y pi@xxx.xxx.xxx.xxx
54
Matplotlib Pyplot
Either enable X11 forwarding with ssh -Y or use VNC Viewer
$ ssh -Y pi@155.246.200.x
pi@piot4:~ $ cd iot
pi@piot4:~/iot $ git pull
pi@piot4:~/iot $ cd lesson8
pi@piot4:~/iot/lesson8 $ ls
boxplot_demo.py plot_lda.py README.md
histogram_demo_extended.py plot_lda_qda.py result2.csv
histogram_demo_features.py plt_cv2.py result.csv
interpolation.py plt_final.py rpidata1.csv
keras_diabetes.py pyplot_annotate.py scatter_demo.py
legend_demo.py pyplot_formatstr.py simple_plot.py
linreg.py pyplot_scales.py test.csv
major_minor_demo1.py pyplot_simple.py ticklabels_demo_rotation.py
pima-indians-diabetes.csv pyplot_text.py titanic_1.py
plot_cv_diabetes.py pyplot_three.py titanic_2.py
plot_cv_predict.py pyplot_two_subplots.py train.csv
pi@piot4:~/iot/lesson8 $ cat pyplot_simple.py
import matplotlib.pyplot as plt
plt.plot([1,2,3,4])
plt.ylabel('some numbers')
plt.show()
pi@piot4:~/iot/lesson8 $ python3 pyplot_simple.py
pi@piot4:~/iot/lesson8 $
55
IPython and Jupyter
%matplotlib inline
56
pyplot_simple.py
57
simple_plot.py
58
pyplot_formatstr.py
59
ticklabels_demo_rotation.py
60
pyplot_three.py
61
pyplot_two_subplots.py
62
pyplot_scales.py
63
symlog: symmetric log, logit or logistic unit: the logarithm of the odds p/(1-p)
pyplot_annotate.py
64
major_minor_demo1.py
65
legend_demo.py
66
scatter_demo.py
67
histogram_demo_features.py
pi@piot4:~/iot/lesson8 $ cat histogram_demo_features.py
import numpy as np
import scipy.stats
import matplotlib.pyplot as plt
# Example data
mu = 100 # mean of distribution
sigma = 15 # standard deviation of distribution
x = mu + sigma * np.random.randn(10000)
num_bins = 50
# The histogram of the data
n, bins, patches = plt.hist(x, num_bins, density=1, facecolor='green', alpha=0.5)
# Add a 'best fit' line
y = scipy.stats.norm.pdf(bins, mu, sigma)
plt.plot(bins, y, 'r--')
plt.xlabel('Smarts')
plt.ylabel('Probability')
plt.title(r'Histogram of IQ: $\mu=100$, $\sigma=15$')
# Tweak spacing to prevent clipping of ylabel
plt.subplots_adjust(left=0.15)
plt.show()
pi@piot4:~/iot/lesson8 $
68
Save Figure
pi@piot4:~/iot/lesson8 $ python3 histogram_demo_features.py
69
Step 2: close this window to exit Python
Step 1: Save as figure_1.png
pyplot_text.py
70
Fig. 1. Filled Steps and Line
pi@piot4:~/iot/lesson8 $ python3 histogram_demo_extended.py
71
Fig. 2. Unequally Spaced Bars
72
Fig. 3. Cumulative Histogram
73
Fig. 4. Normalized Bars
74
Fig. 5. Normalized Stacked Bars
75
Fig. 6. Stacked Filled Steps
76
Fig. 7. Multiple Histograms
77
John Tukey 1915—2000
78
Box Plot
79
Contribution by Nagrajan Chandrasekaran, 2017 Spring
Fig. 1. Basic Box Plot
pi@piot4:~/iot/lesson8 $ python3 boxplot_demo.py
80
Fig. 2. Notched Box Plot
81
Fig. 3. Outlier Point Symbols
82
Fig. 4. Without Outlier Points
83
Fig. 5. Horizontal Box Plot
84
Fig. 6. Change Whisker Length
85
Fig. 7. Multiple Box Plots
86
Linear Regression 0/1
87
Linear Regression 1/1
88
How Regression Got Its Name
89
linreg.py 0/1
pi@piot4:~/iot/lesson8 $ cat linreg.py
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
x = np.random.random(10)
y = np.random.random(10)
slope, intercept, r_value, p_value, std_err = stats.linregress(x,y)
plt.xlabel('x')
plt.ylabel('y')
plt.plot(x,y,'ro')
plt.plot([intercept,intercept+slope])
plt.show()
pi@piot4:~/iot/lesson8 $ python3 linreg.py
90
linreg.py 1/1
91
Interpolation.py 0/1
pi@piot4:~/iot/lesson8 $ cat interpolation.py
import numpy as np
from scipy.interpolate import interp1d
import matplotlib.pyplot as plt
x = np.linspace(0, 10, num=11, endpoint=True)
y = np.cos(-x**2/9.0)
f = interp1d(x, y)
f2 = interp1d(x, y, kind='cubic')
xnew = np.linspace(0, 10, num=41, endpoint=True)
plt.plot(x, y, 'o', xnew, f(xnew), '-', xnew, f2(xnew), '--')
plt.legend(['data', 'linear', 'cubic'], loc='best')
plt.xlabel('x')
plt.ylabel('y')
plt.show()
pi@piot4:~/iot/lesson8 $ python3 interpolation.py
92
Interpolation.py 1/1
93
Overfitting and Underfitting
94
Choose the Right Estimator
95
Classification
pi@piot4:~/iot/lesson8 $ python3 plot_lda.py
pi@piot4:~/iot/lesson8 $ python3 plot_lda_qda.py
96
plot_lda.py
97
plot_lda_qda.py
98
Play Audio/Video on Raspberry Pi
99
Piano Concerto No. 21 — Andante
100
Browsing History on Raspberry Pi
101
Google Sheet Chart — Time Series
102
Google Sheet Chart — Histogram
103
Google Sheet Chart — Histogram
104
Google Sheet Chart — Scatter Plot
105
Time is a lurking variable; Customize > Series > Linear Trendline*
* Contribution by Gina Schnecker, 2017 Fall
Time Series
106
Histogram of CPU Usage
107
Histogram of Temperature
108
Horizontal Box Plot of CPU Usage
109
Vertical Box Plot of Temperature
110
Linear Regression
111
Cross-Validation (CV)
112
K-Fold Cross-Validation
The complete data set is partitioned into K folds, one is left out as the test set in each of the K iterations of training, and the trained model is validated by the test set.
113
Test
Training
Training
Training
Training
Training
Test
Training
Training
Training
Training
Training
Test
Training
Training
Training
Training
Training
Test
Training
Training
Training
Training
Training
Test
Test
Test
Test
Test
Test
Complete Data
Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5
Predicted
Measured
Cross-Validation Prediction
114
Data include CPU usage only
Cross-Validation Prediction
115
Data include both time and CPU usage
plot_cv_predict.py 0/1
pi@piot4:~/iot/lesson8 $ cat plot_cv_predict.py
from sklearn import datasets
from sklearn.model_selection import cross_val_predict
from sklearn import linear_model
import matplotlib.pyplot as plt
lr = linear_model.LinearRegression()
boston = datasets.load_boston()
y = boston.target
print('Number of instances: %d' % (boston.data.shape[0]))
predicted = cross_val_predict(lr, boston.data, y, cv=10)
fig, ax = plt.subplots()
ax.scatter(y, predicted)
ax.plot([y.min(), y.max()], [y.min(), y.max()], lw=2)
ax.set_xlabel('Measured')
ax.set_ylabel('Predicted')
plt.show()
pi@piot4:~/iot/lesson8 $ python3 plot_cv_predict.py
Number of instances: 506
116
plot_cv_predict.py 1/1
117
plot_cv_diabetes.py
118
Lasso Regression
119
Precision and Recall
120
Support Vector Machines (SVMs)
In machine learning, support vector machines (SVMs) are supervised learning models with associated learning algorithms that analyze data for classification, regression, and outliers detection
pi@piot4:~ $ cd ~/iot/lesson8
pi@piot4:~/iot/lesson8 $ python3 traffic.py
[[22 4]
[ 1 40]]
precision recall f1-score support
0 0.96 0.85 0.90 26
1 0.91 0.98 0.94 41
accuracy 0.93 67
macro avg 0.93 0.91 0.92 67
weighted avg 0.93 0.93 0.92 67
121
TensorFlow
122
Keras
123
The Gates of Horn and Ivory
124
Keras/TensorFlow on Raspberry Pi
pi@piot4:~ $ python3
Python 3.7.3 (default, Apr 3 2019, 05:39:12)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
...
>>> hello = tf.constant('hello, world')
>>> sess = tf.Session()
>>> print(sess.run(hello).decode())
hello, world
>>> exit()
pi@piot4:~ $ cd iot/lesson8
pi@piot4:~/iot/lesson8 $ python3 keras_diabetes.py
More Keras examples at GitHub
125
Google Colab and AI Hub
126
Contribution by Alhussain Almarhabi, 2017 Fall
RMS (Royal Mail Ship) Titanic
127
Maiden Voyage of Titanic
128
41°43′ N, 49°56′ W
Thermal Inversion
The Titanic was sailing from Gulf Stream waters into the frigid Labrador Current, where the air column was cooling from the bottom up, creating a thermal inversion: layers of cold air below layers of warmer air. Extraordinarily high air pressure kept the air free of fog.
129
Superior Mirage
A thermal inversion refracts light abnormally and can create a superior mirage: Objects appear higher (and therefore nearer) than they actually are, before a false horizon. The area between the false horizon and the true one may appear as haze.
130
Iceberg Camouflage
The Californian’s radio operator warned the Titanic of ice. But the moonless night provided little contrast, and a calm sea masked the line between the true and false horizons, camouflaging the iceberg. A Titanic lookout sounded the alarm when the berg was about a mile away—too late.
131
Mistaken Identity
Shortly before the collision, the Titanic sailed into the Californian’s view—but it appeared too near and small to be the great ocean liner. Californian captain Stanley Lord knew the Titanic was the only other ship in the area with a radio, and so concluded this ship did not have one.
132
Morse Lamp
Californian captain said he repeatedly had someone signal the ship by Morse lamp “and she did not take the slightest notice of it.” The Titanic, now in trouble, signaled the Californian by Morse lamp, also to no avail. The abnormally stratified air was distorting and disrupting the signals.
133
Distress Rockets Ignored
The Titanic fired distress rockets some 600 feet into the air—but they appeared to be much lower relative to the ship. Those aboard the Californian, unsure of what they saw, ignored the signals. When the Titanic sank at 2:20 am on Monday, 1912-04-15, they thought she might be simply sailing away.
134
titanic_1.py 0/2
pi@piot4:~/iot/lesson8 $ cat titanic_1.py
import numpy as np
import matplotlib.pyplot as plt
import pdb
from pandas import *
data = read_csv('train.csv')
cols = data.columns
print('Data Columns:')
print(cols)
survivors = data.groupby('Sex').Survived.mean()
print('Survived:')
print(survivors)
plt.hist(data.Pclass, bins=[1,2,3,4], align='left', rwidth=0.5)
plt.xticks([0,1,2,3,4])
plt.xlabel('Passenger Class')
plt.ylabel('Number of Passengers')
plt.show()
135
titanic_1.py 1/2
pi@piot4:~/iot/lesson8 $ cp train.csv ~/demo
pi@piot4:~/iot/lesson8 $ cp test.csv ~/demo
pi@piot4:~/iot/lesson8 $ cp titanic_1.py ~/demo
pi@piot4:~/iot/lesson8 $ cp titanic_2.py ~/demo
pi@piot4:~/demo $ python3 titanic_1.py
Data Columns:
Index(['PassengerId', 'Survived', 'Pclass', 'Name', 'Sex', 'Age', 'SibSp',
'Parch', 'Ticket', 'Fare', 'Cabin', 'Embarked'],
dtype='object')
Survived:
Sex
female 0.742038
male 0.188908
Name: Survived, dtype: float64
136
Contribution by Abhinandan Nuli, 2020 Spring
titanic_1.py 2/2
137
titanic_2.py 0/3
pi@piot4:~/demo $ cat titanic_2.py
import numpy as np
from pandas import *
options.mode.chained_assignment = None
data=read_csv('train.csv')
columns=data.columns
look_up=(data.groupby(['Sex', 'Pclass']).Survived.mean())
test=read_csv('test.csv')
test['Prediction']=0
for i in range(len(test)):
test.Prediction[i]=round(look_up[test.Sex[i]][test.Pclass[i]])
test.to_csv('result.csv', index=False)
data['Fare_Bracket']=0
test['Fare_Bracket']=0
na=sum(test.Fare != test.Fare)
wh_badfare=np.flatnonzero(test.Fare != test.Fare)
sv = test.Fare[test.Fare != test.Fare]
test.Fare[test.Fare != test.Fare]=(3-test.Pclass)*11
138
titanic_2.py 1/3
test.Fare_Bracket=np.array([min(int(price/10),3) for price in test.Fare])
test.Fare[wh_badfare]=sv
data.Fare_Bracket=np.array([min(int(price/10),3) for price in data.Fare])
look_up2=(data.groupby(['Sex', 'Pclass', 'Fare_Bracket']).Survived.mean())
for i in range(len(test)):
test.Prediction[i] =
round(look_up2[test.Sex[i]][test.Pclass[i]][test.Fare_Bracket[i]])
test.to_csv('result2.csv', index=False)
print('Survived:')
print(look_up)
print('Number of Missing Fare: %d' % (na))
print('Indices of Missing Fare: %d' % (wh_badfare))
print('Survived:')
print(look_up2)
pi@piot4:~/demo $ python3 titanic_2.py
139
Four fare brackets: ≤£9, £10~19, £20~29, and ≥£30
titanic_2.py 2/3
Survived:
Sex Pclass
female 1 0.968085
2 0.921053
3 0.500000
male 1 0.368852
2 0.157407
3 0.135447
Name: Survived, dtype: float64
Number of Missing Fare: 1
Indices of Missing Fare: 152
140
titanic_2.py 3/3
Survived:
Sex Pclass Fare_Bracket
female 1 2 0.833333
3 0.977273
2 1 0.914286
2 0.900000
3 1.000000
3 0 0.593750
1 0.581395
2 0.333333
3 0.125000
male 1 0 0.000000
2 0.400000
3 0.383721
2 0 0.000000
1 0.158730
2 0.160000
3 0.214286
3 0 0.111538
1 0.236842
2 0.125000
3 0.240000
Name: Survived, dtype: float64
pi@piot4:~/demo$
141
}
Titanic Passengers
142
Decision Trees or Neural Networks
143
Contribution by Noah McDermott, 2020 Spring
Titanic Passengers and Crew
https://upload.wikimedia.org/wikipedia/commons/6/69/Titanic_casualties.svg
144
The New Colossus
145
Charles Minard, Napoleon's March
146
Napoleon’s troops size to Moscow in brown and the way back in black with temperature
Historical Data Visualization: Minard’s Map Vectorized and Revisited
War and Peace
147
Life Expectancy vs. GDP per Capita
148
Hans Rosling 1948—2017
149
GSMA Mobile Connectivity Index
150
Global Human Journey
151
Most Recent Common Ancestor
152
Country Population
153
154
US Wind Map
155
New York City 311 Calls
156
34,522 complaints called into 311 between September 8 and 15, 2010
Nanocubes
157
Super Bowl and Driving Behavior
158
Hard accelerations and breaks from a subset of Automatic drivers
ThingSpeak
159
Kibana
160
Freeboard
161
IBM Bluemix
162
DGLogik
163
Qlik
164
Data Analysis and Report
165
Natural Language Generation (NLG)
166
Wikipedia Editing Bots
167
Parabola
Parabola is a visual coding tool that enables anyone to work with large data sets, use APIs, and automate workflows, all without writing a single line of code
168
Contribution by Kyra DiFrancesco, 2019 Spring
Robotic Process Automation
169
Sentiment Analysis
170
Apache MXNet
171
deeplearning.ai
172
Object Detection
173
Intel Neural Compute Stick (NCS)
174
Contribution by Ziran Gong, 2018 Fall
Acceleration-as-a-Service (AaaS)
Soft DNN processing units (DPUs) based on field-programmable gate arrays (FPGAs) enable Acceleration-as-a-Service such as Accelize, Amazon EC2 F1, and Microsoft Brainwave with access to ready-to-use accelerators in the Cloud
175
CUDA Processing Flow
176
Tensor Processing Unit (TPU)
177
Cerebras Wafer Scale Engine (WSE)
178
Lesson 8 Summary
179