Introduction to Deep Learning with Python
Samar Haider
University of Southern California
11/7/2019
Artificial intelligence, machine learning & deep learning
The AI universe
USC 2019
3
Samar Haider
Goodfellow et. al., “Deep Learning.” MIT Press (2016)
The AI universe
“What's the difference between AI and ML?”
“It's AI when you're raising money, it's ML when you're trying to hire people.”
USC 2019
4
Samar Haider
Machine learning
USC 2019
5
Samar Haider
Data
Machine learning
USC 2019
6
Samar Haider
Learning Algorithm
Data
Machine learning
USC 2019
7
Samar Haider
Hypothesis
Learning Algorithm
Data
Machine learning
USC 2019
8
Samar Haider
Hypothesis
Input
Prediction
Data
Learning Algorithm
Machine learning
USC 2019
9
Samar Haider
Engineer features
Learn mapping
Data
Features
Labels
Deep learning
USC 2019
10
Samar Haider
Learn features
Learn mapping
Data
Features
Labels
The rise of deep learning
USC 2019
11
Samar Haider
The rise of deep learning
USC 2019
12
Samar Haider
Goodfellow et. al., “Deep Learning.” MIT Press (2016)
The rise of deep learning
USC 2019
13
Samar Haider
Dean, “Large-Scale Deep Learning for Intelligent Computer Systems.” WSDM (2016)
The rise of deep learning
USC 2019
14
Samar Haider
The rise of deep learning
“If big data is the new oil, deep learning is the new internal combustion engine.”
– Yann LeCun
(Director, Facebook AI Research)
USC 2019
15
Samar Haider
The rise of deep learning
“AI is the new electricity: Just as electricity transformed almost everything 100 years ago, today I actually have a hard time thinking of an industry that I don’t think AI will transform in the next several years.”
– Andrew Ng
(Founder, deeplearning.ai)
USC 2019
16
Samar Haider
Neural networks
Biological neuron
USC 2019
18
Samar Haider
Artificial neuron
USC 2019
19
Samar Haider
Biological vs artificial neuron
USC 2019
20
Samar Haider
Activation functions
USC 2019
21
Samar Haider
Sze et. al., “Efficient Processing of Deep Neural Networks: A Tutorial and Survey.” arXiv (2017)
Activation functions
USC 2019
22
Samar Haider
LeCun et. al., “Deep Learning.” Nature (2015)
A shallow neural network
USC 2019
23
Samar Haider
A deep neural network
USC 2019
24
Samar Haider
What we want
USC 2019
25
Samar Haider
Good features/representations
Correct predictions
What we want
USC 2019
26
Samar Haider
Goodfellow et. al., “Deep Learning.” MIT Press (2016)
The need for depth
USC 2019
27
Samar Haider
Goodfellow et. al., “Deep Learning.” MIT Press (2016)
… and even more depth
USC 2019
28
Samar Haider
Szegedy et. al., “Going Deeper with Convolutions.” CVPR (2015)
Specialized architectures
Vision: Convolutional Neural Networks
USC 2019
29
Samar Haider
Specialized architectures
Language: Recurrent Neural Networks
USC 2019
30
Samar Haider
Deep learning
The learning process
USC 2019
32
Samar Haider
Minimizing the error
USC 2019
33
Samar Haider
Gradient descent
USC 2019
34
Samar Haider
Gradient descent in higher dimensions
USC 2019
35
Samar Haider
Learning multiple layers
USC 2019
36
Samar Haider
Forward pass
USC 2019
37
Samar Haider
Backward propagation
USC 2019
38
Samar Haider
Backward propagation
USC 2019
39
Samar Haider
Backward propagation
USC 2019
40
Samar Haider
Backward propagation
USC 2019
41
Samar Haider
Backpropagation algorithm in full
USC 2019
42
Samar Haider
LeCun et. al., “Deep Learning.” Nature (2015)
Applications
Object Detection
USC 2019
44
Samar Haider
Ren et. al., “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.” NIPS (2015)
Scene segmentation
USC 2019
45
Samar Haider
Badrinarayanan et. al., “SegNet: A Deep Convolutional Encoder-Decoder Architecture…” PAMI (2016)
Super resolution
USC 2019
46
Samar Haider
Dahl et. al., “Pixel Recursive Super Resolution.” arXiv (2017)
Style transfer
USC 2019
47
Samar Haider
Gatys et. al., “A Neural Algorithm of Artistic Style.” arXiv (2015)
Image translation
USC 2019
48
Samar Haider
Liu et. al., “Unsupervised Image-to-Image Translation Networks.” NIPS (2017)
Image generation
USC 2019
49
Samar Haider
Karras et. al., “Progressive Growing of GANs for Improved Quality, Stability, and Variation.” arXiv (2017)
Image generation
USC 2019
50
Samar Haider
Karras et. al., “Progressive Growing of GANs for Improved Quality, Stability, and Variation.” arXiv (2017)
Learning word representations
USC 2019
51
Samar Haider
Mikolov et. al., “Efficient Estimation of Word Representations in Vector Space.” arXiv (2013)
Learning sentiment representations
USC 2019
52
Samar Haider
Radford et. al., “Learning to Generate Reviews and Discovering Sentiment.” arXiv (2017)
Writing stories
USC 2019
53
Samar Haider
Radford et. al., “Language Models are Unsupervised Multitask Learners.” OpenAI (2019)
Image captioning
USC 2019
54
Samar Haider
Vinyals et. al., “Show and Tell: A Neural Image Caption Generator.” CVPR (2015)
Visual question answering
USC 2019
55
Samar Haider
Yang et. al., “Stacked Attention Networks for Image Question Answering.” CVPR (2016)
Playing games
USC 2019
56
Samar Haider
Silver et. al., “Mastering the Game of Go with Deep Neural Networks and Tree Search.” Nature (2016)
Building better neural networks
USC 2019
57
Samar Haider
Zoph et. al., “Neural Architecture Search with Reinforcement Learning.” ICLR (2017)
Building better software
USC 2019
58
Samar Haider
Kraska et. al., “The Case for Learned Index Structures.” arXiv (2017)
What you need to get started with deep learning
These, pretty much
USC 2019
60
Samar Haider
… plus a handful of other stuff
USC 2019
61
Samar Haider
https://medium.com/towards-data-science/building-your-own-deep-learning-box-47b918aea1eb
Building a deep learning rig
USC 2019
62
Samar Haider
https://pcpartpicker.com/list/FRp8XH
An alternative
USC 2019
63
Samar Haider
Amazon Web Services
USC 2019
64
Samar Haider
Amazon Web Services
USC 2019
65
Samar Haider
Amazon Web Services
USC 2019
66
Samar Haider
Deep learning software ecosystem
USC 2019
67
Samar Haider
https://towardsdatascience.com/deep-learning-framework-power-scores-2018-23607ddf297a
A typical beginner stack
USC 2019
68
Samar Haider
A typical beginner stack
USC 2019
69
Samar Haider
A typical beginner stack
USC 2019
70
Samar Haider
A typical beginner stack
USC 2019
71
Samar Haider
Another beginner stack
USC 2019
72
Samar Haider
Deep learning in a day
USC 2019
73
Samar Haider
USC 2019
74
Samar Haider
USC 2019
75
Samar Haider
USC 2019
76
Samar Haider
USC 2019
77
Samar Haider
USC 2019
78
Samar Haider
USC 2019
79
Samar Haider
USC 2019
80
Samar Haider
$ssh -i ./aws-key.pem ubuntu@ec2-34-211-139-121.us-west-2.compute.amazonaws.com
$jupyter notebook
USC 2019
81
Samar Haider
USC 2019
82
Samar Haider
USC 2019
83
Samar Haider
USC 2019
84
Samar Haider
USC 2019
85
Samar Haider
A free alternative: Google Colab
USC 2019
86
Samar Haider
Where to learn more
Courses
fast.ai
by Jeremy Howard
USC 2019
88
Samar Haider
Courses
USC 2019
89
Samar Haider
deeplearning.ai
by Andrew Ng
Books
USC 2019
90
Samar Haider
Goodfellow
Bengio
& Courville
Books
USC 2019
91
Samar Haider
Michael Nielsen
Books
USC 2019
92
Samar Haider
François Chollet
Papers
USC 2019
93
Samar Haider
★ Yann LeCun, Yoshua Bengio, and Geoffrey Hinton, “Deep Learning.” Nature (2015)
Most Cited Deep Learning Papers
https://github.com/terryum/awesome-deep-learning-papers
Deep Learning Papers Reading Roadmap
https://github.com/songrotek/Deep-Learning-Papers-Reading-Roadmap
Demos
USC 2019
94
Samar Haider
TensorFlow Playground
https://playground.tensorflow.org/
ConvNetJS
https://cs.stanford.edu/people/karpathy/convnetjs/
Quick, Draw!
https://quickdraw.withgoogle.com/
“Software is eating the world, but AI is going to eat software.”
– Jensen Huang
(CEO, Nvidia)
Thank you
@samarhdr