1 of 77

Deep Learning

with TensorFlow 2 and PyTorch

jonkrohn.com/talks

​

github.com/jonkrohn/DLTFpT

Feb-Apr 2022

​

Jon Krohn, Ph.D.

Click to edit Master title style

2 of 77

Deep Learning

with TensorFlow 2 and PyTorch

​

​

Slides: jonkrohn.com/talks

​

Code: github.com/jonkrohn/DLTFpT

​

​

Stay in Touch:

​

jonkrohn.com to sign up for email newsletter

​

linkedin.com/in/jonkrohn

jonkrohn.com/youtube

twitter.com/JonKrohnLearns

3 of 77

ODSC AI+ ML Foundations Series

Subjects…

​

  1. Intro to Linear Algebra
  2. Linear Algebra II: Matrix Operations
  3. Calculus I: Limits & Derivatives
  4. Calculus II: Partial Derivatives & Integrals
  5. Probability & Information Theory
  6. Intro to Statistics
  7. Algorithms & Data Structures
  8. Optimization

​

…are foundational for deeply understanding ML models.

github.com/jonkrohn/ML-foundations

4 of 77

The Pomodoro Technique

Rounds of:

  • 25 minutes of work
  • with 5 minute breaks

​

Questions best handled at breaks, so save questions until then.

​

When people ask questions that have already been answered, do me a favor and let them know, politely providing response if appropriate.

​

Except during breaks, I recommend attending to this lecture only as topics are not discrete: Later material builds on earlier material.

5 of 77

POLL

Where are you?

  • The Americas
  • Europe / Middle East / Africa
  • Asia-Pacific
  • Extra-Terrestrial Space

6 of 77

POLL

What are you?

  • Developer / Engineer
  • Scientist / Analyst / Statistician / Mathematician
  • Combination of the Above
  • Other

7 of 77

POLL

What’s your level of experience with the topic?

  • Little to no exposure to deep learning
  • Some deep learning theory
  • Some deep learning theory + experience with a deep learning library
  • Strong deep learning theory + experience with a deep learning library

8 of 77

Deep Learning Fundamentals

​

  1. The Unreasonable Effectiveness of Deep Learning
  2. Essential Deep Learning Theory
  3. Deep Learning with TensorFlow 2 and PyTorch

Click to edit Master title style

9 of 77

​

Segment 1:

The Unreasonable Effectiveness of Deep Learning

  • Intro to Neural Networks and Deep Learning
  • Deep Learning Families
  • Deep Learning Libraries

Deep Learning Fundamentals

Click to edit Master title style

10 of 77

​

Segment 1:

The Unreasonable Effectiveness of Deep Learning

  • Intro to Neural Networks and Deep Learning (Chapter 1)
  • Deep Learning Families
  • Deep Learning Libraries

Deep Learning Fundamentals

Click to edit Master title style

11 of 77

​

​

35% off orders:

bit.ly/iTkrohn

(use code KROHN during checkout)

​

Book-signing at upcoming non-virtual ODSC events!

​

Click to edit Master title style

12 of 77

Click to edit Master title style

13 of 77

Click to edit Master title style

14 of 77

Click to edit Master title style

15 of 77

Case Study: The History of Vision

Click to edit Master title style

16 of 77

Click to edit Master title style

17 of 77

Click to edit Master title style

18 of 77

Click to edit Master title style

19 of 77

Click to edit Master title style

20 of 77

Case Study: The History of Vision

Click to edit Master title style

21 of 77

Neocognitron (Fukushima, 1980)

Click to edit Master title style

22 of 77

Case Study: The History of Vision

Click to edit Master title style

23 of 77

Yann LeCun and Yoshua Bengio

Click to edit Master title style

24 of 77

LeNet-5 (LeCun et al., 1998)

Click to edit Master title style

25 of 77

Case Study: The History of Vision

Click to edit Master title style

26 of 77

Click to edit Master title style

27 of 77

Traditional ML vs Deep Learning

Click to edit Master title style

28 of 77

Case Study: The History of Vision

Click to edit Master title style

29 of 77

Viola & Jones (2001)

Click to edit Master title style

30 of 77

Case Study: The History of Vision

Click to edit Master title style

31 of 77

Geoff Hinton

Click to edit Master title style

32 of 77

AlexNet (Krizhevsky et al., 2012)

Click to edit Master title style

33 of 77

Click to edit Master title style

34 of 77

Fei-Fei Li

Click to edit Master title style

35 of 77

POLL

If a voice recognition algorithm is fed audio of speech as inputs, given corresponding text as the outputs (labels) to learn, and no features are explicitly programmed, is this a:

  • Traditional Machine Learning Algorithm
  • Deep Learning Algorithm
  • I Don't Know

​

Click to edit Master title style

36 of 77

​

Segment 1:

The Unreasonable Effectiveness of Deep Learning

  • Intro to Neural Networks and Deep Learning
  • Deep Learning Families (Chapters 2-4)
  • Deep Learning Libraries

Deep Learning Fundamentals

Click to edit Master title style

37 of 77

Dense Networks

Click to edit Master title style

38 of 77

The Cart Before the Horse (Chapter 5)

​

​

​

​

​

​

​

​

​

GitHub repo: github.com/jonkrohn/DLTFpT

Interactive Colab demo: Shallow Net in TensorFlow

Click to edit Master title style

39 of 77

ConvNets: Convolutional Networks

Click to edit Master title style

40 of 77

ConvNets: Convolutional Networks

Ren et al. (2015)

Click to edit Master title style

41 of 77

RNNs: Recurrent Neural Networks

Click to edit Master title style

42 of 77

GANs: Generative Adversarial Networks

Click to edit Master title style

43 of 77

GANs: Generative Adversarial Networks

Karros et al. (2018)

Click to edit Master title style

44 of 77

Deep Reinforcement Learning

Click to edit Master title style

45 of 77

Demis Hassabis and David Silver

Click to edit Master title style

46 of 77

POLL

If you were designing an algorithm to learn to play Tetris by maximizing its score, which of these Deep Learning approaches would be most appropriate?

  • Convolutional Neural Network
  • Recurrent Neural Network
  • Deep Reinforcement Learning
  • Generative Adversarial Network

​

Click to edit Master title style

47 of 77

POLL

If you were designing an algorithm to recognise tumours in medical images, which of these Deep Learning approaches would be most appropriate?

  • Convolutional Neural Network
  • Recurrent Neural Network
  • Deep Reinforcement Learning
  • Generative Adversarial Network

​

Click to edit Master title style

48 of 77

POLL

If you were designing an algorithm to predict stock price movements based on time series data, which of these Deep Learning approaches would be most appropriate?

  • Convolutional Neural Network
  • Recurrent Neural Network
  • Deep Reinforcement Learning
  • Generative Adversarial Network

​

Click to edit Master title style

49 of 77

​

Segment 1:

The Unreasonable Effectiveness of Deep Learning

  • Intro to Neural Networks and Deep Learning
  • Deep Learning Families
  • Deep Learning Libraries (Chapter 14)

Deep Learning Fundamentals

Click to edit Master title style

50 of 77

Leading Deep Learning Libraries

Click to edit Master title style

51 of 77

Click to edit Master title style

52 of 77

Click to edit Master title style

53 of 77

​

Caffe

Torch

MXNet

TensorFlow

Language

Python, Matlab

Lua, C

Python, R, C++ Julia, Matlab

JavaScript, Go Scala, Perl

Python, C, C++

Java, Go, JS, Swift (Haskell, Julia, R, Scala, Rust, C#)

Programming Style

Symbolic

Imperative

Imperative

Imperative (since 2.0)

Parallel GPUs: Data

Yes

Yes

Yes

Yes

Parallel GPUs: Model

​

Yes

Yes

Yes

Pre-Trained Models

Model Zoo

Model Zoo

Model Zoo

github.com/tensorflow/models

High-Level APIs

​

PyTorch

in-built

Keras

Particular Strength

CNNs

interactivity

​

production deployment

Leading Deep Learning Libraries

Click to edit Master title style

54 of 77

PyTorch

TensorFlow

“NumPy”, optimized for GPUs

ported to Python from C++

dynamic auto-differentiation (autodiff)

static graph (historically or with Keras)

debugging is easier

​

fast.ai API

Keras API

​

more widely adopted

TorchScript Just-In-Time compilation

TensorFlow Serving, .js, Lite, tf.data, tf.io

more enjoyable for model design

better for production deployments

Learn both! And you can translate: github.com/onnx/onnx

Click to edit Master title style

55 of 77

​

Segment 2:

Essential Deep Learning Theory

  • Learning with Artificial Neurons (Chapters 6-7)
  • TensorFlow Playground

Deep Learning Fundamentals

Click to edit Master title style

56 of 77

“Whiteboarding”!

Click to edit Master title style

57 of 77

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Click to edit Master title style

58 of 77

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Cost Functions

  • quadratic cost

Click to edit Master title style

59 of 77

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Cost Functions

  • quadratic cost
  • cross-entropy

Click to edit Master title style

60 of 77

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Cost Functions

  • quadratic cost
  • cross-entropy

Gradient Descent

Click to edit Master title style

61 of 77

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Cost Functions

  • quadratic cost
  • cross-entropy

Gradient Descent

Backpropagation

Click to edit Master title style

62 of 77

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Cost Functions

  • quadratic cost
  • cross-entropy

Stochastic Gradient Descent

  • mini-batch size
  • learning rate
  • second-order, e.g., Nadam

Backpropagation

Click to edit Master title style

63 of 77

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Cost Functions

  • quadratic cost
  • cross-entropy

Stochastic Gradient Descent

  • mini-batch size
  • learning rate
  • second-order, e.g., Nadam

Backpropagation

Initialization

  • Glorot/He

​

Click to edit Master title style

64 of 77

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Cost Functions

  • quadratic cost
  • cross-entropy

Stochastic Gradient Descent

  • mini-batch size
  • learning rate
  • second-order, e.g., Nadam

Backpropagation

Initialization

  • Glorot

Layers

  • dense
  • convolutional, recurrent, etc.

Click to edit Master title style

65 of 77

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Cost Functions

  • quadratic cost
  • cross-entropy

Stochastic Gradient Descent

  • mini-batch size
  • learning rate
  • second-order, e.g., Nadam

Backpropagation

Initialization

  • Glorot

Layers

  • dense
  • convolutional, recurrent, etc.

Avoiding Overfitting

  • Dropout

Click to edit Master title style

66 of 77

Dropout

Click to edit Master title style

67 of 77

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Cost Functions

  • quadratic cost
  • cross-entropy

Stochastic Gradient Descent

  • mini-batch size
  • learning rate
  • second-order, e.g., Nadam

Backpropagation

Initialization

  • Glorot

Layers

  • dense
  • convolutional, recurrent, etc.

Avoiding Overfitting

  • Dropout
  • Data Augmentation

Click to edit Master title style

68 of 77

TensorFlow Playground

interactive demo: bit.ly/TFplayground

Click to edit Master title style

69 of 77

Part 3:

Deep Learning with TensorFlow

  • Revisiting our Shallow Neural Network
  • Deep Nets in TensorFlow (Chapters 8-9)
  • Deep Nets in PyTorch
  • What to Study Next, Depending on Your Interests

Deep Learning Fundamentals

Click to edit Master title style

70 of 77

Revisiting our Shallow Net

interactive Colab demo: Shallow Net in TensorFlow

Click to edit Master title style

71 of 77

Deep Nets in TensorFlow

interactive Colab demo: Deep Net in TF (bit.ly/deepNetTF)

Click to edit Master title style

72 of 77

Deep Nets in PyTorch

interactive Colab demos:

  • Shallow Net in PyTorch (bit.ly/shallowPT)
  • Deep Net in PyTorch (bit.ly/deepPTsolution)

Click to edit Master title style

73 of 77

Pointers for DL Job Search

  • Read all of Deep Learning Illustrated
  • Understand all of the code, perhaps converting TF into PyTorch
  • Carry out several projects, led by Chapter 14 guidance
  • Carry out a specialized, independent project using inspiration and/or data from jonkrohn.com/resources
  • Next-level prep:
    • Machine Learning Foundations series: github.com/jonkrohn/ML-foundations
    • Chapters 12, 13 & 19 of Geron’s Hands-on Machine Learning
    • Goodfellow et al.’s Deep Learning
  • Research whoever’s interviewing you on LinkedIn, etc. so that you have thoughtful questions / find common ground
  • Work with a specialized recruiter from GQR :)

Click to edit Master title style

74 of 77

ODSC AI+ ML Foundations Series

Optimization builds upon and is foundational for:

​

  • Intro to Linear Algebra
  • Linear Algebra II: Matrix Operations
  • Calculus I: Limits & Derivatives
  • Calculus II: Partial Derivatives & Integrals
  • Probability & Information Theory
  • Intro to Statistics
  • Algorithms & Data Structures
  • Optimization

​

Deeply understanding machine learning models.

github.com/jonkrohn/ML-foundations

75 of 77

POLL

What deep learning topics interest you most?

  • CNNs and Machine Vision
  • Natural Language Processing
  • Generative Adversarial Networks
  • Deep Reinforcement Learning
  • Something Else

​

Click to edit Master title style

76 of 77

​

​

35% off orders:

bit.ly/iTkrohn

(use code KROHN during checkout)

​

Book-signing at upcoming non-virtual ODSC events!

​

Click to edit Master title style

77 of 77

Stay in Touch

jonkrohn.com to sign up for email newsletter

linkedin.com/in/jonkrohn

​

youtube.com/c/JonKrohnLearns

​

twitter.com/JonKrohnLearns