1 of 84

Deep Learning

with PyTorch and TensorFlow

April 23rd, 2024

​

​

Jon Krohn, Ph.D.

Click to edit Master title style

2 of 84

3 of 84

Deep Learning

with PyTorch and TensorFlow

​

​

Slides: jonkrohn.com/talks

​

Code: github.com/jonkrohn/DLTFpT

​

​

Let’s stay connected:

​

jonkrohn.com to sign up for email newsletter

​

linkedin.com/in/jonkrohn

jonkrohn.com/youtube

twitter.com/JonKrohnLearns

4 of 84

The Pomodoro Technique

Rounds of:

  • 25 minutes of work
  • with 5 minute breaks

​

Questions best handled at breaks, so save questions until then.

​

When people ask questions that have already been answered, do me a favor and let them know, politely providing response if appropriate.

​

Except during breaks, I recommend attending to this lecture only as topics are not discrete: Later material builds on earlier material.

5 of 84

POLL

Where are you?

  • The Americas
  • Europe / Middle East / Africa
  • Asia-Pacific
  • Extra-Terrestrial Space

6 of 84

POLL

What are you?

  • Developer / Engineer
  • Scientist / Analyst / Statistician / Mathematician
  • Combination of the Above
  • Other

7 of 84

POLL

What’s your level of experience with the topic?

  • Little to no exposure to deep learning
  • Some deep learning theory
  • Some deep learning theory + experience with a deep learning library
  • Strong deep learning theory + experience with a deep learning library

8 of 84

ODSC AI+ Deep Learning Series

20 hours of content…

​

  1. How Deep Learning Works
  2. Training a Deep Learning Network
  3. Machine Vision and Creativity
  4. NLP
  5. Deep RL and A.I.
  6. PyTorch and Beyond

​

…are abbreviated into today’s half-day training.

aiplus.training

9 of 84

ODSC AI+ ML Foundations Series

Subjects…

​

  • Intro to Linear Algebra
  • Linear Algebra II: Matrix Operations
  • Calculus I: Limits & Derivatives
  • Calculus II: Partial Derivatives & Integrals
  • Probability & Information Theory
  • Intro to Statistics
  • Algorithms & Data Structures
  • Optimization

​

…are foundational for deeply understanding ML models.

github.com/jonkrohn/ML-foundations

jonkrohn.com/youtube

10 of 84

11 of 84

​

  1. The Unreasonable Effectiveness of Deep Learning
  2. Essential Deep Learning Theory
  3. Deep Learning with TensorFlow and PyTorch

Deep Learning Fundamentals

Click to edit Master title style

12 of 84

​

Segment 1:

The Unreasonable Effectiveness of Deep Learning

  • Intro to Neural Networks and Deep Learning
  • Deep Learning Families
  • Deep Learning Libraries

Deep Learning Fundamentals

Click to edit Master title style

13 of 84

​

Segment 1:

The Unreasonable Effectiveness of Deep Learning

  • Intro to Neural Networks and Deep Learning (Chapter 1)
  • Deep Learning Families
  • Deep Learning Libraries

Deep Learning Fundamentals

Click to edit Master title style

14 of 84

​

​

35% off orders:

bit.ly/iTkrohn

(use code KROHN during checkout)

​

Will sign your book at

ODSC Events

Click to edit Master title style

15 of 84

Click to edit Master title style

16 of 84

Click to edit Master title style

17 of 84

Click to edit Master title style

18 of 84

Case Study: The History of Vision

Click to edit Master title style

19 of 84

Click to edit Master title style

20 of 84

Click to edit Master title style

21 of 84

Click to edit Master title style

22 of 84

Click to edit Master title style

23 of 84

Case Study: The History of Vision

Click to edit Master title style

24 of 84

Neocognitron (Fukushima, 1980)

Click to edit Master title style

25 of 84

Case Study: The History of Vision

Click to edit Master title style

26 of 84

Yann LeCun and Yoshua Bengio

Click to edit Master title style

27 of 84

LeNet-5 (LeCun et al., 1998)

Click to edit Master title style

28 of 84

Case Study: The History of Vision

Click to edit Master title style

29 of 84

Click to edit Master title style

30 of 84

Traditional ML vs Deep Learning

Click to edit Master title style

31 of 84

Case Study: The History of Vision

Click to edit Master title style

32 of 84

Viola & Jones (2001)

Click to edit Master title style

33 of 84

Case Study: The History of Vision

Click to edit Master title style

34 of 84

Geoff Hinton

Click to edit Master title style

35 of 84

AlexNet (Krizhevsky et al., 2012)

Click to edit Master title style

36 of 84

Fei-Fei Li

Click to edit Master title style

37 of 84

Click to edit Master title style

38 of 84

QUESTION

If a voice recognition algorithm is fed audio of speech as inputs, given corresponding text as the outputs (labels) to learn, and no features are explicitly programmed, is this a:

  • Traditional Machine Learning Algorithm
  • Deep Learning Algorithm
  • I Don't Know

​

Click to edit Master title style

39 of 84

​

Segment 1:

The Unreasonable Effectiveness of Deep Learning

  • Intro to Neural Networks and Deep Learning
  • Deep Learning Families (Chapters 2-4)
  • Deep Learning Libraries

Deep Learning Fundamentals

Click to edit Master title style

40 of 84

Dense Networks

Click to edit Master title style

41 of 84

The Cart Before the Horse (Chapter 5)

​

​

​

​

​

​

​

​

​

GitHub repo: github.com/jonkrohn/DLTFpT

Interactive Colab demo: Shallow Net in TensorFlow

Click to edit Master title style

42 of 84

ConvNets: Convolutional Networks

Click to edit Master title style

43 of 84

ConvNets: Convolutional Networks

Ren et al. (2015)

Click to edit Master title style

44 of 84

NLP with RNNs Transformers (tomorrow)

Click to edit Master title style

45 of 84

Image Gen. (Transformers/Latent Diffusion)

Jepson Taylor

Michael King

Click to edit Master title style

46 of 84

Image Gen. (Transformers/Latent Diffusion)

Jepson Taylor

GPT-4

�Midjourney/

DALL-E

​

Click to edit Master title style

47 of 84

Deep Reinforcement Learning

Click to edit Master title style

48 of 84

Click to edit Master title style

49 of 84

Demis Hassabis and David Silver

Click to edit Master title style

50 of 84

POLL

If you were designing an algorithm to learn to play Tetris by maximizing its score, which of these Deep Learning approaches would be most appropriate?

  • Convolutional Neural Network
  • Transformer
  • Recurrent Neural Network
  • Deep Reinforcement Learning

​

Click to edit Master title style

51 of 84

POLL

If you were designing an algorithm to recognise tumours in medical images, which of these Deep Learning approaches would be most appropriate?

  • Convolutional Neural Network
  • Recurrent Neural Network
  • Deep Reinforcement Learning
  • Generative Adversarial Network

​

Click to edit Master title style

52 of 84

POLL

If you were designing an algorithm to predict stock price movements based on time series data, which of these Deep Learning approaches would be most appropriate?

  • Convolutional Neural Network
  • Recurrent Neural Network
  • Deep Reinforcement Learning
  • Generative Adversarial Network

​

Click to edit Master title style

53 of 84

​

Segment 1:

The Unreasonable Effectiveness of Deep Learning

  • Intro to Neural Networks and Deep Learning
  • Deep Learning Families
  • Deep Learning Libraries (Chapter 14)

Deep Learning Fundamentals

Click to edit Master title style

54 of 84

Leading Deep Learning Libraries

Click to edit Master title style

55 of 84

Click to edit Master title style

56 of 84

Click to edit Master title style

57 of 84

​

Caffe

Torch

MXNet

TensorFlow

Language

Python, Matlab

Lua, C

Python, R, C++ Julia, Matlab

JavaScript, Go Scala, Perl

Python, C, C++

Java, Go, JS, Swift (Haskell, Julia, R, Scala, Rust, C#)

Programming Style

Symbolic

Imperative

Imperative

Imperative (since 2.0)

Parallel GPUs: Data

Yes

Yes

Yes

Yes

Parallel GPUs: Model

​

Yes

Yes

Yes

Pre-Trained Models

Model Zoo

Model Zoo

Model Zoo

github.com/tensorflow/models

High-Level APIs

​

PyTorch

in-built

Keras

Particular Strength

CNNs

interactivity

​

production deployment

Leading Deep Learning Libraries

Click to edit Master title style

58 of 84

PyTorch

TensorFlow

“NumPy”, optimized for GPUs

ported to Python from C++

dynamic auto-differentiation (autodiff)

static graph (historically or with Keras)

debugging is easier

​

fast.ai API, PyTorch Lightning

Keras API

more prevalent in new packages

more prevalent in legacy code

TorchScript Just-In-Time compilation

TensorFlow Serving, .js, Lite, tf.data, tf.io

more enjoyable for model design

better for production deployments

Learn both! And you can translate: github.com/onnx/onnx

Click to edit Master title style

59 of 84

​

Segment 2:

Essential Deep Learning Theory

  • Learning with Artificial Neurons (Chapters 6-7)
  • TensorFlow Playground

Deep Learning Fundamentals

Click to edit Master title style

60 of 84

“Whiteboarding”!

Click to edit Master title style

61 of 84

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Click to edit Master title style

62 of 84

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Cost Functions

  • quadratic cost

Click to edit Master title style

63 of 84

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Cost Functions

  • quadratic cost
  • cross-entropy

Click to edit Master title style

64 of 84

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Cost Functions

  • quadratic cost
  • cross-entropy

Gradient Descent

Click to edit Master title style

65 of 84

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Cost Functions

  • quadratic cost
  • cross-entropy

Gradient Descent

Backpropagation

Click to edit Master title style

66 of 84

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Cost Functions

  • quadratic cost
  • cross-entropy

Stochastic Gradient Descent

  • mini-batch size
  • learning rate
  • second-order, e.g., Nadam

Backpropagation

Click to edit Master title style

67 of 84

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Cost Functions

  • quadratic cost
  • cross-entropy

Stochastic Gradient Descent

  • mini-batch size
  • learning rate
  • second-order, e.g., Nadam

Backpropagation

Initialization

  • Glorot/He

​

Click to edit Master title style

68 of 84

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Cost Functions

  • quadratic cost
  • cross-entropy

Stochastic Gradient Descent

  • mini-batch size
  • learning rate
  • second-order, e.g., Nadam

Backpropagation

Initialization

  • Glorot/He

Layers

  • dense
  • convolutional, etc.

Click to edit Master title style

69 of 84

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Cost Functions

  • quadratic cost
  • cross-entropy

Stochastic Gradient Descent

  • mini-batch size
  • learning rate
  • second-order, e.g., Nadam

Backpropagation

Initialization

  • Glorot/He

Layers

  • dense
  • convolutional, etc.

Avoiding Overfitting

  • dropout

Click to edit Master title style

70 of 84

Dropout

Click to edit Master title style

71 of 84

Your Arsenal

Activation Functions

  • sigmoid
  • tanh
  • ReLU
  • softmax

Cost Functions

  • quadratic cost
  • cross-entropy

Stochastic Gradient Descent

  • mini-batch size
  • learning rate
  • second-order, e.g., Nadam

Backpropagation

Initialization

  • Glorot/He

Layers

  • dense
  • convolutional, etc.

Avoiding Overfitting

  • dropout
  • data augmentation
  • batch normalization

Click to edit Master title style

72 of 84

TensorFlow Playground

​

​

​

​

interactive demo

Click to edit Master title style

73 of 84

TensorFlow Playground

​

Click to edit Master title style

74 of 84

Part 3:

Deep Learning with TensorFlow

  • Revisiting our Shallow Neural Network
  • Deep Nets in TensorFlow (Chapters 8-9)
  • Deep Nets in PyTorch
  • What to Study Next, Depending on Your Interests

Deep Learning Fundamentals

Click to edit Master title style

75 of 84

Revisiting our Shallow Net

interactive Colab demo: Shallow Net in TensorFlow

Click to edit Master title style

76 of 84

Deep Nets in TensorFlow

interactive Colab demo: Deep Net in TensorFlow

​

​

​

​

​

​

​

​

Hint: See LeNet in TensorFlow for how we could do even better.

Click to edit Master title style

77 of 84

Deep Nets in PyTorch

Click to edit Master title style

78 of 84

Deep Nets in PyTorch

interactive Colab demos:

  • Deep Net in PyTorch
  • Deep Net in PyTorch with GPU

Click to edit Master title style

79 of 84

Pointers for DL Job Search

  • Read Deep Learning Illustrated
    • Understand all of the code, converting TF into PyTorch
      • e.g., convert LeNet notebook to PyTorch with help from GPT-4
    • Carry out several projects, led by Chapter 14 guidance
  • Carry out several specialized, independent projects
    • e.g., using inspiration and/or data from jonkrohn.com/resources
  • Next-level prep:
    • Machine Learning Foundations series: github.com/jonkrohn/ML-foundations
    • Goodfellow et al.’s Deep Learning
    • Chapters 12, 13 & 19 of Geron’s Hands-on ML (2022, 3rd ed.)
  • Research whomever’s interviewing you on LinkedIn, etc. so that you have thoughtful questions / find common ground

Click to edit Master title style

80 of 84

​

  • The Unreasonable Effectiveness of Deep Learning
  • Essential Deep Learning Theory
  • Deep Learning with TensorFlow and PyTorch

Deep Learning Fundamentals

Click to edit Master title style

81 of 84

POLL

What deep learning topics interest you most?

  • Machine Vision
  • Natural Language Processing
  • Multimodal Generative A.I.
  • Deep Reinforcement Learning
  • Something Else

​

Click to edit Master title style

82 of 84

83 of 84

​

​

35% off orders:

bit.ly/iTkrohn

(use code KROHN during checkout)

​

Will sign your book at

ODSC Events

Click to edit Master title style

84 of 84

Stay in Touch

jonkrohn.com to sign up for email newsletter

linkedin.com/in/jonkrohn

​

youtube.com/c/JonKrohnLearns

​

twitter.com/JonKrohnLearns