Machine Learning Foundations

Calc II: Partial Derivatives & Integrals

Using Gradients in

Python to Enable Algorithms to

Learn from Data

​

​

Jon Krohn, Ph.D.

​

jonkrohn.com/talks

​

github.com/jonkrohn/ML-foundations

Machine Learning Foundations

Calc II: Partial Derivatives & Integrals

Slides: jonkrohn.com/talks

​

Code: github.com/jonkrohn/ML-foundations

​

​

Stay in Touch:

​

jonkrohn.com to sign up for email newsletter

​

linkedin.com/in/jonkrohn

jonkrohn.com/youtube

twitter.com/JonKrohnLearns

The Pomodoro Technique

Rounds of:

• 25 minutes of work
• with 5 minute breaks

​

Questions best handled at breaks, so save questions until then.

​

When people ask questions that have already been answered, do me a favor and let them know, politely providing response if appropriate.

​

Except during breaks, I recommend attending to this lecture only as topics are not discrete: Later material builds on earlier material.

POLL

What is your level of familiarity with Calculus?

• Little to no exposure
• Some understanding of the theory
• Deep understanding of the theory
• Deep understanding of the theory and experience applying calculus operations (e.g., differentiation) with code

POLL

What is your level of familiarity with Machine Learning?

• Little to no exposure, or exposure to theory only
• Experience applying machine learning with code
• Experience applying machine learning with code and some understanding of the underlying theory
• Experience applying machine learning with code and strong understanding of the underlying theory

ML Foundations Series

Calculus II builds upon and is foundational for:

​

1. Intro to Linear Algebra
2. Linear Algebra II: Matrix Operations
3. Calculus I: Limits & Derivatives
4. Calculus II: Partial Derivatives & Integrals
5. Probability & Information Theory
6. Intro to Statistics
7. Algorithms & Data Structures
8. Optimization

Calc II: Partial Derivatives & Integrals

​

1. Review of Introductory Calculus
2. Machine Learning Gradients
3. Integrals

Calc II: Partial Derivatives & Integrals

​

• Review of Introductory Calculus
• Machine Learning Gradients
• Integrals

• The Delta Method
• Differentiation with Rules
• AutoDiff: Automatic Differentiation

Segment 1: Review of Introductory Calc

What Calculus Is

• Mathematical study of continuous change
• Two branches:
1. Differential calculus: expanded on in Calc II
2. Integral calculus: a focus of Calc II subject

What Calculus Is

• Mathematical study of continuous change
• Two branches:
• Differential calculus: expanded on in Calc II
• Integral calculus: a focus of Calc II subject

What Differential Calculus Is

• Study of rates of change
• Consider a vehicle traveling some distance d over time t:

What Calculus Is

• Mathematical study of continuous change
• Two branches:
• Differential calculus: expanded on in Calc II
• Integral calculus: a focus of Calc II subject

What Integral Calculus Is

• Study of areas under curves
• Facilitates the “opposite” of differential calculus:

The Delta Method

​

​

​

​

​

Derivative Notation

Derivative of a Constant

​

​

Assuming c is constant:

​

​

Intuition: A constant has no variation so its slope is nothing, e.g.:

The Power Rule

The Constant Product Rule

The Sum Rule

The Chain Rule

• Many applications within ML
• Critical for backpropagation algo used to train neural nets
• Based on nested functions:
• Let’s say y = (5x + 25)³
• We can let u = 5x + 25
• In that case, y = u³
• y is a function of u, and u is a function of x
• Chain rule is easy way to find derivative of nested function:

​

The Chain Rule

Power Rule on a Function Chain

Fitting a Line with Machine Learning

• Line equation y = mx + b as directed acyclic graph (DAG)
• Nodes are input, output, parameters, or operations
• Directed edges (“arrows”) are tensors (N.B.: non-operation nodes can be tensors too)

​

​

​

​

​

​

​

​

​

Machine Learning

Machine Learning

Step 3: Partial Differentiation (the primary focus of Calc II)

Step 4: Descend gradient of cost C w.r.t. parameters m and b

​

​

​

​

​

​

​

​

Hands-on code demo: regression-in-pytorch.ipynb

gradient of C w.r.t. p = 0

Image © 2020 Pearson

Calc II: Partial Derivatives & Integrals

​

• Review of Introductory Calculus
• Machine Learning Gradients
• Integrals

• Partial Derivatives of Multivariate Functions
• The Partial-Derivative Chain Rule
• Quadratic Cost
• Gradients
• Gradient Descent
• Backpropagation
• Higher-Order Partial Derivatives

Segment 2: ML Gradients

Multivariate Functions

Even in a simple regression such as y = mx + b:

y is a function of multiple parameters

— in this case, m and b.

Therefore, we can’t calculate the full derivative dy/dm or dy/db.

​

​

Partial Derivatives

Enable the calculation of derivatives of multivariate equations.

​

Consider the equation z = x² - y²

​

​

Hands-on demo: geogebra.org/3d

​

The partial derivative of z with respect to x is obtained by considering y to be a constant:

​

​

​

​

​

​

​

​

The slope of z along the x axis is twice the x axis value.

​

​

Hands-on code demo

Partial Derivatives

Reconsider z = x² - y²

from the perspective of z w.r.t y

​

Hands-on demo: geogebra.org/3d

​

The partial derivative of z with respect to y is obtained by considering x to be a constant:

​

​

​

​

​

​

​

​

The slope of z along the y axis is twice the y axis value

...and is inverted.

​

Hands-on code demo

Solutions

​

Solutions

​

Solutions

​

Partial Derivatives

Hands-on code demo

Partial Derivatives

Hands-on code demo

Exercises

Find all the partial derivatives of the following functions:

1. z = y³ + 5xy
2. The surface area of a cylinder is described by a = 2πr² + 2πrh.
3. The volume of a square prism with a cube cut out of its center is described by v = x²y - z³.

Solutions

​

Solutions

​

Solutions

​

Partial Derivative Notation

The Chain Rule

Let’s say:

​

Recall that the chain rule for full derivatives would be:

​

​

​

​

With univariate functions, the partial derivative is identical:

The Chain Rule

​

With a multivariate function, the partial derivative is more interesting:

The Chain Rule

​

With multiple multivariate functions, it gets really interesting:

The Chain Rule

Generalizing completely:

Exercises

Find all the partial derivatives of y, where:

Solutions

​

Solutions

​

Solutions

​

Recalling Machine Learning

Recalling Machine Learning

​

Step 3: Automatic differentiation

Step 4: Descend gradient of cost C w.r.t. parameters m and b

​

​

​

​

​

​

​

​

Hands-on code demo: single-point-regression-gradient NB

gradient of C w.r.t. p = 0

Image © 2020 Pearson

Quadratic Cost w.r.t. Predicted y

Predicted y w.r.t. Model Parameters

Quadratic Cost w.r.t. Model Parameters

Hands-on code demo

Recalling Machine Learning

Recalling Machine Learning

​

Step 3: Determine gradient of cost C w.r.t. parameters m and b

Step 4: Descend gradient

​

​

​

​

​

​

​

​

​

Image © 2020 Pearson

gradient of C w.r.t. p = 0

∇C: the Gradient of Cost

Image © 2020 Pearson

​

Hands-on code demo: batch-regression-gradient NB

MSE w.r.t. Predicted y

​

MSE w.r.t. Model Parameters

Hands-on code demo

Regression Line after 1000 Epochs

Backpropagation

Chain rule of partial derivatives of cost w.r.t. model parameters extends to deep neural networks, which may have 1000s of layers:

Higher-Order Derivatives

Higher-Order Partial Derivatives

In ML, used to accelerate through gradient descent. (Optimization)

​

Consider the following first-order partial derivatives...

Higher-Order Partial Derivatives

Higher-Order Partial Derivatives

Higher-Order Partial Derivative Notation

Exercise

Find all the second-order partial derivatives of z = x³ + 2xy.

​

Solution

Calc II: Partial Derivatives & Integrals

​

• Review of Introductory Calculus
• Gradients Applied to Machine Learning
• Integrals

• Binary Classification
• The Confusion Matrix
• The Receiver-Operating Characteristic (ROC) Curve
• Calculating Integrals Manually
• Numeric Integration with Python
• Finding the Area Under the ROC Curve
• Resources for Further Study of Calculus

Segment 3: Integrals

Supervised Learning

• Have x and y
• Goal: learn function that uses x to approximate y
• Examples:
• Regression
• Clinical measure of forgetfulness
• Sales of a product
• Future value of an asset
• Classification
• Multinomial
• Handwritten digits: 10 classes
• Imagenet: 21k classes
• Binomial
• Movie-review sentiment: positive vs negative
• Photos of fast food: Hot dog vs not hot dog

Accuracy at a Single Threshold

​

​

​

​

​

​

​

​

• Doesn’t reflect model quality at other points in output distribution
• If y = 1: Prediction of 0.49 is 100% wrong; 0.51 prediction 100% correct
• Prediction of 0.51 is considered as correct as prediction of 0.99
• Solution: ROC AUC metric

The Confusion Matrix

Four Hot Dog Predictions

Image © 2020 Pearson

Receiver-Operating Characteristic

The ROC Curve

Image © 2020 Pearson

The ROC Curve

Integral Calculus

• Study of areas under curves
• Facilitates the inverse of differential calculus:

​

​

​

​

​

​

​

​

• Also finds areas more generally, volumes, and central points

Integral Calculus Applications in ML

Find area under the curve:

• Receiver operating characteristic (Calc II)
• Probability theory’s “expectation” of random variable is widely used in machine learning, incl. deep learning (Prob. & Info. Thy.)

Image © 2020 Pearson

dx Slice Width

​

​

​

​

​

​

​

​

​

dx indicates slice width (Δx) is approaching zero width

Integral Notation

The Power Rule

The Constant Multiple Rule

The Sum Rule

Exercises

Solutions

Definite Integrals

Hands-on code demo

Exercise

Evaluate the following expression using both pencil and Python:

Solution

Hands-on code demo

Area Under the ROC Curve

Hands-on code demo

Image © 2020 Pearson

Resources for Further Study

• General reference textbook: Michael Spivak’s Calculus
• Differential Calculus:
• Ch. 6 of Deisenroth et al. (2020) Mathematics for ML
• 3Blue1Brown on YouTube
• Integral Calculus:
• ditto
• Appendix 18.5 of Zhang et al.’s (2019) Dive into Deep Learning
• Next steps in the ML Foundations series:
• Probability & Information Theory
• Intro to Statistics
• Optimization

Next Subject: Probability & Info. Thy.

Apply calculus to ascertain how much meaningful signal is present in data.

​

Learn the probability theory-based foundations of stats and ML.

POLL with Multiple Answers Possible

​

What follow-up topics interest you most?

• Linear Algebra
• More Calculus
• Probability / Statistics
• Computer Science (e.g., algorithms, data structures)
• Machine Learning Basics
• Advanced Machine Learning, incl. Deep Learning
• Something Else

​

Stay in Touch

jonkrohn.com to sign up for email newsletter

linkedin.com/in/jonkrohn

​

youtube.com/c/JonKrohnLearns

​

twitter.com/JonKrohnLearns

PLACEHOLDER FOR:

​

5-Minute Timer

PLACEHOLDER FOR:

​

10-Minute Timer

PLACEHOLDER FOR:

​

15-Minute Timer