Conv. + Momentum + Adam +�Stochastic Gradient Descent

Lecture 12

The core algorithm of modern machine learning

Action Packed

Lecture

EECS 189/289, Fall 2025 @ UC Berkeley

Joseph E. Gonzalez and Narges Norouzi

EECS 189/289, Fall 2025 @ UC Berkeley

Joseph E. Gonzalez and Narges Norouzi

Join at slido.com�#3389863

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

3389863

The (Batch) Gradient Descent Alg.

3389863

What are the conditions for convergence?

Learning rate seemed to affect convergence even on a simple parabola.

3389863

Convergence Assuming�Quadratic Approximation

Quadratic Approximations

3389863

Taylor Expansion of the Error Surface

Matrix Equivalent of the Second Derivative

3389863

Hessians (the 2^nd derivative)

3389863

Hessian Exercise

Plots in demo notebook.

3389863

3389863

Taylor Expansion of the Error Surface

Use a change of variables to better understand this part.

3389863

Eigen Decomposition of the Hessian

3389863

Eigen Decomposition of the Hessian

Derivation:

Eigenvector Defn.

Orthonormality

3389863

Taylor Expansion at the Stationary Pts.

3389863

If the eigen values at w* are all positive then

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

3389863

Taylor Expansion at the Stationary Pts.

3389863

Stationary Points and the Hessian

Eigenvalues of the Hessian determine the curvature �at the critical points

3389863

The eigenvector u1 corresponds to the:

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

3389863

Eigenvectors of the Hessian

Elliptical contours of constant error align with the eigenvectors of the Hessian matrix.

Plots in demo notebook.

Direction with Smallest Eigenvalue

Direction with Largest Eigenvalue

3389863

Demo

Taylor Expansions and the Hessian

3389863

3389863

Recap: Quadratic Approx. Error

3389863

Grad. Descent on the Quadratic Approx.

3389863

Grad. Descent on the Quadratic Approx.

3389863

Learning Rate Impact on Convergence

3389863

Condition Number and Loss Surface

3389863

Issues with Gradient Descent (GD)

GD converges slowly when the error surface is relatively flat:

​

​

​

GD oscillates when the error surface is poorly conditioned

​

​

3389863

Demo

Oscillating convergence

3389863

Momentum

3389863

How Momentum Helps with Flat Dir.

3389863

How Momentum Helps with Oscillations

3389863

Demo

Momentum

3389863

Learning Rate Schedules

3389863

Demo

Learning Rate Schedules

3389863

Adaptive Learning Rates – AdaGrad

*Developed at UC Berkeley

Tracking Magnitude

(terms squared)

Scaled Learning Rate

Issue:

Large initial gradients result in overly small learning rates near convergence.

3389863

Adaptive Learning Rates – RMSProp

AdaGrad

RMSProp

3389863

Adam = RMSProp + Momentum

RMSProp

Momentum

Gradient

Update

3389863

Demo

Adam

3389863

Recap: Batch Gradient Descent

3389863

Stochastic Gradient Descent

Approximating Gradient Descent

The Cost of Batch Gradient Descent

3389863

What is the cost for a single gradient update assuming N data points in D dimensions and the basic gradient update equation.

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

3389863

The Cost of Batch Gradient Descent

3389863

The Training Error Estimates the Test Error

3389863

Approximating the Gradient

3389863

Stochastic Gradient Descent

3389863

Stochastic Gradient Descent (Shuffling)

In practice, we often shuffle and then iterate over the entire dataset

One epoch is a complete pass over all the data points.

Often only randomize data once.

3389863

Demo

Stochastic Gradient Descent

3389863

A Better Gradient Approximation

Estimating the gradient using a single data point results in noisy estimates.

?

Noisy but Cheap

Accurate but Expensive

Mini-Batch Stochastic�Gradient Descent

3389863

A Spectrum of Stochastic Gradients

We typically refer to any of the sampling based gradient estimation procedures as Stochastic Gradient Descent (SGD)

Noisy but Cheap

Accurate but Expensive

Stochastic �Gradient Descent

Batch �Gradient Descent

Mini-Batch Stochastic�Gradient Descent

Stochastic Gradient Methods

3389863

Mini-batch Stochastic Gradient Descent

3389863

Shuffling for Minibatch Gradient Descent

In practice, we often shuffle and then iterate over the entire dataset

One epoch is a complete pass over all the data points.

Often only randomize data once.

3389863

If I have N data points and a choose a batchsize B=2. In one epoch how many gradient steps will I take?

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

3389863

How do I Pick the Batch Size?

Larger batch size:

• Better gradient estimate
• Enables hardware parallelism – can compute gradient for each point in parallel

Smaller batch size:

• Cheap to compute 🡪 take more steps
• Greater stochasticity can help with local minima

Typical design decision:

• Increase batch size to saturate hardware parallelism
• Increase learning rate proportional to batch size (take bigger steps)

3389863

Demo

Mini-batch Gradient Descent

3389863

Stochastic Gradient Descent

Lecture 13

Credit: Joseph E. Gonzalez and Narges Norouzi

Reference Book Chapters: Chapter 7