1 of 52

PyTorch Continued &�Backpropagation

Lecture 15

Backpropagation – computing gradients easily and efficiently!

EECS 189/289, Fall 2025 @ UC Berkeley

Joseph E. Gonzalez and Narges Norouzi

EECS 189/289, Fall 2025 @ UC Berkeley

Joseph E. Gonzalez and Narges Norouzi

2 of 52

Join at slido.com�#3940975

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

3940975

3 of 52

Implement Gradient Descent with Optimizer

In PyTorch you implement your own gradient descent loop but you can use built in update rules (e.g., Adam) called optimizers.

loader = DataLoader(tr_data, batch_size=32, shuffle=True)

optimizer = torch.optim.Adam(model.parameters(),eta)

​

for epoch in range(nepochs):

for (x, t) in loader:

optimizer.zero_grad()

loss = loss_fn(model(x), t)

loss.backward()

optimizer.step()

Minibatch SGD DataLoader

Adam optimizer

Epoch is one pass over data

Load one minibatch at a time

Compute the gradient of the loss

Use optimizer to update all the parameters (1-step)

3940975

4 of 52

Demo

3940975

5 of 52

Recap: Stochastic Gradient Descent

  •  

 

3940975

6 of 52

Simple Running Example

Consider the following single hidden layer classification network

 

 

 

 

 

 

 

 

 

 

Inputs

Hidden�Layer

Output

 

Matrix Notation

 

 

 

2

3

3

1

1

No bias at each layer in this example.

3940975

7 of 52

Defining the Error Function

  •  

 

 

 

 

 

 

 

 

Inputs

Hidden�Layer

Output

 

 

Likelihood of the data.

Algebra!

3940975

8 of 52

Computing the Gradient

  •  

 

 

 

 

 

 

 

 

Inputs

Hidden�Layer

Output

 

 

 

2

3

 

3

1

3940975

9 of 52

What is the shape of the gradient?

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

3940975

10 of 52

Computing the Gradient

  •  

 

 

 

 

 

 

 

 

Inputs

Hidden�Layer

Output

 

 

 

2

3

 

3

1

3940975

11 of 52

Computing the Gradient

  •  

 

 

 

 

 

 

 

 

Inputs

Hidden�Layer

Output

 

 

 

 

 

3940975

12 of 52

Numerical Differentiation

13 of 52

Proposal: Numerical Differentiation

  •  

3940975

14 of 52

Proposal: Numerical Differentiation

  •  

3940975

15 of 52

Cost of Numerical Differentiation

  •  

3940975

16 of 52

Symbolic Differentiation

17 of 52

Proposal: Symbolic Differentiation

  •  

3940975

18 of 52

Symbolic Computer Algebra Systems

Computer algebra systems like SymPy can be used to automate the differentiation process.

  • Express functions �symbolically
  • Automatically �construct �derivative �expressions

​

3940975

19 of 52

Demo

SymPy

3940975

20 of 52

Symbolic Computer Algebra Systems

Computer algebra systems like SymPy can be used to automate the differentiation process.

  • Express functions symbolically
  • Automatically construct derivative expressions

Advantages:

  • Automatic: algebraic systems can be applied to “any” expression
    • Some issues with error functions that contain control flow/loops.
  • Exact: Doesn’t require epsilon approximation

Disadvantages:

  • Inefficient: algebraic derivative expressions could be significantly larger and contain many repeated expressions:
    • Why?

3940975

21 of 52

Expression Swell

  •  

 

3940975

22 of 52

Automatic Differentiation

23 of 52

Automatic Differentiation (Autodiff)

  •  

3940975

24 of 52

Forward Tracing the Computation Graph

We construct the graph by evaluating the forward computation:

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Program:

0: v1 = x1

1: v2 = x2

2: v3 = v1 * v2

3: v4 = exp(v3)

4: v5 = sin(v2)

5: v6 = v4 + v3

6: v7 = v6 – v5

In this program we have broken each expression into a separate line.

3940975

25 of 52

 

  •  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Compute this using the �same process

 

 

 

3940975

26 of 52

 

  •  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Compute these using the �same process

 

3940975

27 of 52

 

  •  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Base Case!

3940975

28 of 52

Backward Propagation (Recursion)

  •  

Recursion:

 

 

 

 

Parents

 

 

 

Children

 

 

 

 

3940975

29 of 52

Backward Autodiff Example

  •  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3940975

3940975

30 of 52

 

  •  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3940975

3940975

31 of 52

 

  •  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3940975

3940975

32 of 52

 

  •  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3940975

3940975

33 of 52

 

  •  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3940975

3940975

34 of 52

What is the value of the adjoint variable a_3.

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

3940975

35 of 52

 

  •  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3940975

3940975

36 of 52

 

  •  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3940975

37 of 52

 

  •  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3940975

3940975

38 of 52

Obtaining The Gradient!

  •  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

  1. The final input adjoint variables are the gradient.

​

3940975

39 of 52

Cost Analysis

Backpropagation is efficient when there is a single output and many inputs.

​

​

​

​

​

​

​

​

​

Need to run backpropagation repeatedly for multiple outputs.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Often reuse computation from the forward step.

  • Need to cache or recompute intermediate values

Only need to maintain single adjoint variable at each step.

3940975

40 of 52

Demo

You will do this in your homework.

  • But my example code (and my example bug) may be helpful.

​

3940975

A

B

D

E

F

G

C

Hint:

The adjoint at B can’t be computed until the adjoint at D is computed (topological ordering) but does not depend on the adjoint at C (an inactive path).

3940975

41 of 52

Additional Reading

3940975

42 of 52

PyTorch Continued &�Backpropagation

Lecture 15

Reading: Chapter 8 in Bishop Deep Learning Textbook

43 of 52

Forward Differentiation

Bonus Material

44 of 52

Chain Rule on The Computation Graph

  •  

 

 

 

 

 

 

 

 

 

 

 

3940975

45 of 52

Chain Rule on The Computation Graph

  •  

 

 

 

 

 

 

 

 

 

3940975

46 of 52

Forward Autodiff Example

 

 

 

 

 

 

 

 

 

Derivation

 

 

 

 

 

 

 

 

 

 

 

 

3940975

47 of 52

Forward Autodiff Example

 

 

 

 

 

 

 

 

 

Derivation

 

 

 

 

 

 

 

 

 

 

 

 

 

3940975

48 of 52

Forward Autodiff Example

 

 

 

 

 

 

 

 

 

Derivation

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3940975

49 of 52

Forward Autodiff Example

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3940975

50 of 52

Support multiple outputs

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Additional function output can reuse computation.

Essentially, we computed the gradient at every step!

?

3940975

51 of 52

Expensive for Lots of Variables

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Need to maintain full gradient at every intermediate step.

For models with billions of parameters, need to store and update billion-dimension vector for every arithmetic operation.

3940975

52 of 52

Adjoint vs Tangent Variables

We have now defined tangent and adjoint variables for forward and backward autodiff respectively:

​

​

​

Notice how they build from one side of the graph to the other:

​

​

​

​

 

Tangent Variables

 

Adjoint Variables

 

 

 

 

Parents

 

 

 

Children

 

3940975