1 of 45

Join at slido.com�#2095864

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

2095864

2 of 45

Neural Networks (1)

Lecture 12

Non-linearity, Architecture, Activation Functions, Output Layers, and Loss

EECS 189/289, Fall 2025 @ UC Berkeley

Joseph E. Gonzalez and Narges Norouzi

EECS 189/289, Fall 2025 @ UC Berkeley

Joseph E. Gonzalez and Narges Norouzi

3 of 45

Recap: Linear Models

2095864

4 of 45

Logistic Regression on XOR

2095864

5 of 45

Roadmap

The Neuron
Architecture
Activation Functions
Output Layers and Loss
PyTorch (Optional)

2095864

6 of 45

The Neuron

The Neuron
Architecture
Activation Functions
Output Layers and Loss
PyTorch (Optional)

2095864

7 of 45

From Biological to Artificial Neurons

Biological neuron contains

Inputs
Summation of inputs
Processing unit
Output

Artificial neuron

+

Linear Combination �of Inputs

Processing unit

2095864

8 of 45

Neuron – Activation Function

An artificial neuron is a summation unit + an activation function.
The weighted summation is passed to the activation function to calculate the neuron's activation.

Bias Term

Weights

Output

Linear Combination �of Inputs

Activation

Function

2095864

9 of 45

Perceptron

Assume the activation function to be a step function.

If the weighted combination of inputs is negative, the output is 0 → neuron is not activated.
If the weighted combination of inputs is positive, the output is 1 → neuron is activated.

2095864

10 of 45

Functional Completeness - AND


1	0	0
1	1	1
0	0	0
0	1	0

1

2095864

11 of 45

What are example parameters for the AND gate?�w_0=?, w_1=?, w_2=?

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

2095864

12 of 45

Functional Completeness - AND


1	0	0
1	1	1
0	0	0
0	1	0

1

2095864

13 of 45

Functional Completeness - XOR


1	0	1
1	1	0
0	0	0
0	1	1

1

2095864

14 of 45

Can we build the XOR gate with only one neuron?

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

2095864

15 of 45

Architecture

The Neuron
Architecture
Activation Functions
Output Layers and Loss
PyTorch (Optional)

2095864

16 of 45

XOR with Two Layers


1	0	1
1	1	0
0	0	0
0	1	1

1

Non-linear Decision Boundary

(No linear separability)


1	0	1
1	1	0
0	0	0
0	1	0


1	0	0
1	1	0
0	0	0
0	1	1

1

OR

1

2095864

17 of 45

Hidden Units Are New Features

2095864

18 of 45

Neural Network

Weights

Bias Term

Pre-activation or logits

Activation Function

Output of basis functions

Hidden layer

Input layer

2095864

19 of 45

Now, let's add output layer.

Hidden layer

Input layer

Output layer

2095864

20 of 45

Deep Neural Network

Input layer

Output layer

Hidden layer 1

Hidden layer 2

Hidden layer 3

2095864

21 of 45

Count the Parameters

784

input

128

hidden

64

hidden

10

output

How many weights and biases?

2095864

22 of 45

Count the Parameters

784

input

128

hidden

64

hidden

10

output

100,480

8,256

650

2095864

23 of 45

Universality Theorem

Visual proof: http://neuralnetworksanddeeplearning.com/chap4.html

Can be realized by a network with one hidden layer

(given enough hidden neurons)

Why “Deep” neural network not “Fat” neural network?

Neural networks are said to be universal approximators.

2095864

24 of 45

Why Deep? Analogy

Logic circuits consists of gates.
Two layers of logic gates can represent any Boolean function.
Using multiple layers of logic gates to build some functions are much simpler.

Neural network consists of neurons.
A hidden layer network can represent any continuous function.
Using multiple layers of neurons to represent some functions are much simpler.

less gates needed

Logic circuits

Neural network

less parameters

less data?

2095864

25 of 45

Activation Functions

The Neuron
Architecture
Activation Functions
Output Layers and Loss
PyTorch (Optional)

2095864

26 of 45

Depth Alone Does Not Help

2095864

27 of 45

Why Linear Layers Collapse

2095864

28 of 45

Depth Alone Doesn't Help

Demo in the code

2095864

29 of 45

Activation Functions

The simplest option for a hidden unit activation function is the identity function.

But it is redundant to have a hidden layer if all activations are identity.

Simple non-linear differentiable activation function is sigmoid.

2095864

30 of 45

Other Activation Functions

We need an activation that has non-zero gradient at least when activations are large positive values.

2095864

31 of 45

ReLU and Its Variants

Neurons with negative activations will receive "no error signal".

A small slope for negative inputs keeps the gradient nonzero.

Also called soft ReLU.

A smooth ReLU, used in transformers. It dips slightly below 0 for negative inputs.

2095864

32 of 45

Training a ReLU Network on XOR

2095864

33 of 45

Where Gradients Vanish

2095864

34 of 45

In a ReLU network, one hidden unit's pre-activation a is negative for every training point. What happens to that unit's incoming weights during gradient descent?

The Slido app must be installed on every computer you’re presenting from

Do not edit�How to change the design

2095864

35 of 45

A Dead ReLU Unit

Unit 0 has a negative pre-activation on all 400 training points.

2095864

36 of 45

Output Layers and Loss

The Neuron
Architecture
Activation Functions
Output Layers and Loss
PyTorch (Optional)

2095864

37 of 45

The Output Layer Is a GLM

From linear regression lectures

Neural network

2095864

38 of 45

Match the Output Layer to the Target

Target	Output activation	Likelihood	Loss	PyTorch loss
a real number	identity	Gaussian	squared error	nn.MSELoss
one of 2 classes	sigmoid	Bernoulli	binary cross-entropy	nn.BCEWithLogitsLoss
one of K classes	softmax	categorical	cross-entropy	nn.CrossEntropyLoss

2095864

39 of 45

Multi-Label: One Sigmoid per Label

One of K classes

Any subset of K labels

a digit from 0 to 9

every animal in a photo

2095864

40 of 45

Building It in PyTorch

The Neuron
Architecture
Activation Functions
Output Layers and Loss
PyTorch

2095864

41 of 45

Big Concepts in PyTorch

torch.tensor is the basic datatype used to encode data, weights, and activations and is very similar numpy.ndarrays.
Neural networks are built using the many primitives in the torch.nn module and defining the forward process of making a prediction.
PyTorch uses automatic differentiation to efficiently compute the gradient (Lecture 13).
Training is accomplished by (you) implementing some variation of gradient descent.

2095864

42 of 45

torch.nn.Module

When building a new model you extend the nn.Module

class MLP(nn.Module):

def __init__(self, dims, activation="relu"):

super().__init__()

self.layers = nn.ModuleList(

[nn.Linear(dims[i], dims[i+1]) for i in range(len(dims)-1)])

if activation == "relu": self.act = nn.functional.relu

else: self.act = torch.tanh�

def forward(self, x):

for i, layer in enumerate(self.layers):

x = layer(x)

if i < len(self.layers)-1:

x = self.act(x)

return x

Extend nn.Module

Add modules to your module. These add parameters.

Compute the forward pass using the layer modules.

2095864

43 of 45

Implement Gradient Descent with Optimizer

In PyTorch you implement your own gradient descent loop but you can use built in update rules (e.g., Adam) called optimizers.

loader = DataLoader(tr_data, batch_size=32, shuffle=True)

optimizer = torch.optim.Adam(model.parameters(),eta)

for epoch in range(nepochs):

for (x, t) in loader:

optimizer.zero_grad()

loss = loss_fn(model(x), t)

loss.backward()

optimizer.step()

Minibatch SGD DataLoader

Adam optimizer

Epoch is one pass over data

Load one minibatch at a time

Compute the gradient of the loss

Use optimizer to update all the parameters (1-step)

2095864

44 of 45

Demo

Width, depth, and activation

2095864

45 of 45

Neural Networks (1)

Lecture 12

Credit: Joseph E. Gonzalez and Narges Norouzi

Reference: Bishop & Bishop, Deep Learning: Foundations and Concepts, Ch. 6, Sections 6.1, 6.2.1–6.2.3, 6.3.1, 6.4