Deep Learning (DEEP-0001)�

Prof. André E. Lazzaretti

lazzaretti@utfpr.edu.br

https://sites.google.com/site/andrelazzaretti/graduate-courses/deep-learning-cpgei/2025

4 – Deep Neural Networks

Deep neural networks

• Networks with more than one hidden layer
• Intuition becomes more difficult

Deep neural networks

• Composing two networks
• Combining the two networks into one
• Hyperparameters
• Notation change and general case
• Shallow vs. deep networks

​

Composing two networks.

Network 1:

Network 2:

Composing two networks.

Network 1:

Network 2:

https://udlbook.github.io/udlfigures/

Comparing to shallow with six hidden units

• 20 parameters
• 9 regions

• 19 parameters
• Max 7 regions

Composing networks in 2D

Deep neural networks

• Composing two networks
• Combining the two networks into one
• Hyperparameters
• Notation change and general case
• Shallow vs. deep networks

​

Combine two networks into one

Network 1:

Network 2:

Hidden units of second network in terms of first:

Create new variables

Two-layer network

1

1

Two-layer network as one equation

https://udlbook.github.io/udlfigures/

Deep neural networks

• Composing two networks
• Combining the two networks into one
• Hyperparameters
• Notation change and general case
• Shallow vs. deep networks

​

Hyperparameters

Deep neural networks

• Composing two networks
• Combining the two networks into one
• Hyperparameters
• Notation change and general case
• Shallow vs. deep networks

​

Notation change #1

Notation change #1

Notation change #1

Notation change #1

Notation change #2

Notation change #3

Notation change #3

Bias vector

Weight matrix

General equations for deep network

Example

Deep neural networks

• Composing two networks
• Combining the two networks into one
• Hyperparameters
• Notation change and general case
• Shallow vs. deep networks

​

Shallow vs. deep networks

The best results are created by deep networks with many layers.

• 50-1000 layers for most applications
• Best results in
• Computer vision
• Natural language processing
• Graph neural networks
• Generative models
• Reinforcement learning

​

All use deep networks. But why?

Shallow vs. deep networks

1. Ability to approximate different functions?

​

Both obey the universal approximation theorem.

​

Argument: One layer is enough.

Shallow vs. deep networks

2. Number of linear regions per parameter

​

Number of linear regions per parameter

5 layers

10 hidden units per layer

471 parameters

161,501 linear regions

Shallow vs. deep networks

2. Number of linear regions per parameter

​

• Deep networks create many more regions per parameter;
• But may contain complex dependencies and symmetries.

Shallow vs. Deep Networks

3. Depth efficiency

​

• There are some functions that require a shallow network with exponentially more hidden units than a deep network to achieve an equivalent approximation

​

• This is known as the depth efficiency of deep networks

​

• But do the real-world functions we want to approximate have this property? Unknown.

Shallow vs. Deep Networks

4. Large structured networks

​

• Think about images as input – might be 1M pixels
• Fully connected works not practical
• Answer is to have weights that only operate locally, and share across image
• This leads to convolutional networks
• Gradually integrate information from across the image – needs multiple layers

Shallow vs. Deep Networks

5. Fitting and generalization

​

• Fitting of deep models seems to be easier up to about 20 layers
• Then needs various tricks to train deeper networks, so (in vanilla form), fitting becomes harder
• Generalization is good in deep networks.

​

Shallow vs. Deep Networks

5. Fitting and generalization

​

• Fitting of deep models is also faster

​

​

Where are we going?

• We have defined families of very flexible networks that map multiple inputs to multiple outputs
• Now we need to train them
• How to choose loss functions
• How to find minima of the loss function
• How to do this in particular for deep networks
• Then we need to test them