Reconhecimento de Padrões (RPD-0041)�

Prof. André E. Lazzaretti

lazzaretti@utfpr.edu.br

https://sites.google.com/site/andrelazzaretti/graduate-courses/reconhecimento-de-padrões-cpgei/cronograma-2025

Lecture 10 - Deep Feedforward Neural Networks and Backpropagation

Introduction

• Feedforward: information flows from the input, through the intermediate computations used to define the function, and finally to the output.

​

Introduction

• Feedforward: information flows from the input, through the intermediate computations used to define the function, and finally to the output.
• Networks: layers of functions f(x) =f⁽³⁾(f⁽²⁾(f⁽¹⁾(x)))

​

Introduction

• Feedforward: information flows from the input, through the intermediate computations used to define the function, and finally to the output.
• Networks: layers of functions f(x) =f⁽³⁾(f⁽²⁾(f⁽¹⁾(x)))
• Depth of the model: deep learning arose from this terminology.

​

Introduction

• Feedforward: information flows from the input, through the intermediate computations used to define the function, and finally to the output.
• Networks: layers of functions f(x) =f⁽³⁾(f⁽²⁾(f⁽¹⁾(x)))
• Depth of the model: deep learning arose from this terminology.
• Neural because they are loosely inspired by neuroscience.
• Each unit resembles a neuron - it receives input from many other units and computes its own activation value.

Introduction

• Feedforward: information flows from the input, through the intermediate computations used to define the function, and finally to the output.
• Networks: layers of functions f(x) =f⁽³⁾(f⁽²⁾(f⁽¹⁾(x)))
• Depth of the model: deep learning arose from this terminology.
• Neural because they are loosely inspired by neuroscience.
• Each unit resembles a neuron - it receives input from many other units and computes its own activation value.
• Intrinsically nonlinear.

Basic Structure

Learning XOR

Learning XOR

Learning XOR

1

1

1.1 + 1.1 + 1.(-1.5) = 0.5

1.1 + 1.1 + 1.(-0.5) = 1

1

1

1.(-2) + 1.1 + 1.(-0.5) = -1.5

0

Learning XOR

How to train?

• Designing and training a neural network is not much different from training any other machine learning model.
• Steps:
• Cost Function;
• Architecture: output units, hidden units;
• Training (backpropagation).

Loss Functions: Classification

Multiclass (cross-entropy):

Binary (logistic):

For regression: https://keras.io/api/losses/regression_losses/

Architecture

Hidden Units

Hidden Units

• Rectified Linear Units (ReLU):
• Variations:
• Leaky ReLU (fixes α_i to a small value, e.g. 0.01).
• Parametric ReLU, or PReLU, treats α_i as a learnable parameter.

https://www.youtube.com/watch?v=Xvg00QnyaIY&ab_channel=DeepLearningAI

Output Units

• Linear Units:

​

• Sigmoid Units:

​

• Softmax Units:

Binary!

Multiclass!

Regression!

Effect of depth

L² Parameter Regularization

​

Smooth

Abrupt

Small weights

Large weights

Early Stopping

Dropout

• It consists of 'turning off' random neurons from the network at each iteration, based on a user-defined probability (only during training).
• Force all neurons to contribute in the final result.
• Prevents coadaptation (better when hidden units can detect features independently of each other, i.e., uncorrelated) of neurons.

Dropout

• Generally used on fully connected layers.
• Typical values of dropout rate, or probability of turning off neurons:
• 0.5 for fully connected layers.
• 0.2 for convolutional layers.

How to train? Gradient-Based Learning

​

Backpropagation

• Gradient-based:

​

Backpropagation

• Gradient-based:

​

• Two alternating steps:
• Forward propagation

​

σ(x₄w₄)

σ(x₁w₁)

σ(x₅w₅)

σ(x₆w₆)

σ(x₂w₂)

σ(x₃w₃)

ℇ

Backpropagation

• Gradient-based:

​

• Two alternating steps:
• Forward propagation
• Backward prograpagation

​

∂ℇ/∂w₄

∂ℇ/∂w₁

∂ℇ/∂w₅

∂ℇ/∂w₆

∂ℇ/∂w₂

∂ℇ/∂w₃

ℇ

σ(x₄w₄)

σ(x₁w₁)

σ(x₅w₅)

σ(x₆w₆)

σ(x₂w₂)

σ(x₃w₃)

ℇ

Backpropagation

• Gradient-based:

​

• Two alternating steps:
• Forward propagation
• Backward prograpagation

​

∂ℇ/∂w₄

∂ℇ/∂w₁

∂ℇ/∂w₅

∂ℇ/∂w₆

∂ℇ/∂w₂

∂ℇ/∂w₃

ℇ

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

Backpropagation

• Vectorized Implementation:
• This link in [37:35]. This link may also help.

​

• Example in Python: link.