Deep Learning (DEEP-0001)�

Prof. André E. Lazzaretti

lazzaretti@utfpr.edu.br

https://sites.google.com/site/andrelazzaretti/graduate-courses/deep-learning-cpgei/2025

9 – Performance

Measuring performance

• MNIST1D dataset model and performance
• Noise, bias, and variance
• Reducing variance
• Reducing bias & bias-variance trade-off
• Double descent
• Curse of dimensionality & weird properties of high dimensional space
• Choosing hyperparameters

​

MNIST Dataset

MNIST 1D Dataset

Network

• 40 inputs
• 10 outputs
• 4000 training examples (~400 training examples per class)
• Two hidden layers
• 100 hidden units each
• SGD with batch size 100, learning rate 0.1
• 6000 steps (150 Epochs)

​

Need to use separate test data

The model has not generalized well to the new data

Measuring performance

• MNIST1D dataset model and performance
• Noise, bias, and variance
• Reducing variance
• Reducing bias & bias-variance trade-off
• Double descent
• Curse of dimensionality & weird properties of high dimensional space
• Choosing hyperparameters

​

Regression example

Generated data: Sampling input values in the range [0, 1], passing them through this function, and adding Gaussian noise with a fixed variance.

Toy model

• K hidden units
• First layer fixed so “joints” divide the interval evenly
• Second layer trained

Noise, bias, and variance

• Noise in measurements
• Some variables not observed
• Data mislabeled

​

​

Data generation is noisy, so even if the model exactly replicates the true underlying function (black line), the noise in the test data (gray points) means that some error will remain (gray region represents two standard deviations).

Noise, bias, and variance

The three-region model (cyan line) cannot exactly fit the true function (black line).

Noise, bias, and variance

Depending on the training dataset, the results will vary slightly each time.

Noise, bias, and variance

• Variance is the uncertainty in fitted model due to choice of training set
• Bias is systematic deviation from the mean of the function we are modeling due to limitations in our model
• Noise is inherent uncertainty in the true mapping from input to output

Measuring performance

• MNIST1D dataset model and performance
• Noise, bias, and variance
• Reducing variance
• Reducing bias & bias-variance trade-off
• Double descent
• Curse of dimensionality & weird properties of high dimensional space
• Choosing hyperparameters

​

Variance

Can reduce variance by adding more samples

Measuring performance

• MNIST1D dataset model and performance
• Noise, bias, and variance
• Reducing variance
• Reducing bias & bias-variance trade-off
• Double descent
• Curse of dimensionality & weird properties of high dimensional space
• Choosing hyperparameters

​

Reducing bias

Increasing model capacity also increases the variance term (gray region). This is known as the bias-variance trade-off.

Why does variance increase? Overfitting

Describes the training data better, but not the true underlying function (black curve)

model with three regions

model with ten regions

Bias and variance trade-off

model capacity (number of hidden units / linear regions in range of data)

Measuring performance

• MNIST1D dataset model and performance
• Noise, bias, and variance
• Reducing variance
• Reducing bias & bias-variance trade-off
• Double descent
• Curse of dimensionality & weird properties of high dimensional space
• Choosing hyperparameters

​

Number of datapoints

MNIST-1D for a two-hidden layer network as we increase the number of hidden units

The training error decreases to zero as the number of parameters approaches the number of training examples (vertical dashed line).

​

The test error does not show the expected bias-variance trade-off but continues to decrease even after the model has memorized the dataset.

Double descent

noisier training data

• The training error reduces to zero, although it now takes almost as many parameters as training points to memorize the dataset.

​

• The test error shows the predicted bias/variance trade-off; it decreases as the capacity increases but then increases again as we near the point where the training data is exactly memorized.
• However, it subsequently decreases again and ultimately reaches a better performance level. This is known as double descent.

• Note that train data is very close to zero.
• Whatever is happening isn’t happening at training data points
• Must be happening between the data points??

​

Potential explanation:

• can make smoother functions with more hidden units
• being smooth between the datapoints is a reasonable thing to do

But why?

• All of these solutions are equivalent in terms of loss.
• Why should the model choose the smooth solution?
• Tendency of model to choose one solution over another is inductive bias

Measuring performance

• MNIST1D dataset model and performance
• Noise, bias, and variance
• Reducing variance
• Reducing bias & bias-variance trade-off
• Double descent
• Curse of dimensionality & weird properties of high dimensional space
• Choosing hyperparameters

​

Curse of dimensionality

• As dimensionality increases, the volume of space grows so fast that the amount of data needed to densely sample it increases exponentially. This phenomenon is known as the curse of dimensionality.

​

Measuring performance

• MNIST1D dataset model and performance
• Noise, bias, and variance
• Reducing variance
• Reducing bias & bias-variance trade-off
• Double descent
• Curse of dimensionality & weird properties of high dimensional space
• Choosing hyperparameters

​

Choosing hyperparameters

• Don’t know bias or variance
• Don’t know how much capacity to add
• How do we choose capacity in practice?
• Or model structure
• Or training algorithm
• Or learning rate
• Third data set – validation set
• Train models with different hyperparameters on training set
• Choose best hyperparameters with validation set
• Test once with test set

​

https://www.youtube.com/watch?v=OTc2Q17GK4c

K-Fold Cross-Validation

Grid Search