OUR DISCUSSION

Batch Normalization

INTRODUCTION

• Batch normalization (aka batch norm) is used to make the training of artificial neural networks faster and more stable.

​

• Proposed by Sergey Ioffe and Christian Szegedy in 2015.

​

• It mitigates the problem of internal covariate shift.

​

• A phenomenon where the distribution of the activations of a neural network layer changes during training.
• The parameters of the earlier layers are updated during the learning process, the subsequent layers receive inputs with varying distributions, which can slow down training and make it harder to optimize the network effectively.
• Batch Normalization, by normalizing the activations within each mini-batch during training, mitigates the effects of internal covariate shift, leading to faster and more stable convergence.

​

Internal Covariate Shift

It is standard practice to normalize data to zero mean and unit variance when inputting it to a deep learning model. This means that the mean of each feature is set to zero and the variance is set to one.

The rationale for normalizing the data is that it can help to improve the performance of the model. When the data is normalized, the features are all on the same scale, which can help the model to learn more effectively.

Normalization is typically done for each feature column separately. The mean and variance of each feature column are computed over the entire dataset.

Normalizing Input Data

Fig: How we normalize.

The image below shows the effect of normalizing data. The original values (in blue) are now centered around zero (in red). This ensures that all the feature values are now on the same scale.

​

The original values are scattered around a wide range of values. This can make it difficult for the model to learn effectively. However, after normalization, the values are all centered around zero and have a variance of one. This makes it easier for the model to learn and improve its performance.

Fig: What normalized data looks like.

Consider any of the hidden layers in a network. The activations from the previous layer are simply the inputs to this layer. For example, from the perspective of Layer 2 in the image below, if we "blank out" all the previous layers, the activations coming from Layer 1 are no different from the original inputs.

The same logic that requires us to normalize the input for the first layer also applies to each of these hidden layers.

Therefore, batch normalization can be used to normalize the activations of any layer in a neural network, not just the input layer.

In other words, if we are able to somehow normalize the activations from each previous layer then the gradient descent will converge better during training. This is precisely what the Batch Norm layer does for us.

The need for BatchNorm

Fig: The inputs of each hidden layer are the activations from the previous layer, and must also be normalized.

Just like the parameters (eg. weights, bias) of any network layer, a BatchNorm layer also has parameters of its own:

• Two learnable parameters are called beta and gamma.
• Two non-learnable parameters (Mean Moving Average and Variance Moving Average) are saved as part of the ‘state’ of the Batch Norm layer.

These parameters are per the Batch Norm layer.

​

Parameters and Working

Fig: Parameters of a Batch Norm layer.

If there are, say, three hidden layers and three Batch Norm layers in the network, we would have three learnable beta and gamma parameters for the three layers. Similarly for the Moving Average parameters.

Fig: Each Batch Norm layer has its own copy of the parameters.

During training, the network is feeded, one mini-batch of data at a time. During the forward pass, each layer of the network processes that mini-batch of data. The Batch Norm layer processes its data as follows:

Fig: Calculations performed by Batch Norm layer.

During training, BatchNorm starts by calculating the mean and variance for a mini-batch. However, during Inference, we have a single sample, not a mini-batch.

In such case, here is where the two Moving Average parameters come in — the ones that we calculated during training and saved with the model. We use those saved mean and variance values for the Batch Norm during Inference.

​

Calculating and saving the mean and variance of the full data is expensive, so the Moving Average is used as a proxy. It is more efficient because only the most recent Moving Average needs to be remembered.

Fig: Batch Norm calculation during Inference.

Order of Placement of BatchNorm layer

​

There are two opinions on where the Batch Norm layer should be placed in the architecture — before and after activation.

Fig: Batch Norm can be used before or after activation.

The batch_normalize() function first calculates the mean and variance of each feature in the batch of data. Then, it normalizes the data by subtracting the mean and dividing by the standard deviation, plus a small epsilon value to avoid dividing by zero. The normalized data is then returned.

​

The epsilon parameter is a small value that is added to the variance to avoid dividing by zero. This is important because the variance of a batch of data can sometimes be zero, especially if the batch is small. Adding a small epsilon value ensures that the denominator of the normalization equation is never zero.

​

Running the code will print the original data and the normalized data. The normalized data will be centered around zero and have a standard deviation of 1. This will help to stabilize the training of a deep neural network that uses the data array as input.

Examples

The nn.BatchNorm1d class in PyTorch implements batch normalization for 1D data. The affine=False argument specifies that the batch normalization layer should not have learnable parameters. This means that the mean and variance of the data will be fixed during training.

The torch.randn() function in PyTorch creates a random tensor with a normal distribution. The 22, 102 argument specifies that the tensor should have 22 samples and 102 features.

The n(inputval) line normalizes the input tensor using the batch normalization layer. The normalized output is then printed out.

The output of the batch normalization layer changes every time because the input tensor is random. The torch.randn() function in PyTorch creates a random tensor with a normal distribution. This means that the mean and variance of the input tensor will be different every time.

​

The first line imports the torch and nn modules from PyTorch.

The second line creates a BatchNorm2d layer with 120 features. The affine=False argument tells the layer to not learn any affine parameters.

The third line creates a 4D tensor with 20 batches, 120 channels, 55 rows, and 65 columns.

The fourth line passes the inputs tensor to the BatchNorm2d layer.

The fifth line prints the output of the BatchNorm2d layer.

Advantages

Real World

Applications

Limitations

Requires a large batch size.

This is because the moving average and variance are calculated over the entire batch. If the batch size is small, then the moving average and variance may not be accurate, which can lead to problems with the training process.

Can make the network less robust to changes in the input data.

This is because batch normalization normalizes the activations, which can make the network less sensitive to changes in the input data. This can be a problem if the input data is not normally distributed.

Can make the network more difficult to interpret.

This is because batch normalization adds a non-linear transformation to the network, which can make it difficult to understand how the network works.

Alternatives to BatchNorm

THANK YOU

FOR YOUR TIME