Bayesian neural networks

Philip Smolenski-Jensen�Working with Marek Cygan, Piotr Tempczyk, Ksawery Smoczyński and Rafał Michaluk

Bayesian neural networks vs standard neural networks

Components of BNN

• Functional model (neural net)
• Initial distributions (priors)
• Observed variables
• Latent variables
• Parameters

Math behind BNN’s

• Priors:

​

• Bayes theorem:

​

• Prediction:

Predictions in Bayesian Neural Nets

• In practice, during prediction, we can sample weights from posterior distribution and run neural net with sampled weights.
• When we sample multiple times (independently) the outputs form distribution that approximates p(y).

Advantages of BNN’s

• BNN’s tend to perform better when estimating classification uncertainty
• BNN’s can provide confidence intervals for regression tasks.
• They allow to distinguish between epistemic uncertainty (of the model) and aleatoric uncertainty (of the data)
• Priors work as regularization (so BNN’s do not overfit)
• Can choose the most valuable datapoint for future learning (useful when labels are expensive to get)
• Widely used in economy, medicine etc.

Finding posterior distribution

​

• The posterior distribution is high-dimensional (as there are many weights) and non-convex, which makes computing the denominator of RHS difficult
• We must be able to sample from posterior distribution to make predictions
• This is intractable using standard methods.
• Two most popular methods to approximate posterior are Monte Carlo Markov Chains (MCMC) and Variational Inference (VI). We will focus on the latter.

Variational Inference: general idea

• Establish a family of distributions that are easy to deal with (e.g. some subset of parameterized distributions) and try to compute member (called variational distribution) q such that KL(q(z) || p(z|x,y)) is minimal.

VI intro: KL divergence

• Kullback-Leibler divergence is a measure between probability distributions given by formula:

​

​

• It originates from information theory, it represents the expected number of additional bits to encode sample from p given code optimized for q.
• Intuitively, the smaller the KL divergence the “closer” the distributions.

Variational inference: ELBO

We have:

​

​

​

after substitution we get:

Variational Inference: ELBO

​

​

• The first term on the RHS is called Evidence Lower Bound (ELBO). If we optimize ELBO by changing q, the LHS will be constant (as it’s independent from q) hence the KL-divergence will be minimized.
• For some sufficiently simple families of distribution it is possible to maximize ELBO directly but we will focus on stochastic methods

Stochastic Variational Inference

Elbo has the form of:

We can compute the gradient of E[f(z)] in general case:

​

​

​

The estimator of ELBO gradient tends to have high variance.

​

Alternative: Reparameterization trick

Suppose:

Then we can write:

​

​

Where

This allows us to produce more stable ELBO gradient estimator. Such a function g exists for multiple distributions. This is called the reparameterization trick.

What about non-Bayesian parameters?

• The model might contain deterministic parameters
• We would like to optimize those parameters as well.

​

​

• If we maximize ELBO wrt. to theta, we maximize p(y|x) as well.
• We converge to theta such that the likelihood of y given x is maximal which is analogous to training standard neural networks.

Variational Inverence: Full algorithm

What families of variational distributions can one use?

​

• In case of Bayesian Neural Networks the weights’ distribution is often parametrized as normal distribution.
• In general case one can try to capture the dependance between weights, this requires to learn the whole covariance matrix (with O(n^2) parameters)
• To speed up training and reduce memory usage one can assume that the weights are independent.
• We end up with modeling weights as independent, normally distributed variables, this may make the variational family not expressive enough.

Motivation behind the project

• One can try to modify the neural network in such a way that it’s Bayesian counterpart is better at uncertainty modeling.
• For instance, experiments conducted on standard neural networks show log likelihood distribution is closer to normal distribution when Leaky ReLU is used as activation function (compared to ReLU).
• This suggests that the posterior distributions might be closer to normal in Bayesian nets with LeakyRelu.
• That in turn might lead to better accuracy and calibration.

Log Likelihood distribution for ReLU vs Leaky ReLU

Expected Calibration Error

• The interval [0,1] is divided into K bins B1, B2, …, Bk
• For fixed i acc_i and conf_i denote the accuracy and average confidence for datapoints in bin Bi.

Experiments

• Datasets: MNIST, FashionMNIST
• Models:
• MLEClassify (Fully connected neural network with 1 hidden layer)
• DeepMLECLassify (Fully connected neural network with 3 hidden layers)
• ConvClassify (2 convolutional layers with 128 and 256 channels respectively + fully connected layer)
• Activations:
• LeakyReLU with negative slope from -1 to 1 in 0.1 intervals
• SiLU, SoftPlus, ELU.

Sample LeakyReLU results: MLEClassify, FashionMNIST

Sample LeakyReLU results: ConvClassify, MNIST

​

​

Some observations:

• There is often a spike around slope 0 in ECE.
• For slope values in range [−1, 0] the accuracy tends to be non-increasing (and oftentimes decreasing) function of slope and the ECE is non-decreasing (and mostly increasing) function of slope.
• For slope values in range [0, 1] the accuracy is decreasing with slope → 1 as the functional model converges to linear model that is less expressive.
• I’ve picked slope -1 (corresponding to absolute value function) for further comparisons.

Sample Results: DeepMLEClassify, MNIST

Sample Results: ConvClassify, FashionMNIST

Full Results

Decalibration with ReLU

No decalibration with LeakyReLU

​

Conclusions

​

• LeakyRelu leads to better calibration of Bayesian Neural Networks with similar (or even slightly better accuracy).
• The results are consistent across datasets and models.
• This might be caused by the fact that likelihood distribution is flatter for ReLU which can lead to decalibration if the flat part is close to mode.

​

Next steps

• Run experiments on bigger architectures and datasets (ResNet, CIFAR10).
• Verify if similar improvements occur in regression tasks (in particular, try computer vision tasks like depth estimation etc.)
• Try to modify the model so it learns the best slope of LeakyReLU during training.

​

Thank you for the attention!