Uncertainty modeling from 50M to 1B

Dustin Tran

A pervasive problem in the field

• Most papers fix an architecture and try to improve log-likelihood.

​

• Is it reasonable to compare 100 MCMC samples to the original model?�Is it reasonable to compare an ensemble of size 10 to the original model?

​

• Why are we ignoring the original model scaled up 10-100x?

All that matters is p(y | x).

The uncertainty-robustness frontier

Quality of uncertainty & robustness

Image source: NeurIPS 2020 tutorial on Uncertainty & Robustness

# Parameters

0.1M

1M

10M

100M

10B

1B

Ensembles as a Giant Model

​

• Paths between subnetworks are independent ⇒ SGD-trained models have independent predictions by construction.
• Bridge the gap from single model to ensembles by sharing parameters, learning how to decorrelate predictions during training.

Image source: NeurIPS 2020 tutorial on Uncertainty & Robustness

Efficient Ensembles by Sharing Parameters

Parameterize each weight matrix as a new weight matrix W multiplied by the outer product of two vectors r and s.��

There is an independent set of r and s vectors for each ensemble member; W is shared.

Known as BatchEnsemble.

[Wen+ 2020]

Efficient Ensembles by Sharing Parameters

BatchEnsemble has a convenient vectorization.�

Duplicate each example in a given mini-batch K times.

​

The model yields K outputs for each example.

Can interpret rank-1 weight perturbations as feature-wise transformations.

[Wen+ 2020]

The uncertainty-robustness frontier

Quality of uncertainty & robustness

# Parameters

0.1M

1M

10M

100M

10B

1B

Efficient ensembles

[BatchEnsemble]

Architecture�[ResNet, ViT]

The value of Bayes is the prior distribution.��Rank-1 BNNs place priors over BatchEnsemble’s rank-1 weights p(r), p(s).

Rank-1 priors induce distribution over all weights.

Can we improve ensembles with Bayes?

[Dusenberry+ 2020]

��Rank-1 BNNs use mixture posterior distributions to combine multimodal representations with distributional uncertainty.

Can we improve ensembles with Bayes?

[Dusenberry+ 2020]

The uncertainty-robustness frontier

Quality of uncertainty & robustness

Priors�[Rank-1 BNNs]

# Parameters

0.1M

1M

10M

100M

10B

1B

Efficient ensembles

[BatchEnsemble]

Architecture�[ResNet, ViT]

Does data augmentation work? Well..

[Wen+ 2021]

BatchEnsemble

MC Dropout

Deep Ensemble

BatchEnsemble

MC Dropout

Deep Ensemble

BatchEnsemble

MC Dropout

Deep Ensemble

BatchEnsemble

MC Dropout

Deep Ensemble

Does data augmentation work? Well..

• Ensembles reduce overconfidence.
• Mixup makes model underconfident.

​

⇒ Ensemble + Mixup are even more unconfident!

[Wen+ 2021]

Data augmentation conflates model and data uncertainty.

[Wen+ 2021]

ResNet-50 on ImageNet

With adjustment, ensembles + DA reach state-of-the-art calibration.

The uncertainty-robustness frontier

Quality of uncertainty & robustness

# Parameters

0.1M

1M

10M

100M

10B

1B

Priors & Invariances�[Rank-1 BNNs, Mixup]

Efficient ensembles

[BatchEnsemble]

The uncertainty-robustness frontier

Quality of uncertainty & robustness

?

# Parameters

0.1M

1M

10M

100M

10B

1B

}

What should we expect according to the literature?

• Gains in accuracy and OOD generalization [Djolonga+ 2021]
• But... “modern neural networks … �are poorly calibrated.”
• “...increasing depth and width … negatively affect model calibration.”
• (Guo et al. 2017; 1500 citations)
• Many similar reports.

Experiment study

• Models.
• Weak supervision (BiT, ResNext WSL) with non-convolutional architectures (MLP-Mixer, ViT).
• Semi-supervised (SimCLR).
• Natural language (CLIP).
• Classic (AlexNet).

​

• Datasets. ImageNet, ImageNetV2, ImageNet-C, ImageNet-R, ImageNet-A.

​

​

[Minderer+ 2021]

1: Recent architectures are not miscalibrated.

[Minderer+ 2021]

1: Recent architectures are not miscalibrated.

[Minderer+ 2021]

2: Larger models deteriorate in-dist. but improve on OOD.

[Minderer+ 2021]

2: Larger models deteriorate in-dist. but improve on OOD.

[Minderer+ 2021]

2: Larger models deteriorate in-dist. but improve on OOD.

• Accuracy and calibration deteriorate with distribution shift.
• Larger models deteriorate less.

[Minderer+ 2021]

3: Accuracy predicts calibration.

Fit power law:

Takeaways

We need to trace the uncertainty-robustness frontier: understand progress relative to scale.�

• Efficient ensembles are a dominant strategy for uncertainty & robustness performance.

​

• Strategies with Bayes and data augmentation further improve the frontier.

​

• At the giant model regime: �“Modern neural networks tend to make overconfident predictions.” ← Not true!

Thank you!

Rafael Müller

​

​

​

Matthias Minderer

Ghassen Jerfel

​

​

​

​

​

Josip Djolonga

Mike Dusenberry

Rob Romijnders

Jasper Snoek

Balaji Lakshminarayanan

Frances Hubis

Xiaohua Zhai

Neil Houlsby

Mario Lucic

Katherine Heller

Yian Ma

Jimmy Ba