Multimodal and Contrastive Learning

AIDA Technical workshop

Erik Ylipää

Outline

• 13:00 – 13:30 Representations and the softmax
• 13:30 – 14:00 Notebook 1, part 1
• 14:00 – 14:15 Break
• 14:15 – 14:45 Notebook 1, part 2
• 14:45 – 15:10 Contrastive learning
• 15:10 – 15:45 Notebook 2

Has Deep Learning solved Computer Vision?

3

Betteridge's law of headlines: "Any headline that ends in a question mark can be answered by the word no."

Computer vision using Deep Learning

4

Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. "Explaining and harnessing adversarial examples." arXiv preprint arXiv:1412.6572 (2014).

https://www.eff.org/files/AI-progress-metrics.html

Note, “human performance” has N=1: Andrej Karpathy

http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/

Excellent generalization to a test set

Brittle in surprising ways (adversarial examples)

What features are used?

The current incarnation of deep neural networks exhibit a tendency to learn surface statistical regularities as opposed to higher level abstractions in the dataset. For tasks such as object recognition, due to the strong statistical properties of natural images, these superficial cues that the deep neural network have learned are sufficient for high performance generalization, but in a narrow distributional sense.

​

Jo, Jason, and Yoshua Bengio. "Measuring the tendency of cnns to learn surface statistical regularities." arXiv preprint arXiv:1711.11561 (2017).

What features are used?

• To solve classification problems, often superficial high frequency features correlate strongly with the “semantic” features
• How can we suppress learning of these superficial features and encourage the model to “understand” images in more “semantically” meaningful explanatory factors?

Wang, Haohan, et al. "High-frequency component helps explain the generalization of convolutional neural networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

Learning Representations using Supervised Learning

• Supervised Learning is only tasked with finding patterns which solves the problem
• Often, there might be superficial features which is enough to solve the problem
• Consider a model trained on separating images of lions from leopards
• This problem could be solved by the lowest layers of a CNN just recognizing the leopards spots
• While these patterns help solve the lion vs leopard problem, the representations would not transfer well to other image problems

Representation space

Representation Learning

8

Artificial Intelligence (AI) and Deep Learning

• It’s important to highlight that Deep Learning is a subset of Representation Learning
• Also, Deep Learning is not the same thing as Deep Neural Networks (DNN)
• DNNs are a way to implement the ideas of Deep Learning, but should not be conflated with it
• Just like Random Forest should not be conflated with Ensemble Learning

9

Deep Learning

Example: Recurrent Neural Networks

Representation Learning

Exemple: Word vectors (Word2Vec)

Machine Learning

Exemple: Random Forest

Artificial Intelligence

Example: Expert system

Bengio, Yoshua, Ian Goodfellow, and Aaron Courville. Deep learning. Vol. 1. MIT press, 2017. https://www.deeplearningbook.org/ 

Deep Learning - Complex data can be modeled as a hierarchy of simpler factors through learning

• One of the first models dubbed Deep Learning was Deep Boltzmann Machines, networks formed by stacking Restricted Boltzmann Machines
• These were generative models built from undirected networks of binary stochastic variables trained using greedy layerwise pretraining within an energy minimization framework
• They were often then used as initial starting points for fine tuning regular Feed-Forward Network with gradient descent

10

Salakhutdinov, Ruslan, and Hugo Larochelle. "Efficient learning of deep Boltzmann machines." Proceedings of the thirteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings, 2010.

Representation is fundamental for AI

• How things are represented directly determines how easily we can devise algorithms to solve problems
• In this toy example we can see that the same dataset represented in polar coordinates can be solved by a simpler (linear) model than when it’s represented in cartesian coordinates
• Historically, much of machine learning research was about figuring out transformations from complex representations to simpler ones (often referred to as feature engineering)

11

Representation Learning

• Use data to learn an encoder from the raw, complex representation to a simple one
• Solved the problem in the simple representation space, typically with a linear model (e.g. logistic regression)
• The encoder here is most often a Deep Neural Network

Transfer learning

Representation Learning can be thought of as an “inverse” problem - example inverse rendering

14

Yildirim, Ilker, et al. "Efficient inverse graphics in biological face processing." Science advances 6.10 (2020): eaax5979. https://www.science.org/doi/10.1126/sciadv.aax5979

Dynamically computed “layers”

15

Focus on the head of the model

16

Typical focus for CNNs for computer vision

Our focus for now

Intuiting the dot-product

• The fundamental work horse of Deep Neural Networks is matrix multiplications, which even more fundamentally can be understood as dot products

​

17

Highlighting the dot product

18

A softmax “layer”

19

MNIST Example

20

Dot product and angles

• Recall from linear algebra, the fundamental connection between the dot product the angle of two vectors

​

​

​

​

​

​

​

• A unit in a neural network gets highly activated when the incoming vector has a small angle to its weight vector

21

Slight detour - Capacity of high dimensional space

22

For a deeper dive, see the first chapter of Blum, A., Hopcroft, J., & Kannan, R. (2020). Foundations of Data Science. Cambridge: Cambridge University Press. doi:10.1017/9781108755528. Preprint online version here.

Neural Networks solves problems by aligning vectors

23

Softmax regression and the dot product

• When we train a network for a categorical problem, each vector associated with the output can be though of as a prototype vector
• The learning procedure tries to align the mapped inputs to these “softmax vectors”, essentially by maximizing the dot product of the representation of examples which belong to that categoical value to its corresponding softmax vector
• If the magnitude of vectors where fixed to unit length, this will map examples to points along the unit sphere and the problem is solved by minimizing the angle to the softmax vector
• Since the softmax vectors are parameters of the model, we need to decide on how many we should have, and what categorical values they should correspond to when designing the model
• We can’t easily add new classes

24

Notebook 1

Contrastive Learning

Let’s do away with the softmax vectors

27

What can we do with this “non-parametric” softmax?

• We can use examples as implicit softmax vectors
• This allows us to apply the model to new classes on the fly, needing only some reference image

28

Wu, Zhirong, et al. "Unsupervised feature learning via non-parametric instance discrimination." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

Triplet Loss

29

Contrastive Learning

30

Tian, Yonglong, Dilip Krishnan, and Phillip Isola. "Contrastive multiview coding." arXiv preprint arXiv:1906.05849 (2019).

Multimodal Learning

31

The way in which something happens or is experienced.

• Modality refers to a certain type of information and/or the representation format in which information is stored.
• Sensory modality: one of the primary forms of sensation, as vision or touch; channel of communication.

https://sites.google.com/site/multiml2016cvpr/MMML-Tutorial-Part1.pdf?attredirects=0

Multimodal Learning

32

Arandjelovic, Relja, and Andrew Zisserman. "Look, listen and learn." Proceedings of the IEEE International Conference on Computer Vision. 2017.

Multimodal learning - CLIP

33

https://openai.com/blog/clip/

Notes on training: Full dataset is 400 million (text, image) pairs scraped from the internet. Each training batch contains ~32000 images.

Multimodal learning - CLIP

34

Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International Conference on Machine Learning. PMLR, 2021.

The problem of annotated data

• While these contrastive learning methods we’ve looked at seem to learn better representations than from pure supervision, they still rely on semantically meaningfull pairs of data
• Pairs of images of the same person for face detection
• Different modalities of the same scene for multi modallearning
• We would like to get away from this pesky detail of manually annotating data

35

Notebook 2

36

Self-supervised Learning

37

Supervised vs Unsupervised learning

• Supervised learning is often framed as the task of modeling some conditional distribution P(Y | X), where predicting Y helps us solve a problem

• Unsupervised learning is often framed as modeling the distribution of P(X), with the hope of learning something about the data

38

• This is not a true dichotomy, we can often reformulate a supervised learning problem as an unsupervised and vice versa
• What often matters is the data we have: supervised learning often requires manually annotated data to act as our supervision target
• This scales very poorly with improvements in compute
• What if we could automatically create supervision targets?
• This idea is referred to as self-supervised learning (auto = "self, same")

Self-supervised learning through prediction

39

Yann LeCun, https://www.youtube.com/watch?v=7I0Qt7GALVk

Self-supervised learning through prediction

• Come up with a task which encourages sensible representation learning
• Auto encoding
• Predict how an image has been transformed
• Order pieces of a puzzle
• Paint in parts of image given a context

40

The Autoencoder

• Learn to copy the input to the output
• Self-encoding, the target is the input
• Some kind of obstacle is introduced - a bottleneck layer or a corrupted input
• When trained with MSE reconstruction loss it’s very similar to PCA
• Learns to encode images in a representation which captures the most variance in data space
• The encoder can be used for transfer learning

41

Issues with autoencoders�reconstructing pixels might learn the wrong thing

42

Bengio, Yoshua, Ian Goodfellow, and Aaron Courville. Deep learning. Vol. 1, page 542. MIT press, 2017. https://www.deeplearningbook.org/ 

Gato

43

Reed, Scott, et al. "A Generalist Agent." arXiv preprint arXiv:2205.06175 (2022).

Self-supervision and Contrastive Learning

44

https://ai.googleblog.com/2020/04/advancing-self-supervised-and-semi.html

Gordon, Daniel, et al. "Watching the world go by: Representation learning from unlabeled videos." arXiv preprint arXiv:2003.07990 (2020).

Thank you!

erik.ylipaa@scilifelab.se