Overview of Self-supervised Learning (SSL)

Jongbum Won

01815150

1. RECAP: �Supervised Learning

Supervised Learning

2. Introduction:�What do models learn?

Representation Learning

Representation Learning

• In machine learning, feature learning or representation learning is a set of techniques that allows a system to automatically discover the representations needed for feature detection or classification from raw data (https://en.wikipedia.org/wiki/Feature_learning).

​

• Can be categorized into 3 types:

1. Supervised (e.g., DNN, CNN)

2. Unsupervised (e.g., clustering, PCA)

3. Self-supervised (e.g., autoencoder, word embedding)

Representation Learning

• Example: Convolutional Neural Network (CNN)’s architecture

Representation Learning

• Example: Convolutional Neural Network (CNN)’s architecture

Feature map or

Image feature space

(latent space)

Representation Learning

• Example: Convolutional Neural Network (CNN)’s architecture

Feature map or

Image feature space

(latent space)

Representation Learning

Then, why do we compress data?

Representation Learning

Then, why do we compress data,

even losing information?

Representation Learning

Then, why do we compress data,

even losing information?

🡺 A model tries to learn important (compact) information

Latent space (embedding space)

Point (0.4, 0.3, 0.8) graphed in 3D space

This ‘compressed state’ is the Latent Space Representation of our data.

Latent space (embedding space)

Point (0.4, 0.3, 0.8) graphed in 3D space

​

​

​

Data points will be pointed in the latent space.

“Similar” data points will be pointed “closer” to each other within certain subspace (= manifold).

Latent space (embedding space)

Point (0.4, 0.3, 0.8) graphed in 3D space

​

​

​

Data points will be pointed in the latent space.

“Similar” data points will be pointed “closer” to each other within certain subspace (= manifold).

​

Q1. What does it mean by “similar”?

Q2. What is manifold?

Latent space (embedding space)

​

​

​

Q1. What does it mean by “similar” (in the latent space)?

Data points in the pixel space:

Latent space (embedding space)

​

​

​

Q1. What does it mean by “similar”?

Data points in the pixel space:

4 legs

Latent space (embedding space)

​

​

​

Q1. What does it mean by “similar”?

back-rest, no drawer, connections between legs

4 legs

Latent space (embedding space)

​

​

​

Q1. What does it mean by “similar”?

back-rest, no drawer, connections between legs

4 legs

color

Latent space (embedding space)

​

​

​

put closer

Latent space (embedding space)

​

​

​

put closer

Metric to measure “closeness” is not necessarily Euclidian distance…

There are various ways to measure it.

Latent space (embedding space)

Manifold:

Groups or subsets of data that are ‘similar’ (closer each other based on certain metric) in some way

Above manifold learning (i.e., finding subspace of similar datapoints) is subset of representation learning

​

Q2. What is manifold?

Latent space (embedding space)

Latent space (embedding space)

Interpolation in pixel space

​

Interpolation in latent space

​

Latent space (embedding space)

Autoencoder

CNN

3. Main Topic:�What is Self-supervised Learning?

What is Self-supervised Learning (SSL)?

In self-supervised learning, the system learns to predict part of its input from other parts of it input. In other words, a portion of the input is used as a supervisory signal to a predictor fed with the remaining portion of the input.

https://www.facebook.com/722677142/posts/10155934004262143/

What is Self-supervised Learning (SSL)?

​

• Self-Supervised Learning (SSL) is a special type of representation learning that enables learning good data representation from unlabelled dataset.

​

• The term “self-supervised learning” was first introduced in robotics, where the training data is automatically labeled by finding and exploiting the relations between different input sensor signals. (Liu et al., 2021)

​

• LeCun’s classical definitions:

1. Obtain “labels” from the data itself by using a “semi-automatic” process.

2. Predict part of the data from other parts.

Supervised VS Unsupervised

• In supervised learning, human-annotated labels (target value) is used for training
• In unsupervised learning, there is no labels but analyzes and clusters unlabeled data sets, discovering hidden patterns in data without the need for human intervention

Examples) Clustering, association, dimensionality reduction, …

​

Goal is also different: Predictive learning (regression, classification, …) VS Representation learning (later…)

Self-supervised VS Unsupervised

• Self-supervised learning is a subset of unsupervised learning

​

4. Main Topic:�Why Self-supervised Learning?

Limitation of Supervised Learning

• Example: Convolutional Neural Network (CNN)’s architecture

Limitation of Supervised Learning

• Example: Convolutional Neural Network (CNN)’s architecture

Limitation of Supervised Learning

• Various annotated (image) datasets are available…

Motivation of SSL

Lack of data in supervised learning also can result in Out-Of-Distribution (OOD) problem

​

🡪 Model could misbehave with unseen data

5. Main Topic:�How does Self-supervised work?

How SSL works

1. Define a pretext task
2. Train the network using unlabeled data to solve the pretext task
3. Assume, if the pretext task is appropriate, the network (or model) will learn good representations of the given data
4. Transfer the pre-trained network (from step 3) for the downstream task (real problem we want to solve)

How SSL works

1. Define a pretext task
2. Train the network using unlabeled data to solve the pretext task
3. Assume, if the pretext task is appropriate, the network (or model) will learn good representations of the given data
4. Transfer the pre-trained network (from step 3) for the downstream task (real problem we want to solve)

SSL Methods: Pretext Tasks

Goal: By solving a given problem regarding the data,

we expect meaningful representation to be learned from the data.

SSL Methods

1. Self-prediction
2. Contrastive learning

(Open AI NeurIPS 2021 Tutorial)

SSL Methods

Self-prediction: Given an individual data sample, the task is to predict one part of the

sample given the other part. The part to be predicted pretends to be missing.

​

Contrastive learning: Given multiple data samples, the task is to predict the

relationship among them.

The multiple samples can be selected from the dataset based on some known logics

(e.g. the order of words / sentences), or fabricated by altering the original version.

​

SSL Method 1: Self-prediction

1. Autoregressive generation: predicts future behavior based on past behavior

​

Example:

• Autoregressive language modeling (GPT)

I like ___ 🡺 I like apple ___ 🡺 I like apple and ___ 🡺 I like apple and banana.

​

SSL Method 1: Self-prediction

2. Masked generation:

We mask a random portion of information and pretend it is missing, irrespective of the natural sequence.

Example:

• Masked language modeling (BERT) 🡺 I like ___ and banana.
• Images with masked patch (denoising autoencoder, context autoencoder, colorization)

​

SSL Method 1: Self-prediction

3. Innate relationship prediction:

Some transformation (e.g. segmentation, rotation) of one data sample should maintain the original information or follow the desired innate logic.

Example:

• Order of image patches (e.g., relative position, jigsaw puzzle)
• Image rotation

Example: Context Prediction (Doersch et al., 2015)

1. Define a pretext task
2. Train the network using unlabeled data to solve the pretext task
3. Assume, if the pretext task is appropriate, the network (or model) will learn good representations of the given data
4. Transfer the pre-trained network (from step 3) for the downstream task

Example: Context Prediction (Doersch et al., 2015)

1. Define a pretext task
2. Train the network using unlabeled data to solve the pretext task
3. Assume, if the pretext task is appropriate, the network (or model) will learn good representations of the given data
4. Transfer the pre-trained network (from step 3) for the downstream task

Automatically labeled

(i.e., pseudo-label)

Example: Context Prediction (Doersch et al., 2015)

1. Define a pretext task
2. Train the network using unlabeled data to solve the pretext task
3. Assume, if the pretext task is appropriate, the network (or model) will learn good representations of the given data
4. Transfer the pre-trained network (from step 3) for the downstream task

Automatically labeled

(i.e., pseudo-label)

This label can be generated within the dataset itself, without human intervention

Example: Context Prediction (Doersch et al., 2015)

1. Define a pretext task
2. Train the network using unlabeled data to solve the pretext task
3. Assume, if the pretext task is appropriate, the network (or model) will learn good representations of the given data
4. Transfer the pre-trained network (from step 3) for the downstream task

Example: Context Prediction (Doersch et al., 2015)

1. Define a pretext task
2. Train the network using unlabeled data to solve the pretext task
3. Assume, if the pretext task is appropriate, the network (or model) will learn good representations of the given data
4. Transfer the pre-trained network (from step 3) for the downstream task

Example: Context Prediction (Doersch et al., 2015)

SSL Method 2: Contrastive Learning

The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. The key idea is to find common feature information from different points of view on certain datapoint.

​

Categorized in 3 types:

1. Inter-sample classification

2. Feature clustering

3. Multiview coding

​

​

​

SSL Method 2: Contrastive Learning

The goal of contrastive representation learning is to learn such an embedding space in which similar sample pairs stay close to each other while dissimilar ones are far apart. The key idea is to find common feature information from different points of view on certain datapoint.

​

Categorized in 2 types:

1. Inter-sample classification

2. Feature clustering

​

​

SSL Method 2: Contrastive Learning

1. Inter-Sample Classification

Given both similar (“positive”) and dissimilar (“negative”) candidates, to identify which ones are similar to the anchor data point is a classification task.

​

There are creative ways to construct a set of data point candidates:

1. The original input and its distorted version

2. Data that captures the same target from different views

​

SSL Method 2: Contrastive Learning

1. Inter-Sample Classification

Given both similar (“positive”) and dissimilar (“negative”) candidates, to identify which ones are similar to the anchor data point is a classification task.

​

​

​

Since a model knows if the given input (e.g., augmented images) comes from the same anchor data point or not, labeling process is not required.

Architecture of SimCLR

SSL Method 2: Contrastive Learning

1. Inter-Sample Classification

A model learns to find latent space in which similar data points are put closer, whereas negative pairs are put farther as having a lower boundary distance (i.e., margin).

Various loss functions are developed:

​

​

SSL Method 2: Contrastive Learning

1. Inter-Sample Classification

A model learns to find latent space in which similar data points are put closer, whereas negative pairs are put farther as having a lower boundary distance (i.e., margin).

​

​

​

SSL Method 2: Contrastive Learning

2. Feature Clustering

Rather than using positive/negative pair, use clustering algorithm and put pseudo-labels to data points, so that we can run intra-sample classification (contrastive learning) again.

​

​

6. Future directions

Future directions

In general, the performance of SSL models are lower than current well-studied supervised learning models.

🡪 State-of-the-art SSL models now have a smaller gap between .

​

To improve SSL performance:

1. Choose appropriate (self-supervised) pretext task

Ex) Masked (self) prediction vs Contrastive learning

Masked prediction 🡪 Intra-sample prediction 🡪 representation about local-level features

Contrastive prediction 🡪 Inter-sample prediction 🡪 representation about overall-level features

​

2. Combine different pretext task

​

3. Improve training efficiency / Reduce training cost

​

​

Thank you