CSE 5524: �Representation learning

1

HW 3

• Due: 4/8/2025
• Focus on the region proposal network (RPN).
• A bit more implementation, so start earlier.

​

Final project (30%)

​

• Project proposal: 4/1 (3%)
• Instructions will be released soon

Today (30)

• Representation learning

​

​

​

4

Representation learning

• Now we have learned multiple neural networks

​

​

​

​

​

​

​

• They all can output a “vector” to present an image

The usefulness of representation

6

Retrieval, image-to-image search

The representation learning setup

​

​

​

​

​

​

​

• How can we teach the neural network encoder to capture image relationships?

​

​

​

​

​

​

​

​

​

Image

The representation learning setup

​

​

​

​

​

​

​

​

• What properties should z possess?
• What is the objective function to learn f to achieve it?

​

​

​

​

​

​

​

​

​

The representation learning setup

​

​

​

​

​

​

​

​

• z has lower dimensionality
• P(z) ha a simple structure (simple distribution)
• Dimensions are disentangled

​

​

​

​

​

​

​

​

What makes a good representation?

• Compression:
• Less memory
• Invariant to nuisance
• Occam’s razor: Less is more

​

​

• Prediction:
• Predictive of the future/past
• Predictive of any question you can ask about the image

​

An overview

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Autoencoder

Objective function:

​

Design principle: z with lower dimensionality

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Autoencoder

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Example

Training data

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Predictive encoding

Pretext tasks:

A good representation should solve these tasks well

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Predictive encoding

Pretext tasks:

A good representation should solve these tasks well

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Predictive encoding

Pretext tasks:

A good representation should solve these tasks well

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Predictive encoding

Pretext tasks:

A good representation should solve these tasks well

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Predictive encoding

Encoder

Predictive encoding

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Object detectors emerge from scene recognition

Self-supervised learning

• Come up with a “pretext” target from raw data itself

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Self-supervised learning: context encoder

[Pathak et al., CVPR 2016]

Self-supervised learning: masked autoencoder (MAE)

[He et al., CVPR 2022]

Imputation

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Imputation

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Clustering

• So far, we consider “continuous output” vector

​

​

​

​

• We can also consider learning a “one-hot” “discrete” representation

​

​

• Lightweight, abstract representation
• More aligned with language

Clustering

• Classification is “supervised” clustering

​

​

​

​

​

​

• Without labels, we need a certain way to group data

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

K-means

• Iterative clustering:
• Initialize “cluster center”
• Assign each data to its closest center
• Re-calculate “cluster means”

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

K-means

• Iterative clustering:
• Initialize “cluster center”
• Assign each data to its closest center
• Re-calculate “cluster means”

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

K-means

• Iterative clustering:
• Initialize “cluster center”
• Assign each data to its closest center
• Re-calculate “cluster means”

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

K-means

• Iterative clustering:
• Initialize “cluster center”
• Assign each data to its closest center
• Re-calculate “cluster means”

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

K-means

• Algorithm

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

K-means + deep learning

• Deep Clustering for Unsupervised Learning of Visual Features [Caron et al., ECCV 2018]

K-means + deep learning

• Neural Discrete Representation Learning [Aaron van den Oord et al., 2018]
• Algorithm known as VQ-VAE

Question: what do different algorithms learn?

Contrastive learning

• Making you the representation “invariant” to some transformation (views)

​

​

• f is the neural network encoder
• T is a certain transformation (such as image translation)

​

Data transformation

37

[Chen et al., A Simple Framework for Contrastive Learning of Visual Representations, ICML 2020]

Contrastive learning

• Making you the representation “invariant” to some transformation (views)

​

​

• f is the neural network encoder
• T is a certain transformation (such as image translation)

​

​

• Question: there is a trivial solution, which is f always outputs 0

Contrastive learning: transformation

• Solution:
• Positive pairs:

​

• Negative pairs:

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Contrastive learning: co-occurence

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Contrastive learning

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Contrastive learning: exemplar losses

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

43

[Google AI blog, Advancing Self-Supervised and Semi-Supervised Learning with SimCLR]

Potential final project

• Design your transformation for invariance

[Figure credit: A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision.]

Section 30.10.2

Figure 30.13

Research example

Rethinking pre-training for object detector

Goal: Bridge the input point cloud and output bounding boxes

​

Our idea: Help the backbone cluster points into object instances or parts

46

Feature

backbone

Detection

head

Our proposal: leverage color information

Motivation: each object instance or part often possesses a coherent color and has a sharp contrast to the background

​

​

​

​

​

Hypothesis: Learning to colorize LiDAR point clouds would equip the backbone with the semantic cues to segment points

47

Region growing [Preetha et al., 2012]

Superpixel [Liu et al., 2011]

Our solution: “grounded” point colorization (GPC)

Provide colors of a subset of points as hints

​

​

​

​

​

​

48

Our solution: “grounded” point colorization (GPC)

Provide colors of a subset of points as hints

​

​

​

​

​

​

49

GPC Pre-training + fine-tuning

50

color

decoder

Output embeddings indicate which points should be colored/segmented together

Pre-training = Point-wise color regression problem

GPC Pre-training + fine-tuning

51

Output embeddings indicate which points should be colored/segmented together

GPC Pre-training + fine-tuning

52

backbone

feature

color

decoder

detection

head

pre-training

fine-tuning

concat

hints

Pre-Training LiDAR-Based 3D Object Detectors Through Colorization, ICLR 2024

What does GPC really learn?

​

​

​

​

​

Predict the exact color of each point

​

Predict which points possess the same color and are segmented together

​

53