A Subtle Introduction to Deep Representation Learning

Applied Machine and Deep Learning

190.015

M.Sc. Fotios (Fotis) Lygerakis

October 2023

Chair of Cyber-Physical-Systems

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Outline

• Recap on Neural Networks
• Introduction to Representation Learning
• Core Techniques and Approaches
• Case Studies
• Coding Examples
• Conclusion and Q&A

2

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Recap on Neural Networks

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Recap: Neuron & Neural Network

4

Perceptron

https://www.allaboutcircuits.com/technical-articles/how-to-train-a-basic-perceptron-neural-network/

Deep Neural Network

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Recap: BackProp and Gradient Descent

5

Gradient Descent

https://www.analyticsvidhya.com/blog/2023/01/gradient-descent-vs-backpropagation-whats-the-difference/

Backpropagation

https://www.3blue1brown.com/lessons/backpropagation-calculus

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Representation Learning

https://classic.csunplugged.org/activities/image-representation/

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Definition

“Representation Learning is a process in machine learning where algorithms extract meaningful patterns from raw data to create representations that are easier to understand and process. These representations can be designed for interpretability, reveal hidden features, or be used for transfer learning.”

Reads

• Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation Learning: A Review and New Perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35, 8 (August 2013), 1798–1828. https://doi.org/10.1109/TPAMI.2013.50

7

https://paperswithcode.com/task/representation-learning

h = f(x)

x

h

original data

representation

function

f

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Feature Engineering vs Representation Learning

GO* Feature Engineering

• Manually defined features
• Requires domain expertise
• time-consuming/not scalable with more data

Representation Learning

• Automatically learns the most important features
• No need for explicit programming
• More efficient and can handle complex, high-dimensional data

8

*GO: good old

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Why is it important?

• Highlight Essential Features: Focuses the model’s attention on the most important aspects of the data
• Dimensionality Reduction: Speeds up learning and helps removing noise, enhancing performance
• Computational Efficiency: Train faster
• Improved Generalization: Facilitates better understanding of unseen data, building robust models

9

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Today’s Focus on Representation Learning

• Fundamentals
• Core Techniques & Approaches
• Applications in various domains

Automated feature engineering

Improved Performance

Real-World Applications

10

WHY Deep?

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Representation Learning

Encoder

Raw Data

Information

What is?

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Representation Learning

Encoder

What is?

Raw Data

Information

Low

High

Activation

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

13

Raw Data

Pixel Values

Information

Representation Vector

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

14

Encoder

Neural Network

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Kinds of Representations Learning

Unsupervised

Supervised

15

YES!

No!

Maybe?

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Which flavor to choose?

Unsupervised

Supervised

16

Abundance of labeled data

High precision tasks

Exploratory analysis

Data with hidden structures

Self-Supervised

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Self-Supervised Learning (SSL)

• Supervision is derived from the input data itself
• No explicit external labels needed
• Training by designing auxiliary tasks (reconstruction, regularization, etc)
• Given a portion of the data the model is trained to predict or reconstruct the remaining part.
• Once the model is trained in this manner, the learned representations can be used for downstream tasks, often with a separate fine-tuning step.

17

That’s what we are going to discover today!

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Which flavor to choose?

Unsupervised

Supervised

18

Unlabeled data

labeled data

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Representation Learning

How do we get the representations?

Encoder

Raw Data

Classifier

Labels

Supervised Learning

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Representation Learning

How do we get the representations?

Encoder

Raw Data

Self-Supervised Learning (SSL)

Inherent Learning Signal

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Representation Learning

SSL

Encoder

Raw Data

Decoder

Reconstruction

Reconstruction

(Autoencoding)

D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” 2014.

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Representation Learning

SSL

Encoder

Raw Data

Joint Embeddings

(Contrastive, Regularization, EBM)

Encoder

Positive Sample

A. van den Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” 2018

K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” 2020

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations, 2020

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Kinds of Representations Learning

Unsupervised

Supervised

23

YES!

No!

Maybe?

Representations

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Which flavor to choose?

Unsupervised

Supervised

• Benefits:
• Learning from labeled data.
• Often provides better performance with adequate labeled data.

• Benefits:
• Learning from unlabeled data, which is more abundant.
• Can uncover latent structures and features not evident from labels.

24

• Best for:
• Tasks with a substantial amount of labeled data.
• Situations where high accuracy is essential and labeled examples are clear.

• Best for:
• Tasks with limited or no labeled data.
• Understanding the underlying structure of the data.

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Representation Learning

Why bother with SSL?

• Utilize abundant, unlabeled data.
• Representations
• robust
• generalizable / task-agnostic
• efficient

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Real World Applications!

• Image Denoising
• Anomaly Detection
• Dimensionality Reduction
• Data Compression
• Visual Representation Learning
• Pre-training for Downstream Tasks
• Image Retrieval

• Image Generation
• Data Augmentation
• Super-Resolution
• Style Transfer
• Question Answering
• Multimodal Learning
• Robot Learning

26

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Real World Applications!

27

https://openai.com/chatgpt

https://openai.com/dall-e-2

https://www.midjourney.com

DINOv2

RT2

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Real World Applications!

28

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Core Techniques and Approaches

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Autoencoders

30

Encoder(img)

Decoder(z)

original

z

reconstructed

Training

1. Forward pass (left-to-right)
2. MSE(Original, Reconstructed) MSE: Mean Squared Error
3. Backpropagate the MSE loss

Reads

https://www.v7labs.com/blog/autoencoders-guide

https://www.geeksforgeeks.org/implementing-an-autoencoder-in-pytorch/

https://www.tutorialspoint.com/how-to-implementing-an-autoencoder-in-pytorch

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Contrastive Learning

31

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Contrastive Learning

32

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Contrastive Learning

33

make similar

make dissimilar

Aäron van den Oord, Yazhe Li, & Oriol Vinyals (2018). Representation Learning with Contrastive Predictive Coding. CoRR, abs/1807.03748.

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Contrastive Learning

34

Encoder(anc)

Anchor

zanc

Reads

https://www.v7labs.com/blog/contrastive-learning-guide

https://encord.com/blog/guide-to-contrastive-learning/

https://www.youtube.com/watch?v=sftIkJ8MYL4

Encoder(pos)

​

Positive

zpositive

Encoder(neg)

Negative

znegative

pull apart

pull together

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Contrastive Learning

35

Training

• Data Augmentation:
• Forward Pass:
• Compute zq
• Compute zk
• Compute Similarity
• Contrastive Loss LNCE
• Backward Pass

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Transformers

36

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Masked Autoencoders

37

Reads

https://medium.com/dair-ai/papers-explained-28-masked-autoencoder-38cb0dbed4af

https://towardsdatascience.com/into-the-transformer-5ad892e0cee

https://towardsdatascience.com/illustrated-guide-to-transformers-step-by-step-explanation-f74876522bc0

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Basics of Convolution

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Convolutions Basics: Kernel(filter)

39

https://courses.cs.washington.edu/courses/cse446/21au/sections/08/convolutional_networks.html

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Convolutions Basics: Stride

40

https://www.analyticsvidhya.com/blog/2022/03/basics-of-cnn-in-deep-learning/

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Convolutions Basics: Padding

41

https://towardsdatascience.com/types-of-convolutions-in-deep-learning-717013397f4d

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Examples

42

Kernel:

2x2

Stride:

1

Padding:

0

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Examples

43

Kernel:

3x3

Stride:

2

Padding:

0

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Examples

44

Kernel:

3x3

Stride:

1

Padding:

1

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Coding Examples

Github Repository

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Case Studies

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Contrastive Regularized VAE (CR-VAE)

​

​

47

F. Lygerakis, E. Rueckert, CR-VAE: Contrastive Regularization on Variational Autoencoders for Preventing Posterior Collapse, 7th Asian Conference on Artificial Intelligence (ACAIT), 2023

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Combining Vision and Touch with SSL: MViTac

48

V.Dave*, F. Lygerakis*, E. Rueckert (*Equal Contribution), Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Training, IEEE International Conference on Robotics and Automation (ICRA), 2024

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Let the robots touch what they see: M2CURL

49

F. Lygerakis, V.Dave, E. Rueckert, M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulation, 2024, 21st International Conference on Ubiquitous Robots (UR2024)

Best Student Paper Award

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Wrapping Up

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Wrap up

• Representation Learning is important
• Highlight Essential Features
• Dimensionality Reduction
• Computational Efficiency
• Improved Generalization
• Exploit big unlabeled data

​

51

• Real-World Applications
• Core Techniques
• Autoencoding
• Contrastive Learning

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Resources & Extra Reads

Autoencoders

Introduction to Pytorch

Contrastive Learning

Masked Autoencoders(Transformers)

52

https://www.v7labs.com/blog/autoencoders-guide

https://www.geeksforgeeks.org/implementing-an-autoencoder-in-pytorch/

https://www.tutorialspoint.com/how-to-implementing-an-autoencoder-in-pytorch

https://lilianweng.github.io/posts/2021-05-31-contrastive/

https://arxiv.org/abs/2002.05709

https://arxiv.org/abs/1911.05722

https://arxiv.org/abs/2006.07733

https://arxiv.org/abs/2006.09882

https://arxiv.org/abs/2105.04906

https://medium.com/dair-ai/papers-explained-28-masked-autoencoder-38cb0dbed4af

https://towardsdatascience.com/into-the-transformer-5ad892e0cee

https://towardsdatascience.com/illustrated-guide-to-transformers-step-by-step-explanation-f74876522bc0

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Thank you for your attention!

Fotios (Fotis) Lygerakis

Chair of Cyber-Physical-Systems

Montanuniversität Leoben

Franz-Josef-Straße 18,

8700 Leoben, Austria

​

E-mail: fotios.lygerakis@unileoben.ac.at

Web: https://cps.unileoben.ac.at/fotios-lygerakis-m-sc/

​

53

Disclaimer: The lecture notes posted on this website are for personal use only. The material is intended for educational purposes only. Reproduction of the material for any purposes other than what is intended is prohibited. The content is to be used for educational and non-commercial purposes only and is not to be changed, altered, or used for any commercial endeavor without the express written permission of Professor Rueckert.

Presentation

Link

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS

Thank you for your attention!

Fotios (Fotis) Lygerakis

Chair of Cyber-Physical-Systems, Montanuniversität Leoben

E-mail: fotios.lygerakis@unileoben.ac.at

Website: www.lygerakis.com

54

Disclaimer: The lecture notes posted on this website are for personal use only. The material is intended for educational purposes only. Reproduction of the material for any purposes other than what is intended is prohibited. The content is to be used for educational and non-commercial purposes only and is not to be changed, altered, or used for any commercial endeavor without the express written permission of Professor Rueckert.

MONTANUNIVERSITÄT LEOBEN

CYBER-PHYSICAL-SYSTEMS