CSE 5524: �Transfer learning & Stereo

1

HW 3 & HW 4 & quizzes

• HW 3
• Caution: Please re-download the data
• Due: 4/11/2025

​

• HW 4
• Plan: A lighter homework
• Due: 4/21/2025

​

• Quizzes:
• Two quizzes to be released in the next two weeks --- True/False, multiple choices, unlimited tries

​

​

​

​

Today (37 & 40)

• Recap
• Transfer learning & adaptation
• Stereo vision

​

​

​

3

Recap: Domain gap

Domain gap

Recap: Domain gap

Recap: Data augmentation

• Increate the amount, diversity, and coverage of the data

Recap: Data augmentation

Today (37 & 40)

• Recap
• Transfer learning & adaptation
• Stereo vision

​

​

​

8

Data is important …

The existence of domain gaps implies that we need to “re-collect” training data

Data is important …

The existence of new tasks implies that we need to “re-collect” training data

Bird species

Dog breeds

Car brands & styles

Neural networks are data hungry…

Sufficient labeled data

​

ImageNet-1K (ILSVRC)

1,000 object classes

1,000 training images per class

Humans do not re-learn from scratch …

• Once we learn certain skills (e.g., recognizing bird species), those skills can usually be “transferred” to other related tasks (e.g., recognizing dog breeds).

​

• We typically only need a few examples (some images of different dog breeds) to equip us with the capability for the new tasks

​

• Can neural networks do so?

Transfer learning

• Transfer knowledge learned from prior tasks to new tasks

Transfer learning

• Transfer knowledge learned from prior tasks to new tasks

Transfer learning

• Transfer knowledge learned from prior tasks to new tasks

How to achieve “adaptation”?

Pre-training and fine-tuning

CNN revisit

17

Pre-training and fine-tuning

Probably don’t need to change a lot!

Pre-training and fine-tuning

Pre-training and fine-tuning

Algorithm

Questions?

22

This paradigm “pre-training + fine-tuning” is everywhere

This paradigm “pre-training + fine-tuning” is everywhere

This paradigm “pre-training + fine-tuning” is everywhere

25

Ficus auriculata

Onoclea

sensibilis

Onoclea hintonii

CLIP (Contrastive Language-Image Pretraining) matches images to text,

• capture hierarchy relationship between classes
• enable few-shot or even zero-shot learning

This paradigm “pre-training + fine-tuning” is everywhere

Fine-tuning a subset of neural networks

Feature encoder

Prediction head

Feature vector

Image

Label

Fine-tuning a subset of neural networks

Feature encoder

Feature vector

Image

Label

Prediction head

Fine-tuning a subset of neural networks

Feature encoder

Feature vector

Image

Label

Prediction head

Fine-tuning a subset of neural networks

Feature encoder

Feature vector

Image

Label

• Full fine-tuning: Update everything

​

• (Linear) probing: Only update the (linear) prediction head

​

• Parameter-efficient fine-tuning: Update a subset of parameters

Prediction head

Parameter efficient fine-tuning

Parameter efficient fine-tuning

Parameter efficient fine-tuning

Questions?

34

Learning from a teacher

• Typically: learning from data

​

​

​

​

• What if we have a very powerful but computationally expensive pre-trained model?

Learning from a teacher: knowledge distillation

Learning from a teacher: knowledge distillation

Learning from a teacher: knowledge distillation

Learning from a teacher: knowledge distillation

​

Teachers could:

• Instruct semantic relatedness
• Denoise

​

Questions?

40

Prompting

Prompting

Prompting

Prompting

Prompting

Prompting

Prompting

• Provide the “context of the task” in the input data!

Visual Prompt Tuning

48

Visual Prompt Tuning [Menglin Jia et al.]

Questions?

49

3D reconstruction

• How can we reconstruct 3D from image(s)?

Depth estimation and 3D reconstruction

3D reconstruction

• Making “assumptions”

​

​

​

​

​

​

• Parallel projections
• Flat surfaces
• …

3D reconstruction

• How do humans do?

​

​

​

​

• Learning from “experience” – monocular depth estimation
• Leverage “two eyes” – stereo depth estimation

Stereo depth estimation

53

I_l

I_r

D

Z

disparity

depth

Focal length Baseline

Stereo depth estimation

54

Left

Right

Depth Map

Stereo depth estimation

55

Left

Right

Similarity

Disparity

Stereo depth estimation

56

Left

Right

Pyramid stereo matching network (PSMNet)

57

[Chang et al., Pyramid stereo matching network, 2018]

General architecture

General architecture

Cost volume

Continuous Disparity Network

60

Left

Right

[Garg et al., Wasserstein Distances for Stereo Disparity Estimation, 2020]

Continuous Disparity Network

61

Left

Right

[Garg et al., Wasserstein Distances for Stereo Disparity Estimation, 2020]

62

Output disparity

= mode

Output disparity

= shifted mode

0

0

Questions?

63

What if we have more images?

What if we have more images?

Can we synthesize images from other views?