Cross-Modal Fine-Tuning:

Align then Refine

Junhong Shen^1,2, Liam Li², Lucio Dery¹, Corey Staten², Mikhail Khodak¹,

Graham Neubig¹, Ameet Talwalkar^1.2

1 Carnegie Mellon University 2 Hewlett Packard Enterprise

Outline

• Motivation: why do we study cross-modal transfer?
• ORCA: align-then-refine workflow
• Empirical results
• Discussion & future directions

2

Pretrained transformers are changing ML fields

Language, vision, audio

3

Pretrained transformers are changing ML fields

4

Pretrained transformers are changing ML fields

5

What about other modalities?

6

What about other modalities?

7

Everything Else

— “Diverse Tasks”

Internet: search

Mobile: Assistants

Transportation: Autonomous vehicle

Language, Vision, Audio

Solving diverse tasks in less-studied modalities

​

• Design specialized networks ➔ domain knowledge & ML expertise
• AutoML (NAS)
• General-purpose architectures

​

8

Challenges:

Existing approaches: neural architecture search

• Search for task-specific networks from a predefined search space and then train the network, e.g., XD [Roberts et al., 2021], DASH [Shen et al., 2022]

​

​

​

​

​

​

​

9

Existing approaches: general-purpose models

• Provide a general architecture that can be trained with a variety of data formats, e.g., Perceiver and Perceiver IO [Jaegle et al., 2021]

​

​

​

​

​

10

image

point cloud

optical flow

audio

video

classification labels

audiovisual sequences

optical flow fields

Solving diverse tasks in less-studied modalities

• Design specialized networks ➔ domain knowledge & ML expertise
• AutoML (NAS)
• General-purpose architectures
• Limited label data

​

11

Challenges:

Require training from scratch

How can existing pretrained models help?

• No need for training from scratch
• Alleviate modeling & data concerns
• Reduce the human effort needed to develop high-quality task-specific models

12

Applying models pretrained in data-rich modalities to new problems

How can existing pretrained models help?

• No need for training from scratch
• Alleviate modeling & data concerns
• Reduce the human effort needed to develop high-quality task-specific models

13

Applying models pretrained in data-rich modalities to new problems

Cross-Modal Transfer Learning

?

Early evidence: language models can be adapted to…

​

​

​

Recognize images [Kiela et al., 2019] Solve tabular tasks [Dinh et al., 2022]

​

​

​

Play referential games [Li et al., 2020] Reinforcement learning [Reid et al., 2022]

14

But these methods cannot be applied to arbitrary tasks

Many of them are task-specific/ad-hoc

• Hand-crafted architecture components, manual prompt engineering

​

​

​

​

15

A more general method

Frozen Pretrained Transformers (FPT) [Lu et al., 2022]

​

• Image/protein sequence/math symbol ➔ sequence features
• Fine-tune layernorms ➔ prevent overfitting
• No task/semantic-specific component ➔ not effective enough

16

Our goal

General

+

Effective (accounts for modality difference)

=

ORCA

17

ORCA: motivation

• Different data have distinct patterns
• Given a pretrained model, we know on what datasets it performs well, can we manipulate the target data into something similar to those datasets?

Insight: data alignment

• Dimensionality
• Distribution

​

18

Notation

​

​

• Target data {x^t , y^t } ~ P_t in target domain D_t
• Transformer m_s in source domain D_s
• Reference data {x^s , y^s } ~ P_s in source domain D_s

​

​

Goal: learn a new model m_t based on m_s to minimize the expected loss on P_t

​

19

ORCA: workflow

20

Dimensionality alignment ➔ general-purpose

​

Distribution alignment

➔ task-specific

Stage 1: dimensionality alignment

Decompose a transformer

• Embedder f ^t : raw input ➔ sequence features
• R^cxhxw ➔ R^{seq_len x embed_dim}
• Model body g^t: sequence ➔ sequence
• R^{seq_len x embed_dim} ➔ R^{seq_len x embed_dim}
• Prediction head h^t: sequence features ➔ output
• Classification: 1D adaptive pooling + linear
• Dense prediction: reshape + adaptive pooling

21

Stage 2: embedder learning for data alignment

​

If we have a metric d to measure the distance between two distributions

• Minimize d to align the embedded feature distribution of the target dataset with that of the reference dataset

22

• ​

Stage 2: embedder learning for data alignment

If we have a metric d to measure the distance between two distributions

• Minimize d to align the embedded feature distribution of the target dataset with that of the reference dataset

23

?

Stage 2: embedder learning for data alignment

​

Optimal Transport Dataset Distance (OTDD) [Alvarez-Melis & Fusi, 2020]

• Use label information: model each class as a distribution over features

​

​

Pairwise l2, cosine distance, maximum mean discrepancy, optimal-transport, …

24

Stage 3: refine the remaining weights

Compute task loss, back propagate to update

• embedder
• model body
• prediction head

25

Comparison with previous work

26

Empirical results

Two backbones: RoBERTa [Liu et al., 2019], Swin Transformer [Liu et al., 2021]

Reference proxy datasets: CoNLL-2003 (entity recognition), CIFAR-10

​

• A breadth of modalities : NAS-Bench-360
• Two benchmarks that we analyze in depth
• PDEBench
• OpenML-CC18 Tabular Classification
• Compare with existing cross-modal work on drug response & tabular prediction

​

​

​

27

Can pretrained models transfer across diverse modality?

NAS-Bench-360: 10 tasks with diverse inputs (1D/2D), outputs (point/dense), and modalities (vision, audio, electrocardiogram, PDE, protein, genomics, cosmic-ray)

3 Classes of baselines: hand-designed architectures, general-purpose models, AutoML (NAS) methods

​

​

​

​

28

…

Wait…which distance metric should we use?

ORCA + different embedder learning metrics on NAS-Bench-360

29

Evaluate aggregate performance

Performance profile [Dolan and More, 2002]

• Normalize metrics for different tasks (0-1 error, MAP, F1 score, MAE, …)
• Compute fraction of tasks where a method has distance from optimality less than τ

30

Wait…which distance metric should we use?

ORCA + different embedder learning metrics on NAS-Bench-360

31

Can pretrained models transfer across diverse modality?

Yes. ORCA achieves the lowest error on 7/10 tasks

• Outperforms/matches hand-designed architectures on all tasks

​

• Outperforms all AutoML baselines on 8/10 tasks
• 2nd DeepSEA, 3rd onNinaPro

​

• Improvements come at a small computational cost
• Embedder learning time is a small portion (11%) of the fine-tuning time

32

ORCA attains the best aggregate performance

​

ORCA being far in the top left corner indicates it is rarely suboptimal and is often the best

​

33

Behind the scene

• Distribution alignment is the key
• Full fine-tuning is better than partial fine-tuning
• Suitable pretrained model selection is important

34

Key 1: distribution alignment

​

​

​

​

• ORCA consistently outperforms naive fine-tuning ➔ we need alignment
• Train-from-scratch is worse than ORCA but better than fine-tuning on ECG, Satellite, and DeepSEA ➔ naive fine-tuning without alignment may even hurt performance

35

Key 1: distribution alignment

​

How does optimizing the OTDD to various levels of convergence affect performance?

​

​

​

​

​

36

Key 2: full fine-tuning is better than partial tuning

Recall FPT:

1. Does not pretrain the embedding layers for task-specific adaptation
2. Only fine-tunes the layer norms

​

​

​

​

37

Key 2: full fine-tuning is better than partial tuning

​

​

​

​

​

• ORCA-layernorm outperforms FPT, but performance gain is smaller than full fine-tuning
• Data alignment boosts the cross-modal performance of FPT
• Full fine-tuning takes better advantage of the aligned embeddings
• Fine-tuning just the layer norms only results in less than 2x speedups

38

Key 3: pretraining modality affects performance

​

​

​

​

​

• To select pretrained models from a predefined model hub for each task: compare the optimized distribution distance and pick the one with the smallest value

39

Application: data-limited regimes

Recall motivation: utilize existing model resources to help regimes where training models from scratch is challenging

​

​

40

Application: data-limited regimes

Hypothesis: obtain a good feature embedder can reduce the difficulty of fine-tuning

• Fine-tuning suffer from limited data, but ORCA can considerably alleviate the problem

• ORCA allows us to match the performance of standard fine-tuning with 3x the amount of data

41

Empirical results

Two backbones: RoBERTa [Liu et al., 2019], Swin Transformer [Liu et al., 2021]

Reference proxy datasets: CoNLL-2003 (entity recognition), CIFAR-10

​

• A breadth of modalities : NAS-Bench-360
• Two benchmarks that we analyze in depth
• PDEBench
• OpenML-CC18 Tabular Classification
• Compare with existing cross-modal work on drug response & tabular prediction

​

​

​

42

PDEBench for scientific ML

• ORCA outperforms PINN and U-Net on all evaluated datasets and beats FNO on half

43

PDEBench for scientific ML

• ORCA achieves zero-shot super-resolution when using RoBERTa + pointwise conv embedder

44

OpenML Tabular Classification

• ORCA ranks 1st on 12/30 tasks, matches the performance of AutoGluon

45

Empirical results

Two backbones: RoBERTa [Liu et al., 2019], Swin Transformer [Liu et al., 2021]

Reference proxy datasets: CoNLL-2003 (entity recognition), CIFAR-10

​

• A breadth of modalities : NAS-Bench-360
• Two benchmarks that we analyze in depth
• PDEBench
• OpenML-CC18 Tabular Classification
• Compare with existing cross-modal work on drug response & tabular prediction

​

​

​

46

Compare with task-specific cross-modal methods

• IGTD [Zhu et al., 2021]: convert tabular drug-gene features into pixel representation & apply CNN to predict drug response

47

Compare with task-specific cross-modal methods

• LIFT [Dinh et al., 2022] transforms tabular data into text to prompt a pretrained GPT-3

48

Future directions

• Why ORCA works?
• High-dimensional problems and reinforcement learning
• ORCA 2.0: automating architecture & weight selection

49

Discussion: pretrained models for scientific ML

• Develop modality-specific ones
• Architecture?
• Annotated data?
• Unsupervised learning scheme?
• Synthetic data?

​

• Exploit resources in other modalities
• Models (e.g. LLMs for protein sequence representation learning [Vinod et al., 2023])
• Data

50

Thanks for listening!

Paper: https://arxiv.org/abs/2302.05738

Code: https://github.com/sjunhongshen/ORCA

​

51