Adaptive Budget Allocation for �Parameter-Efficient Fine Tuning��Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheg He, Yu Cheng, Weizhu Chen, Tuo Zhao ��ICLR 2023 � �

Presented by: Meghna Kalra

Electrical and Computer Engineering

Introduction to Language model and fine tuning

• Pretrained language model (PLM)

- Model trained on a vast corpus of text to understand language patterns/contexts

- Example – ChatGPT , Bert.

​

• Fine tuning

- Adjusting a pre-trained model to specialize/adapt in a particular task.

​

• Challenges of Fine tuning

- Memory cost (separate copy for each fine – tuned task )

- Computational cost (resource intensive , billions of parameters)

​

​

Fine Tuning all the parameter uniformly is not efficient !

Parameter-Efficient Fine-Tuning Methods�

Low-Rank Adaptation (LoRA)

Instead of changing the entire weight matrix, only two smaller matrices are

adjusted !

Why Low rank works ?

The core function/behavior learned by the model resides in

a much lower-dimensional space than number of its parameters.

Applications: signal processing, optimization, computer algebra, machine learning ……..

• Low rank approximation of an image
• Full rank = 256, even with rank = 5,

we can identify Joseph Fourier.

​

Madathil, Baburaj, et al. "Tensor low rank modeling and its applications in signal processing.

Limitation of LoRA

Fine tuning FFN modules achieve better performance than self-attention.

Weight matrices in top layers are more important than those in bottom layers.

​

​

How can we allocate the parameter budget adaptively according to importance of modules to improve the performance of parameter-efficient fine-tuning ? �

ADALoRA – Adaptive Low-rank Update

• The core idea is to represent the incremental updates to the pre-trained weight matrices by using low rank approximation of Singular Value Decomposition (SVD).

​

ADALoRA – Adaptive Low-rank Update

Notations

Importance – Aware Rank Allocation

Equation for global budget scheduler

Designing sensitivity-based “importance scores”

Refining Sensitivity for Robustness

AdaLoRA Algorithm

Experimental results

• Natural language understanding - AdaLoRA applied to DeBERTaV3-base, evaluated on GLUE (General Language Understanding Evaluation) benchmark.
• Compared with the following Baseline algorithms:
• Full Fine-Tuning
• All parameters updated
• BitFit
• Only fine-tunes bias vectors – parameter efficient method
• Adapter Tuning
• Inserts two-layer adapters between Transformer blocks
• Houlsby Adapter: Between self-attention and FFN
• Pfeiffer Adapter: After FFN and LayerNorm, more efficient
• LoRA (Low-Rank Adaptation)
• Incremental updates with two small matrices

​

AdaLoRA outperforms or equals existing methods across most datasets and budget levels

Comparison to low-rank parameterization. �

AdaLoRA outperforms or equals existing methods across all low rank parameterizations

Importance scores:

Budget Distribution

AdaLoRA - Resulting rank of each incremental matrix

​

LoRA

This validates that the proposed importance metric in AdaLoRA focuses on crucial modules and layers.

​

Conclusion

IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuning��Feiyu Zhang, Liangzhi Li, Junhao Chen, Zhouqiang Jiang, Bowen Wang, and Yiming Qian��� �

Presented by: Seonho Kim

Electrical and Computer Engineering

Recap

• LoRA : gives the equal rank to each module matrix.

​

- Importance of each matrix is ignored.

​

- needs to give more rank to more important matrix.

​

​

​

​

​

​

​

​

​

• AdaLoRA : mitigate these limitations using SVD form and pruning with importance scores.

​

- upper bound on the rank for each matrix is determined by the initial rank (the sum of ranks of total matrices decreases over iteration).

​

- cannot give more rank on a matrix than the initial rank even if the matrix is very important.

​

​

​

​

​

​

​

​

​

Basic idea of IncreLoRA

• Increase the total rank automatically by giving more rank to important matrices without upper bound on rank.

​

​

​

​

​

​

​

​

​

​

​

​

​

Basic idea of IncreLoRA

• Increase the total rank automatically by giving more rank to important matrices without upper bound on rank.

​

​

​

​

​

​

​

​

​

​

​

​

​

Methodology

​

​

​

​

​

​

​

​

​

​

​

​

​

• Formulation

​

Pretrained

Weights

Methodology

​

​

​

​

​

​

​

​

​

​

​

​

​

• SVD form

​

Methodology

• Parameter Update

​

• Parameter updates are applied to all linear layers.

Methodology

• Incremental Parameter Allocation

​

SS with an exponential moving average helps to mitigate the effect of variability in the gradients across training iterations.

Methodology

• Incremental Parameter Allocation

​

Methodology

• Advanced Learning

​

Methodology

Experiments

Transformer : DeBERTaV3-base

Dataset : GLUE

​

• Observation : IncreLoRA at the same budget level is likely to occupy more ranks to FFN modules, which have four times parameters than other modules.

Experiments

• Different Budget Levels

​

• IncreLoRA at the lowest budget outperforms the all performances of LoRA in considered budgets.

Experiments

• Ablation Experiment

Experiments

• Ablation Experiment

Conclusion

• IncreLoRA improve the parameter efficiency compared with AdaLoRA.

• IncreLoRA incrementally assign trainable parameters while improving training effect and stability using advance learning.

• IncreLoRA outperforms the existing methods such as LoRA and AdaLoRA.

Remaining Questions

• Is the Importance Scoring necessary?

-> 1. It depends on hyperparameters.

2. increase the computational cost.

3. Ignore the importance of singular values.

• Is the Frobenius norm for orthogonalization the best choice?

• The regularization to promote the low-rank structure has not been explored yet.