LoRA on the Go:
Instance-level Dynamic LoRA Selection and Merging
Seungeon Lee, Soumi Das, Manish Gupta and Krishna P. Gummadi
Why training & data-free?
- No single LoRA fits all real inputs�- Real-world queries are unseen or span multiple domains.
- One input often needs several LoRAs�- combining related adapters helps more than any single one.
Given a pool of pre-trained LoRAs, how can we select and combine the right ones for each input, without any training or per-task data?
- Run one probe pass with all adapters attached
- Score each adapter by its activation signal
- Select top-k adapters and merge by signal-weighted sum
- Evolving LoRA pool�- Adapters are added or deprecated constantly, so any trained router must be re-trained.�
- Per-LoRA data maintenance is inefficient�- Selection by retrieval/similarity needs each adapter's training data on hand.
- Similar tasks activate adapters similarly: clear block-diagonal pattern (red boxes).
- Simple signals (norm / entropy) capture this relevance.
- We train 260 LoRAs* per model on FLAN-v2 tasks, across 3 models (LLaMA-3.1-8B, Qwen-2.5-7B, DeepSeek-7B).
- We evaluate on 27 datasets spanning 5 task categories.
* Trained LoRAs released on HuggingFace (huggingface.co/archon159/LoGo-loras-collection)
Results on LLaMA-3.1-8B backbone. The reported results are averaged over each dataset category. The best results are in bold, the second-best are underlined.
- LoGo outperforms baselines on most categories, and remains comparable on the rest.
- LoGo assigns larger weights to more similar tasks (left).
- Probe-pass overhead is amortized over long generations (right).