Relevance-aware Multi-context Contrastive Decoding for Retrieval-augmented Visual Question Answering
�Presented at WACV 2026�
Jongha Kim, Byungoh Ko, Jeehye Na, Jinsung Yoon, Hyunwoo J. Kim
1
Korea University
MLV Lab
NeurIPS 2024
Retrieval-Augmented Generation for Visual Question Answering
In Retrieval-Augmented Generation (RAG), relevant contexts are retrieved from external knowledge bases and fed to an MLLM to supplement missing or incomplete knowledge.
[1] Chen et al. ”Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions”, EMNLP 2023
Question samples of preivous VQA dataset (top) and InfoSeek (bottom)
MLV Lab
Korea University
MLV Lab
NeurIPS 2024
Not all retrieved contexts are equally relevant
Although multiple contexts are retrieved, their relevance varies across samples: some are highly relevant, while others are only weakly relevant or entirely irrelevant.
[1] Wang et al. ”REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering”, EMNLP 2024
Example of retrieved results with different level of relevance
MLV Lab
Korea University
MLV Lab
NeurIPS 2024
Goal: Effectively leveraging multiple retrieved contexts
Previous MLLMs treat every retrieved context equally, agnostic to their relevance. Thus, we propose a new decoding method leveraging multiple contexts considering their relevance.
Accuracy of baseline and baseline + RMCD (ours) by the number of retrieved contexts
MLV Lab
Korea University
MLV Lab
NeurIPS 2024
RMCD: Relevance-aware Multi-context Decoding
Using the retrieval scores, we compute context weights that reflect each context’s relative importance: relevant contexts receive larger weights, while irrelevant ones receive smaller or even negative weights.
Overall pipeline of RMCD
MLV Lab
Korea University
MLV Lab
NeurIPS 2024
RMCD: Relevance-aware Multi-context Decoding
Then, we aggregate the output logits from each retrieved context using the computed weights, amplifying the influence of relevant contexts while suppressing (or deflecting) irrelevant ones.
Overall pipeline of RMCD
MLV Lab
Korea University
MLV Lab
NeurIPS 2024
Quantitative Results
By replacing decoding methods of baselines, RMCD consistently improves accuracy across multiple MLLMs and knowledge-intensive benchmarks.
Accuracy on InfoSeek / Encyclopedic-VQA (left) and OK-VQA (right)
MLV Lab
Korea University
MLV Lab
NeurIPS 2024
Cost Analysis
While utilizing same number of contexts, RMCD maintain or improves latency, while improving QA accuracy. Moreover, RMCD does not require additional training, as it is only applied during inference.
Accuracy on OK-VQA (left) and latency comparison with baselines (right)
MLV Lab
Korea University
MLV Lab
NeurIPS 2024
Performance by retrieval quality
RMCD consistently outperforms every other decoding methods regardless of the retrieval quality, showing the best accuracy under both the best and worst retrieval results.
Retrieval results (left) and accuracy by decoding methods (right) by knowledge base sizes.�Retrieval gets difficult in larger knowledge base as portion of irrelevant corpus increases.
MLV Lab
Korea University
MLV Lab
NeurIPS 2024
Performance with oracle retriever
RMCD yields additional accuracy gains even with an oracle retriever, where the ground-truth is always ranked top-1, demonstrating that it complements retriever improvements.
Accuracy with oracle retriever, where ground-truth corpus is always set as top-1 retrieval result.
MLV Lab
Korea University
MLV Lab
NeurIPS 2024
Qualitative Results
Higher weights are assigned to relevant contexts, while negative weights are assigned to irrelevant contexts which distract baseline decoding methods.
MLV Lab
Korea University
MLV Lab
NeurIPS 2024
Conclusion
Github
Paper
MLV Lab
Korea University
MLV Lab
NeurIPS 2024