1 of 21

Building Large-Scale Visual Augmented Q&A with Vision Language Models

Abhijeet Kumar

2 of 21

Agenda

01. Issue with Text-RAG

Pain points. �Use-cases: Visual augmented Q&A�

Multi-modal LLM & Large context length problem

02. Developing Vision-RAG

Vision Language Models.

Vision Embeddings LMs like ColPali, ColQwen

State-of-the-art Late Interaction

03. Challenges in Scaling

1. Storing multi-vectors ?

2. In-memory Late Interaction

3. Inflated memory�

04. Building Scalable Visual Q&A

Making conventional Vector-DB work.

Late interaction Re-ranker

Pooling vectors�

05. Benchmark Results & Takeaways

Benchmark results on ViDoRe

State of vector-DBs

2

July 18, 2025

3 of 21

Problems with Text-RAG

3

July 18, 2025

AT&T sustainability document

How much carbon reduction was enabled in Transportation by AT&T in 2023 ?

Humans convey complex information to readers using infographics (visual elements of information) such as tables, charts, graphs, images etc.

4 of 21

Problems with Text-RAG

4

July 18, 2025

Alphabet 10-Q Form (2024)

What is the quarterly revenue from Google Cloud for 2023 and 2024 ?

Text-RAG systems can not process beyond text.

Traditionally, large effort went into OCR, document layout detection, and table extraction etc.

5 of 21

Multi-modal LLMs: Understands visual elements

5

July 18, 2025

AT&T sustainability document

How much carbon reduction was enabled in Transportation by AT&T in 2023 ?

The pie chart shows that the transportation sector accounts for 17.6% of the total carbon reductions enabled by AT&T. To find the specific amount of carbon reduction in the transportation sector, we need to calculate 17.6% of the total 39.1 million metric tons CO2e.

Answered by Qwen 2.5VL-3B

6 of 21

Multi-modal LLMs: Understands visual elements

6

July 18, 2025

Alphabet 10-Q Form (2024)

What is the quarterly revenue from Google Cloud for 2023 and 2024 ?

The revenue from Google Cloud for 2023 was $7,454 million, and for 2024 it was $9,574 million.

Answered by Qwen 2.5VL-3B

7 of 21

Visual LMs: Encodes visual elements

7

July 18, 2025

Source: Qdrant blog

Visual LMs encodes a page into vision embedding.
Visual LMs are jointly encoded on text and image embeddings during pre-training.

How ?

Converts a page in fixed sized 32X32 patches.
Each patch is encoded by ColPali model into 128 dim embeddings.
Each page (chunk) is a multi-vector representation i.e 1024X128.
Text query is represented by tokens X 128

Late Interaction Scoring Mechanism�

Computes retrieval score using multi-vector representation of both the text query and document patches. (Colpali)

Late Interaction is an in-memory operation i.e. all page patches and text vectors are required for similarity computation.

8 of 21

Transitioning from text-RAG to Vision-RAG

8

July 18, 2025

All models deteriorate after a certain number of tokens. (Lost in the middle, RULER, Long context - LLM Performance)
Finding needle in the haystack problem (High search space)

Question

Text embedding models (qwen3-embedding, gemini-embedding, text-embedding-3-large, cohere & jina)

Corpus

Large Language Model

(Claude, GPT4, Llama, Mixtral, DeepSeek)

Response Generation

Vector DB

Conventional RAG system

9 of 21

Transitioning from text-RAG to Vision-RAG

9

July 18, 2025

All models deteriorate after a certain number of tokens. (Lost in the middle, RULER, Long context - LLM Performance)
Finding needle in the haystack problem (High search space)

Question

Text embedding models (qwen3-embedding, gemini-embedding, text-embedding-3-large, cohere & jina)

Corpus

Large Language Model

(Claude, GPT4, Llama, Mixtral, DeepSeek)

Response Generation

Vector DB

Conventional RAG system

Can we replace this with Multi-modal LLMs ?

Multi-modal LLMs

10 of 21

Transitioning from text-RAG to Vision-RAG

10

July 18, 2025

All models deteriorate after a certain number of tokens. (Lost in the middle, RULER, Long context - LLM Performance)
Finding needle in the haystack problem (High search space)

Question

Text embedding models (qwen3-embedding, gemini-embedding, text-embedding-3-large, cohere & jina)

Corpus

Multi-modal LLMs

(Sonnet 3.5, Qwen-2.5 VL, Llama 3.2 Vision, GPT4o, Gemini Pro)

Response Generation

Vector DB

Conventional RAG system

Are there Embedding Models which can encode Visual Information along with text ?

Vision Language models encodes visual elements along with textual information which can be used for complex documents retrieval.

11 of 21

Developing Vision-RAG based Q&A System

11

July 18, 2025

Question

Vision embedding models (ColPali, ColQwen, Jina Clip-V1, Nomic-AI, Azure-AI multimodal embeddings text)

Corpus

Multi-modal LLMs

(Sonnet 3.5, Qwen-2.5 VL, Llama 3.2 Vision, GPT4o, Gemini Pro)

Response Generation

Vector DB

Vision based RAG system

RETRIEVER

GENERATOR

12 of 21

Scaling Vision-RAG

13 of 21

Challenges with Scaling

13

July 18, 2025

1 parameter = 2 bytes (16-bit float)

For each page,

1024 X 128 parameters = 2 x 128KB = 256KB

Corpus of 1000 pages,

1000 X 256 KB per page = 256MB

How about a million pages ?

Let’s do a quick math....

There are majorly two problems with scaling Visual Language Models

How do we store multi-vector embeddings ?

Earlier a text chunk is stored as single dim vector.
Easy cosine scoring, Matching meant relevant text chunk found.
What does a patch match mean here?
Can we aggregate matched patches to page ?

Needs huge in-memory computation.

State-of-the-art Late interaction mechanism needs all vectors in memory (check Math)
How do we leverage scalable ANN indexing methods like HSNW ?

Inflated Index memory ?

In text-RAG, 1 page is ~ 2 KB
In vision-RAG, 1 page is 256 KB
Can we reduce vector DB storage ?

14 of 21

Scaling Visual Q&A: Using Vector DB

14

July 18, 2025

Question

Query Embeddings

Multi-Modal LLM

ColPali

Re-ranker

Response (source)

ColPali VLM

2D Patch Embeddings

Open Search

PDF Reports

Document Indexing Process

GPT4o, Claude Sonnet 3.5

1D patches with patch id, page id

similar pages

Top pages

Hybrid search

patch embedding.

patch num.

page num.

document name

15 of 21

Scaling Visual Q&A: Using Vector DB

15

July 18, 2025

Question

Query Embeddings

Multi-Modal LLM

ColPali

Re-ranker

Response (source)

ColPali VLM

2D Patch Embeddings

Open Search

PDF Reports

Document Indexing Process

GPT4o, Claude Sonnet 3.5

1D patches with patch id, page id

similar pages

Top pages

Hybrid search

patch embedding.

patch num.

page num.

document name

Let’s uncover the workaround

16 of 21

Uncovering the workaround

16

July 18, 2025

ColPali Late Interaction Re-ranker

For each token embedding in query

Fetch Page ID of matching patches

&

Add page ID to list

Open Search

flattened patch embedding & metadata with patch id, page id

Match query

(Hybrid search)

patch embedding.

patch num.

page num.

document name

Top scoring patches (>0.90)

Yes

stop

end of token

iteration end

Filter Query

(Page IDs)

Top pages

Issue of Inflated memory of Index remains !!

17 of 21

Index Optimization: Token Pooling

Token Pooling: Reducing Memory Footprint�

Concept

Many image patches share redundant information, e.g. white background patches.

By pooling these patches together, we can reduce the amount of embeddings while retaining most of the page's signal.

Clustering based pooling - based on the cosine distance between vectors.

Pooling Factor 2 → 50% reduction in memory.

We observe a starker degradation from pooling factors of 4 onwards.

17

July 18, 2025

Research showed a pooling factor of 3 achieves an average performance degradation of less than 2%, despite reducing storage requirements by over 66%.

18 of 21

ViDoRe Benchmark

18

July 18, 2025

ViDoRe, composed of various page-level retrieval tasks spanning multiple domains, languages, and practical settings.

There are 10 datasets consisting on academic and practical retrieval tasks with 8K pages.

We did following experiments:

ColPali as retriever (in-memory)
Open-search as retriever (1st pass), ColPali as in-memory re-ranker (2nd pass)
Open-search (retriever) - Colpali (re-ranker) with Pool Factor 2

19 of 21

Benchmark Results: ViDoRe

19

July 18, 2025

	ArxivQ	DocQ	InfoQ	TabF	TATQ	Shift	AI	Energy	Gov	Health	Avg Score
Colpali 1.2 (in-memory)	69.2	42.4	73.2	83.57	52.61	60.6	96	92	84	92	74.5
Colpali 1.2 - OpenSearch	68.4	42.4	73.2	83.57	52.07	62	96	92	84	92	74.5
ColPali 1.3 (in-memory)	77.6	49.4	79.8	81.4	57.1	63	97	94	91	93	78.33
ColPali 1.3 (in-memory) - pool factor 2	77.8	44.6	75.4	76.4	55.8	61	97	98	87	94	76.7
Colpali 1.3 - opensearch - pool factor 2	77.8	44.6	75.4	76.4	55.3	61	96	93	87	94	76.05

With open-search as 1st pass retriever & 2nd pass in-memory re-ranker, there is negligible performance degradation.�
A pooling factor of 2 achieves an average performance degradation of ~2%, despite reducing storage requirements by over 50%.

20 of 21

Current State of Vector DBs

Qdrant vector DB has an offering with ColPali, ColQwen integration (Jan, 2025)

Weaviate DB started offering multi-vector embedding support from v1.29 (Feb, 2025)

Milvus vector DB completed integration with ColPali (multi-vector) by May, 2025.

Elasticsearch has rolled out ColPali implementation in March, 2025

20

July 18, 2025

21 of 21

Thank you

21

July 18, 2025