1 of 17

Multimodal Needle in a Haystack:�Benchmarking Long-Context Capability of�Multimodal Large Language Models

Hengyi Wang, Haizhou Shi, Shiwei Tan,

Weiyi Qin, Wenyuan Wang, Tunyu Zhang,�Akshay Nambi, Tanuja Ganu, Hao Wang�

06/27/24

2 of 17

4.

3.

2.

1.

Introduction

MMNeedle Benchmark

Experiments

Conclusion

CONTENTS

3 of 17

Introduction

1.

4 of 17

Needle-in-a-Haystack Test

Input: July 2010What hard liquor, cigarettes, heroin, and crack have in common is that they're all more concentrated forms of less addictive predecessors.

Most if not all the things we describe as addictive are. And the

scary thing is, the process that created them is accelerating.We wouldn't want to stop it. It's the same process that cures

diseases: technological progress. The best thing to do in San Francisco is eat a sandwich and sit in Dolores Park on a sunny day. Technological progress means making things do more of what we want. When the thing we want is something we want to want, we consider technological progress good.

If some new technique makes solar cells x% more efficient, that

seems strictly better. When progress concentrates something we

don't want to want—when it transforms opium into heroin—it seems

bad. But it‘s the same process at work.

Question: What is the best thing to do in San Francisco?

5 of 17

Needle-in-a-Haystack Test

6 of 17

Challenges in Multimodal LLMs

Insufficient image context length

Costly annotations on sub-image objects
Our solution – Image Stitching

7 of 17

MMNeedle Benchmark

2.

8 of 17

Key Components

Needle Sub-Image
Haystack Image Inputs
Text Inputs (Instructions and Caption)
LLM Outputs

MMNeedle Evaluation Overview. Correct answers are marked with checkmark (✓), while the incorrect answers are marked with cross (×). Our evaluation setup involves the following key components: (a) Needle Sub-Image: The needle sub-image to be retrieved based on the given caption. (b) Haystack Image Inputs: The long-context visual inputs consist of M images, each stitched from N × N sub-images. (c) Text Inputs (Instructions and Caption): Detailed instructions to MLLMs, followed by a caption describing the needle, i.e., sub-image 20. (d) LLM Outputs: The answers from different MLLMs, indicating their ability to accurately locate the needle in the haystack based on the given caption. The expected output is composed of the model's identification of the index, row, and column of the matching sub-image. The results showcase the comparative performance of various models: GPT-4o correctly predicts the exact location of the needle; Gemini Pro 1.5 only correctly predicts the image index of the needle; other API models predict incorrect locations; open-source models often output with wrong formats.

�

9 of 17

Experiments

3.

10 of 17

Long-Context Capability

11 of 17

Hallucination on Negative Examples

Table 3 shows the existence accuracy (defined in Sec. 3.5) for negative�samples (defined in Sec. 3.3). For API-based models, Claude 3 Opus and Gemini Pro 1.0 perform�well across different configurations, suggesting robustness in handling varied context-length for�the negative samples. On the other hand, GPT-4V and GPT-4o achieve inferior accuracy on more�complex settings, including multi-image inputs (M = 10) and/or large stitching size (N = 4 or�N = 8). These results reveal that: (1) Even top API-based models severely suffer from hallucination;�they incorrectly believe the needle exists in the haystack when it does not. (2) API-based models�with stronger needle-retrieval performance, e.g., GPT-4o, tend to suffer more from hallucination.�The performance of open-source models varies significantly, with some generally underperforming�compared to API-based models (e.g., CogVLM-17B, Fuyu-8B, InstructBLIP and LLaVA-Llama-�3), while others demonstrate high existence accuracy (e.g., CogVLM2-Llama-3, mPLUG-Owl-v2,�IDEFICS2-8B). Notably, IDEFICS2-8B achieves the highest accuracy of 62.00% on M = 1, N = 8�samples, indicating a low level of hallucination in this setting.

12 of 17

Multi-Needle Evaluation

13 of 17

Statistitical Significance

14 of 17

Effect of Context Length

15 of 17

Conclusion

4.

16 of 17

Links and Resources

Learderboard: https://paperswithcode.com/sota/long-context-understanding-on-mmneedle
Code: https://github.com/Wang-ML-Lab/multimodal-needle-in-a-haystack/tree/main
Paper: https://arxiv.org/pdf/2406.11230
MMNeedle Dataset: https://drive.google.com/drive/folders/1D2XHmj466e7WA4aY7zLkbdTmp3it2ZPy?usp=sharing
Hugging Face: https://huggingface.co/papers/2406.11230

We propose MMNeedle, a benchmark to evaluate MLLMs’ long-context capabilities. MMNeedle includes a comprehensive dataset and establishes diverse settings as well as a systematic set of coarse-to-fine evaluation metrics. We reveal that while API-based models, such as GPT-4o, outperform open-source models in long-context scenarios, they still struggle with hallucination issues in negative samples and challenges in large stitching size/multi-needle retrieval. A limitation of our MMNeedle evaluation is the assumption that the MLLM takes both images and texts as inputs and supports multiple-image inputs. However, we argue that these are necessary requirements for an ideal MLLM.

Please feel free to use our code and benchmark. Any suggestions, github issues, and code requests will be greatly appreciated!!

17 of 17

References

[1] G. Kamradt. Needle in a haystack - pressure testing llms.

https://github.com/gkamradt/LLMTest_NeedleInAHaystack, 2023.

[2] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.