All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages �(CVPR 2025 Highlight)

Ashmal Vayani, Dinura Dissanayake, Hasindri Watawana, Noor Ahsan, Nevasini Sasikumar, Omkar Thawakar, … , Aman Chadha, Hisham Cholakkal, Rao Muhammad Anwer, Michael Felsberg, Jorma Laaksonen , Thamar Solorio, Monojit Choudhury , Ivan Laptev, Mubarak Shah, Salman Khan, Fahad Shahbaz Khan

​

University of Central Florida, Mohamed bin Zayed University of AI, Amazon, Aalto University, Australian National University, Linköping University

https://mbzuai-oryx.github.io/ALM-Bench/

• Motivation
• Problem Statement
• Dataset Comparison
• Dataset Curation
• Experimental Setup
• Evaluation and Results
• Qualitative Figures

Overall Picture

• LMMs lack global inclusivity - biased toward high-resource languages.
• Current benchmarks lacks evaluation of underrepresented cultures.
• Lack of diverse question types, cultural domains and language coverage.
• Need for natively verified, multilingual benchmarks to ensure cultural inclusivity.

Motivation

• Existing benchmarks focus on high-resource languages, overlooking cultural richness of low-resource languages.
• The benchmarks are often unfair - limited to a single question type only (MCQ).
• Existing works ignores the cultural inclusivity in LMMs from Multilingual perspective.
• ALM-Bench fills this gap: Global visual cultural benchmark for 100 languages.

​

​

Problem

ALM-Bench Overview

• Contains 22.7K culturally diverse multilingual LMM questions in 100 languages.
• Contains 19 generic and culture-specific categories.
• Four diverse question types (MCQs, Short QAs, Long QAs, True/False.)
• Over 800 hours of native-language expert annotations.
• Over 60 volunteers from 53 countries.
• Benchmarked 16 LMMs (open-source and closed-source), identifying performance gaps and areas for improvement.

Various Question Types

Benchmark Comparison

Dataset Statistics

• ALM-Bench Spans:
• 100 Languages
• 73 Countries
• 24 Language Scripts
• 15 Language Families

Dataset Curation

Dataset Curation – Generic QAs

• Generic VQA Pairs:
• LLaVA-Bench-in-the-Wild dataset. [1]
• 6 Categories (Indoor, Outdoor, Food Items, Memes, Painting, Sketch).

​

Outdoor

Food

Meme

Painting

Indoor

Sketch

[1] Liu, Haotian, et al. "Visual instruction tuning." Advances in neural information processing systems 36 (2024).

Dataset Curation

Dataset Curation – Cultural QAs

Dataset Curation – Cultural QAs

• Web-scraping using Google Search Engine.
• Pass a Country-Language Pair in search query.
• Eg. “{language} {cultural domain description} in {country}.”
• Selected images only from public and open licenses.

Dataset Curation – Cultural QAs

Dataset Curation – Cultural QAs

Dataset Curation – Cultural QAs

• Manual Filtration of irrelevant cultural images.
• Blurring PIDs (such as face, watermarks) using PicdeFacer tool.
• Fetch other metadata (eg. Image title, Image captions) from retrieved images.

PicdeFacer: https://picdefacer.com/en/

Dataset Curation – Cultural QAs

Dataset Curation – Cultural QAs

Prompt Includes:

• Image
• Added Caption
• Language
• Country
• Cultural Category

Dataset Curation – Cultural QAs

Dataset Curation – Cultural QAs

User Interface

Guidelines:

• Question Relevance: Question cannot be answered without the use of VLM.
• Where is UCF Located? ❌
• Where is the place shown in the image located? ✅

​

• Translation Verification: If either translation (Q/A) is incorrect,
• Identify the translation error.
• Rewrite the correct one.

GPT-4o Translation Issues - Human Verification

Most Errors

Evaluation Setup

Various Question Types:

• Different prompts for different question types.

​

Evaluation Criterion:

• MCQs: Accuracy
• True/False: Accuracy
• Short QA: Correctness
• Long QA: Consistency, Fluency, Relevance

​

​

Evaluation Judge

Scoring

• GPT-4o as a Judge.
• Score between 0-10.
• Compare ground truth with predicted answer.
• Results verification from native experts.

Benchmarking LMM on ALM-Bench

Benchmarking LMM on ALM-Bench

• List of 100 languages (x-axis).
• List of open-source and closed-source models (y-axis).
• Performance of model on a language in each respective box.
• Higher color intensity represents higher results and vice versa.
• 16 LMMs (14 open-source, 2 closed-source).

Benchmarking LMM on ALM-Bench

Analysis of ALM-Bench Evaluation

• Overall Results:
• Closed-Source models (GPT-4o and Gemini-1.5 Pro) performed higher than the open-source models.
• Best open-source model was GLM-4V-9B (51.9%)
• Best close-source model was GPT-4o (78.8%)
• Both open- and closed- sourced models struggled with Amharic, Kinyarwanda, Burmese, and Sanskrit.

Analysis of ALM-Bench – Language Scripts

• Both GPT-4o and Qwen2-VL struggles on low-resource languages.

​

• For eg. Ge’ez (Amharic), Sinhalese (Sinhala), Oriya (Odia), Myanmar (Myanmar-Burmese).

​

• Native speakers Error Analysis on the results for analysis.

Error Analysis on Language Scripts

Qualitative Examples of Error Analysis on Scripts

Correct Cases

Incorrect Cases

Qualitative Examples of Error Analysis on Scripts

Incorrect Cases

Analysis of ALM-Bench – Language Families

• Both GPT-4o and Qwen2-VL struggles on African Language Families.

​

• Eg. Language family African-Congo (Igbo, Kinyarwanda, Swahili, Yoruba).

​

• Evaluated 15 Language Families.

Analysis of ALM-Bench – Question Types

• Closed-source models perform better on Long VQAs (LVQA).

​

• Open-source models perform better on Short VQAs (SVQAs).

​

• Both fare better on Decision Making Questions (MCQs, T/F).

Analysis of ALM-Bench – Cultural Categories

• Closed-source model, GPT-4o outperforms with 80.3%.

​

• GPT-4o achieves 83.7% on Education and heritage but drops to 72.7% on Notable key figures category.

​

• Categories such as Notable Key Figures and Customs are often culturally specific and under-represented in training datasets.

Analysis of ALM-Bench – Location Aware Prompts

• Adding country related information in the prompts for LMM evaluation.

​

• Closed-source models like GPT-4o and Gemini-1.5-Pro better utilizes the added geographical context.

​

• Open-source models do not utilize this information well.

ALM-Bench Insights

• Near equal distribution for all the cultural categories.

​

• Economy category was difficult to find cultural samples.

​

• Verification done by native speakers, removing culturally irrelevant information.

Annotators Demographics

• Over 60 volunteers for the research project.

​

• Volunteers represent over 50 countries.

​

• One-Fourth presence of female volunteers.

​

• Over 46% fall in 18-25 age limit.

​

• Over 80% speakers were either native speakers or bilingual.

​

• Over 88.5% annotators are culturally familiar with their languages.

​

• Over 87.9% of annotators have lived in the country from which cultural examples are from.

Qualitative Examples of ALM-Bench

Qualitative Examples of ALM-Bench

Qualitative Examples of ALM-Bench

Word Cloud of Categories

Summary

• We introduce ALM-Bench, novel multilingual multimodal cultural benchmark.

​

• We propose over 22.7k humanly verified examples across 19 domains.

​

• ALM-Bench spans across 73 countries, 24 language scripts, 15 language families.

​

• We benchmark 16 LMMs with four different question types (MCQs, T/F, SVQA, LVQA).

​

• We highlight several important insights to curate a better culturally diverse pre-training dataset.

Thank you for listening.��Any questions?

Ashmal Vayani

University of Central Florida

Masters in Computer Vision

linkedin.com/in/ashmal-vayani/

https://ashmalvayani.github.io/