CIRAL @ FIRE 2023:

Track Overview

Track Organizers: Mofetoluwa Adeyemi, Akintunde Oladipo, Xinyu Crystina Zhang, David Alfonso-Hermelo, Mehdi Rezagholizadeh, Boxing Chen, Jimmy Lin

David R. Cheriton School of Computer Science, University of Waterloo

Noah’s Ark Lab, Huawei Technologies

2

CIRAL: Cross-lingual Information Retrieval for African Languages

​

​

Goal: To promote involvement and research in CLIR for African languages.

Background

Cross-lingual Information Retrieval (CLIR): Retrieval where query is in one language and documents in another.

3

1. Ranar Arafah biki ne na Musulunci wanda ya zo a ranar 9 ga watan Zu al-Hijjah na…
2. Ranar tara ga Zul Hijjah ita ce ranar Arafah. Yana kan wannan…
3. ….

When is the day of Arfa?

English Query

Hausa Documents

Background

CLIR Test Collections with African languages:

• IARPA MATERIAL (2)
• Large Scale CLIR (1)
• CLIRMatrix (5)
• AfriCLIRMatrix (15)

CLIR Tasks in Information Retrieval:

• TREC 1997 - 2001
• NTCIR
• CLEF
• FIRE
• NeuCLIR @ TREC

However;

• There were no such CLIR tasks at conferences or workshops for African languages.
• Existing collections were mostly collected synthetically.

4

Research Questions

1. What retrieval methods are most effective in CLIR for African languages?
2. How effective is indigenous textual data in CLIR for African languages, and in the curation of test collections?
3. What is the effect of machine translation techniques, and how do they compare to methods implementing end-to-end CLIR?
4. How do multi-stage approaches, i.e., reranking, compare to single-stage approaches, i.e., dense, sparse and hybrid in CLIR for African languages?

5

Task

• Cross-lingual Passage Ranking task

• English and 4 African Languages:
• Hausa
• Somali
• Swahili
• Yoruba
• Binary relevance: 1 for relevant, 0 for non-relevant.
• Metrics: nDCG@20, Recall@100, MRR@10 and MAP

6

Dataset

Corpora

• News articles obtained from Indigenous websites for each language
• Articles chunked to passage-level:
• Hausa: 715,355
• Somali: 827,552
• Swahili: 949,013
• Yoruba: 82,095

7

docid: Unique identifier

title: Article title

text: Passage body

url: Link to news article

Dataset

Queries

• Queries developed as natural language factoid questions.
• Queries model the interests of the language speaker.
• Developed in African language using the MasakhaNews dataset, then translated to English.

​

Train and Test splits

• 10 Sample Train queries along with qrels per language
• Test queries:
• Hausa: 85
• Somali: 100
• Swahili: 85
• Yoruba: 100

8

Relevance Assessment

• Manual assessment of submissions to form query pools
• Depth k = 20 for submissions.
• Binary relevance also used for assessments
• Average query pool size per query is 83 to 95 assessments.

9

Participation and Methods

• Participating Teams
• John Hopkins University HLTCOE
• Masakhane
• h2oloo

• 84 runs submitted, 21 for each language.
• Runs consisted mostly of dense and reranking methods, with minimal sparse and hybrid submissions.

10

Participation and Methods

• John Hopkins University HLTCOE
• JH Polo Technique
• Translate-train on ColBERT-X models for African languages.
• Masakhane
• Multiple Afrocentric language models, i.e., AfriBERTa, AfriBERTA-v2, AfroXLMR, trained as mDPRs.
• h2oloo
• Hybrids of dense methods.
• Reranking approaches with multilingual T5 models.

11

Evaluation and Results

12

Evaluation and Results

13

Summary

• The CIRAL track held for the first time at the Forum for Information Retrieval Evaluation, and is a step towards shared tasks for CLIR in African languages.
• CIRAL covered four (4) African languages, for which test collection was curated.
• Diversity in submissions and participation is very important in the development of methods and curation of the test collection.

14

Thank You!

15

Questions?

16