�Birds of a Feather�Information Retrieval

NAACL 2021

Luca Soldaini �(he/him)�Amazon Alexa AI

Sean MacAvaney�(he/him)�University of Glasgow

(recently: IR Lab @ Georgetown)

Session Structure

• Intro and code of conduct (3 mins)
• Overview of suggested topics (7 minutes)
• Two breakout sessions (20 minutes each)
• Final discussions and closing remarks

Code of Conduct

NAACL 2021 adheres to the ACL’s code of ethics and ACL’s anti-harassment policy. If you would like to raise a complaint under the anti-harassment policy or have related concerns you would like to discuss, follow instructions at bit.ly/naacl_coc. Concerns regarding possible violations of the code of ethics may be brought to any current member of the NAACL Executive Board.

Goals of a BoF session

• Meet researchers in the field
• Exchange ideas, discover new topics to work on
• Share what you are working on, get feedback
• Find new collaborators/mentors

Topic #1: What is left in neural IR?

• A lot of ad-hoc retrieval work has been focused on neural models
• Both full ranking and re-ranking models have been extensively studied
• What do you think are the next opportunities in Neural IR?
• Are efficiency problems “solved” with dense retrieval?
• Neural rankers for specific tasks?
• Can neural IR improve other NLP tasks?

Topic #2: Where is the training data?

• Ad-hoc IR gravitates around the same few datasets, particularly for training (e.g., MSMARCO, TREC)
• What biases might these datasets introduce?
• What are the challenges in creating new datasets?
• Can collaborative efforts be set up among institutions to obtain new datasets?
• Can heuristics and distant supervision be used to obtain reliable data?

Topic #3: Where’s non-English IR?

• Compared other text-based AI work, IR has remained shockingly centered around English
• Why aren’t there more non-English tasks?
• What are some interesting approaches in for IR in other languages? What are the challenges?
• Is there hope for multilingual and cross-lingual models?
• Beyond text: sign language retrieval?

Topic #4: Interesting domain-specific IR

• Many successful applications of IR to different fields
• E.g., in the last year, retrieval for COVID literature
• What are some interesting domain specific applications worth studying?
• Any success stories?
• Interesting collaborations with out-of-domain experts?
• Can there be a “BM25 of Neural IR” or do deep learned models inherently enforce a task/domain?

Topic #5: Users in IR

• Many IR applications naturally revolve around users: Conversational IR, Session Search, etc.
• What are the current challenges in user-based IR? Data acquisitions? Simulations? Any approaches to recommend here?
• How can we best support research in this area? Are shared tasks worth it?

Topic #6: Bias and Interpretability in IR Models

• Traditional IR axioms are being deposed by neural IR models that make less interpretable and more biased decisions
• What makes models “interpretable”? How might the proposed EU AI regulations affect this area of research?
• Is there a “right” way to measure these biases?
• What are the most important biases to focus our effort correcting?
• Can we fix biased models without affecting ad-hoc ranking effectiveness? How biased are our test collections?

Breakout Rooms

Time: 20 minutes

• Brief round of intros (name, pronouns, affiliation, area of research)
• Discuss one or more topics highlighted in the presentation, or bring your own!
• Focus on sharing knowledge and encourage all to join �the conversation