1 of 13

Personalised Information Retrieval (PIR) 2024

Kasela Pranav, Braga Marco, Effrosyni Sokli, Gian Carlo Milanese, Georgios Peikos, Sandip Modha, Alessandro Raganato, Marco Viviani, Gabriella Pasi

Department of Informatics, Systems, and Communication (DISCo)

University of Milano-Bicocca

Milan, Italy

2 of 13

Overview

Introduction

Tasks

Evaluation

Participations

Results

Future of PIR

3 of 13

Problem

Personalization in Information Retrieval (IR):

Personalized Search is one of the main developments of IR, finalized at providing search results that better suit user’s needs.

Personalization can improve the user's perceived quality of service.

Current datasets are not suited to evaluate personalization from a user perspective

4 of 13

Task Definition

Personalized Community Question Answering (cQA)

Each user can select the best answer to his/her question

IR approaches to cQA: the question is seen as a query, and the answers are retrieved from the pool of past answers indexed for the purpose.

5 of 13

Dataset

1M questions and 2M answers provided by 600K users.

50 communities of the Stack Exchange dump between 2008-09-10 and 2022-09-25.

6 of 13

Dataset

The training, validation, and test split are done temporally to avoid any kind of data leakage

There are a total of 1 125 407 questions in the dataset

Finally, we have 822 974 training questions, 78 854 validation questions, and 99 878 test questions.

7 of 13

Task 1: Standard IR

The cQA task will be tackled as a standard ad-hoc IR task, where the questions are going to be considered as the queries, and the collection, from which the answers will be retrieved, is composed by all the answers available in the dataset

In this case, personalization can be tackled using any standard or novel technique to create a user profile and inject it into the retrieval model

Possible Approaches:

Both classical techniques such as BM25, and neural approaches based on BERT-like models

Re-rankers, using cross-encoders, like Mono-T5

Personalized baselines: using of a mix of tags and historical documents related to the users and weighted according to their importance for the current question

8 of 13

Task 2: Prompt-based IR

Prompt-based IR personalizes the results by using models like GPT, or more recent LLMs, with prompts such as the following one:

“To which degree between 0 and 1 does the document [DOCUMENT] answer the question [QUESTION], and is relevant to a user with the following profile [USER PROFILE]”,

where the [USER PROFILE] is a series of user interests that are inferred from their activities and ordered according to their timestamp (most recent first) and importance.

9 of 13

Evaluation

For both the tasks, we use traditional evaluation metrics in the IR literature that can be applied also to personalized search:

Precision (P)
Recall (R)
Mean Average Precision (MAP)
Mean Reciprocal Rank (MRR)
(normalized) Discounted Cumulative Gain (nDCG)

10 of 13

Partecipation

Registrations: 15+ registrations

2 runs submitted

2 papers submitted

11 of 13

Results: Task 1

Results of Team: Word Wizards

12 of 13

Future of PIR

We plan to offer PIR in 2025:

More tasks, i.e. Personalized Text Classification

Focus on personalization across varied domains