What are we missing in Algorithmic Fairness?��Discussing Open Challenges for Fairness Analysis in�User Profiling with Graph Neural Networks

Erasmo Purificato and Ernesto William De Luca

Fourth International Workshop on Algorithmic Bias in Search and Recommendation (Bias 2023) @ ECIR 2023

2^nd April 2023, Dublin, Ireland

Background

2

E. Purificato, E.W. De Luca. What are we missing in Algorithmic Fairness? Discussing Open Challenges for Fairness Analysis in User Profiling with Graph Neural Networks. In Proceedings of the 4^th International Workshop on Algorithmic Bias in Search and Recommendation. April 2^nd, Dublin, Ireland.

Background

Algorithmic Fairness

Uncovering and rectifying biases in statistical and machine learning models, which comes from the unjustified differences in models’ performance along social axes, e.g. race and gender.

(Mitchell et al., 2021)

3

E. Purificato, E.W. De Luca. What are we missing in Algorithmic Fairness? Discussing Open Challenges for Fairness Analysis in User Profiling with Graph Neural Networks. In Proceedings of the 4^th International Workshop on Algorithmic Bias in Search and Recommendation. April 2^nd, Dublin, Ireland.

Background

Potential roots of unfairness

​

1. Biased data

4

Potential roots of unfairness

​

2. Algorithms receptive to the biases already present in the datasets used for training

​

E. Purificato, E.W. De Luca. What are we missing in Algorithmic Fairness? Discussing Open Challenges for Fairness Analysis in User Profiling with Graph Neural Networks. In Proceedings of the 4^th International Workshop on Algorithmic Bias in Search and Recommendation. April 2^nd, Dublin, Ireland.

Background

Graph Neural Networks (GNNs) falls in the second category.

5

GNN

E. Purificato, E.W. De Luca. What are we missing in Algorithmic Fairness? Discussing Open Challenges for Fairness Analysis in User Profiling with Graph Neural Networks. In Proceedings of the 4^th International Workshop on Algorithmic Bias in Search and Recommendation. April 2^nd, Dublin, Ireland.

Background

As any ML system trained on historical data, GNNs are prone to learn biases in such data and reveal them in their output.

6

E. Purificato, E.W. De Luca. What are we missing in Algorithmic Fairness? Discussing Open Challenges for Fairness Analysis in User Profiling with Graph Neural Networks. In Proceedings of the 4^th International Workshop on Algorithmic Bias in Search and Recommendation. April 2^nd, Dublin, Ireland.

Background

User profiling

Infer an individual’s interests, personality traits or behaviours from generated data to create an efficient representation, i.e. a user model.

7

E. Purificato, E.W. De Luca. What are we missing in Algorithmic Fairness? Discussing Open Challenges for Fairness Analysis in User Profiling with Graph Neural Networks. In Proceedings of the 4^th International Workshop on Algorithmic Bias in Search and Recommendation. April 2^nd, Dublin, Ireland.

Background

Existing approaches evaluate user profiling models as a classification task.

8

E. Purificato, E.W. De Luca. What are we missing in Algorithmic Fairness? Discussing Open Challenges for Fairness Analysis in User Profiling with Graph Neural Networks. In Proceedings of the 4^th International Workshop on Algorithmic Bias in Search and Recommendation. April 2^nd, Dublin, Ireland.

Challenges

Common characteristics in most fairness assessments

• The fairness metrics are applied in classification scenarios where both the target class and the sensitive attribute are binary;
• The capability of a model to produce fair results is evaluated considering the absolute difference of the scores of the two sensitive groups considered.

9

E. Purificato, E.W. De Luca. What are we missing in Algorithmic Fairness? Discussing Open Challenges for Fairness Analysis in User Profiling with Graph Neural Networks. In Proceedings of the 4^th International Workshop on Algorithmic Bias in Search and Recommendation. April 2^nd, Dublin, Ireland.

Fairness metrics adopted

Statistical Parity (SP)

​

10

Equal opportunity (EO)

​

E. Purificato, E.W. De Luca. What are we missing in Algorithmic Fairness? Discussing Open Challenges for Fairness Analysis in User Profiling with Graph Neural Networks. In Proceedings of the 4^th International Workshop on Algorithmic Bias in Search and Recommendation. April 2^nd, Dublin, Ireland.

Analysed contributions

Dai and Wang (WSDM 2021)

​

Model proposed and analysed:

• FairGNN

​

​

11

Purificato et al. (CIKM 2022)

​

Models analysed:

• CatGCN (Chen et al., 2021)

​

​

​

• RHGN (Yan et al., 2021)

E. Purificato, E.W. De Luca. What are we missing in Algorithmic Fairness? Discussing Open Challenges for Fairness Analysis in User Profiling with Graph Neural Networks. In Proceedings of the 4^th International Workshop on Algorithmic Bias in Search and Recommendation. April 2^nd, Dublin, Ireland.

Case study – absolute difference

12

Ethical implication 1. Considering the absolute difference score in fairness analysis can be hazardous. From both a system and user perspective, with this practice, we cannot clearly figure out the disadvantaged groups for every specific combination of model, dataset and fairness metrics. Thus, we cannot make in place any tailored intervention to mitigate the issue in a real-world scenario.

E. Purificato, E.W. De Luca. What are we missing in Algorithmic Fairness? Discussing Open Challenges for Fairness Analysis in User Profiling with Graph Neural Networks. In Proceedings of the 4^th International Workshop on Algorithmic Bias in Search and Recommendation. April 2^nd, Dublin, Ireland.

Case study – binary scenarios

13

Ethical implication 2. It is essential to evaluate fairness by examining the actual distribution of sensitive groups. Firstly, if the system at hand is not as effective for certain groups, they will end up receiving less effective services, such as targeted advertisements or recommendations. Secondly, reducing the different classes and groups into a binary representation can lead to an incorrect evaluation of the fairness of models, potentially distorting the original data conditions.

E. Purificato, E.W. De Luca. What are we missing in Algorithmic Fairness? Discussing Open Challenges for Fairness Analysis in User Profiling with Graph Neural Networks. In Proceedings of the 4^th International Workshop on Algorithmic Bias in Search and Recommendation. April 2^nd, Dublin, Ireland.

Conclusion and future directions

• Posed and discussed two potential open challenges in recent studies on Algorithmic Fairness.
• Conducted two case studies on GNN-based models for User Profiling.
• Presented our position arguing in favour of a multiclass assessment with a clear understanding of the disadvantaged groups.
• Exposed some ethical implications which derive from the experimental results.
• In future work, we will explore the discussed aspects of fairness analysis by proposing solutions for multi-class and multi-group scenarios.

14

E. Purificato, E.W. De Luca. What are we missing in Algorithmic Fairness? Discussing Open Challenges for Fairness Analysis in User Profiling with Graph Neural Networks. In Proceedings of the 4^th International Workshop on Algorithmic Bias in Search and Recommendation. April 2^nd, Dublin, Ireland.

Thanks!

I am Erasmo Purificato

You can find me at:

https://erasmopurif.com

@erasmopurif11

erasmo.purificato@ovgu.de

​

PAPER