1 of 24

End-user Algorithmic Auditing for Music Discoverability: A Research Roadmap

Lorenzo Porcaro (he/him)

MuRS: Music Recommender Systems Workshop 2024, Bari, 14.10.2024

Grant Agreement Number 101148443

2 of 24

Who am I?

  • PhD @ Music Technology Group, UPF, Barcelona�
  • European Centre for Algorithmic Transparency (EC, JRC)�
  • Marie Skłodowska-Curie Actions Postdoctoral Fellows

Grant Agreement Number 101148443

3 of 24

Discoverability

3

Grant Agreement Number 101148443

4 of 24

Discoverability

  • Policy concept (from French: découvrabilité) �“Discoverability of an item is its availability online and its ability to be found among a wide range of other content, particularly by someone who was not specifically looking for it”*�
  • Design concept�Discoverability refers to how easily users can find features, information, or functionalities within a product, service, or system.**

​

* Franco-Quebec mission on the online discoverability of French-language cultural content, Technical Report, Bibliothèque et Archives nationales du Québec, 2020. URL: https://www.vie-publique.fr/rapport/277472-rapport-sur-decouvrabilite-en-ligne-des-contenus-culturels-francophones

** Norman, D. A. (2013). The design of everyday things. The MIT Press.

4

Grant Agreement Number 101148443

5 of 24

Discoverability (cont’d)

  • Policy concept (from French: découvrabilité) �“Discoverability of an item is its availability online and its ability to be found among a wide range of other content, particularly by someone who was not specifically looking for it”*�
  • Design concept�Discoverability refers to how easily users can find features, information, or functionalities within a product, service, or system.**

​

* Franco-Quebec mission on the online discoverability of French-language cultural content, Technical Report, Bibliothèque et Archives nationales du Québec, 2020. URL: https://www.vie-publique.fr/rapport/277472-rapport-sur-decouvrabilite-en-ligne-des-contenus-culturels-francophones

** Norman, D. A. (2013). The design of everyday things. The MIT Press.

5

Grant Agreement Number 101148443

6 of 24

Low Discoverability

6

Grant Agreement Number 101148443

7 of 24

High Discoverability

7

Grant Agreement Number 101148443

8 of 24

What about music discoverability?

8

Grant Agreement Number 101148443

9 of 24

9

Grant Agreement Number 101148443

10 of 24

Algorithmic Auditing

  • Algorithm audits are a specific subset of audit studies focused on studying algorithmic systems and content.*�
  • An algorithmic audit involves the collection and analysis of outcomes from a fixed algorithm or defined model within a system. [...] these audits can uncover problematic patterns in models of interest. **�
  • An empirical study investigating a public algorithmic system for potential problematic behavior.***

​

* Metaxa, D., Park, J. S., Robertson, R. E., Karahalios, K., Wilson, C., Hancock, J., & Sandvig, C. (2021). Auditing algorithms. Foundations and Trends in Human-Computer Interaction, 14(4), 272–344. https://doi.org/10.1561/1100000083

** Raji, I. D., & Buolamwini, J. (2019). Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products. AIES ’19: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society.

*** Bandy, J. (2021). Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1). https://doi.org/10.1145/3449148

10

Grant Agreement Number 101148443

11 of 24

Algorithmic Auditing (cont’d)

  • Algorithm audits are a specific subset of audit studies focused on studying algorithmic systems and content.*�
  • An algorithmic audit involves the collection and analysis of outcomes from a fixed algorithm or defined model within a system. [...] these audits can uncover problematic patterns in models of interest. **�
  • An empirical study investigating a public algorithmic system for potential problematic behavior.***

​

* Metaxa, D., Park, J. S., Robertson, R. E., Karahalios, K., Wilson, C., Hancock, J., & Sandvig, C. (2021). Auditing algorithms. Foundations and Trends in Human-Computer Interaction, 14(4), 272–344. https://doi.org/10.1561/1100000083

** Raji, I. D., & Buolamwini, J. (2019). Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products. AIES ’19: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society.

*** Bandy, J. (2021). Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1). https://doi.org/10.1145/3449148

11

Grant Agreement Number 101148443

12 of 24

Algorithmic Auditing (cont’d)

  • Algorithm audits are a specific subset of audit studies focused on studying algorithmic systems and content.*�
  • An algorithmic audit involves the collection and analysis of outcomes from a fixed algorithm or defined model within a system. [...] these audits can uncover problematic patterns in models of interest. **�
  • An empirical study investigating a public algorithmic system for potential problematic behavior.***

​

* Metaxa, D., Park, J. S., Robertson, R. E., Karahalios, K., Wilson, C., Hancock, J., & Sandvig, C. (2021). Auditing algorithms. Foundations and Trends in Human-Computer Interaction, 14(4), 272–344. https://doi.org/10.1561/1100000083

** Raji, I. D., & Buolamwini, J. (2019). Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products. AIES ’19: Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society.

*** Bandy, J. (2021). Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1). https://doi.org/10.1145/3449148

12

Grant Agreement Number 101148443

13 of 24

https://www.loudandquiet.com/short/man-what-a-song-how-algorithms-reinforce-musics-gender-imbalance/

https://www.billboard.com/pro/martina-mcbride-spotify-female-country-artists-interview/

https://www.music-tomorrow.com/blog/fairness-and-diversity-in-music-recommendation-algorithms

https://exclaim.ca/music/article/james-blake-on-tik-tok-streaming-economics-the-brainwashing-worked-and-now-people-think-music-is-free

https://www.justice.gov/usao-sdny/pr/north-carolina-musician-charged-music-streaming-fraud-aided-artificial-intelligence

13

Grant Agreement Number 101148443

14 of 24

Research Questions

14

Grant Agreement Number 101148443

15 of 24

RQ1. What is the potential of auditing music recommender systems, and what obstacles must be overcome?

​

RQ2. How can we audit recommender systems’ mechanisms affecting music discoverability?

​

RQ3. How can we design recommender systems for promoting cultural diversity, by limiting their problematic behaviours?

15

Grant Agreement Number 101148443

16 of 24

RQ1. What is the potential of auditing music recommender systems, and what obstacles must be overcome?

​

RQ2. How can we audit recommender systems’ mechanisms affecting music discoverability?

​

RQ3. How can we design recommender systems for promoting cultural diversity, by limiting their problematic behaviours?

16

Grant Agreement Number 101148443

17 of 24

RQ1. What is the potential of auditing music recommender systems, and what obstacles must be overcome?

​

RQ2. How can we audit recommender systems’ mechanisms affecting music discoverability?

​

RQ3. How can we design recommender systems for promoting cultural diversity, by limiting their problematic behaviours?

17

Grant Agreement Number 101148443

18 of 24

Methodology

18

Grant Agreement Number 101148443

19 of 24

End-user|User-driven|Everyday Algorithmic Auditing

Everyday algorithm auditing is the ways everyday users detect, understand, and/or interrogate problematic machine behaviors via their day-to-day interactions with algorithmic systems.

​

​

Initiation → Awareness Raising → Hypothesizing & Testing → Remediation

​

​

​

Shen, H., Devos, A., Eslami, M., & Holstein, K. (2021). Everyday Algorithm Auditing: Understanding the Power of Everyday Users in Surfacing Harmful Algorithmic Behaviors. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2). https://doi.org/10.1145/3479577

Devos, A., Dhabalia, A., Shen, H., Holstein, K., & Eslami, M. (2022, April 29). Toward User-Driven Algorithm Auditing: Investigating users’ strategies for uncovering harmful algorithmic behavior. Proceedings of the ACM on Human-Computer Interaction. https://doi.org/10.1145/3491102.3517441

Lam, M. S., Gordon, M. L., Metaxa, D., Hancock, J. T., Landay, J. A., & Bernstein, M. S. (2022). End-User Audits: A System Empowering Communities to Lead Large-Scale Investigations of Harmful Algorithmic Behavior. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2). https://doi.org/10.1145/3555625

19

Grant Agreement Number 101148443

20 of 24

End-user|User-driven|Everyday Algorithmic Auditing (cont’d)

Everyday algorithm auditing is the ways everyday users detect, understand, and/or interrogate problematic machine behaviors via their day-to-day interactions with algorithmic systems.

1. Initiation →

2. Awareness Raising →

3. Hypothesizing & Testing →

4. Remediation

​

Shen, H., Devos, A., Eslami, M., & Holstein, K. (2021). Everyday Algorithm Auditing: Understanding the Power of Everyday Users in Surfacing Harmful Algorithmic Behaviors. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2). https://doi.org/10.1145/3479577

Devos, A., Dhabalia, A., Shen, H., Holstein, K., & Eslami, M. (2022, April 29). Toward User-Driven Algorithm Auditing: Investigating users’ strategies for uncovering harmful algorithmic behavior. Proceedings of the ACM on Human-Computer Interaction. https://doi.org/10.1145/3491102.3517441

Lam, M. S., Gordon, M. L., Metaxa, D., Hancock, J. T., Landay, J. A., & Bernstein, M. S. (2022). End-User Audits: A System Empowering Communities to Lead Large-Scale Investigations of Harmful Algorithmic Behavior. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2). https://doi.org/10.1145/3555625

20

Grant Agreement Number 101148443

21 of 24

Proposed Methodology

Literature

review

Think-aloud

interviews

Diary

study

Web-tool

design

Workshop

Usability

test

Audit

analysis

Synthesis

EVIDENCE GATHERING

FRAMEWORK DEVELOPMENT

ANALYSIS / SYNTHESIS

21

Grant Agreement Number 101148443

22 of 24

Proposed Methodology (cont’d)

Literature

review

Think-aloud

interviews

Diary

study

Web-tool

design

Workshop

Usability

test

Audit

analysis

Synthesis

EVIDENCE GATHERING

FRAMEWORK DEVELOPMENT

ANALYSIS / SYNTHESIS

22

Grant Agreement Number 101148443

23 of 24

Proposed Methodology (cont’d)

Literature

review

Think-aloud

interviews

Diary

study

Web-tool

design

Workshop

Usability

test

Audit

analysis

Synthesis

EVIDENCE GATHERING

FRAMEWORK DEVELOPMENT

ANALYSIS / SYNTHESIS

23

Grant Agreement Number 101148443

24 of 24

Algorithmic Auditing for Music Discoverability

https://lorenzoporcaro.me

https://aa4md.carrd.co�https://cordis.europa.eu/project/id/101148443

Grant Agreement Number 101148443