1 of 28

Towards Richer Online Music Public-domain Archives

EU H2020 TROMPA Project

Emilia Gómez (PI)

www.trompamusic.eu

TROMPA: Towards Richer Online Music Public-domain Archives

2 of 28

2013-2016 - PHENICX

TROMPA: Towards Richer Online Music Public-domain Archives

3 of 28

Motivation

3

TROMPA: Towards Richer Online Music Public-domain Archives

4 of 28

Traditional enrichment tool

4

TROMPA: Towards Richer Online Music Public-domain Archives

5 of 28

However, it was not scalable

Lack of good quality digital music data
Limited accuracy of technologies for classical music
Expensive expert human annotation/corrections
Closed derived data → limited reusability

5

TROMPA: Towards Richer Online Music Public-domain Archives

6 of 28

TROMPA - Towards Richer Online Music Public-domain Archives

6

Disciplines, communities:

Musicology
Music Institutions
Technological partners
Music users
Content owners

UNDERSTANDING EUROPE – PROMOTING THE EUROPEAN PUBLIC AND CULTURAL SPACE

2018–2021

TROMPA: Towards Richer Online Music Public-domain Archives

7 of 28

TROMPA vision

7

TROMPA: Towards Richer Online Music Public-domain Archives

8 of 28

TROMPA vision

8

TROMPA: Towards Richer Online Music Public-domain Archives

9 of 28

Five music communities

9

TROMPA: Towards Richer Online Music Public-domain Archives

10 of 28

Interests, enjoyment, contributions

10

	Interest - data	Enjoyment - technologies	Contributions
Music scholars	Digital scores	Sharing perspectives	Annotations
Piano players	Digital scores, recordings	Performance analysis, alignment	Rehearsal practice, performance annotations
Choirs	Digital scores, recordings	Performance analysis, score sonification, alignment	Rehearsal practice, performance annotations
Orchestras	Digital scores	Score digitization	Score annotations
Music enthusiasts	Recordings	Music emotion recognition, recommendations	Music emotion annotations

TROMPA: Towards Richer Online Music Public-domain Archives

11 of 28

TROMPA vision

11

TROMPA: Towards Richer Online Music Public-domain Archives

12 of 28

Machine intelligence

Advance Music Information Retrieval (MIR) Technologies
From knowledge-driven to data-driven approaches
Address MIR limitations for classical music:

Acoustic and musical complexity:

Choir singing
Orchestra scores: digitization
Expressive piano performances

Specific datasets for training

12

TROMPA: Towards Richer Online Music Public-domain Archives

13 of 28

Machine intelligence

Automatic description of music:

Audio: music emotion recognition, music performance rating (pitch, timing, dynamics)
Image: visual analysis of scores
Audio to score alignment

Automatic synthesis of music:

Score sonification: choir singing
Sound source separation

13

Aus der Tiefen rufe ich, Herr, zu dir, BWV 131 (Herreweghe)

TROMPA: Towards Richer Online Music Public-domain Archives

14 of 28

TROMPA vision

14

TROMPA: Towards Richer Online Music Public-domain Archives

15 of 28

Human intelligence

Involve the crowd (at varying expertise levels) in unlocking knowledge and expressing own perspectives on music material.
Exploit crowd labelled data for algorithm improvement.
Crowd + algorithm work together to enrich music artifacts.

15

(Crowd +

Expert +

End Users)

Labellers

ML

+

Human Computation

Enriched music content

Music content

TROMPA: Towards Richer Online Music Public-domain Archives

16 of 28

Incentivisation

Engagement with...

Experts, users, volunteers
From 5 communities

… for different contribution Actions

annotation, comment, validation
Target tasks: emotions in music, transcription of scores, difficulty of playing a piano piece or singing a choir part

Understand incentivisation mechanisms:

“Support” tool: learning, open access, recommendations.

TROMPA: Towards Richer Online Music Public-domain Archives

The TROMPA model for incentivisation of crowds follows the principles of the most relevant motivation theories from social sciences, namely the Expectancy theory [3] and SDT [4]. Expectancy theory provides insights on the relevant aspects to determine user’s behavior, while SDT provides the concept of motivation as a continuum in which users can be more self-determined to perform specific activities using the correct extrinsic motivations (autonomous motivation).

self-efficacy (person belief in their ability to successfully perform a behavior), control (person belief of control over the expected outcomes according to his/her performance) and trust (in the community members and processes). The effect of these personal characteristics on user’s performance are moderated by environmental factors such as usability and user responsibilities (related to community roles). The second element that influence user’s participation is the importance given by the users to the community goals, and how these goals are synchronized to their personal goals.

All these previous elements provide an initial state of users’ motivations, which can be enhanced or developed during the implementation of the project through different sets of incentives or rewards.

17 of 28

Hybrid human-machine intelligence

17

TROMPA: Towards Richer Online Music Public-domain Archives

18 of 28

TROMPA vision

18

TROMPA: Towards Richer Online Music Public-domain Archives

19 of 28

Reusable components

Provide machine and human intelligence into a set of reusable components for different use cases:

A database: distributed data, metadata, human-machine contributed data - Contributor Environment (CE).
A set of music description algorithms run on the backend, triggered to any data type - �Processing Library (PL)
A set of components for data navigation and annotation, music score viewing, analysis of human performances - Annotation tool, Performance Assessment

19

FAIR data & principles: �Findability, Accessibility, Interoperability, Reuse

TROMPA: Towards Richer Online Music Public-domain Archives

20 of 28

TROMPA vision

20

TROMPA: Towards Richer Online Music Public-domain Archives

21 of 28

Five music communities

21

TROMPA: Towards Richer Online Music Public-domain Archives

22 of 28

Music scholars

Added value:

Support and innovate musicological research processes.

Technologies: Digital Score Edition (DSE) for annotation using the Music Encoding Initiative’s MEI format.
Annotations: textual, links to media.
Repertoire:

Tempo and expression marks in� Mahler’s 4th symphony �(Dr Paul Banks) .
Early vocal music.

22

https://trompamusic.eu/music-scholars

Prof. Tim Crawford

TROMPA: Towards Richer Online Music Public-domain Archives

23 of 28

Piano players

CLARA (Companion for Long-term Analyses of Rehearsal Attempts): tool for rehearsal progress.

Repertoire: solo piano performances.

Technologies to capture, gather, and compare rehearsal recordings.

Alignment of rehearsals (video) and digital score encoded (MEI).
Allowing visualization of performance aspects (e.g. tempo) against the musical score.

Added value:

Insights on performance temporal evolution.
Immediate feedback on stylistic and technical aspects.

23

https://trompa.mdw.ac.at/

Dr. David Weigl, Dr. Werner Goebel

TROMPA: Towards Richer Online Music Public-domain Archives

24 of 28

Choir singers

Goals: support singers in their choir practice. Amateur choirs.
Technologies:

Singing voice performance analysis.
Multilingual singing voice synthesis technology for score sonification.

Added value:

Access to the repertoire (digital form), listen to the synthetic pieces.
Record, automatic analysis of the interpretation.
Virtual choir.

24

http://trompa,netlify.app

https://voctrolabs.bitbucket.io/virtualchoir/

TROMPA: Towards Richer Online Music Public-domain Archives

25 of 28

Orchestras

Community and added-value: public-domain digital scores, student and youth orchestras based in The Netherlands.
Technologies: score digitisation and annotation, public-domain musical scores, connection to IMLSP.
Example: transcription of excerpts of Beethoven Wind Sextet, op. 71.

25

https://trompamusic.eu/content-owners

Dr. Cynthia Liem

TROMPA: Towards Richer Online Music Public-domain Archives

26 of 28

Music enthusiasts

Repertoire: Stichting Centrale Discotheek (CDR) NL muziekweb.nl
Goals:

Gather and study of the so-called “ground truth” data for machine learning models of emotions in music.
Enhance interdisciplinary discussions on music & emotion.

Added value:

learning resources on general notions of musical properties and their relation to emotions, and
reward users with music recommendations.

26

https://ilde.upf.edu/trompa/

TROMPA: Towards Richer Online Music Public-domain Archives

27 of 28

Conclusions

Five communities, FAIR data, reusable components and public-domain archives
Hybrid human and machine intelligence.
Covid has emphasized the need for the digital transformation of the cultural sector.
Still many research challenges to address on human and machine intelligence and how technologies change the way we experience culture.

27

TROMPA: Towards Richer Online Music Public-domain Archives

28 of 28

Thanks!

Scientific publications: https://trompamusic.eu/research/results/publications
Public project deliverables: https://trompamusic.eu/research/results/deliverables
https://twitter.com/TrompaMusic

28

TROMPA: Towards Richer Online Music Public-domain Archives