From Speech to Gesture: �Theoretical Tensions in the Notion of Prominence

Patrick L. Rohrer

Serra Húnter Fellow

Universitat Rovira i Virgili

01/07/2026

The Notion of Prominence

A linguistic element that stands out from neighboring elements

​

(Streefkerk, 2002; Himmelmann & Primus, 2015; �von Heusinger & Schumacher, 2019; Cangemi & Baumann, 2020; Grice & Kügler, 2021 )

Prosodic prominence generally refers to the relative acoustic and perceptual salience of a linguistic unit within its surrounding context through the manipulation of acoustic features such as pitch, intensity and duration.

Lexical/syntactic

Prosodic

See Ladd & Arvaniti (2023) for a review

Phonological Pitch Accents vs. Relative Salience

Mainstream American English (MAE) ToBI pitch accent inventory (Veilleux et al., 2006)

Data taken from English M3D-TED Corpus (Rohrer et al., 2023),

see also Baumann & Winter (2018) for similar results in German

PARADIGMATIC�PERSPECTIVE

SYNTAGMATIC PERSPECTIVE

“… within its surrounding context…”

“… standing out relative to neighboring elements…”

The perception of prominence can be modulated by:

• The different combinations of acoustic cues
• Position within a phrase
• Relative pitch height and relative distance between different accents

See Lorenzen et al. (2024) and references therein

Annotating Prominence

Tones and Breaks Index (ToBI): Only indirectly assesses prominence (pitch accented syllables)

Rapid Prosody Transcription (RPT): Crowdsourced binary judgements from multiple non-specialist participants

Rhythm and Pitch (RaP): Two degrees of prominence (weak/strong) + pitch accented

Deutsche Intonation, Modellierung und Annotation (DIMA): Prominence Tier

See, e.g., Jun (2006, 2014)

Dilley & Brown (2005)

Cole & Shattuck-Hufnagel (2015)

Kügler & Baumann (2020)

MOVING FROM SPEECH TO GESTURE

Gestural Prominence

See Rohrer et al. (2025)

The Notion of Prominence

A linguistic element that stands out from neighboring elements

​

(Streefkerk, 2002; Himmelmann & Primus, 2015; �von Heusinger & Schumacher, 2019; Cangemi & Baumann, 2020; Grice & Kügler, 2021 )

Prosodic prominence generally refers to the relative acoustic and perceptual salience of a linguistic unit within its surrounding context through the manipulation of acoustic features such as pitch, intensity and duration.

Lexical/syntactic

Prosodic

See Ladd & Arvaniti (2023) for a review

Gestural

The degree of visual saliency ("standing out") of the target hand gesture, head movement, or eyebrow movement in relation to neighboring gestural movements.

Applying Gestural Prominence with M3D

Three gestures where the final gesture differs in prominence

Gestural prominence is coded perceptually according to the degree of visual saliency of the target hand gesture stroke and/or head or eyebrow movement in relation to the preceding and following strokes. That is, every stroke is assessed in relation to the neighboring strokes to determine how much the stroke stands out from the others, as well as taking into account the general gestural style of the speaker.

Similar approach to what is done in Speech Prosody

Largely adapted from DIMA

It is important to note that this is preliminary work… and further research is ongoing in order to better assess the different kinematic cues to prominence and further refine the labeling guidelines currently proposed.

Gestural prominence and its relationship with prosodic prominence in focus marking

Sara Coego

Núria Esteve-Gibert

Pilar Prieto

Pilar Prieto

Frank Kügler

Alina Gregori

Paula

Sánchez-Ramón

Adult speakers of Catalan and German

Children acquiring Catalan

Methodology

Focus Elicitation Task

Based on Esteve-Gibert et al. (2022)

Image taken from Coego & Sánchez-Ramón (2024)

Coding of prosodic and gestural prominence with DIMA and M3D (Scale from 0 to 3)

Kügler & Baumann (2020), Rohrer et al. (2025)

0

1

2

3

Examples of gestural prominence ratings

Applying Gestural Prominence

Development

Coego, S., Esteve-Gibert, N., & Prieto, P. (2025). Preschoolers Mark Focus Types Through Multimodal Prominence: Further Evidence for the Precursor Role of Gestures. Languages, 10(5), 92. https://doi.org/10.3390/languages10050092

Adult - Catalan

Sánchez-Ramón, P. G., Gregori, A., Kügler, F., & Prieto, P. (2025). Multimodal prominence as an indicator of increasing degrees of contrastivity: Focus type marking in Catalan and German (No. 6114386). SSRN. https://dx.doi.org/10.2139/ssrn.6114386

The notion of gestural prominence is a useful tool to better understanding the link between speech prosody and gesture!

Challenges surrounding Gestural Prominence

In gesture, does “more” = “more” ?

Cross-modal “phonetic” cues to prominence

AND HIGHER-ORDER INTERACTIONS!

In speech:

• Limited number of cues which generally correlate
• In speech, increasing a cue leads to greater perception of prominence

Unlike speech, gesture varies simultaneously along many dimensions, creating substantially more degrees of freedom in how prominence may be perceived.

Challenges surrounding Gestural Prominence

Dealing with spontaneous speech

Challenges surrounding Gestural Prominence

Dealing with spontaneous speech

Where do we go from here?

MultIS2 Project

• Identify the relative contribution of individual cues to perceived gestural prominence �(e.g., see Coego et al., 2026)
• Investigate the interaction between cues in shaping prominence judgements �(e.g., see Trujillo & Holler, 2024 on compositional meaning in facial signals)
• Empirically assess the current M3D guidelines for gestural prominence
• Publish a refined version of the M3D guidelines, assess reliability, and develop training material

The notion of perceptual prominence has clear parallels across speech and gesture, making gestural prominence a promising avenue for multimodal research, as recent studies have demonstrated.�

At the same time, the theoretical foundations that support prominence in speech have developed over decades, and introducing this notion to the visual modality introduces additional complexities and methodological challenges.

​

The next challenge is not whether gestural prominence exists, but how to build the theoretical foundations, annotation practices, and empirical evidence needed to understand it as rigorously as prominence has been understood in speech.

Discussion Questions

• Should we move towards a more standardized approach for annotating gestural prominence that is similar to DIMA or ToBI for speech, which takes various structural and kinematic (“phonetic”) features into account?
• Or should we remain with a perceptual, holistic approach potentially to the detriment of objective justification?

Methodology: Can we standardize gestural prominence?

• Should gestural prominence be defined primarily in syntagmatic terms (relative to neighboring gestures) or paradigmatic terms (relative to the set of possible realizations)? Both?
• At what point do there become too many degrees of freedom for "prominence" to remain a useful theoretical construct?
• Can prominence be treated as a single, modality-independent perceptual construct, or should each modality develop its own specific notion of prominence?
• Should gesture researchers continue adopting concepts inherited from speech prosody, or should gesture develop it’s own theoretical vocabulary?

Theoretical: How should prominence be conceptualized?

• Given the many interacting kinematic parameters that may contribute to prominence, is reliable annotation realistically achievable?
• If extensive training is necessary, what does that imply about the perceptual reality of gestural prominence?

Reliability: Can humans reliably annotate this?