Serpil Karabüklü�Toyota Technological Institute at Chicago�skarabuklu@ttic.edu�

How to Capture Differences in Nonmanuals: Behavioral and Computational Analyses

Simultaneity in Sign Languages

• Sign languages’ simultaneous nature (hands, face, body) can encode multiple information.
• Sign languages have high information content due to their dynamic three-dimensional signal.
• Age of acquisition and length of exposure affect the production and perceptual abilities of signers.

​

​

​

(Pfau & Quer, 2010, Napoli & Sutton-Spence 2018, Slonimska et al. 2020, Karabüklü & Brentari in rev., Borneman et al. 2018, Malaia et al., 2020, Boudreault & Mayberry 2006, Bosworth et al. 2019, Krebs et al. 2020, Wilbur 2021)

​

1

Taken from Slonimska et al. 2020

Simultaneity in Sign Languages

Hands

(manual)

2

Facial & body movements

(nonmanuals)

Space

Nonmanuals in Sign Languages

Nonmanuals, including the same nonmanual, are discussed to function across linguistic domains:

• Phonology
• Morphology
• Syntax
• Semantics – Pragmatics

​

​

​

(Coulter 1978, Liddell 1980, 1986, Wilbur 1994, Neidle et al., 2000, Shaffer 2004, Pfau & Quer, 2010, Wilbur 2011, Herrmann 2013, Wilbur 2021, Karabüklü 2022, 2024)

​

3

Case Study - Head Nod

• Cross-linguistically, head nod has been reported mainly as

(i) function across domains – prosody vs. semantics

single head nod = boundary marker in ASL vs. repetitive ones = certainty in ASL

(ii) different realizations are subcategories - semantics – pragmatics

smaller, repetitive nods > single, sharp nod

lower certainty 🡨---------🡪 higher certainty

Are they the same head nod?

​

4

(Wilbur 2000, Shaffer 2004, Akahori et al. 2013, Herrmann 2013, Brentari et al. 2018, Özsoy et al. 2018, Karabüklü 2022, 2024, Gürer & Karabüklü 2023)

Case Study - Data

Participants: 10 TİD, 9 ASL signers

Stimuli: 16 target objects X 4 event types = 64

Procedure: Watching the video and signing what is happening

​

5

(Karabüklü & Brentari 2024, 2025, in rev.)

Case Study - Head Nod

• Nuanced differences between different realizations of “same nonmanual”
• Do head nods labelled as “continuation rise” and “success/completion” differ in their phonetic features?
• Do ASL and TİD use this distinction similarly?

​

6

C1 head nod – “continuation rise”

C2 head nod – “success”

Behavioral Annotations

• Let’s look at annotations of head nod in different annotation schemas.

​

7

Kentner et al. 2022, Oomen et al. 2023, Esselink et al. 2025, Oomen et al. 2025, Roelofsen et al. 2025

https://nonmanuals.net/

Behavioral Annotations

8

We found higher proportions of head nod in these conditions (NFN) compared to other, but we still don’t know whether they are the same head nod.

(Karabüklü & Brentari 2024, 2025, in rev., Karabüklü , 2025)

Behavioral Annotations

• Did we answer our research questions?
• Current human annotations are:
• Binary
• Requires immense time
• Subjective
• Inconsistent

​

9

Head nod

Head thrust

??

Chin down

Kentner et al. 2022, Oomen et al. 2023, Esselink et al. 2025, Oomen et al. 2025, Roelofsen et al. 2025

Computational Tools: Key Points Extraction

• By using EnvisionBox code, MediaPipe pipeline was used to extract pose estimations of head movements on x axis.
• Amplitude, velocity, peak prominence, and trough prominence were measured

​

10

Bauer et al. 2024, Lugasai et al. 2019, Pouw 2024, Pouw & Akamine, 2025, Akamine et al. 2026, Karabüklü 2025, in prep.

Computational Tools: Key Points Extraction

• By using keypoint estimation, I show that head nods occurring in different syntactic positions in ASL and TİD have distinct trajectories.

​

​

11

Head nod

Bauer et al. 2024, Lugasai et al. 2019, Pouw 2024, Pouw & Akamine, 2025, Akamine et al. 2026, Karabüklü 2025, in prep.

C1 Head nod

C1 Head nod

C2 Head nod

C2 Head nod

DTW Analysis – Trajectory Comparsions

12

​

​

​

• Dynamic Time Warping analysis (DTW) and dimension differences are used to analyze the similarities in co-speech gestures.

​

Sakoe & Chiba 1978, Kruskal & Wish 1978, Beecks et al. 2015, Pouw & Dixon 2019, Pouw et al. 2021, Trujillo et al. 2023, Akamine et al. 2026 Karabüklü 2025, in prep.

DTW Analysis – Trajectory Comparisons

• Head nods at the end of sentence (C2) clustered similarly in ASL and TİD than the ones at the end of clause (C1).
• C1 head nods clustered significantly distinctly from C2 head nods and in each language.

​

​

​

13

Conclusion

14

• We have complementing results from behavioral analysis and computational analysis.
• While providing formal analysis of nonmanuals, we should first capture their differences in form-meaning mappings.
• Their function does not have to be prosodic, but their prosodic features can be mapped on specific functions.
• Available computer vision tools can enable us to capture the nuanced distinctions between nonmanuals.

​

​

Challenges

15

• Human annotations requires significant time.
• Humans can miss nuanced differences.
• Computer vision tools fail when hands are in front of the face.
• They do not show the same performance in all nonmanuals (i.e. brow movements).

​

​

​

​

(Kuznetsova & Kimmelman 2024, Sargano et al. 2024)

Discussion Questions

16

• What would be the best/idealistic ways to capture nonmanuals, their differences?
• Could it be possible to combine behavioral and computational methods?
• Would it be possible to have ToBI or Praat for nonmanuals one day?

​

​

​

​

Thank You

17

Behavioral Annotations

• There is a current international team working on an annotation guidelines for both gestures and nonmanuals
• https://nonmanuals.net/

​

18

Computational Tools: Key Points Extraction

19

​

​

​

Common measures (amplitude, velocity, peaks prominence and trough prominence) hinted the difference between peak prominence and trough prominence.

• Trough prominence > peak prominence in C1
• Peak prominence > trough prominence in C2

Karabüklü 2025, in prep.

References

• Akahori, H., Yano, U., Matsuoka, K., & Oka, N. (2013). Expressing modality: A descriptive study of Japanese Sign Language. Proceedings of 147th Meeting of Linguistic Society of Japan (pp. 1-6). Kobe, Japan: University of Foreign Studies.
• Akamine, S., Dingemanse, M., & Özyürek, A. (2026). Validating dynamic time warping asameasureofgesture formsimilarity. Behavior Research Methods, 58(4), 110. https://doi.org/10.3758/s13428-026-02975-5.
• Bauer, A., Kuder, A., Schulder, M., & Schepens, J. (2025). Phonetic differences between affirmative and feedback head nods in German Sign Language (DGS): A pose estimation study. Plos One, 20(3), e0321229, https://doi.org/10.1371/journal.pone.0304040.
• Borneman, J. D., Malaia, E. A., & Wilbur, R. B. (2018). Motion characterization using optical flow and fractal complexity. Journal of Electronic Imaging, 27(05), 1, https://doi.org/10.1117/1.JEI.27.5.051229.
• Bosworth, R. G., Wright, C. E., & Dobkins, K. R. (2019). Analysis of the visual spatiotemporal properties of American Sign Language. Vision Research, 164, 34–43, https://doi.org/10.1016/j.visres.2019.08.008.
• Boudreault, P., & Mayberry, R. I. (2006). Grammatical processing in American Sign Language: Age of first-language acquisition effects in relation to syntactic structure. Language and cognitive processes, 21(5), 608-635.
• Brentari, D., Falk, J., Giannakidou, A., Herrmann, A., Volk, E., & Steinbach, M. (2018). Production and Comprehension of Prosodic Markers in Sign Language Imperatives. Frontiers in Psychology, 9, 770, https://doi.org/10.3389/fpsyg.2018.00770.
• Coulter, G. (1978). Raised eyebrows and wrinkled noses: The grammatical function of facial expression in relative clauses and related constructions. Proceedings of the second National Symposium on Sign Language Research and Teaching. Coronado, CA.
• Esselink, Lyke, Marloes Oomen & Floris Roelofsen. 2025. STEADY: A toolbox for analyzing the reliability of timed-event sequential data. Preprint available at https://doi.org/10.17605/OSF.IO/2KEFQ
• Gürer, A., & Karabüklü, S. (2022). Non-manual Focus Markers in Turkish Sign Language. Proceedings of the Workshop on Turkic and Languages in Contact with Turkic, (pp. 32-45, https://doi.org/10.3765/ptu.v7i1.5321).

​

20

References

• Herrmann, A. (2013). Modal and Focus Particles in Sign Languages: A Cross-Linguistic Study. De Gruyter.
• Karabüklü, S. (2022). Modal Signs and Co-occurring Nonmanual Markers in Turkish Sign Language (TİD). Purdue University, Dissertation.
• Karabüklü, S. (2024). Simultaneity of certainty in Turkish Sign Language (TİD). Journal of Pragmatics, 232, 141-166, https://doi.org/10.1016/j.pragma.2024.08.010.
• Karabüklü, S. (2025) Form and Meaning Relations of Head Nods Across Sign Languages: A Pose Estimation Study. Talk presented at Sign Language Grammars, Parsing Models, & the Brain, Nov 6-7.
• Karabüklü, S. (in prep.) Beyond the Label: Capturing Nuanced Head Nod Distinctions in ASL and TİD via Pose Estimation and DTW
• Karabüklü, S., & Brentari, D. (2024). More cues are not always "better": Resolving instrument atypicality in visual domain. The 37th Annual Conference on Human Sentence Processing (HSP 2024)
• Karabüklü, S., & Brentari, D. (under rev.). Simultaneous Articulation Enhances Semantic Richness in Sign Languages.
• Karabüklü, S., & Gürer, A. (2024). Prosody of focus in Turkish Sign Language. Language and Cognition, 16(4), 1238-1271.
• Kentner, A., Karabüklü, S. & Wilbur, R. B. (2022). An Articulatory Model for Annotating Non-manual Markers in Sign Languages, the 96th Annual Meeting of Linguistic Society of America, Washington, DC., USA, Jan 6-9.
• Krebs, J., Malaia, E., Wilbur, R. B., & Roehm, D. (n.d.). Interaction between topic marking and subject preference strategy in sign language processing. Language, Cognition and Neuroscience, 35(4), 466-484.
• Kuznetsova, A., & Kimmelman, V. (2024). Testing MediaPipe Holistic for linguistic analysis of nonmanual markers in sign languages. arXiv preprint arXiv:2403.10367.
• Kuznetsova, A., Imashev, A., Mukushev, M., Sandygulova, A., & Kimmelman, V. (2021). Using Computer Vision to Analyze Non-manual Marking of Questions in KRSL. Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), (pp. 49-59. ).

​

​

​

21

References

• Liddell, S. K. (1980). American Sign Language Syntax. Berlin, Boston: De Gruyter Mouton. https://doi.org/10.1515/9783112418260.
• Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., . . . Grundmann, M. (2019). MediaPipe: A Framework for Perceiving and Processing Reality. Third Workshop on Computer Vision for AR/VR at IEEE Computer Vision and Pattern Recognition (CVPR) 2019, (pp. 1-4). Long Beach, CA.
• Malaia, E., Krebs, J., Roehm, D., & Wilbur, R. B. (2020). Age of acquisition effects differ across linguistic domains in sign language: EEG evidence. Brain and Language, 200, 104708.
• Napoli, D. J., & Sutton-Spence, R. (2010). Limitations on simultaneity in sign language. Language, 86(3), 647-662.
• Oomen, Marloes, Cindy van Boven, Lyke Esselink, Tobias de Ronde & Floris Roelofsen. 2025. ANM: An Annotation standard for Non-Manual elements in visual communication. Proof-of-concept and prototype development. Preprint available at https://doi.org/10.17605/OSF.IO/2KEFQ
• Pfau, R., & Quer, J. (2010). Nonmanuals: Their prosodic and grammatical roles. In D. Brentari, Sign Languages (pp. 381-402). Cambridge University Press.
• Pouw, W. (2024). EnvisionBOX modules for social signal processing (Version 1.0.0) [Computer software] https://github.com/WimPouw/envisionBOX_modulesWP.
• Pouw, W., & Dixon, J. A. (2019). Gesture networks: Introducing dynamic time warping and network analysis for the kinematic study of gesture ensembles. Discourse Processes, 57(4), 301-319. https://doi.org/10.1080/0163853X.2019.1678967.
• Roelofsen, Floris, Florence Baills, Gemma Barberà, Anastasia Bauer, Cindy van Boven, Raquel Veiga Busto, Brendan Costello, Lyke Esselink, Johannes Heim, Annika Herrmann, Serpil Karabüklü, Vadim Kimmelman, Andrea Lackner, Clara Lombart, Cornelia Loos, Nina-Kristin Meister, Liona Paulus, Alexandra Navarrete-González, Pilar Prieto, Sophie Repp, Patrick Louis Rohrer, Tobias de Ronde, Rosalee Wolfe, Rebecca Woods, Giorgia Zorzi & Marloes Oomen. 2025. Developing an annotation standard for non-manual elements in visual communication: International and cross-disciplinary collaboration. Paper presented at International Society for Gesture Studies (ISGS) 10, Nijmegen.
• Shaffer, B. (2004). Information ordering and speaker subjectivity: Modality in ASL. Cognitive Linguistics, 15(2), 175-195. https://doi.org/10.1515/cogl.2004.007.

​

​

​

​

22

References

• Slonimska, A., Özyürek, A., & Capirci, O. (2020). The role of iconicity and simultaneity for efficient communication: The case of Italian Sign Language (LIS). Cognition, 200, 104246.
• Trujillo, J. P., Dideriksen, C., Tylén, K., Christiansen, M. H., & Fusaroli, R. (2023). The dynamic interplay of kinetic and linguistic coordination in Danish and Norwegian conversation. Cognitive Science,47(6), Article e13298. https://doi.org/10.1111/cogs.13298
• Wilbur, R. B. (1994). Eyeblinks & ASL Phrase Structure. Sign Language Studies, 84, 221-240. https://doi.org/10.1353/sls.1994.0019.
• Wilbur, R. B. (2000). Phonological and prosodic layering of nonmanuals in American Sign Language. In K. Emmorey, & H. L. Lane, The Signs of Language Revisited: An Anthology To Honor Ursula Bellugi and Edward Klima (pp. 215-244. https://doi.org/10.4324/9781410604972). New York: Lawrence Erlbaum.
• Wilbur, R. B. (2011). Nonmanuals, semantic operators, domain marking, and the solution to two outstanding puzzles in ASL. Sign Language & Linguistics, 14(1), 148-178.
• Wilbur, R. B. (2013). Phonological and prosodic layering of nonmanuals in American Sign Language. In The signs of language revisited (pp. 196-220). Psychology Press

​

​

​

​

​

23