What does AU17 ‘flat chin’ mean?

RONNIE B. WILBUR¹

L.R. NIK NIKOLAI^1,2

¹PURDUE UNIVERSITY; ²UNIVERSITY OF MICHIGAN

Acknowledgements

PURDUE TEAM

• Ashley Kentner
• Serpil Karabüklü
• Margaret Ruth Crabtree

​

​

COLLABORATORS & FUNDING

• Aleix Martínez, Ohio State
• Daniel Hole, Stuttgart
• Fabian Bross, Stuttgart

​

​

• NIH #R01DC014498 jointly to A. Martínez & R. B. Wilbur

2

What are we looking at?

3

Introduction and Methods

4

Be careful what you wish for!

Feb 23, 2017

14.4 million numbers

​

​

5

Why so many?

• 1 file, unformatted
• 60 columns, 1 for each AU
• 12,000+ rows, 1 for each frame in the video
• 60*12000= 720,000 numbers per file
• 20 files
• 5 files for each video
• 4 videos in the Face of ASL published series, MJ Bienvenu signer
• Total 20*720,000 = 14,400,000 numbers [14.4 million]

6

But there’s good news …

• Only 12 columns have relevant numbers [0,1]; all others are 999
• 12000*12=144,000 relevant numbers per file
• 144000*20=2,880,000 relevant numbers in 20 files
• Whew!! Only 2.88 million!

7

Why ‘Face of ASL’?

• Published video series (4 half-hour tapes)
• Clear signing and face for teaching purposes
• Formulate hypotheses on basis of these videos, confirm with corpus data of other signers

​

​

8

Procedure

• Face of ASL (FoA) videos sent to Aleix’s OSU team
• Purdue team put FoA videos into ELAN
• Segmented signs and marked syntax (developed for our previous AR project)
• OSU team sent AR identification of AU presence ‘1’ or absence ‘0’
• Purdue team identified target AU for annotation (about which more shortly)
• Annotator(s) did not see data from OSU team
• Worked in ELAN independently identifying AU presence and duration
• Summarized AU function(s) in FoA [Sample 1]
• Tested AU function hypotheses against independently elicited data from Deaf native signer [Sample 2]

​

9

The 12 AUs

• Chosen for computer program benefit (easy to detect)

​

10

The 12 AUs

• Chosen for computer program benefit (easy to detect)

​

11

NEXT – AU detections into ELAN

• Read each column of numbers for each AU as a track
• Can display multiple tracks or selected tracks
• Enables ELAN to show where in video computer program thinks it saw an AU
• Could allow a human to compare computer detection with human annotation
• BUT a computer can do that faster and better

12

13

12 AU tracks, �showing 1=detected, 0=not detected

14

Today’s Target AU17

15

AU17 present – line UP�AU17 absent – line DOWN

16

Why AU17 chin raiser?

• Long history of brow studies [AU1,2,4]
• Wanted something new
• Nose wrinkle [AU9] most frequent
• Extensively studied by Sandra Wood while she was at Purdue
• Cheek raiser [AU6] Baker, Padden, Wood
• Mouth [AU12,15,20,25]
• Studied by Kadir Gökgöz while at Purdue
• Upper lid raiser [AU5] few and hard to see (human)
• Chin raiser [AU17] !!

17

More about AU17

• Basically unstudied
• Chin thrust [forward] (Liddell, 1978)
• Chin up (Schalber, 2004 for Austrian SL)
• But NOT chin itself
• In this 2006 chart, I only included chin options in In/Out axis

​

18

AU17 – What does it mean?

19

Start discussion based on Sample 1

• Sample 1 is 6 eaf files from Face of ASL [constructed by video length]
• 5 files are related to FoA Conditionals and Relative Clauses
• 1 file related to FoA Basic Questions
• 275 sentences annotated (approximately 25 minutes video)
• 114 sentences (41.5%) contained AU17
• AU17 appears 158 times in Sample 1
• AU17 occurs average 1.39 times/sentence (of those containing AU17)
• Initial hypothesis: Related to negation (Benitez-Quiroz, Wilbur, Martínez 2016)

20

The NOT face: AU17 in affective non-signing facial expressions

21

The NOT face �(Benitez-Quiroz, Wilbur, Martinez 2016)

22

NOT

The NOT face < from affective faces�(Benitez-Quiroz, Wilbur, Martinez 2016)

23

NOT

Hypotheses for AU17 in ASL:

• H1: AU17 is related to presence of negation
• Finding in Sample 1: 6 appearances (3.8%) of AU17 in negative clauses (Nikolai, 2018)
• New hypotheses based on Sample 1:
• H2: AU17 related to speaker evaluation of scalarity
• H3: AU17 related to discussion of NMMs
• H4: AU17 related to partitivity
• H5: AU17 related to uncertainty
• H6: AU17 related to specific words

24

Evaluate hypotheses

25

H2: AU17 related to speaker evaluation of scalarity (much/little)

• Most prevalent function in Sample 1 (28% of occurrences)
• AU17 occurs where signer makes a judgment (good/bad/etc) about what they are stating

​

26

AU17

#RUG #IF GERMAN STRONG

'If the rug is German, it is strong.'

H3: AU17 is used when discussing facial expressions/NMM (16% of occurrences)

AU17

PEOPLE SEE ASL STRONG FACIAL EXPRESSION+++

'People see that ASL has strong facial expressions.'

27

H4: AU17 related to partitivity �(13% of occurrences of AU17)

AU17

NOW VIDEOTAPE SPLIT FEW CL:1+5 CUT

'Now, this videotape is cut into a few parts.'

28

29

H5: AU17 related to uncertainty�(13% of occurrences of AU17)

• Signer is uncertain about what they are saying or are offering a possibility for the future that they are not certain of
• Notice scope is over entire sentence

AU17

MAYBE OFFER IDEA MAYBE CAN HAPPEN WILL HAPPEN NOT-KNOW

'Maybe you introduce an idea that can or will happen - I don't know.'

​

30

H6: AU17 related to specific lexical items

• 3-15% of occurrences

31

Questions raised by Sample 1 analysis

• Are uncertainty and speaker evaluation separate functions of AU17?
• Are there subtypes of speaker evaluation that can be categorized?
• Does the degree of AU17 vary based on its function?
• Are all the functions of AU17 possibly derived from one core function?
• How does AU17 interact with other NMM, specifically mouth such as AU25 & AU26?
• Also may have connection to puffed cheeks and/or wrinkled nose. Both appear frequently nearby or with AU17.

​

32

Quick side note from Sample 1: �AU25 Parted lips & AU26 Jaw drop

• Out of 44 conditionals in Sample 1, only 3 did NOT have strong AU25 or slight AU26 in their protasis.
• Out of 54 ‘adverbial fronted’ clauses in Sample 1, only 4 did not contain AU25 or AU26.
• Possible difference in size or quality of AU25 and AU26 based on type of adverb used.
• Is this tied to adverbs in general or just fronted adverbs?
• Almost all fingerspelled words (99 words out of 101) exhibited AU25 or AU26
• Lexicalized (‘ASL’, ‘style’, but not ‘LSD’ [Louisiana School for the Deaf]) and borrowed words (‘grammar’ or ‘speed’)
• This may be irrelevant if she is attempting to mouth English words while fingerspelling.

​

33

Back to AU17

• We can account for 73% of the occurrences of AU17 in Sample 1
• Questions raised (again):
• Are uncertainty and speaker evaluation separate functions of AU17?
• Are there subtypes of speaker evaluation that can be categorized?
• Are all the functions of AU17 possibly derived from one core function?
• Use different data, test hypotheses

​

34

Test Hypotheses with Sample 2

[CORPUS DATA]

35

Corpus/Elicited Data & Hypotheses

• Data from a native Deaf ASL signer for other purposes several years ago
• Tested two hypotheses
• H1: AU17 is obligatory in ‘speaker evaluation of scalarity’ contexts
• H2: AU17 is obligatory in partitive contexts
• All sentences with evaluation of scalarity and partitivity were investigated

36

Results (Nikolai, 2019)

• H1: AU17 was obligatory in evaluation contexts (100%)
• H2: AU17 was not obligatory in partitivity contexts (37.5%)
• Obvious question: What is AU17 really doing in the ‘partitivity’ contexts?
• Where do we go from here?

​

37

Making sense of this

38

Theoretical framework > Cinque Hierarchy (1999, 2006): Sequence of functional projections for operators

39

Bodily Mapping Hypothesis�(Bross & Hole 2017, for German SL, DGS)

• “... the wider/higher the scope of a clausal operator [is], the more likely its expression will occur with a high body part by way of layering” in SLs

​

40

Separation of projections by body height (DGS)

41

High categories (i. e., above T) are expressed nonmanually.

Intermediate (below T above Voice) expressed manually, concatenating from L-to-R or R-to-L.

Karabüklü et al (2018) �Turkish SL (TİD)

• Like DGS, TİD is SOV language
• TİD does NOT allow modal sign sequence left-to-right, only right-to-left
• NMM eye squint and head tilt (upper face) give epistemic flavor, whereas hands show force (existential POSSIBLE or universal NECESSARY)
• Other similarities to DGS in keeping with Bross & Hole Bodily Mapping Hypothesis

​

​

42

Bross (2018) dissertation (DGS)�Added categories and new option

43

AU17 is on the lower face (chin) -�If B&H correct, should be ‘scalarity’

44

AU17 does not start at beginning of sentence, thus below Speech Act, Mirative, & Epistemic.

AU17

ɪx-3 ᴡʀɪᴛᴇ ᴛʜʀᴇᴇ ʙᴏᴏᴋ ᴡᴏᴡ ɢᴏᴏᴅ

‘He has written three books, and that’s really impressive!’

​

AU17

ʟɪᴏɴ ɪx-3 ᴡᴏᴀʜ ʜᴇᴀᴠʏ ᴡᴇɪɢʜ ᴍᴏʀᴇ-ᴛʜᴀɴ ᴛᴡᴏ-ʜᴜɴᴅʀᴇᴅ

‘Wow, lions weigh even more than 200 pounds.’

​

AU17

ɪx-3 ʜᴇʏ sᴇᴇ ɪx-3 CL:F "spots" sᴇᴇ ɴᴏɴᴇ

‘He doesn’t see any spots there.’

45

[NB: AU17 is a monadic operator.]

Summary

• AU17 occurs frequently in evaluation of scalarity contexts
• AU17 is located in the right place if Bross & Hole (2017) are correct
• If this is on the right track, then use of AU17 in uncertainty contexts might be cases of ‘possibility’, which would also be lower face

​

46

Cinque Hierarchy (1999, 2006)

47

Summary

• AU17 occurs frequently in evaluation of scalarity contexts
• AU17 is located in the right place if Bross & Hole (2017) are correct
• If this is on the right track, then use of AU17 in uncertainty contexts might be cases of ‘possibility’, which would also be lower face
• The Bodily Mapping Hypothesis gives us predictions, which we are more than happy to test and confirm or dispute as we fill in the pieces on this puzzle.

48

There’s one more major problem

• HSR:
• HUMAN Sign Recognition!
• After several decades of NMM research, it’s really, really clear that there’s no good guide to how humans should annotate NMMs.
• We’ve made several attempts for specific projects
• Now in development of a full-scale PURDUE TEMPLATE for NMM annotation
• Project is led is by Ashley Kentner and Serpil Karabüklü
• Will include template for ELAN
• Will include instruction manual for annotators
• STAY TUNED FOR THE NEXT EXCITING EPISODE …

​

49

References

• Benitez-Quiroz, C. F., Wilbur, R. B., & Martínez, A. M. (2016). The NOT face: A grammaticalization of facial expressions of emotion. Cognition, 150, 77-84.
• Bross, F., Hole, D. (2017). Scope taking strategies in German Sign Language. Glossa 2(1): 76, 1–30.
• Cinque, G. (1999). Adverbs and functional heads: A cross-linguistic perspective. New York: Oxford University Press.
• Ekman, P., & Friesen, W. V. (1978). Facial action coding system: A technique for the measurement of facial movement. Palo Alto, California: Consulting Psychologists Press.
• Ekman, P., Sorenson, E. R., & Friesen, W. V. (1969). Pan-Cultural elements in facial displays of emotion. Science 164 (3875), 86-88.
• Karabüklü, S., Bross, F., Wilbur, R.B. & Hole, D. (2018). Modal signs and scope relations in T˙ID. FEAST 2: 82-92.
• Nikolai, L.R. (2018). Progress report on AU17, Spring 2018.
• Nikolai, L. R., Wilbur, R.B. (2018). Facial expressions in sign language grammar: What does flat chin mean? 7^th Meeting, Signed and Spoken Language Linguistics, Osaka, Japan.

50

THANK YOU!!

ANY QUESTIONS?

51

52

Extras for use if needed

53

Relevant Tiers in FoA ELAN

• English Transcription: contains exact transcription of voiceover which gives rough English translation of what signer says. Annotation lines up to beginning and end of each of speaker’s sentences. 
• English Translation: Contains more accurate English translation of what signer says. Annotation lines up to beginning and end of each of signer’s sentences. Annotation “XXXX” used for unknown sign.
• Complex: Indicate when a sentence was complex, that it contained more than one clause. Annotation states ‘complex,’ and stretches length of sentence.
• Simple: Indicate individual clauses. Annotation stretched over whole sentence; if a sentence is ‘complex’, multiple annotations in this tier within one sentence to indicate boundaries between clauses. 
• Clause/Phrase Type: Several things: (1) If sentence is topic fronted (start with topic) or adverbial fronted (start with adverbial). (2) If sentence is assertion or question. Questions then split by type: polarity questions or wh-questions. Wh-questions variations: wh-postposed, wh-q, and wh-cleft. Wh-postposed used when wh-word at end of question with lowered brows. Wh-q when wh-word left in-situ or doubled in question. Wh-cleft used when wh-word was used in statement rather than question. Annotation stretched for length of clause or phrase in question.
• Embedded: Child tier to Clause/Phrase Type; indicate subordination. Annotation stretches from beginning to end of clause, and contains note re behavior - i.e. relative clause or if signer quoted a previous example she had given. 
• Polarity: ‘positive’ or ‘negative’. Special attention re: AU17 & negation (Benitez-Quiroz, Wilbur, & Martínez, 2016) 
• Conditional: Indicate if hypothetical (ex. “If it rains, the picnic will be canceled.”) or counterfactual (ex. “If I had been there, I would have known.”). If neither, left blank.
• AU-17: Indicate presence. Annotations stretches from beginning to end of each AU17 appearance and labeled ‘AU17.’
• AU-25: Indicate presence. Annotations stretches from beginning to end of each AU25 appearance and labeled ‘AU25.’ 
• AU-26: Indicate presence. Annotations stretches from beginning to end of each AU26 appearance and labeled ‘AU26.’ 
• Notes: Notes about patterns or anomalies in data or thoughts on annotations (esp. tiers AU17, AU25, & AU26)

54