1 of 30

Tap to Sign: Towards using American Sign Language for

Text Entry on Smartphones

Saad Hassan[1], Abraham Glasser[2], Max Shengelia[3], Thad Starner[4], Sean Forbes[5], Nathan Qualls[5], Sam S. Sepah[4]

[1] Tulane University, [2] Gallaudet University

[3] Rochester Institute of Technology, [4] Google, [5] Deaf Professional Artist Network

2 of 30

Motivation

  • Over 70 million Deaf and hard of hearing (DHH) people in the world
  • American Sign Language (ASL) is used by about 500,000 people as a primary form of communication
  • With the advent of smartphones and being able to communicate “on-the-go”, some Deaf users have begun altering their signing style while using smartphone cameras [ A. Morris, 2022; Keating and Mirus, 2003]
  • 53% Deaf users reported holding their smartphone in one hand while signing with the other [K. Mack et al., 2020]
  • Interactions with personal assistants is one of the most requested features by DHH individuals [Glasser et al., 2021]

2

3 of 30

Motivation

  1. Is smartphone text entry (specifically fingerspelling) a desirable use of sign recognition over current practice?

​

  • Is commanding the mobile Assistant in ASL desirable (fingerspelling or word-level ASL) over current practice?

​

3

4 of 30

Tap to Sign Text Entry Emulation

4

Similar to how speech is used with the Assistant on Android currently.

5 of 30

Push to Sign Text Entry Emulation

5

Similar to early “push-to-talk” speech recognition and walkie-talkies.

Recognition is easier because each word is isolated.

6 of 30

Tap to Sign: Towards using American Sign Language for

Text Entry on Smartphones

​

​

6

Part 1

Part 2

Part 3

Training Required in a Fingerspelling Text Entry Interface

Is Fingerspelling Faster and more Preferred vs. Virtual Keyboard?

Smartphone Signing vs. Fingerspelling vs. Virtual Keyboard for Commanding a Mobile Assistant

7 of 30

Experimental Study

12 Deaf Participants

9 Females, 3 Males

Average Age: 33.6

RQ1: How much training is required to reach expertise in a fingerspelling text entry interface?

​

7

8 of 30

Findings: Learning Curve

8

Mackenzie-Soukoreff phrase set: all 500 phrases repeated twice

9 of 30

Tap to Sign: Towards using American Sign Language for

Text Entry on Smartphones

​

​

9

Part 1

Part 2

Part 3

Training Required in a Fingerspelling Text Entry Interface

Is Fingerspelling Faster and more Preferred vs. Virtual Keyboard?

Smartphone Signing vs. Fingerspelling vs. Virtual Keyboard for Commanding a Mobile Assistant

10 of 30

Experimental Study

12 Deaf Participants

​

1-hour Study

​

Split into 3 sessions

RQ2 Are the Tap to Sign and Push to Sign fingerspelling prototypes faster and more preferred for text entry versus a virtual keyboard for Deaf users trained to expertise with the fingerspelling interface?

10

11 of 30

Virtual Keyboard Prototype

11

12 of 30

Findings: Text Entry Speeds over Time

12

13 of 30

Findings: Text Entry Speeds over Time

13

Condition

Speed (WPM)

Corrected

Error Rate

Uncorrected Error Rate

Bandwidth

Throughput

Tap to Sign

39.3

(10.1)

4.97%

(6.19%)

0.761%

(1.23%)

94.0%

(3.52%)*

13.2

(3.28)

Push to Sign

42.5

(11.3)*

4.02%

(4.27%)

0.866%

(1.13%)

95.4%

(3.40%)*

14.2

(3.78)*

Virtual

Keyboard

31.9

(10.1)

6.46%

(2.26%)

0.770%

(0.372%)

88.3%

(3.60%)

10.9

(3.45)

* indicates statistical significance over virtual keyboard

14 of 30

Findings: NASA TLX Questionnaire

14

15 of 30

Findings: NASA TLX Questionnaire

15

16 of 30

Findings: NASA TLX Questionnaire

16

17 of 30

Findings: Subjective Feedback Questionnaire

17

Half preferred Tap to Sign; half virtual keyboard

​

18 of 30

Tap to Sign: Towards using American Sign Language for

Text Entry on Smartphones

​

​

18

Part 1

Part 2

Part 3

Training Required in a Fingerspelling Text Entry Interface

Is Fingerspelling Faster and more Preferred vs. Virtual Keyboard?

Smartphone Signing vs. Fingerspelling vs. Virtual Keyboard for Commanding a Mobile Assistant

19 of 30

In-the-Wild Study

12 Deaf and Hard of Hearing Participants

RQ3: In a walk-up usability evaluation, do Deaf users prefer fingerspelling and/or smartphone- signing for text entry over typing on a virtual smartphone keyboard to command an emulated virtual assistant?

19

20 of 30

Smartphone Signing Emulated Mobile Assistant

20

21 of 30

Fingerspelling Emulated Mobile Assistant

21

22 of 30

Virtual Keyboard Emulated Mobile Assistant

22

23 of 30

Findings: Naturalness and Subjective Preference

23

24 of 30

Findings: Subjective Feedback Questionnaire

24

Condition

System Usability Scale

"User-friendliness"

Score

Net Promoter Score

Smartphone Signing

75.8

(13.7)

6.17

(1.03)

83.3%

Fingerspelling Only

63.5

(16.9)

4.75

(2.30)

0%

Virtual Keyboard

79.8

(11.9)

6.10

(0.994)

100%

25 of 30

Findings: Qualitative

25

Category

Use-case

Navigation

Navigating to an address (P2, P3, P6), Viewing traffic (P11), Restaurants (P9, P11), Ride-sharing services (P9), Food-ordering services

Communication

Communication with others (P1), Messages (P7), Email (P8)

Information

Color of a flag (P4), Weather (P6, P7, P8, P10), News (P7), Technology release (P7), Recipes (P9), Gas Prices (P12), Deaf Events (P12)

Device Control

Smartphone commands (P5), Temperature control (P5), Home information, e.g. lights on, doors locked (P5), Open Apps (P12), Record (P10)

Personal Use

Appointments (P6, P7), Calendar (P6), Reminders (P6), Calendar (P6, P7), Productivity stats (P6), Reservations (P7), Tasks (P7), Paying Bills (P7), Movie releases (P7), Flights (P7), Timezones (P7), Health (P7), Pandemic (P7), Scriptures (P7), Documents (P8), Grocery List (P9)

26 of 30

Findings: Qualitative

“I would use this everyday. I hope this could be ready today."

“It is a fantastic resource, even when I am moving or traveling"

​

“Will it pan to follow them, if they set it down?"

“while its dark, if you’re in bed"

“Can it read from a distance? or from up really close?"

​

​

ASL drops words [Signs a, the, and is]. It is hard to not sign when told to fingerspell. English grammar causes contrived signing and English word order.”

26

Excitement

Feedback

Concerns

27 of 30

Summary and Takeaway

  • If users are given time to acclimatize to a text entry interface based on fingerspelling, their speeds and subjective preferences regarding using fingerspelling as a text entry method improve over time.
  • Study 1 showed that with experience, half of the participants preferred fingerspelling for text entry, and fingerspelling proved faster.
  • Study 2 showed that walk-up users prefer signing and the virtual keyboard over fingerspelling for interacting with a mobile assistant.
  • Fingerspelling recognition does seem to be a worthwhile endeavor for sign recognition efforts for interfaces where the user can be enticed to gain sufficient experience with the interaction or as part of a more general signing system for interacting with a mobile assistant

27

28 of 30

Kaggle Fingerspelling Dataset

https://www.kaggle.com/competitions/asl-fingerspelling

28

>3 million fingerspelled characters

>100 Deaf signers

29 of 30

Thank You

Saad Hassan (saadhassan@tulane.edu)

Abraham Glasser (abraham.glasser@gallaudet.edu)

Max Shengelia (mas5627@rit.edu)

Thad Starner (thadstarner@google.com)

Sean Forbes (sean@dpan.tv)

Nathan Qualls (nathan@dpan.tv)

Sam S. Sepah (sepah@google.com)

29

30 of 30

Tap to Sign: Towards using American Sign Language for

Text Entry on Smartphones

Saad Hassan[1], Abraham Glasser[2], Max Shengelia[3], Thad Starner[4], Sean Forbes[5], Nathan Qualls[5], Sam S. Sepah[4]

[1] Tulane University, [2] Gallaudet University

[3] Rochester Institute of Technology, [4] Google, [5] Deaf Professional Artist Network