1 of 9

Sign Spotter: Design and Initial Evaluation of an Automatic Video-based ASL Dictionary System

ASSETS’23 DEMO

Matyas Bohacek (Stanford University) & Saad Hassan (Tulane University)

CRASH

YELLOW

UGLY

LAUGH

CUTE

THEORY

MAKE

GOVERNMENT

LAUGH

LAUGH

LAUGH

LAUGH

LAUGH

LAUGH

LAUGH

LAUGH

LAUGH

2 of 9

Introduction & Motivation

  • Over 70 million Deaf and hard of hearing (DHH) individuals world-wide use sign languages [ 1 ]
  • While recent HCI research has explored ASL learners’ interaction with prototype ASL dictionaries, there is limited integration of these findings with functioning recognition systems [ 2 ]

​

  • In our ongoing research, we are developing a video-based ASL dictionary system using state-of-the-art sign recognition technology [ 3 ]

BOHACEK & HASSAN

ASSETS’23

3 of 9

System Design

  • We designed a web-application consisting of two primary web pages:

(a) the main dictionary interface

(b) the detailed results

​

  • The model’s confidence in each prediction is converted into a human-understandable label

BOHACEK & HASSAN

ASSETS’23

4 of 9

System Design

BOHACEK & HASSAN

ASSETS’23

Fig 1.

Main dictionary

interface page

5 of 9

System Design

BOHACEK & HASSAN

ASSETS’23

Fig 2.

Detailed results page

6 of 9

System Design

BOHACEK & HASSAN

ASSETS’23

Fig 3.

Handshape filter

selection modal

7 of 9

Initial Feedback & Findings

  • Participants provided feedback on the system design through a structured interview; all four participants expressed interest in using our system
  • Some of the responses include:

“I believe... I would make regular use of it, as many individuals in my class already engage in recording themselves and sharing it with multiple people to determine the understanding of certain signs.”

“To be honest, I might actually end up using something like this in its current state, because I feel like in the detailed analysis page, you can really get what you’d like.”

BOHACEK & HASSAN

ASSETS’23

8 of 9

Thanks for your attention!

See you at the demo!

BOHACEK & HASSAN

ASSETS’23

9 of 9

References

  1. Ross Mitchell, Travas Young, Bellamie Bachleda, and Michael Karchmer. 2006. How Many People Use ASL in the United States? Why Estimates Need Updating. Sign Language Studies 6 (03 2006). https://doi.org/10.1353/sls.2006.0019
  2. Saad Hassan, Oliver Alonzo, Abraham Glasser, and Matt Huenerfauth. 2021. Effect of Sign-Recognition Performance on the Usability of Sign-Language Dictionary Search. ACM Trans. Access. Comput. 14, 4, Article 18 (oct 2021), 33 pages. https: //doi.org/10.1145/3470650
  3. Matyáš Boháček and Marek Hrúz. 2022. Sign Pose-based Transformer for Word-level Sign Language Recognition. 2022 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) (2022), 182–191.

BOHACEK & HASSAN

ASSETS’23