1 of 14

An end-to-end working prototype that detects sign language meanings in images/videos and generate equivalent, realistic voice of words communicated by the sign language, in real-time.

Steven Kolawole

CS Undergrad @FUNAAB, Nigeria

Previously @#3

2 of 14

Next steps @#3

  • Retrain on the TFOD APIs method using a better architecture
  • Increase dataset’s range to support 50+ sign phrases
  • Support for African languages using Machine Translation
  • Write a paper???

https://github.com/SteveKola/Sign-Language-to-Speech-with-DeepStack-Custom-API

Contact:

kolawolesteven99@gmail.com

3 of 14

Sign-to-Speech Model for Sign Language Understanding using Object Detection:

A Case Study of Nigerian Sign Language

Steven Kolawole

​

​

​

September 22, 2021

​

​

#4

4 of 14

Synopsis

To develop a lightweight sign-to-speech machine learning application that works in real-time to reduce the communication barrier between the hearing impaired community and the larger society with a focus on sub-Saharan Africa, which is the region with most cases of hearing disabilities.

5 of 14

Dataset

Creating low-resource dataset with Nigerian Sign Language as a case study;

  • Dataset creation by a sign language TV broadcaster from the Ogun State Broadcasting Corporation (OGBC).
  • Students and teachers from two special education schools in Lagos and Abeokuta, both cities in Nigeria.

​

​

6 of 14

Dataset

Creating low-resource dataset with Nigerian Sign Language as a case study;

  • 8k+ static images of 135+ signs words/phrases with 20+ individuals.
  • Images captured in diverse backgrounds

​

7 of 14

Dataset

Image Annotation using LabelImg;

  • Dataset labelled in both YOLO (for YOLO model) and PASCAL VOC (for SSD) format.

​

8 of 14

Architecture

  • YOLOv5

https://zenodo.org/record/4679653#.YUshfLpKhNg

9 of 14

Evaluation

10 of 14

Evaluation

11 of 14

Evaluation

12 of 14

Challenges

  • Annotation was stressful! :(
  • Model cannot capture sequences of spellings
  • Compute resources constraints 😑

13 of 14

Next steps

  • Finish training SSD model & compare results
  • Finish writing the camera-ready draft
  • Submit to ML4D workshop at NeurIPS
  • Start thinking of graduate school 😅

Contact:

kolawolesteven99@gmail.com

Link to abstract paper

14 of 14