Gesto.AI: Sign Language Translation

Team 46: Shivam Arora, Daniel Firebanks-Quevedo, Pablo Oberhauser, Dhruv Shah, Ibrahim Sherif, Samuel Tang

Motivation

• Sign languages are visual languages that use two types of features to convey information: manual (handshape, palm orientation) and non-manual (head movement, facial grammar)
• Most technologies support spoken/written language, not sign language
• Communication barrier between hearing people and deaf/hard-of-hearing people
• Recent progress in machine learning has been promising!
• Challenges
• Limited data
• ASL (American Sign Language) is not a direct representation of English
• Limited models

Source: How2Sign

Idea: What we set ourselves to do

• Build a web application that translates ASL from a video into English text,
• Level 1: Image (letter) to text
• Level 2: Video (word) to text
• Level 3: Video (sentence) to text
• Level 4: Real-time video stream (sentence) to text
• Features:
• Data storage connected to UI
• Fine-tuning pipeline and model monitoring
• Model deployment and predictor service
• User interface for ease of usage
• Incorporate user feedback by giving them the option to provide a correct label when the model makes a mistake.
• Incorporate multiple datasets + videos uploaded by users to fine-tune the model
• Use a second model for auto-completing a sentence

What we’ve actually done (in blue)

• Build a web application that translates ASL from a video into English text,
• Level 1: Image (letter) to text
• Level 2: Video (word) to text
• Converted original model to torchscript (made it more compact/lighter)
• Wrote PyTorch Lightning version + added training/evaluation logic
• Level 3: Video (sentence) to text
• Level 4: Real-time video stream (sentence) to text
• Features:
• Data storage connected to UI
• Fine-tuning pipeline and model monitoring
• Model deployment and predictor service
• User interface for ease of usage
• Incorporate user feedback by giving them the option to provide a correct label when the model makes a mistake.
• Incorporate multiple datasets + videos uploaded by users to fine-tune the model
• Use a second model for auto-completing a sentence

System Diagram

Deploy

Feedback

Re-train or fine-tune

Request a prediction

Response from predictor service

Dataset: Word-Level American Sign Language (WLASL)

• Input: Video containing a person doing a word/phrase in sign language
• 2,000-21,000 videos
• Label: Word in text.
• 100-2000 words
• Links
• Paper
• Paper Repository
• Dataset homepage

​

Model: Inception I3D

• Inception trained on ImageNet, fine-tuned on Kinetics-400, with initialized 3D filters.
• Further fine-tuned on WLASL, we use the pretrained weights directly
• Original paper
• Original Repository

Future work

• Integrate the dataset from How2Sign
• Real time video stream processing
• Experiment with different model architectures
• Incorporate the following
• Experiment tracking
• Model Monitoring
• Automated training
• Automated model deployment
• Feedback system for the model
• Improve model prediction speed

Demo: Streamlit App

Screenshots