1 of 6

RU IEEE S25

AI/ML Submission

Aaryan, Bhavya, Eric, Mihir

Date

SIGNSENSE

Confidential

2 of 6

Two Pronged Problem: ASL

A fractionally small population of the United States is deaf, and ASL isn’t taught as widely as other languages

Considerably less deaf people are in the workforce, lack of tools for things such as meeting

Online services such as Zoom have no support for deaf people other than typing, which is slower than signing

Confidential

Our Idea

Strategy

An AI-powered ASL-to-speech converter using a webcam to translate ASL into spoken language in real time.�
Seamless, hands-free communication for ASL users.�
No additional hardware—works with any standard webcam.�

Confidential

Training model

We trained the datasets by hand, generating np.array files representing armature transformations over the duration of 30 frames (~1 sec), and then running those values through an LSTM in order to create a classifying model to classify frames
30 Iterations of 30 second frames were used to train gestures
The LSTM is a 5 layer LSTM model that uses the densest neurons to finally make a prediction

Confidential

Stack

The front end was build on React while the backend was built on Express (node.js)
API calls were used to run python files that would predict each frame passed to the backend
LLMs using Ollama were ran as well
The models were trained utilizing tensorflow

Confidential

Execution

The webcam in react (utilzing npm package react-webcam) would send frames every 33 ms (~ 30fps) to the backend via websocket in base64 encoding, and then that would be passed to the spawned python process
The python file utilized openCV to convert back to a frame, extract armature, and then return a post processed json object that contains the concatenated string generated from the video stream
After that, the string/sentence gets passed to a strict grammar checker to return a more english accurate statement, with sentiment and connotations
From there, it is passed to a TTS transformer based model that returns a path to an audio file at which it is displayed in the front end for the user to play
Due to processing power restrictions (running it all on local host and cloud computing being too expensive for a hackathon) as well as lack of access to multimodal LLMs, the translation cannot be fully live, however it is very accurate for what it is

Confidential