Research

Turning a movement into a spoken sentence.

At the Biomedical AI Lab at the University of North Texas, I worked on a system that lets people with motor disabilities speak through gestures. A person makes a movement, a neural network recognizes it, and their phone says the phrase out loud. It runs on the phone itself, on Android and iOS.

Gesture to speech · try it

Pick a gesture. Its movement becomes a signal over time, the network reads that signal step by step, and the result is a spoken phrase.

1 · The movement, as a signal over time

2 · The LSTM reads it one step at a time, remembering what came before

3 · The phone speaks

Illustrative demo. The gestures, signals and phrases here are examples, not the ones from the study. On some browsers the phrase is also spoken aloud.

Why an LSTM

The problemA gesture is a sequence, not a snapshotThe same position can mean different things depending on the movement before it.
The modelAn LSTM keeps a memory as it readsIt is a kind of neural network built for sequences, so it can tell movements apart by how they unfold.
The resultIt runs on the phoneNo internet connection is needed, which matters for something a person relies on to communicate.

Publication

IEEE ICHI 2023An LSTM-based Gesture-to-Speech Recognition SystemR. B. Rafiq, S. A. Karim, M. V. Albert. IEEE International Conference on Healthcare Informatics.