How this works
This is a small experimental AI model that recognizes
two American Sign Language signs: COOKIE
and PIE.
Surprisingly, the model was trained using only
19 examples:
- 7 COOKIE examples
- 12 PIE examples
The training videos came from several different people,
so the model has to learn what the signs have in common
rather than simply memorize one person's movements.
1. Find the hands and body
Each video frame is analyzed with MediaPipe. Instead of
giving the AI the original image, MediaPipe converts the
person's hands and body into numerical landmark coordinates.
That part is done on your browser at 30 frames per second! I
then transmit the list of coordinates to my back-end server.
So, your video or images of you are never sent.
2. Turn movement into numbers
I calculate 303 features for every frame.
These describe things such as hand shape, finger positions,
palm orientation, movement, the relationship between the
two hands, and the hands' relationship to the body. As mentioned,
I capture 30 frames per second, so altogether for the 1.2
seconds of recording, I'm collecting almost 11,000 data points.
Eventually I will need more data points. As I add other signs,
facial cues, and relative positioning to things other than
your other hand, that number will grow even higher. In all,
I'm estimating I will need to catpure well over 30,000 data points
every second.
3. Watch the sign happen over time
Each sign is converted into a sequence of 30 frames.
This means the AI doesn't just see a hand pose. It sees
how the hands move and change during the sign.
4. Let a transformer learn the difference
Those sequences were used to train a small transformer
neural network with about 89,000 trainable
parameters.
I did not program rules saying what COOKIE or PIE should
look like. The transformer learned the differences from
the examples.
5. Try someone it has never seen
When you perform a sign here, your short webcam recording
goes through the same landmark and feature extraction
process. The transformer then decides whether your movement
looks more like COOKIE or PIE.
6. Why COOKIE and PIE?
These two signs are very similar in their movement and
body and hand positions. That makes distinguishing them
a much more interesting test than choosing two obviously
different signs, such as THANK YOU and COOKIE.
Besides, if you're going to learn new words in sign
language, is there anything better than COOKIE and PIE?
I'm pretty sure those are survival words.
7. What's next? NDML - N-Dimension Machine Learning
The transformer experiment led me to another question: is there a different way to recognize complex patterns across many dimensions without requiring every recognizable class to be trained together?
I'm experimenting with a new approach to multidimensional pattern recognition that I'm calling NDML - N-Dimension Machine Learning. Essentially, I'm building a concept for a totally new AI model - one designed to recognize high-dimensional patterns while tolerating the natural variation that occurs between observations - whether that variation occurs over time or across other dimensions.
One of the goals of the architecture is for recognizable classes to be learned independently. In principle, that could allow new classes to be added, modified, or removed without retraining everything that has already been learned.
Gesture and ASL recognition will be the first proof of concept. If the underlying approach works, the same ideas may have applications anywhere complex multidimensional patterns need to be recognized, including manufacturing, sensor analysis, and medical imaging.
NDML is currently an experimental idea, and I'm still hashing out the details, though the basic architecture is nearly complete. The next step is to build the proof of concept and find out whether the theory survives reality - and if I can get this done before my head explodes trying to come up with a math formula to index structured relationships across domains.
This demo is only a proof of concept. Nineteen examples and
two words are nowhere near enough to recognize ASL generally.
The interesting experiment is whether a very small amount
of training data can teach a model enough about hand shape,
movement, and body relationships to recognize a sign from
someone it has never seen before.