gest
Hand gestures as an input device
Why
For health related reasons, I had to stop using a mouse and a keyboard. Talon allowed me to type with my voice and move the cursor with my eyes. This project was started to complement this setup with hand gestures.
Development status
The project is in an early stage of development. I use it on daily basis, so it should be good enough for some.
What is implemented:
- pinching gesture recognition, in one hand orientation
- heatmap output, separate for left and right hand, indicating pinched point position
- demo for testing recognition models
- example script for simulating mouse clicks and scrolling
- scripts for producing and reviewing training data
Bias
The gesture recognition model was trained on images of my hands, taken with my hardware in my working environment, so it is probably heavily biased. I hope people who want to use it, but recognition quality prevents them from it, would capture some images of their hands using included tooling and donate it to the project, so that over time it works well for everyone.
Installation
Use Python 3.6, 3.7 or 3.8 and in a virtual environment run
pip install gest
If you clone this repository, you can get the exact versions of required libraries that I am using with Poetry
poetry install
Walkthrough
Demo
First check how the included model works for you. Run
python -m gest.demo
and see if it recognizes your gestures as here:
If you have multiple cameras, you can pick one like
python -m gest.demo --camera 2
Camera numbers are not necessarily consecutive. Two cameras may be accessible as 0 and 2. This option is supported by other commands as well.
Example script
In the presentation on top I am running
python -m gest.examples.two_handed_scroll_and_click
It only acts if it detects both hands pinching and based on their relative position:
- double clicks if you cross your hands
- scrolls up or down if your hands pinch at different heights
- left clicks if your hands (almost) touch
- right clicks if your hands are on the same height, but not close horizontally (this action is delayed by a fraction of a second to prevent accidental use)
Controlling CPU load
For everyday use, you don't want to dedicate too much resources to gesture recognition. You can control it by setting OMP_NUM_THREADS, as in
OMP_NUM_THREADS=2 python -m gest.examples.two_handed_scroll_and_click
Try different values to find balance between responsiveness and CPU load.
Custom scripts
The demo and example scripts serve two additional purposes: they can be used as templates for custom scripts and they define the public API for the purpose of semantic versioning.
Training data annotation
Capturing
python -m gest.annotation.capture --countdown 5 data_directory
will help you create annotated images.
Once you start automatic annotation (press a to start/stop) it will ask you to pinch a given point with your left or right hand, or to not pinch ("background").
You will have 5 seconds before the image is captured (the --countdown).
You will also see the last annotated image for quick review. It can be deleted with d.
Reviewing
python -m gest.annotation.review --time 1 data_directory closed_pinch_left
will let you review all images annotated as left hand pinch in data_directory, showing you each for 1 second if you start/stop automatic advancing with a. Otherwise you can go to the next/previous image with n/p. Delete incorrectly annotated images with d.
You should also review closed_pinch_right and background.
Annotation guidelines
It makes sense to annotate realistic training data that the model performs poorly on, like if
- it mistakenly detects a pinch when you pick up the phone,
- it doesn't detect pinching when you wear a skin colored shirt.
If it performs poorly overall, it's good to capture the images in many short sessions, with different lighting, clothes, background, camera angle.
The point isn't though to look for tricky cases or stretch the definition of a pinching gesture to include a different hand orientation (eg. with pinching fingers pointing towards the camera).
Donating annotated data
Contact me [email protected]