HoloLens with Leap Motion controller

This repository contains the solution developed for combining the HoloLens with a Leap Motion Controller (LMC). The work was done as a Master's Thesis work for Aalto University at Futurice and the complete work detailing how everything works can be found here. The purpose is to enable the creation of better hand and gesture based interactions for the HoloLens by making use of the hand tracking data provided by the LMC.

Requirements

The Leap Motion client requires the Orion SDK and OpenCV to run. The Skin Detection Trainer only requires OpenCV to run. The OpenCV version used is 3.2.

The Unity application uses Zenject for dependency injection and UniRx for Reactive extensions, but these are already included in the project.

Getting started

Prerequisites

1. Make sure the OpenCV build\x64\your_version_of_visual_studio (e.g. vc14) folder is in the system path.
2. Make sure the Leap SDK is in the path.

The skin color detection trainer

1. Create C++ console application with name "SkinColorDetectionTrainer" in base directory.
2. Delete all autogenerated header and source files from the project.
3. Move all files from SkinColorDetectionTrainerSources to your project folder.
4. Add the existing header and source files to the project.
5. Switch solution platform to x64.
6. Open project properties.
7. Under C/C++ - General, add OpenCV "include" folder to Additional Include Directories.
8. Under Linker - General, add the OpenCV "lib" folder to Additional Library Directories.
9. Under Linker - Input, add "opencv_world320.lib" and "opencv_world320d.lib" to Additional Dependencies.
10. Under C/C++ - Precompiled Headers, make sure Precompiled Header is set to Use.
11. Close project properties.
12. Open properties for "stdafx.cpp".
13. Under C/C++ - Precompiled Headers, make sure Precompiled Header is set to Create.

The Leap Motion client

1. Create C++ console application with name "LeapMotionClient" in base directory.
2. Delete all autogenerated header and source files from the project.
3. Move all files LeapMotionClientSources to your project folder.
4. Add the existing header and source files to the project.
5. Switch solution platform to x64.
6. Open project properties.
7. Under C/C++ - General, add OpenCV and Leap SDK "include" folders to Additional Include Directories.
8. Under Linker - General, add the OpenCV and Leap SDK (make sure it is the x64 folder) "lib" folders to Additional Library Directories.
9. Under Linker - Input, add "opencv_world320.lib", "opencv_world320d.lib", and "Leap.lib" to Additional Dependencies.
10. Under Build Events - Post-Build Event, add the following to Command Line: xcopy /yr "$(LEAP_SDK)\lib\x64\Leap.dll" "$(TargetDir)"
11. Under C/C++ - Precompiled Headers, make sure Precompiled Header is set to Use.
12. Close project properties.
13. Open properties for "stdafx.cpp".
14. Under C/C++ - Precompiled Headers, make sure Precompiled Header is set to Create.

The HoloLens Unity project

Open the base folder in Unity.

Solution details

The entire solution is made up of three separate applications (the Unity application running on the HoloLens, the Leap Motion client that provides the Leap Motion data, and the Skin Color Detection Trainer that calculates the parameters used for skin detection) and solves three key problems. These problems are:

1. Calibrating the HoloLens and the LMC. This is done with Perspective-n-Point and uses the user's fingertips in 1 or more images in combination with the 3D fingertip coordinates provided by the LMC. The implementation of the hand detection in an RGB image can be found here and the fingertip detection can be found here.
2. Calculating the parameters needed for the hand detection. Skin colour detection is done using a combination of k-means clustering and Mahalanobis distances. The required paramaters are calculated based on a set of training images with corresponding ground truths. The trainer can be found here, the training set here, and the ground truths here.
3. Streaming and converting the Leap Motion data. Once calibration is done the data provided by the LMC are streamed using UDP to minimise latency. The data are converted on the HoloLens side to the HoloLens's coordinate system.

Current limitations

The biggest current limitation is the skin colour detection. Since this project was done as a proof-of-concept, the training data for the skin colour detection was gathered only from a single person. If the hand detector doesn't seem to be working properly or accurately enough, this might be the cause. One option is then to collect additional training data using the HoloLens. But one problem that will probably still persist is that pure colour-based detection has problems with some colours (see chapters 4.1.5 and 5.1.1 of the thesis for more info). The optimal solution would be to change to a more reliable way of doing hand detection.

Another thing to note is that the results are not 100% accurate and further investigation of the source(s) of the error is needed. See chapter 5.1.2 of the thesis for a discussion of this error.

Third, the current networking code was written quite hastily, so any improvement suggestions to this are very welcomed. The HoloLens's UI is also very quickly thrown together, so any improvement suggestions to this are also very welcomed.

Feedback, suggestions, and questions

Any feedback, suggestions for improvements, and questions about how something works are extremly welcome. Send any of these to [email protected].

License

This software is licensed under Apache License 2.0.

UniRx is licensed under MIT license, © 2014 Yoshifumi Kawai.

Zenject is licensed under MIT license, © 2016 Modest Tree Media Inc.