mpatacchiola / Deepgaze

Updates

Update 22/01/2020 You may be interested in following my new Youtube channel for weekly videos about Computer Vision, Machine Learning, Deep Learning, and Robotics.

Update 16/07/2019 Stable version of Deepgaze 2.0 is available on branch 2.0.

Update 20/03/2019 Started the porting on Python/OpenCV 3.0, check the branch 2.0 for a preliminary version.

Update 10/06/2017 The PDF of the article "Head pose estimation in the wild using Convolutional Neural Networks and adaptive gradient methods" is available for free download in the next 50 days using this special link

Update 04/06/2017 Article "Head pose estimation in the wild using Convolutional Neural Networks and adaptive gradient methods" have been accepted for publication in Pattern Recogntion (Elsevier). The Deepgaze CNN head pose estimator module is based on this work.

Update 31/05/2017 Implementation of the new package saliency_map.py. The package contains an implementation of the FASA algorithm for saliency detection [example] [wiki]

Update 22/03/2017 Fixed a bug in mask_analysis.py and almost completed a more robust version of the CNN head pose estimator.

What is Deepgaze?

Deepgaze is a library for human-computer interaction, people detection and tracking which uses Convolutional Neural Networks (CNNs) for face detection, head pose estimation and classification. The focus of attention of a person can be approximately estimated finding the head orientation. This is particularly useful when the eyes are covered, or when the user is too far from the camera to grab the eye region with a good resolution. When the eye region is visible it is possible to estimate the gaze direction, which is much more informative and can give a good indication of the FOA. Deepgaze contains useful packages for:

• Head pose estimation (Perspective-n-Point, Convolutional Neural Networks)
• Face detection (Haar Cascade)
• Skin and color detection (Range detection, Backprojection)
• Histogram-based classification (Histogram Intersection)
• Motion detection (Frame differencing, MOG, MOG2)
• Motion tracking (Particle filter)
• Saliency map (FASA)

Deepgaze is based on OpenCV and Tensorflow, some of the best libraries in computer vision and machine learning. Deepgaze is an open source project and any contribution is appreciated, feel free to fork the repository and propose integrations.

This library is the result of a recent work, if you use the library in academic work please cite the following paper:

Patacchiola, M., & Cangelosi, A. (2017). Head pose estimation in the wild using Convolutional Neural Networks and adaptive gradient methods. Pattern Recognition, http://dx.doi.org/10.1016/j.patcog.2017.06.009.

Why should I use Deepgaze?

Because Deepgaze makes your life easier! The implementation of many algorithms such as face detectors, pose estimators and object classificators can be painful. Deepgaze has been designed to implement those algorithms in a few lines of code. Deepgaze is helpful for both beginners and advanced users who want to save time. All the code contained in Deepgaze is optimised and it is based on state-of-the-art algorithms.

What is a Convolutional Neural Network?

A convolutional neural network (CNN, or ConvNet) is a type of feed-forward artificial neural network in which the connectivity pattern between its neurons is inspired by the organization of the animal visual cortex, whose individual neurons are arranged in such a way that they respond to overlapping regions tiling the visual field. Convolutional networks were inspired by biological processes and are variations of multilayer perceptrons designed to use minimal amounts of preprocessing. They have wide applications in image and video recognition, recommender systems and natural language processing [wiki]

Main contributors

This is an updated list of the main contributors of the project. We are looking for contributors! If you want to contribute adding a new module or improving an existing one, send an email to our team!

• Massimiliano Patacchiola: project leader and main contributor
• Joel Gooch: head pose estimation
• Ishit Mehta: CNN-cascade face detection
• Luca Surace: Haar-cascade multi-face detection
• Hrishikesh Kamath: version 2.0 porting, notebooks, test scripts

Prerequisites

The current version of Deepgaze is based on Python 2.7, a porting for Python 3.0 has been scheduled for the next year.

To use the libray you have to install:

• Numpy [link]

sudo pip install numpy

• OpenCV 2.x (not compatible with OpenCV >= 3.x) [link]

sudo apt-get install libopencv-dev python-opencv

• Tensorflow [link]

sudo pip install tensorflow

Some examples may require additional libraries:

• dlib [link]

Installation

ATTENTION: this version is obsolete, please check the branch 2.0 on this repository

Download the repository from [here] or clone it using git:

git clone https://github.com/mpatacchiola/deepgaze.git

To install the package you have to enter in the Deepgaze folder and run the setup.py script (it may require root privileges):

cd deepgaze
sudo python setup.py install

If you want to track all the installed files you can record the installation process in a text file using the --record flag:

sudo python setup.py install --record record.txt

Done! Now give a look to the examples below.

Examples

• Head Pose Estimation using the Perspective-n-Point algorithm in OpenCV [code] [video]

• Head Pose Estimation in-the-wild using Perspective-n-Point and dlib face detector [code] [video]

• Head Pose Estimation in images using Convolutional Neural Networks [code]

• Color detection using the Histogram Backprojection algorithm [blog] [code]

• Skin detection using the HSV range color detector [code]

• Face detection using the HSV range color detector [code]

• Corner detection comparison of four algorithms on a video streaming [code] [video]

• Motion detection and tracking using frame differencing on a video streaming [code]

• Motion detection and tracking comparison of three algorithms on a video streaming [code] [video]

• Motion tracking with unstable measurements using Particle Filter [code] [video]

• Motion tracking with multiple backprojection for playing chrome's dinosaur game [blog] [code] [video]

• Classify object using their colour fingerprint (histogram intersection) [blog] [code]

• Implementation of the FASA (Fast, Accurate, and Size-Aware Salient Object Detection) algorithm [code] [wiki] [link]

Acknowledgements

• The example "head pose estimation using Perspective-n-Point" is partially based on the C++ version you can find here, and on the workshop "Developing an attention system for a social robot" which was part of the 2nd International Summer School on Social Human-Robot Interaction.

• To implement the Bayes and Particle Filters I followed the great repository of rlabbe which you can find here