To test our system with data closer to reality we added support for ETS or ATS. As the simulator is not free, you can opt whether to compile the project with this feature or not. The communication between the simulator and our program is done via a ROS2 client and it provides the following info:

1. Engine Status (On/Off)

2. Trailer Status (Connected/Disconnected).

3. Speed.

4. RPM.

5. Acceleration.

6. Position (Coordinates).

7. Gear (-1 for Reverse, >0 the rest).

We also plan to send the data through MQTT using AWS IoT-Core, to produce a dashboard with the trucks positions, alarms, etc. Again, using AWS may incur in a cost, so this will also be optional for you to compile with/without it.

Using the image detected inside the face ROI (region of interest), we feed a facial landmarks detector to identify points of iterest. Using 6 points for each eye and 6 points for the mouth it is possible to calculate 'Eye Aspect Ratio (EAR)' that gives 2 values for eye/mouth open or closed (based on this paper).

At the moment of writing this guide, the facial landmarks detection model included with OpenVino (facial-landmarks-35-adas-0001) has not enough points to run this calculations. We are using dlib’s facial landmarks detector instead.

Once we have a positive detection for blink/yawn, we count frames of those events and trigger an alarm when they hit a threshold.

Using the face’s ROI, we feed a head-pose detector model provided by OpenVino (head-pose-estimation-adas-0001). Analizing the output of that model we can easily detect when the face is not centered or not looking to the front.

• Target and development platforms meeting the requirements described in the "System Requirements" section of the OpenVINO™ toolkit documentation which may be found at: https://software.intel.com/en-us/openvino-toolkit

Note: While writing this tutorial, an Intel® i7-8550U with Intel® HD graphics 520 GPU was used as both the development and target platform.

• Optional:

  • Intel® Movidius™ Neural Compute Stick

  • USB UVC camera

  • Intel® Core™ CPU with integrated graphics.

• OpenVINO™ toolkit supported Linux operating system. This tutorial was run on 64-bit Ubuntu 16.04.1 LTS updated to kernel 4.15.0-43 following the OpenVINO™ toolkit installation instructions. Also tested on Ubuntu 18.04.

• The latest OpenVINO™ toolkit installed and verified. Supported versions 2019 R2. (Older versions are supported on another branch).

• Git(git) for downloading from the GitHub repository.

• BOOST library. To install on Ubuntu, run:

• LibAO and libsndfile to play some beeping sounds. On Ubuntu, run:

• [Optional] ETS or ATS simulator. Install it through Steam on Ubuntu.

• [Optional] AWS Crt Cpp

By now you should have completed the Linux installation guide for the OpenVINO™ toolkit, however before continuing, please ensure:

• That after installing the OpenVINO™ toolkit you have run the supplied demo samples

• If you have and intend to use a GPU: You have installed and tested the GPU drivers

• If you have and intend to use a USB camera: You have connected and tested the USB camera

• If you have and intend to use a Myriad: You have connected and tested the USB Intel® Movidius™ Neural Compute Stick

• That your development platform is connected to a network and has Internet access. To download all the files for this tutorial, you will need to access GitHub on the Internet.

1. Clone the repository at desired location:

2. The first step is to configure the build environment for the OpenCV toolkit by sourcing the "setupvars.sh" script.

For older versions than 2019 R1, OpenVINO was installed in a different dir, run this instead:

3. Change to the top git repository:

4. OpenVINO’s Release R2020.1 or greater compatibility

If you are using OpenVINO’s Release R2020.1 or greater you don’t need download the models.

If you are using OpenVINO’s Release >= R2 and < R2020.1 you will need to execute the following script:

In case of using the OpenVINO’s 2019 R1.0, before continuing, if you have not manually downloaded all the models before, it is necessary to download the following models.

5. Create a directory to build the tutorial in and change to it.

6. Before running each of the following sections, be sure to source the helper script. That will make it easier to use environment variables instead of long names to the models:

7. Compile:

8. Move to the executable’s dir:

In order to run the simulator you will need to install:

• ROS2 following Installing ROS2 via Debian Packages.

• colcon following Colcon Tutorial.

• ETS or ATS using Steam.

• ets_ros2-plugin from this repo.

Follow the plugin’s instructions to install everything, you can test the ros client is working through the sample application provided there. Once that is working we can build or program.

1. Clone the repository at <ros2_workspace>/src/ets_ros2 location:

2. Source everything!

3. Change to <ros2_ws> location and compile:

N being the number of cores to build (like make’s -jN flag). We recommend using 1 as it is a bit memory intensive.

4. Copy the plugin to the corresponding folder as described in the plugin repo:

or the ATS folder:

5. Lastly, source our workspace:

6.1.1. Face Detection

./driver_behavior -m $face132 -i ../../../data/img_1.jpg

./driver_behavior -m $face132 -i ../../../data/video1.mp4

./driver_behavior -m $face132 -i ../../../data/video1.mp4 -async

./driver_behavior -m $face132 -d GPU -i ../../../data/video1.mp4

./driver_behavior -m $face132 -d MYRIAD -i ../../../data/video1.mp4

6.1.2. Drowsiness detection

./driver_behavior -m $face232 -dlib_lm -i ../../../data/video2.mp4

6.1.3. Driver 'eyes on the road' detection

./driver_behavior -m $face232 -m_hp $hp32 -i ../../../data/video3.mp4

6.1.4. Realtime feed

./driver_behavior -m $face232
./driver_behavior -m $face232 -dlib_lm
./driver_behavior -m $face232 -d GPU -dlib_lm -async
./driver_behavior -m $face232 -m_hp $hp32

6.1.5. Driver recognition

cd scripts/
python3 create_list.py ../drivers/

./driver_behavior -m $face232 -d CPU -m_hp $hp32 -d_hp CPU -dlib_lm -d_recognition -fg ../../../scripts/faces_gallery.json

6.1.6. Simulator

We integrated our program to the Intel® IoT DevCloud platform. This developer tool enabled us to run the inference proccess on different hardware targets. The following is the comparison graph where greater is better:

Driver Assistance has been optimized for having compatibility with OpenVINO’s releases 2018’s (R4, R5) and 2019’s (Lastest version tested 2019 R1.0.1). It is important for the user to be aware that some changes regarding detection models had been introduced between releases from 2018 and 2019. In first instance, 2019 releases do not have the detection model’s binaries included within the toolkit; the user will have to follow the instructions described in the Open Model Zoo link suggested at the “Foreword” section of this installation guide. Be aware that if you choose to download them in a different path than the default, our “scripts/setupenv.sh” will not fully work and you will have to add the path to the models yourself when running the program. In case of using the OpenVINO’s 2019 R1.0 or greater, before continuing, it is necessary to manually download all the models.

If you are using OpenVINO’s Release >= R2 and < R2020.1 you will need to execute the following script:

After, the user will be able to initiate the building process and to start using Driver Assistance System

Firstly, in order to successfully execute the building process, please make sure that all the declared prerequisites –hardware and software- have been met. In particular, regarding software prerequisites, it is fundamental that the OpenVINO’s toolkit version had been downloaded by following the Intel’s intrucctions described in the following links:

• Version 2019 R2 (latest) https://docs.openvinotoolkit.org/latest/_docs_install_guides_installing_openvino_linux.html

• Version 2019 R1.1 https://docs.openvinotoolkit.org/2019_R1.1/_docs_install_guides_installing_openvino_linux.html

• Version 2019 R1.01 https://docs.openvinotoolkit.org/2019_R1.01/_docs_install_guides_installing_openvino_linux.html

• Version 2019 R1.0 https://docs.openvinotoolkit.org/2019_R1/_docs_install_guides_installing_openvino_linux.html

• Version 2018 R5.0 https://docs.openvinotoolkit.org/2018_R5/_docs_install_guides_installing_openvino_linux.html

Secondly, make sure that “BOOST” library has been downloaded. If not, execute the following commands:

In third place, it is fundamental for the building process to configure de build environment for the OpenCV toolkit by executing the following command:

Finally, before executing the compilation process be sure to source the helper script. That will make it easier to use environment variables instead of long names to the models: source ../scripts/setupenv.sh

If you encounter a problem like this:

It’s DLib.

DLib has its own BLAS library which tries to compile if it can’t find any of them installed (openblas, intel mkl, libblas). When this happens, it needs to compile its own libjpeg and throws the error mentioned above. There are 2 ways to solve this:

Lightweight solution, install another libjpeg (on Ubuntu):

Recommended solution, install a full BLAS library as it will boost the program’s performance a bit. We recommend installing Intel’s MKL as it works faster and takes advantage of your Intel’s hardware.

You could also install openblas:

or libblas (untested):

With that, DLib shouldn’t compile the file that’s causing the trouble.

• ✓ Short README with usage examples

• ✓ Travis + Sonarcloud

• ✓ Include diagrams and images

• ✓ Elaborate on the wiki

• ✓ Try with different models

• ✓ Face detection

• ✓ Dlib landmark idetification integration

• ✓ Blink/Yawn detection

• ✓ Blink/Yawn time

• ✓ 'Eye out of road' detection

• ✓ Face identification

• ❏ Heart rate + speed/acceleration patterns risk