Gaze Preserving CycleGAN (GPCyceGAN) & Other Driver Gaze Estimation Models and Datasets
PyTorch implementation of the training and inference procedures described in the papers:
- On Generalizing Driver Gaze Zone Estimation using Convolutional Neural Networks," IEEE Intelligent Vehicles Symposium, 2017
- Driver Gaze Zone Estimation using Convolutional Neural Networks: A General Framework and Ablative Analysis," IEEE Transactions on Intelligent Vehicles, 2018
- Driver Gaze Estimation in the Real World: Overcoming the Eyeglass Challenge, IEEE Intelligent Vehicles Symposium, 2020.
- Gaze Preserving CycleGANs for Eyeglass Removal & Persistent Gaze Estimation, arXiv:2002.02077, 2020.
Parts of the CycleGAN code have been adapted from the PyTorch-CycleGAN respository.
Installation
- Clone this repository
- Install Pipenv:
pip3 install pipenv- Install all requirements and dependencies in a new virtual environment using Pipenv:
cd GPCycleGAN
pipenv install- Get link for desired PyTorch and Torchvision wheel from here and install it in the Pipenv virtual environment as follows:
pipenv install https://download.pytorch.org/whl/cu100/torch-1.2.0-cp36-cp36m-manylinux1_x86_64.whl
pipenv install https://download.pytorch.org/whl/cu100/torchvision-0.3.0-cp36-cp36m-linux_x86_64.whlDatasets
LISA Gaze Dataset v0
This is the dataset introduced in the papers On Generalizing Driver Gaze Zone Estimation using Convolutional Neural Networks and Driver Gaze Zone Estimation using Convolutional Neural Networks: A General Framework and Ablative Analysis. To use this dataset, do the following:
- Download the complete RGB dataset for driver gaze classification using this link.
- Unzip the file.
LISA Gaze Dataset v1
This is the second dataset introduced in the paper Driver Gaze Zone Estimation using Convolutional Neural Networks: A General Framework and Ablative Analysis. To use this dataset, do the following:
- Download the complete RGB dataset for driver gaze classification using this link.
- Unzip the file.
LISA Gaze Dataset v2
This is the dataset introduced in the paper Driver Gaze Estimation in the Real World: Overcoming the Eyeglass Challenge. To use this dataset, do the following:
- Download the complete IR+RGB dataset for driver gaze classification using this link.
- Unzip the file.
- Prepare the train, val and test splits as follows:
python prepare_gaze_data.py --dataset-dir=/path/to/lisat_gaze_data_v2Training (v0 RGB data)
The best performing SqueezeNet gaze classifier can be trained using the following command:
pipenv shell # activate virtual environment
python gazenet.py --dataset-root-path=/path/to/lisat_gaze_data_v0/ --version=1_1 --snapshot=./weights/squeezenet1_1_imagenet.pth --random-transformsTraining (v1 RGB data)
The best performing SqueezeNet gaze classifier can be trained using the following command:
pipenv shell # activate virtual environment
python gazenet.py --dataset-root-path=/path/to/lisat_gaze_data_v1/ --version=1_1 --snapshot=./weights/squeezenet1_1_imagenet.pth --random-transformsTraining (v2 IR data)
The prescribed three-step training procedure can be carried out as follows:
Step 1: Train the gaze classifier on images without eyeglasses
pipenv shell # activate virtual environment
python gazenet.py --dataset-root-path=/path/to/lisat_gaze_data_v2/ir_no_glasses/ --version=1_1 --snapshot=./weights/squeezenet1_1_imagenet.pth --random-transformsStep 2: Train the GPCycleGAN model using the gaze classifier from Step 1
python gpcyclegan.py --dataset-root-path=/path/to/lisat_gaze_data_v2/ --data-type=ir --version=1_1 --snapshot-dir=/path/to/trained/gaze-classifier/directory/ --random-transformsStep 3.1: Create fake images using the trained GPCycleGAN model
python create_fake_images.py --dataset-root-path=/path/to/lisat_gaze_data_v2/ir_all_data/ --snapshot-dir=/path/to/trained/gpcyclegan/directory/
cp /path/to/lisat_gaze_data_v2/ir_all_data/mean_std.mat /path/to/lisat_gaze_data_v2/ir_all_data_fake/mean_std.mat # copy over dataset mean/std information to fake data folderStep 3.2: Finetune the gaze classifier on all fake images
python gazenet-ft.py --dataset-root-path=/path/to/lisat_gaze_data_v2/ir_all_data_fake/ --version=1_1 --snapshot-dir=/path/to/trained/gpcyclegan/directory/ --random-transforms
exit # exit virtual environmentTraining (v2 RGB data)
The prescribed three-step training procedure can be carried out as follows:
Step 1: Train the gaze classifier on images without eyeglasses
pipenv shell # activate virtual environment
python gazenet.py --dataset-root-path=/path/to/lisat_gaze_data_v2/rgb_no_glasses/ --version=1_1 --snapshot=./weights/squeezenet1_1_imagenet.pth --random-transformsStep 2: Train the GPCycleGAN model using the gaze classifier from Step 1
python gpcyclegan.py --dataset-root-path=/path/to/lisat_gaze_data_v2/ --data-type=rgb --version=1_1 --snapshot-dir=/path/to/trained/gaze-classifier/directory/ --random-transformsStep 3.1: Create fake images using the trained GPCycleGAN model
python create_fake_images.py --dataset-root-path=/path/to/lisat_gaze_data_v2/rgb_all_data/ --snapshot-dir=/path/to/trained/gpcyclegan/directory/
cp /path/to/lisat_gaze_data_v2/rgb_all_data/mean_std.mat /path/to/lisat_gaze_data_v2/rgb_all_data_fake/mean_std.mat # copy over dataset mean/std information to fake data folderStep 3.2: Finetune the gaze classifier on all fake images
python gazenet-ft.py --dataset-root-path=/path/to/lisat_gaze_data_v2/rgb_all_data_fake/ --version=1_1 --snapshot-dir=/path/to/trained/gpcyclegan/directory/ --random-transforms
exit # exit virtual environmentInference (v0 RGB data)
Inference can be carried out using this script as follows:
pipenv shell # activate virtual environment
python infer.py --dataset-root-path=/path/to/lisat_gaze_data_v0/ --split=val --version=1_1 --snapshot-dir=/path/to/trained/rgb-model/directory/ --save-viz
exit # exit virtual environmentInference (v1 RGB data)
Inference can be carried out using this script as follows:
pipenv shell # activate virtual environment
python infer.py --dataset-root-path=/path/to/lisat_gaze_data_v0/ --split=val --version=1_1 --snapshot-dir=/path/to/trained/rgb-model/directory/ --save-viz
exit # exit virtual environmentInference (v2 IR data)
Inference can be carried out using this script as follows:
pipenv shell # activate virtual environment
python infer.py --dataset-root-path=/path/to/lisat_gaze_data_v2/ir_all_data/ --split=test --version=1_1 --snapshot-dir=/path/to/trained/ir-models/directory/ --save-viz
exit # exit virtual environmentInference (v2 RGB data)
Inference can be carried out using this script as follows:
pipenv shell # activate virtual environment
python infer.py --dataset-root-path=/path/to/lisat_gaze_data_v2/rgb_all_data/ --split=val --version=1_1 --snapshot-dir=/path/to/trained/rgb-models/directory/ --save-viz
exit # exit virtual environmentPre-trained Weights
You can download our pre-trained model weights using this link.
Config files, logs, results, snapshots, and visualizations from running the above scripts will be stored in the GPCycleGAN/experiments folder by default.
Citations
If you find our data, code, and/or models useful in your research, please consider citing the following papers:
@inproceedings{vora2017generalizing,
title={On generalizing driver gaze zone estimation using convolutional neural networks},
author={Vora, Sourabh and Rangesh, Akshay and Trivedi, Mohan M},
booktitle={2017 IEEE Intelligent Vehicles Symposium (IV)},
pages={849--854},
year={2017},
organization={IEEE}
}
@article{vora2018driver,
title={Driver gaze zone estimation using convolutional neural networks: A general framework and ablative analysis},
author={Vora, Sourabh and Rangesh, Akshay and Trivedi, Mohan Manubhai},
journal={IEEE Transactions on Intelligent Vehicles},
volume={3},
number={3},
pages={254--265},
year={2018},
publisher={IEEE}
}
@article{rangesh2020driver,
title={Gaze Preserving CycleGANs for Eyeglass Removal & Persistent Gaze Estimation},
author={Rangesh, Akshay and Zhang, Bowen and Trivedi, Mohan M},
journal={arXiv preprint arXiv:2002.02077},
year={2020}
}