Masked Face Representation

Masked face recognition focuses on identifying people using their facial features while they are wearing masks. We introduce benchmarks on face verification and facial attribution prediction based on masked face images corresponding to multiple published articles.

Data and Ethics

All models, code and research presented in this repo is provided for reproducibility purposes. Our work is intended for use for helping security-related applications while maintaining COVID-safe protocols (keep your mask on at airport security to maintain a lower chance of spreading COVID) and not for invading individual privacy. We invite researchers to consider the broader implications of perfecting state of the art result work in areas such as masked identification. We believe everyone has the right to their privacy and researchers should consider the broader implications of their work and potential for misuse.

The accuracy and performance of models released in this repo should be sufficient for health-related applications (such as safety protocols) but would not (we hope) be usable for tracking an individuals' movements.

If you are interested in using this work for COVID-Safety applications we are happy to consult on a pro bono basis.

Features

• Masked face identification
  • Generate Synthetic masks to CelebA, Fei Face, georgia_tech, SoF, YoutubeFaces and LFW datasets
  • Apply synthetic mask to a face image
  • Apply synthetic masks to a folder of images
  • [DICTA2021] Regenerate results in the paper : Multi-Dataset Benchmarks for Masked Identification using Contrastive Representation Learning
    • Regenerate benchmark 1 results from the models
    • Regenerate benchmark 2 results from scores files
    • Regenerate benchmark 2 results from the models
• Masked face privacy
  • Generate Synthetic masks to UTKFace dataset
  • Split the dataset with proper distribution among training, validation and test datasets
  • [AJCAI2021] Regenerate results in the paper : Does a Face Mask Protect my Privacy?: DeepLearning to Predict Protected Attributes fromMasked Face Images
    • Regenerate the overall accuracy for models
    • Regenerate the results of masked and unmasked faces

Masked face identification

Conference Venue: DICTA2021

Download the benchmark datasets

Downloading instructions

celeba

1. Go to the google drive link
2. Go inside img_align_celeba_png folder
3. Download img_align_celeba_png.7z001 - img_align_celeba_png.7z016 all files
4. Combine them and unzip it using 7zip
5. Unzip the file to celeba/images folder
6. Download annotations identity_CelebA.txt from the following link https://drive.google.com/drive/u/1/folders/0B7EVK8r0v71pOC0wOVZlQnFfaGs?resourcekey=0-pEjrQoTrlbjZJO2UL8K_WQ
7. Place annotations identity_CelebA.txt under celeba/

youtube_faces

1. Fill the form by providing your name and email.
2. Get the username and password to login to the provided website.
3. Click on frame_images_DB then go to YouTube Faces(YTF) data set.
4. Download YoutubeFaces.tar.gz
5. Unzip under youtube_faces

Folder Structure after downloading the datasets

base_folder : Base folder for all the datasets

• celeba
  • images
  • identity_CelebA.txt
• fei_face_frontal
• fei_face_original
• georgia_tech
• lfw
• sof_original
• youtube_faces

Generate Synthetic masks to benchmark datasets

Install python 3.6. Then run the following command to install the requirements.

pip install -r requirements.txt

To apply synthetic masks on the datasets

python generate_mask_to_datasets.py --base_folder ../base_folder --new_dataset_folder ../benchmark_dataset

Summary of the datasets for DICTA2021

Experiments

Our network architecture (Siamese Network Architecture)

Benchmark 1

Results

Regenerate benchmark 1 results

cd DICTA2021
python regenerate_benchmark_1_from_models.py --base_folder ../base_folder

Benchmark 2

Results

Regenerate benchmark 2 results

cd DICTA2021
python regenerate_experiment_2_results_from_models.py --base_folder ../base_folder

Overall benchmark: results of Exp1(trained on CelebA for 1015k steps) and Ensemble(trained on 4 datasets) on the synthetic unmasked-masked datasets generated

Regenerate overall benchmark

cd DICTA2021
python regenerate_table_6_results_from_models.py --base_folder ../base_folder

FPR/TPR curve, for all datasets using CP1 model

The relative difficulty of different datasets can be visualized based on CP1 model.

Results

Additional Results

FNMR1000 rates for experiment 2

FNMR0 rates for experiment 2

These results indicate the performance of the explored models on the different datasets

Evaluation on using more data points for analysis from each dataset.

We run the multi dataset-trained models through a larger test-set from each dataset. We draw a single image at random for each identity to serve as the reference, and a single authentic and imposter image to act as probes for each identity. So, this test uses 2 tests per identity for each identity in each dataset. We present the 1000-sample test set in the paper as a) the results are easily reproducible and b) the size of the test is fixed (except for in-house data, which is only used for validation purposes).

Results on all identities (2*n pairs of images with n identities)

Result on 1000 pairs

Masked face privacy

Conference Venue: AJCAI2021

Download the benchmark datasets

Downloading instructions

UTKFace

1. Download the dataset using the given URLs.
2. Download part1.tar.gz, part2.tar.gz and part3.tar.gz
3. Unzip the files and get the images to UTKFace folder

Folder Structure after downloading the datasets

base_folder : Base folder for the dataset (This is where you unzip and copy UTKFaces folder)

• UTKFace

Generate Synthetic masks to benchmark datasets

Install python 3.6. Then run the following command to install the requirements.

pip install -r requirements.txt

To apply synthetic masks on the datasets

python generate_mask_to_datasets.py --base_folder ../base_folder --new_dataset_folder ../benchmark_dataset

Summary of the datasets for AJCAI2021

Age buckets used for classification

Overall prediction comparison with confusion matrices

Overall accuracy for models

Results

Regenerate attribute prediction accuracy results with transforms

Results for masked and no mask faces

Results

Regenerate overall accuracy results

build Sex, Race, Age and Age Classification datasets using UTKFaces. Then execute utkfaces_dataset.py. dataset is the original UTKFACE dataset folder. output is the new masked dataset output directory.The types are age, race, sex and age_classification

cd AJCAI2021/dataset_util
python utkfaces_dataset.py --dataset "./utkfacesdataset_path" --output "../new_masked_utk_face_dataset_path" --type age

Download the models to AJCAI2021/dataset_util folder. Then configure the config.json for each model related dataset path, model path and accuracy type (mae or percentage). Then execute regenerate_overall_accuracy_results.py as follows.

cd AJCAI2021
python regenerate_overall_accuracy_results.py

License

MIT

Citation

If you use the repository for masked face identification, please use the following citation:

@INPROCEEDINGS{seneviratne2021multidataset, author={Seneviratne, Sachith and Kasthuriarachchi, Nuran and Rasnayaka, Sanka}, booktitle={2021 Digital Image Computing: Techniques and Applications (DICTA)}, title={Multi-Dataset Benchmarks for Masked Identification using Contrastive Representation Learning}, year={2021}, volume={}, number={}, pages={01-08}, doi={10.1109/DICTA52665.2021.9647194}}

If you use the repository for masked face privacy, please use the following citation:

@misc{seneviratne2021does, title={Does a Face Mask Protect my Privacy?: Deep Learning to Predict Protected Attributes from Masked Face Images}, author={Sachith Seneviratne and Nuran Kasthuriarachchi and Sanka Rasnayaka and Danula Hettiachchi and Ridwan Shariffdeen}, year={2021}, eprint={2112.07879}, archivePrefix={arXiv}, primaryClass={cs.CV} }

Acknowledgements

• This project is supported by National Health and Medical Research Grant GA80134.
• This research was undertaken using the LIEF HPC-GPGPU Facility hosted at the University of Melbourne. This Facility was established with the assistance of LIEF Grant LE170100200.
• This research was undertaken using University of Melbourne Research Computing facilities established by the Petascale Campus Initiative.