Face Image Quality Assessment

15.05.2020 SER-FIQ (CVPR2020) was added.

18.05.2020 Bias in FIQ (IJCB2020) was added.

SER-FIQ: Unsupervised Estimation of Face Image Quality Based on Stochastic Embedding Robustness

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2020

• Research Paper
• Implementation on ArcFace
• Video

Table of Contents

• Abstract
• Key Points
• Results
• Installation
• Bias in Face Quality Assessment
• Citing
• Acknowledgement
• License

Abstract

Face image quality is an important factor to enable high-performance face recognition systems. Face quality assessment aims at estimating the suitability of a face image for recognition. Previous works proposed supervised solutions that require artificially or human labelled quality values. However, both labelling mechanisms are error-prone as they do not rely on a clear definition of quality and may not know the best characteristics for the utilized face recognition system. Avoiding the use of inaccurate quality labels, we proposed a novel concept to measure face quality based on an arbitrary face recognition model. By determining the embedding variations generated from random subnetworks of a face model, the robustness of a sample representation and thus, its quality is estimated. The experiments are conducted in a cross-database evaluation setting on three publicly available databases. We compare our proposed solution on two face embeddings against six state-of-the-art approaches from academia and industry. The results show that our unsupervised solution outperforms all other approaches in the majority of the investigated scenarios. In contrast to previous works, the proposed solution shows a stable performance over all scenarios. Utilizing the deployed face recognition model for our face quality assessment methodology avoids the training phase completely and further outperforms all baseline approaches by a large margin. Our solution can be easily integrated into current face recognition systems and can be modified to other tasks beyond face recognition.

Key Points

• Quality assessment with SER-FIQ is most effective when the quality measure is based on the deployed face recognition network, meaning that the quality estimation and the recognition should be performed on the same network. This way the quality estimation captures the same decision patterns as the face recognition system.
• To get accurate quality estimations, the underlying face recognition network for SER-FIQ should be trained with dropout. This is suggested since our solution utilizes the robustness against dropout variations as a quality indicator.
• The provided code is only a demonstration on how SER-FIQ can be utilized. The main contribution of SER-FIQ is the novel concept of measuring face image quality.
• If the last layer contains dropout, it is sufficient to repeat the stochastic forward passes only on this layer. This significantly reduces the computation time to a time span of a face template generation.

Results

Face image quality assessment results are shown below on LFW (left) and Adience (right). SER-FIQ (same model) is based on ArcFace and shown in red. The plots show the FNMR at FMR as recommended by the best practice guidelines of the European Border Guard Agency Frontex. For more details and results, please take a look at the paper.

Installation

We recommend Anaconda to install the required packages. This can be done by creating an virtual environment via

conda env create -f environment.yml

or by manually installing the following packages.

conda create -n serfiq python=3.6.9
conda install cudatoolkit
conda install cudnn
conda install tensorflow=1.14.0
conda install mxnet
conda install mxnet-gpu
conda install tqdm
conda install -c conda-forge opencv
conda install -c anaconda scikit-learn
conda install -c conda-forge scikit-image
conda install keras=2.2.4

After the required packages have been installed, also download the Insightface codebase at the needed git point in the repository history to a location of your choice and extract the archive if necessary.

We will refer to this location as $Insightface in the following.

The path to the Insightface repository must be passed to the InsightFace class in face_image_quality.py. To avoid any problems, absolute paths can be used. Our InsightFace class automatically imports the required dependencies from the Insightface repository.

insightface = InsightFace(insightface_path = $Insightface) # Repository-path as parameter

Please be aware to change the location in our example code according to your setup.

A pre-trained Arcface model is also required. We recommend using the "LResNet100E-IR,This can be downloaded from the Insightface Model-Zoo.

$Insightface/models/

@inproceedings{DBLP:conf/cvpr/TerhorstKDKK20,
  author    = {Philipp Terh{\"{o}}rst and
               Jan Niklas Kolf and
               Naser Damer and
               Florian Kirchbuchner and
               Arjan Kuijper},
  title     = {{SER-FIQ:} Unsupervised Estimation of Face Image Quality Based on
               Stochastic Embedding Robustness},
  booktitle = {2020 {IEEE/CVF} Conference on Computer Vision and Pattern Recognition,
               {CVPR} 2020, Seattle, WA, USA, June 13-19, 2020},
  pages     = {5650--5659},
  publisher = {{IEEE}},
  year      = {2020},
  url       = {https://doi.org/10.1109/CVPR42600.2020.00569},
  doi       = {10.1109/CVPR42600.2020.00569},
  timestamp = {Tue, 11 Aug 2020 16:59:49 +0200},
  biburl    = {https://dblp.org/rec/conf/cvpr/TerhorstKDKK20.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

@article{DBLP:journals/corr/abs-2004-01019,
  author    = {Philipp Terh{\"{o}}rst and
               Jan Niklas Kolf and
               Naser Damer and
               Florian Kirchbuchner and
               Arjan Kuijper},
  title     = {Face Quality Estimation and Its Correlation to Demographic and Non-Demographic
               Bias in Face Recognition},
  journal   = {CoRR},
  volume    = {abs/2004.01019},
  year      = {2020},
  url       = {https://arxiv.org/abs/2004.01019},
  archivePrefix = {arXiv},
  eprint    = {2004.01019},
  timestamp = {Wed, 08 Apr 2020 17:08:25 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-2004-01019.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}