Video Face Manipulation Detection Through Ensemble of CNNs

This is the official repository of Video Face Manipulation Detection Through Ensemble of CNNs, accepted to ICPR2020 and currently available on arXiv.

We participated as the ISPL team in the Kaggle Deepfake Detection Challenge. With this implementation, we reached the 41st position over 2116 teams (top 2%) on the private leaderboard.

This repository is currently under maintenance, if you are experiencing any problems, please open an issue.

Getting started

Prerequisites

• Install conda
• Create the icpr2020 environment with environment.yml

$ conda env create -f environment.yml
$ conda activate icpr2020

Quick run

If you just want to test the pre-trained models against your own videos or images:

• Video prediction notebook
• Image prediction notebook

The whole pipeline

You need to preprocess the datasets in order to index all the samples and extract faces. Just run the script make_dataset.sh

$ ./scripts/make_dataset.sh

Please note that we use only 32 frames per video. You can easily tweak this parameter in extract_faces.py

Celeb-DF (v2)

Altough we did not use this dataset in the paper, we provide a script index_celebdf.py to index the videos similarly to DFDC and FF++. Once you have the index, you can proceed with the pipeline starting from extract_faces.py.

Train

In train_all.sh you can find a comprehensive list of all the commands to train the models presented in the paper. Please refer to the comments in the script for hints on their usage.

Training a single model

If you want to train some models without lunching the script:

• for the non-siamese architectures (e.g. EfficientNetB4, EfficientNetB4Att), you can simply specify the model in train_binclass.py with the --net parameter;
• for the siamese architectures (e.g. EfficientNetB4ST, EfficientNetB4AttST), you have to:
  1. train the architecture as a feature extractor first, using the train_triplet.py script and being careful of specifying its name with the --net parameter without the ST suffix. For instance, for training the EfficientNetB4ST you will have to first run python train_triplet.py --net EfficientNetB4 --otherparams;
  2. finetune the model using train_binclass.py, being careful this time to specify the architecture's name with the ST suffix and to insert as --init argument the path to the weights of the feature extractor trained at the previous step. You will end up running something like python train_binclass.py --net EfficientNetB4ST --init path/to/EfficientNetB4/weights/trained/with/train_triplet/weights.pth --otherparams

Test

In test_all.sh you can find a comprehensive list of all the commands for testing the models presented in the paper.

Pretrained weights

We also provide pretrained weights for all the architectures presented in the paper. Please refer to this Dropbox link. Each directory is named $NETWORK_$DATASET where $NETWORK is the architecture name and $DATASET is the training dataset. In each directory, you can find bestval.pth which are the best network weights according to the validation set.

Additionally, you can find notebooks for results computations in the notebook folder.

Datasets

• Facebook's DeepFake Detection Challenge (DFDC) train dataset | arXiv paper
• FaceForensics++ | arXiv paper
• Celeb-DF (v2) | arXiv paper (Just for reference, not used in the paper)

References

• EfficientNet PyTorch
• Xception PyTorch

Credits

Image and Sound Processing Lab - Politecnico di Milano

• Nicolò Bonettini
• Edoardo Daniele Cannas
• Sara Mandelli
• Luca Bondi
• Paolo Bestagini