Shallow Optical Flow Three-Stream CNN For Macro and Micro-Expression Spotting From Long Videos

Framework of Proposed SOFTNet approach

Overall framework:

Mainly four phases involved:

• Feature Extraction - Extract the optical flow features (u, v, ε) that represents each frame.
• Pre-processing - Remove global head motion, eye masking, ROI selection, and image resampling.
• SOFTNet - Three-stream shallow architecture that takes inputs (u, v, ε) and outputs a spotting confidence score.
• Spotting - Smoothing spotting confidence score, then perform thresholding and peak detection to obtain the spotted interval for evaluation.

Training

Tensorflow and Keras are used in the experiment. Two datasets with macro- and micro-expression are used for training and testing purposes:

CAS(ME)² - http://fu.psych.ac.cn/CASME/cas(me)2-en.php

SAMM Long Videos - http://www2.docm.mmu.ac.uk/STAFF/M.Yap/dataset.php

Results

Evaluation

Comparison between the proposed approaches against baseline and state-of-the-art approaches in Third Facial Micro-Expression Grand Challenge (MEGC 2020) in terms of F1-Score:

Visualization

Samples visual results for SOFTNet:

Discussion

The proposed SOFTNet approach outperforms other methods on CAS(ME)² while ranked second on SAMM Long Videos. To better justify the effectiveness of the SOFTNet approach, we experimented with a similar framework but without SOFTNet, the results show that the framework with SOFTNet is much more efficient overall.

Visually, SOFTNet activation units show our intuition to concatenate the optical flow features (u, v, ε) from three-stream. The spatio-temporal motion information is captured when macro and micro-expression occur. After the concatenation, action unit 4 (Brow Lower) is triggered when a disgust emotion is elicited.

Reproduce the results for SOFTNet approach (with python script)

Step 1) Download datasets, CAS(ME)² (CASME_sq) and SAMM Long Videos (SAMMLV) and placed in the structure as follows:

├─SOFNet_Weights
├─Utils
├─extraction_preprocess.py
├─load_images.py
├─load_label.py
├─main.py
├─requirements.txt
├─training.py
├─CASME_sq

├─CAS(ME)^2code_final.xlsx
├─cropped
├─rawpic
├─rawvideo
└─selectedpic

├─SAMMLV

├─SAMM_longvideos
└─SAMM_LongVideos_V1_Release.xlsx

Step 2) Installation of packages using pip

pip install -r requirements.txt

Step 3) SOFTNet Training and Evaluation

python main.py

Note for parameter settings

  --dataset_name (CASME_sq or SAMMLV)
  --expression_type (micro-expression or macro-expression)
  --train (True or False)
  --show_plot (True or False)

Note for pre-trained weights

The pre-trained weights for CAS(ME)²and SAMM Long Videos with macro and micro-expression separately are located under folder SOFTNet_Weights. You may load the weights for evaluation. However, the result is slightly different from the result given in the table shown above.

Link to research paper

If you find this work useful, please cite the paper: https://arxiv.org/pdf/2106.06489.pdf

@inproceedings{liong2021shallow,
title={Shallow optical flow three-stream CNN for macro-and micro-expression spotting from long videos},
author={Liong, Gen-Bing and See, John and Wong, Lai-Kuan},
booktitle={2021 IEEE International Conference on Image Processing (ICIP)},
pages={2643--2647},
year={2021},
organization={IEEE}
}

Please email me at [email protected] if you have any inquiries or issues.