Engagement Recognition Model

🤗 TensorFlow & TFLearn implementation of Automatic Recognition of Student Engagement using Deep Learning and Facial Expression:

Engagement is a key indicator of the quality of learning experience, and one that plays a major role in developing intelligent educational interfaces. Any such interface requires the ability to recognise the level of engagement in order to respond appropriately; however, there is very little existing data to learn from, and new data is expensive and difficult to acquire. This work presents a deep learning model to improve engagement recognition from images that overcomes the data sparsity challenge by pre-training on readily available basic facial expression data, before training on specialised engagement data. In the first of two steps, a facial expression recognition model is trained to provide a rich face representation using deep learning. In the second step, we use the model's weights to initialize our deep learning based model to recognize engagement; we term this the engagement model. We train the model on our new engagement recognition dataset with 4627 engaged and disengaged samples. We find that the engagement model outperforms effective deep learning architectures that we apply for the first time to engagement recognition, as well as approaches using histogram of oriented gradients and support vector machines.

Reference

🤗 if you use our code or model, please cite our paper:

@inproceedings{nezami2019automatic,
  title={Automatic recognition of student engagement using deep learning and facial expression},
  author={Nezami, Omid Mohamad and Dras, Mark and Hamey, Len and Richards, Deborah and Wan, Stephen and Paris, C{\'e}cile},
  booktitle={Joint European Conference on Machine Learning and Knowledge Discovery in Databases},
  pages={273--289},
  year={2019},
  organization={Springer}
}

Requiremens

1. Python 2.7.12

   sudo apt update
   sudo apt upgrade
   sudo apt install python2.7 python-pip
   

2. Numpy 1.15.2

   pip install numpy==1.15.2
   

3. DLib, Pillow and OpenCV

   pip install dlib
   pip install pillow
   pip install opencv-python
   

4. Tensorflow 1.8.0 and TFLearn

   pip install tensorflow==1.8.0
   pip install tflearn
   

Data

We used two main datasets:

First, we pre-train our model on the facial expression recognition (FER) dataset. The dataset includes images, labeled {happiness, anger, sadness, surprise, fear, disgust, neutral}. It contains 35,887 samples (28,709 for the training set, 3589 for the public test set and 3589 for the private test set), collected by the Google search API. The samples are in grayscale at the size of 48-by-48 pixels.

Second, we train the model on our new engagement recognition (ER) dataset with 4627 engaged and disengaged samples. We split the ER dataset into training (3224), validation (715), and testing (688) sets, which are subject-independent (the samples in these three sets are from different subjects).

Content

1. CNN Model Source Code for training a basic CNN architecture on the ER dataset or your engagement dataset
2. VGG Model Source Code for training a VGG architecture on the ER dataset or your engagement dataset
3. Engagement Model Source Code for fine-tuning a pre-trained VGG architecture on the ER dataset or your engagement dataset

🤗 Please see below for more detail about data preparation, engagement model training and testing

Pre-Processing and Data Preparation

We applied a similar pre-processing step for both FER and ER datasets:

We used a CNN based face detection algorithm to detect the face of each sample. If there is more than one face in a sample, we choose the face with the biggest size. Then, the face is transformed to grayscale and resized into 48-by-48 pixels.

# Place your images in the data directory
cd code/
# Download and unzip the mmod_human_face_detector.dat file from "http://dlib.net/files/mmod_human_face_detector.dat.bz2"
python preprocess.py

This will generate the preprocess images as numpy files in the the preprocess_data directory. (Here, for testing purposes, we only place 5 images from FER dataset in the data directory and create a sample numpy file for training (X_train.npy) in the preprocess_data directory with all zero labels (Y_train.npy). You need to apply this step for all your images with correct labels. You also need to create testing and validation numpy files.)

Train from Scratch

1. The CNN Model:

   # Specify the name of the saved model in 'CNN_const.py' e.g., SAVE_MODEL_FILENAME = CNN_model
   cd code/
   python CNN_model.py train
   

   or

2. The VGG Model:

   # Specify the name of the saved model in 'VGG_const.py' e.g., SAVE_MODEL_FILENAME = VGG_model
   cd code
   python VGG_model.py train
   

Train Engagement Model

1. Download and untar the pretrained model on the ER dataset

   wget https://cloudstor.aarnet.edu.au/plus/s/LqqdgwJ69NEdnDS/download
   tar -xzf download
   

2. Create a directory called 'model' beside 'code' directory
3. Place the downloaded model in the directory

   mv TF_start/* model/
   rm -r TF_start
   

4. Run the model's script:

   # Specify the name of the saved model in 'ER_const.py' e.g., SAVE_MODEL_FILENAME = TF_final
   # uncomment self.model.load( join(SAVE_DIRECTORY, SAVE_MODEL_FILENAME), weights_only=True ) and restore=False in build_network(self) function
   cd code/
   python ER_model.py train
   

Test Engagement Model

1. Download and untar the trained model on the ER dataset

   wget https://cloudstor.aarnet.edu.au/plus/s/i3oPqcjXhG7Ymva/download
   tar -xzf download
   

2. Create a directory called 'model' beside 'code' directory if you didn't before
3. Place the files of the downloaded model in the the directory

   mv TF_final/* model/
   rm -r TF_final
   

4. Run the model's script:

   cd code/
   python ER_model.py test
   

5. Check predictions on sample data

   python2
   
   import numpy as np
   predictions= np.load('test_f.npy')
   print predictions
   

Results on the ER Test Set

The CNN Model is inspired from Emotion recognition with CNN.