attentive-modality-hopping-for-SER
This repository contains the source code used in the following paper,
Attentive Modality Hopping Mechanism for Speech Emotion Recognition, [paper]
[Notice]
I recently found that I use the "precision" metric for the model evaluation. When I change the metric from "precision" to "accuracy," models show similar performance for the "weighted" case. However, models show lower performance for the "unweighted" case. This behavior is similarly observed for other models (MHA, MDRE).
I already revised the source code. You can change the metric at the "project_config.py."
USE_PRECISION = True --> "precision" metric
USE_PRECISION = False --> "accuracy" metric
Precision (previously misreported as accuracy)
| Model | Modality | Weighted | Unweighted |
|---|---|---|---|
| MDRE[9] | A+T | 0.557 ± 0.018 | 0.536 ± 0.030 |
| MDRE[9] | T+V | 0.585 ± 0.040 | 0.561 ± 0.046 |
| MDRE[9] | A+V | 0.481 ± 0.049 | 0.415 ± 0.047 |
| MHA[12] | A+T | 0.583 ± 0.025 | 0.555 ± 0.040 |
| MHA[12] | T+V | 0.590 ± 0.017 | 0.560 ± 0.032 |
| MHA[12] | A+V | 0.490 ± 0.049 | 0.434 ± 0.060 |
| MDRE[9] | A+T+V | 0.602 ± 0.033 | 0.575 ± 0.046 |
| AMH(ours) | A+T+V | 0.624 ± 0.022 | 0.597 ± 0.040 |
Accuracy (revised results)
| Model | Modality | Weighted | Unweighted |
|---|---|---|---|
| MDRE[9] | A+T | 0.498 ± 0.059 | 0.418 ± 0.077 |
| MDRE[9] | T+V | 0.579 ± 0.015 | 0.524 ± 0.021 |
| MDRE[9] | A+V | 0.477 ± 0.025 | 0.376 ± 0.024 |
| MHA[12] | A+T | 0.543 ± 0.026 | 0.491 ± 0.028 |
| MHA[12] | T+V | 0.580 ± 0.019 | 0.526 ± 0.024 |
| MHA[12] | A+V | 0.471 ± 0.047 | 0.371 ± 0.042 |
| MDRE[9] | A+T+V | 0.564 ± 0.043 | 0.490 ± 0.056 |
| AMH(ours) | A+T+V | 0.617 ± 0.016 | 0.547 ± 0.025 |
[requirements]
tensorflow==1.14 (tested on cuda-10.1, cudnn-7.6)
python==3.7
scikit-learn>=0.20.0
nltk>=3.3
[download data corpus]
- IEMOCAP [link] [paper]
- download IEMOCAP data from its original web-page (license agreement is required)
[preprocessing (our approach)]
-
Get the preprocessed dataset [application link]
If you want to download the "preprocessed dataset," please ask the license to the IEMOCAP team first.
-
For video modality:
- We first split each video frame into two sub-frames so that each segment contains only one actor.
- Then we crop the center of each frame with size 224*224 to focus on the actor and to remove background in the video frame.
- Finally, we extract 2,048-dimensional visual features from each video data using pretrained ResNet-101 at a frame rate of 3 per second.
-
Format of the data for our experiments:
Audio: [#samples, 1000, 120] - (#sampels, sequencs(max 10s), dims)
Text (index) : [#samples, 128] - (#sampels, sequencs(max))
Video: [#samples, 32, 2048] - (#sampels, sequencs (max 10.6s), dims) -
Emotion Classes :
class #samples angry 1,103 excited 1,041 happy 595 sad 1,084 frustrated 1,849 surprise 107 neutral 1,708 -
If you want to use the same processed-data of our experiments, please drop us an email with the IEMOCAP license.
-
We cannot provide ASR-processed transcription due to the license issue (commercial API); however, we assume that it is moderately easy to extract ASR-transcripts from the audio signal by oneself (we used google-cloud-speech-API).
[source code]
- repository contains code for following models
Attentive Modality Hopping (AMH)
[training]
- refer to the "model/train_reference_script.sh"
- Results will be displayed in the console.
- The final test result will be stored in "./TEST_run_result.txt"
[cite]
- Please cite our paper, when you use our code | model | dataset
@inproceedings{yoon2020attentive,
title={Attentive modality hopping mechanism for speech emotion recognition},
author={Yoon, Seunghyun and Dey, Subhadeep and Lee, Hwanhee and Jung, Kyomin},
booktitle={ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
pages={3362--3366},
year={2020},
organization={IEEE}
}