GitPlanet

Cheap and reliable Node.js hosting starts at $3/month, and $1/month static HTML hosting

Created with love in Canada, visit hostnodejs.com today

Feel like to post an Ad? Learn Details
All Projects → declare-lab → BBFN

declare-lab / BBFN

Licence: MIT license
This repository contains the implementation of the paper -- Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis

Programming Languages

python
139335 projects - #7 most used programming language
shell
77523 projects

Labels

multimodal-sentiment-analysismultimodal-deep-learningmultimodal-representationhuggingface-transformers

Projects that are alternatives of or similar to BBFN

MSAF
Offical implementation of paper "MSAF: Multimodal Split Attention Fusion"
Stars: ✭ 47 (+11.9%)
Mutual labels:  multimodal-sentiment-analysis, multimodal-deep-learning
Self-Supervised-Embedding-Fusion-Transformer
The code for our IEEE ACCESS (2020) paper Multimodal Emotion Recognition with Transformer-Based Self Supervised Feature Fusion.
Stars: ✭ 57 (+35.71%)
Mutual labels:  multimodal-sentiment-analysis, multimodal-deep-learning
Social-IQ
[CVPR 2019 Oral] Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence
Stars: ✭ 37 (-11.9%)
Mutual labels:  multimodal-deep-learning
TensorFusionNetwork
EMNLP 2017 (Oral): Tensor Fusion Network for Multimodal Sentiment Analysis Code
Stars: ✭ 55 (+30.95%)
Mutual labels:  multimodal-sentiment-analysis
Multimodal-Future-Prediction
The official repository for the CVPR 2019 paper "Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction"
Stars: ✭ 38 (-9.52%)
Mutual labels:  multimodal-deep-learning
attentive-modality-hopping-for-SER
TensorFlow implementation of "Attentive Modality Hopping for Speech Emotion Recognition," ICASSP-20
Stars: ✭ 25 (-40.48%)
Mutual labels:  multimodal-deep-learning
muscaps
Source code for "MusCaps: Generating Captions for Music Audio" (IJCNN 2021)
Stars: ✭ 39 (-7.14%)
Mutual labels:  multimodal-deep-learning
MultiGraphGAN
MultiGraphGAN for predicting multiple target graphs from a source graph using geometric deep learning.
Stars: ✭ 16 (-61.9%)
Mutual labels:  multimodal-deep-learning
vista-net
Code for the paper "VistaNet: Visual Aspect Attention Network for Multimodal Sentiment Analysis", AAAI'19
Stars: ✭ 67 (+59.52%)
Mutual labels:  multimodal-sentiment-analysis
MISE
Multimodal Image Synthesis and Editing: A Survey
Stars: ✭ 214 (+409.52%)
Mutual labels:  multimodal-deep-learning
hateful memes-hate detectron
Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge. https://arxiv.org/abs/2012.12975
Stars: ✭ 35 (-16.67%)
Mutual labels:  multimodal-deep-learning
multimodal-deep-learning-for-disaster-response
Damage Identification in Social Media Posts using Multimodal Deep Learning: code and dataset
Stars: ✭ 43 (+2.38%)
Mutual labels:  multimodal-deep-learning
scarches
Reference mapping for single-cell genomics
Stars: ✭ 175 (+316.67%)
Mutual labels:  multimodal-deep-learning
mmd
This repository contains the Pytorch implementation for our SCAI (EMNLP-2018) submission "A Knowledge-Grounded Multimodal Search-Based Conversational Agent"
Stars: ✭ 28 (-33.33%)
Mutual labels:  multimodal-deep-learning
hfusion
Multimodal sentiment analysis using hierarchical fusion with context modeling
Stars: ✭ 42 (+0%)
Mutual labels:  multimodal-sentiment-analysis
circDeep
End-to-End learning framework for circular RNA classification from other long non-coding RNA using multimodal deep learning
Stars: ✭ 21 (-50%)
Mutual labels:  multimodal-deep-learning
Robust-Deep-Learning-Pipeline
Deep Convolutional Bidirectional LSTM for Complex Activity Recognition with Missing Data. Human Activity Recognition Challenge. Springer SIST (2020)
Stars: ✭ 20 (-52.38%)
Mutual labels:  multimodal-deep-learning
finetune-gpt2xl
Guide: Finetune GPT2-XL (1.5 Billion Parameters) and finetune GPT-NEO (2.7 B) on a single GPU with Huggingface Transformers using DeepSpeed
Stars: ✭ 353 (+740.48%)
Mutual labels:  huggingface-transformers
referit3d
Code accompanying our ECCV-2020 paper on 3D Neural Listeners.
Stars: ✭ 59 (+40.48%)
Mutual labels:  multimodal-deep-learning
slp
Utils and modules for Speech Language and Multimodal processing using pytorch and pytorch lightning
Stars: ✭ 17 (-59.52%)
Mutual labels:  multimodal-deep-learning
View All Similar Projects ➔

Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis

🔥🔥 BBFN has won the best paper award honourable mention at ICMI 2021!

This repository contains official implementation of the paper: Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis (ICMI 2021)

💎 If you would be interested in other multimodal works in our DeCLaRe Lab, please visit the clustered repository

Model Architecture

Overview of our Bi-Bimodal Fusion Network (BBFN). It learns two text-related pairs of representations, text-acoustic and text-visual by enforcing each pair of modalities to complement mutually. Finally, the four (two pairs) head representations are concatenated to generate the final prediction.

Alt text

A single complementation layer: two identical pipelines (left and right) propagate the main modality and fuse that with complementary modality with regularization and gated control.

Alt text

Results

Results on the test set of CMU-MOSI and CMU-MOSEI dataset. Notation: △ indicates results in the corresponding line are excerpted from previous papers; † means the results are reproduced with publicly visible source code and applicable hyperparameter setting; ‡ shows the results have experienced paired t-test with 𝑝 < 0.05 and demonstrate significant improvement over MISA, the state-of-the-art model.

Alt text

Usage

  1. Set up conda environemnt
conda env create -f environment.yml
conda activate BBFN

  1. Install CMU Multimodal SDK

  2. Set sdk_dir in src/config.py to the path of CMU-MultimodalSDK

  3. Train the model

cd src
python main.py --dataset <dataset_name> --data_path <path_to_dataset>

We provide a script scripts/run.sh for your reference.

Citation

Please cite our paper if you find our work useful for your research:

@article{han2021bi,
  title={Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis},
  author={Han, Wei and Chen, Hui and Gelbukh, Alexander and Zadeh, Amir and Morency, Louis-philippe and Poria, Soujanya},
  journal={ICMI 2021},
  year={2021}
}

Contact

Should you have any question, feel free to contact me through [email protected]

Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].