inspur-hsslab / iMIX

ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration

Utils and modules for Speech Language and Multimodal processing using pytorch and pytorch lightning

A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)

[TPAMI Special Issue on ICCV 2021 Best Papers, Oral] Just Ask: Learning to Answer Questions from Millions of Narrated Videos

An alternative EQA paradigm and informative benchmark + models (BMVC 2019, ViGIL 2019 spotlight)

A PyTorch implementation of VIOLET

GAKG is a multimodal Geoscience Academic Knowledge Graph (GAKG) framework by fusing papers' illustrations, text, and bibliometric data.

MultiGraphGAN for predicting multiple target graphs from a source graph using geometric deep learning.

MinkLoc++: Lidar and Monocular Image Fusion for Place Recognition

Official Tensorflow Implementation of the AAAI-2020 paper "Temporally Grounding Language Queries in Videos by Contextual Boundary-aware Prediction"

A paper list of research conducted based on wikiHow

Official Repository for the paper Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approach published in ECCV 2020 (https://arxiv.org/abs/2007.12553)

[ICCV 2021- Oral] Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.

Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.

A Pytorch implementation of CVPR 2020 paper: Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text

LipNet with gluon

Offical implementation of paper "MSAF: Multimodal Split Attention Fusion"

[CVPR 2019 Oral] Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence

TensorFlow implementation of "Attentive Modality Hopping for Speech Emotion Recognition," ICASSP-20

CALVIN - A benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks