All Categories → No Category → multimodal

Top 24 multimodal open source projects

A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)

✭ 4,713

python deep-learning pytorch dialog pretrained-models vqa captioning multimodal multi-tasking textvqa hateful-memes

A CLI tool/python module for generating images from text using guided diffusion and CLIP from OpenAI.

✭ 260

python Jupyter Notebook deep-learning artificial-intelligence openai image-generation multimodality text-to-image diffusion multimodal text-to-image-synthesis openai-clip

Modality-Transferable-MER

Modality-Transferable-MER, multimodal emotion recognition model with zero-shot and few-shot abilities.

✭ 36

python nlp deep-learning pytorch zero-shot emotion-recognition few-shot multimodal cmu-mosei iemocap

slp

Utils and modules for Speech Language and Multimodal processing using pytorch and pytorch lightning

✭ 17

python natural-language-processing pytorch multimodal-learning multimodal multimodal-deep-learning pytorch-lightning wandb

LAVT-pytorch

LAVT: Language-Aware Vision Transformer for Referring Image Segmentation

✭ 16

Jupyter Notebook python pytorch state-of-the-art multimodal referring-image-segmentation

MVGL

TCyb 2018: Graph learning for multiview clustering

✭ 26

matlab clustering unsupervised-learning multimodality multiview-learning multiview-clustering multiple-features multimodal

NER-Multimodal-pytorch

Pytorch Implementation of "Adaptive Co-attention Network for Named Entity Recognition in Tweets" (AAAI 2018)

✭ 42

python tweets pytorch named-entity-recognition ner coattention multimodal attention-network

docarray

The data structure for unstructured data

✭ 561

python shell EJS graphql deep-learning protobuf sqlite data-structures nearest-neighbor-search cross-modal unstructured-data multimodal nested-data weaviate neural-search qdrant docarray

Diverse-Structure-Inpainting

CVPR 2021: "Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE"

✭ 131

python tensorflow attention generative-adversarial-networks inpainting multimodal vq-vae autoregressive-neural-networks

RSTNet

RSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words (CVPR 2021)

✭ 71

python pytorch transformer image-captioning multimodal cvpr2021

nemar

[CVPR2020] Unsupervised Multi-Modal Image Registration via Geometry Preserving Image-to-Image Translation

✭ 120

python shell matlab deep-learning cnn pytorch multi-modal image-registration affine-transformation stn image-to-image-translation multimodal deformable-transformation multi-modal-learning cvpr2020 registartion multimodal-image-registration

Deep-multimodal-subspace-clustering-networks

Tensorflow implementation of "Deep Multimodal Subspace Clustering Networks"

✭ 62

python clustering tensorflow subspace multimodal affinity-fusion

iMIX

A framework for Multimodal Intelligence research from Inspur HSSLAB.

✭ 21

python framework vqa multimodal vision-and-language multimodal-deep-learning

lipnet

LipNet with gluon

✭ 16

python Jupyter Notebook mxnet gluon multimodal lipreading

img2dataset

Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.

✭ 1,173

python Jupyter Notebook shell image big-data deep-learning dataset image-dataset download-images multimodal

gakg

GAKG is a multimodal Geoscience Academic Knowledge Graph (GAKG) framework by fusing papers' illustrations, text, and bibliometric data.

✭ 21

python shell knowledge-graph multimodal

mix-stage

Official Repository for the paper Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approach published in ECCV 2020 (https://arxiv.org/abs/2007.12553)

✭ 22

python HTML style-transfer generative-model multimodal gesture-generation

MinkLocMultimodal

MinkLoc++: Lidar and Monocular Image Fusion for Place Recognition

✭ 65

python computer-vision deep-learning point-cloud metric-learning place-recognition 3d-vision 3d-convolutional-network multimodal minkowski-engine

pytorch-multimodal sarcasm detection

It is the implementation of paper "Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model"

✭ 3

Jupyter Notebook python pytorch sarcasm-detection multimodal

Kaleido-BERT

(CVPR2021) Kaleido-BERT: Vision-Language Pre-training on Fashion Domain.

✭ 252

python shell fashion e-commerce bert multimodal pre-training vision-language

pykale

Knowledge-Aware machine LEarning (KALE): accessible machine learning from multiple sources for interdisciplinary research, part of the 🔥PyTorch ecosystem

✭ 381

python data-science machine-learning computer-vision deep-learning pytorch transfer-learning graph-analysis domain-adaptation meta-learning medical-image-analysis multimodal-learning multimodal knowledge-aware-learning

Fengshenbang-LM

Fengshenbang-LM(封神榜大模型)是IDEA研究院认知计算与自然语言研究中心主导的大模型开源体系，成为中文AIGC和认知智能的基础设施。

✭ 1,813

python shell C++transformers pytorch chinese-nlp pretrained-models distributed-training multimodal aigc

VideoNavQA

An alternative EQA paradigm and informative benchmark + models (BMVC 2019, ViGIL 2019 spotlight)

✭ 22

python shell benchmark machine-learning deep-neural-networks video navigation vqa question-answering visual-reasoning multimodal embodied cross-modality conditioning videonavqa

tsflex

Flexible time series feature extraction & processing

✭ 252

python Makefile data-science time-series pandas feature-extraction multivariate feature-engineering multimodal window-stride

1-24 of 24 multimodal projects