If you want to reproduce the pretraining, download HowToVQA69M:

bash download/download_howtovqa.sh <DEFAULT_DATASET_DIR>

This requires having about 6Gb free in DEFAULT_DATASET_DIR. You will also need to download features for videos from HowTo100M from the data providers in HOWTO_FEATURES_PATH.

If you want to reproduce the pretraining, download WebVidVQA3M:

bash download/download_webvidvqa.sh <DEFAULT_DATASET_DIR>

This requires having about 1Gb free in DEFAULT_DATASET_DIR. You will also need to download videos from WebVid from the data providers, and extract features following the instructions below in WEBVID_FEATURES_PATH.

The following folders should be created in DEFAULT_DATASET_DIR, and should also contain a video subfolder containing the videos downloaded from each dataset.

HowToVQA69M: We provide the HowToVQA69M dataset at this link. The HowToVQA69M folder should contain howtovqa.pkl, train_howtovqa.csv and val_howtovqa.csv.

iVQA: We provide the iVQA dataset at this link. The iVQA folder should contain train.csv, val.csv and test.csv.

MSRVTT-QA: Download it from the data providers. The MSRVTT-QA folder should contain train_qa.json, val_qa.json, test_qa.json, and also train_val_videodatainfo.json and test_videodatainfo.json. The two last files are from the MSR-VTT dataset, and are used to filter out video IDs in HowTo100M that are in the validation and test sets of MSRVTT-QA.

MSVD-QA: Download it from the data providers. The MSVD-QA folder should contain train_qa.json, val_qa.json, test_qa.json and youtube_mapping.txt. The last file is used to filter out videos IDs in HowTo100M that are in the validation and test sets of MSVD-QA.

ActivityNet-QA: Download it from the data providers. The ActivityNet-QA folder should contain train_q.json, train_a.json, val_q.json, val_a.json, test_q.json and test_a.json.

How2QA: Download it from the data providers. The How2QA folder should contain how2QA_train_release.csv and how2QA_val_release.csv.

HowTo100M: Download it from the data providers. The HowTo100M folder should contain caption_howto100m_with_stopwords.pkl and s3d_features.csv. Note that for the VQA-T pretraining on HowTo100M baseline, we also do zero-shot validation on YouCook2 and MSR-VTT video retrieval. We followed MIL-NCE for the preprocessing of these datasets. You should have in the YouCook2 folder a pickle file with processed data and features youcook_unpooled_val.pkl, and in the MSR-VTT folder a file of processed data MSRVTT_JSFUSION_test.csv and a file of features msrvtt_test_unpooled_s3d_features.pth.

WebVid2M Download it from the data providers. The WebVid folder should contain train_2M_results.csv and val_2M_results.csv.

VideoQA: To process data for each VideoQA dataset, use:

python preproc/preproc_ivqa.py
python preproc/preproc_msrvttqa.py
python preproc/preproc_msvdqa.py
python preproc/preproc_activitynetqa.py
python preproc/preproc_how2qa.py

This will save train, validation and test dataframe files (train.csv, val.csv, test.csv), and the vocabulary map (vocab.json) in the open-ended setting, in each dataset folder. Note that the How2QA preprocessing script should be used after feature extraction (see below) and will also merge features into one file.

HowTo100M: To preprocess HowTo100M by removing potential intersection with the validation and test sets of VideoQA datasets, and removing repetition in the ASR data, use:

python preproc/howto100m_remove_intersec.py
python preproc/howto100m_remove_repet.py

This will save caption_howto100m_sw_nointersec.pickle, caption_howto100m_sw_nointersec_norepeat.pickle and s3d_features_nointersec.csv in HOWTO_PATH.

We provide in the extract folder the code to extract features with the S3D feature extractor. It requires downloading the S3D model weights available at this repository. The s3d_howto100m.pth checkpoint and s3d_dict.npy dictionary should be in DEFAULT_MODEL_DIR.

Extraction: You should prepare for each dataset a csv with columns video_path (typically in the form of <dataset_path>/video/<video_path>), and feature_path (typically in the form of <dataset_path>/features/<video_path>.npy). Then use (you may launch this script on multiple GPUs to fasten the extraction process):

python extract/extract.py --csv <csv_path>

Merging: To merge the extracted features into a single file for each VideoQA dataset, use (for ActivityNet-QA that contains long videos, add --pad 120):

python extract/merge_features.py --folder <features_path> \
--output_path <DEFAULT_DATASET_DIR>/s3d.pth --dataset <dataset>

For HowTo100M, the features should be stored in HOWTO_FEATURES_PATH, one file per video. Similarly, for WebVid2M, the features should be stored in WEBVID_FEATURES_PATH, one file per video. SSD_PATH should preferably on a SSD disk for optimized on-the-fly reading operation time during pretraining.

This requires downloading the pretrained BRNN model weights from Punctuator2. The INTERSPEECH-T-BRNN.pcl file should be in DEFAULT_MODEL_DIR.

Punctuating: First, we punctuate the speech data at the video level and split the video into clips temporally aligned with infered sentences (you may launch this script on multiple CPUs to fasten the process):

python videoqa_generation/punctuate.py

Merging infered speech sentences: Second, we merge the punctuated data into one file:

python videoqa_generation/merge_punctuations.py

Extracting answers: Third, we extract answers from speech transcripts. This requires having cloned this repository in QG_REPO_DIR. Then use (you may launch this script on multiple GPUs to fasten the process):

python videoqa_generation/extract_answers.py

Merging extracted answers: Fourth, we merge the extracted answers into one file:

python videoqa_generation/merge_answers.py

Generating questions: Fifth, we generate questions pairs from speech and extracted answers. Use (you may launch this script on multiple GPUs to fasten the process):

python videoqa_generation/generate_questions.py

Merging generated question-answer pairs: Finally, we merge the generated question-answer pairs into one file (this will save howtovqa.pkl, train_howtovqa.csv and val_howtovqa.csv):

python videoqa_generation/merge_qas.py

This requires downloading the pretrained BRNN model weights from Punctuator2. The INTERSPEECH-T-BRNN.pcl file should be in DEFAULT_MODEL_DIR.

Extracting answers: First, we extract answers from speech transcripts. This requires having cloned this repository in QG_REPO_DIR. Then use (you may launch this script on multiple GPUs to fasten the process):

python videoqa_generation/extract_answers_webvid.py

Merging extracted answers: Second, we merge the extracted answers into one file:

python videoqa_generation/merge_answers_webvid.py

Generating questions: Third, we generate questions pairs from speech and extracted answers. Use (you may launch this script on multiple GPUs to fasten the process):

python videoqa_generation/generate_questions_webvid.py

Merging generated question-answer pairs: Finally, we merge the generated question-answer pairs into one file (this will save webvidvqa.pkl, train_webvidvqa.csv and val_webvidvqa.csv):

python videoqa_generation/merge_qas_webvid.py

The pretraining of QA-T on HowToVQA69M is done with the previous command complemented with --baseline qa. To train VQA-T on HowTo100M with MLM and cross-modal matching objectives (it takes less than 2 days on 8 NVIDIA Tesla V100), run:

python main_htm.py --dataset="howto100m" --epochs=10 --checkpoint_dir="pthtm" \ 
--batch_size=128 --batch_size_val=3500 --n_pair=32 --freq_display=10

Note that the previous command runs a zero-shot video retrieval validation on YouCook2 and MSR-VTT once per epoch.

Finetuning: To finetune a pretrained model on a downstream VideoQA dataset (for MSRVTT-QA, which is the largest downstream dataset, it takes less than 4 hours on 4 NVIDIA Tesla V100), run:

python main_videoqa.py --checkpoint_dir=ft<dataset> --dataset=<dataset> --lr=0.00001 \ 
--pretrain_path=<CKPT_PATH>

Feature probe: VQA-T feature probing is simply obtained by running the previous script with --probe.

Training from scratch: VQA-T trained from scratch is simply obtained by running the previous script with no pretrain_path set.

You can also find there the html code used for iVQA data collection on Amazon Mechanical Turk.

Moreover, you can find the manually evaluated samples from generated data at this link.

Finally, you can find the html and python code for the online demo.