FastFusionNet
Overview
This repo contains the code of FastFusionNet: New State-of-the-Art for DAWNBench SQuAD.
News
We now support PyTorch version >=0.4.1 in a new branch. However, it is slightly slower.
Requirements
torch==0.3.1
spacy==1.9.0
numpy
pandas
tqdm
tesnorboardX
oldsru
Please also install the SRU version 1 (oldsru) from here.
Please download GloVe (Pennington et al., EMNLP 2014) and CoVe (McCann et al., NIPS 2017) by
bash download.shPreprocessing
Preprocessing the data set. This takes about 10 minutes.
PATH_TO_SQAUD_TRAIN should be the path to train-v1.1.json and PATH_TO_SQAUD_DEV should be the path to dev-v1.1.json. This will generate the preprocessed data file at data/squad/data-fusion.pth.
mkdir -p data/squad
python prepro.py --train PATH_TO_SQAUD_TRAIN --dev PATH_TO_SQUAD_DEVTraining
To train FastFusionNet (Wu et al., arXiv 2019):
SAVE='save/fastfusionnet'
mkdir -p $SAVE
python train.py --model_type fusionnet --hidden_size 125 --end_gru \
--dropout_rnn 0.2 --data_suffix fusion --save_dir $SAVE \
-lr 0.001 -gc 20 -e 100 --batch_size 32 \
--rnn_type sru --fusion_reading_layers 2 --fusion_understanding_layers 2 --fusion_final_layers 2To train FusionNet (Huang et al., ICLR 2018):
SAVE='save/fusionnet'
mkdir -p $SAVE
python train.py --model_type fusionnet --hidden_size 125 --end_gru \
--dropout_rnn 0.4 --data_suffix fusion --save_dir $SAVE \
-lr 0.001 -gc 20 -e 100 --batch_size 32 \
--rnn_type lstm --fusion_reading_layers 1 --fusion_understanding_layers 1 --fusion_final_layers 1 --use_coveTo train GLDR-DrQA (Wu et al., arXiv 2017):
python train.py --model_type gldr-drqa --hidden_size 128 \
--dropout_rnn 0.2 --data_suffix fusion --save_dir $SAVE \
-lr 0.001 -gc 20 -e 100 --batch_size 32 \
-doc_layers 17 --question_layers 9Evalutation
To evaluate the best trained model in 'save/fastfusionnet' and get the latency (batch size=1):
python eval.py --save_dir save/fastfusionnet --resume best_model.pt --eval_batch_size 1Pre-trained model
FastFusionNet model link dev EM: 73.58 F1: 82.42
Reference
@article{wu2019fastfusionnet,
title={FastFusionNet: New State-of-the-Art for DAWNBench SQuAD},
author={Wu, Felix and Li, Boyi and Wang, Lequn and Lao, Ni and Blitzer, John and and Weinberger, Kilian Q.},
journal={arXiv preprint arXiv:1902.11291},
url={https://arxiv.org/abs/1902.11291},
year={2019}
}
Acknowledgement
This is based on the v0.3.1 version of Runqi Yang's excellent DrQA code base as well as the official FusionNet on NLI implementation. Lots of Runqi's code is borrowed from Facebook/ParlAI under an MIT license.