TransTQA data and code
This repository contains the code package for the EMNLP'20 paper:
A Technical Question Answering System with Transfer Learning. Wenhao Yu (ND), Lingfei Wu (IBM), Yu Deng (IBM), Ruchi Mahindru (IBM), Qingkai Zeng (ND), Sinem Guven (IBM), Meng Jiang (ND).
-
Our demonstration website is avaiable at http://159.89.9.22:5800/
-
We also provide a video for the paper at https://vimeo.com/431118548
System overall and Framework
Figure: High level architecture of our proposed TransTQA system. First, the pre-trained ALBERT model is fine tuned with unstructured source technical corpus with masked language model (MLM) task, i.e., θ → θ+. Second, a siamese ALBERT takes fine tuned ALBERT and fine tunes with source technical QA, i.e., θ+ → θ++. Third, the siamese ALBERT further fine tunes with target QA, i.e., θ++ → θ+++. Our deployed system takes θ+++. Given a query, our system first calculates similarity scores between the query and each candidate answer, then ranks all scores from highest to lowest. Finally, the system returns top-3 ranked answers.
Environment settings
A detailed dependencies list can be found in requirements.txt and can be installed by:
pip install -r requirements.txt
If you want to run with fp16, you need to install Apex
Run the code
For fine-tuning BERT masked language model on tech corpus (before you run the model, you have to first download the technical corpus from here and put it into mlm/corpus folder ):
./script/run_mlm.sh
For pre-training the model (we use askubuntu for technical domain QA pre-training as default):
./script/run_pretrain.sh
For model transfer learning (we have two target datasets: stackunix and techqa (ACL 2020)):
./script/run_transfer.sh
Note that you should specify the path of pre-trained model and dataset.
Citation
If you find this repository useful in your research, please consider to cite our paper:
@inproceedings{yu2020technical,
title={A Technical Question Answering System with Transfer Learning},
author={Yu, Wenhao and Wu, Lingfei and Deng, Yu and Mahindru, Ruchi and Zeng, Qingkai and Guven, Sinem and Jiang, Meng},
booktitle={Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
year={2020}
}