Integrating Transformer and Paraphrase Rules for Sentence Simplification
Paper Link: http://www.aclweb.org/anthology/D18-1355
Note that some improvement from original EMNLP paper:
- we modified the code to allow supporting subword and the model performs well.
- we found replacing name entities might not be a good idea (i.e. replace John to person0) since it lose some information. Instead, subword is helpful for reducing the huge vocabulary coming from name entities.
- we found the context(memory) addressing is probably redundant. Without it, the model can achieve same(even better) performance.
Data Download:
https://drive.google.com/open?id=132Jlza-16Ws1DJ7h4O89TyxJiFSFAPw7
Pretrained Model Download:
https://drive.google.com/open?id=16gO8cLXttGR64_xvLHgMwgJeB1DzT93N
Command to run the model:
python model/train.py -ngpus 1 -bsize 64 -fw transformer -out bertal_wkori_direct -op adagrad -lr 0.01 --mode transbert_ori -nh 8 -nhl 6 -nel 6 -ndl 6 -lc True -eval_freq 0 --fetch_mode tf_example_dataset --subword_vocab_size 0 --dmode wk --tie_embedding all --bert_mode bert_token:bertbase:init --environment aws --memory direct python model/eval.py -ngpus 1 -bsize 256 -fw transformer -out bertal_wkori_direct -op adagrad -lr 0.01 --mode transbert_ori -nh 8 -nhl 6 -nel 6 -ndl 6 -lc True -eval_freq 0 --subword_vocab_size 0 --dmode wk --tie_embedding all --bert_mode bert_token:bertbase:init --environment aws
Arugument instruction
- bsize: batch size
- out: the output folder will contains log, best model and result report
- tie_embedding: all means tie the encoder/decoder/projection w embedding, we found it can speed up the training
- bert_mode: the mode of using BERT bert_token indicates we use the subtoken vocabulary from BERT; bertbase indicates we use BERT base version (due to the memory issue, we did not try BERT large version yet)
- environment: the path config of the experiment. Please change it in model/model_config.py to fit to your system
More config you can check them in util/arguments.py
Citation
Zhao, Sanqiang, et al. "Integrating Transformer and Paraphrase Rules for Sentence Simplification." Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.
@article{zhao2018integrating,
title={Integrating Transformer and Paraphrase Rules for Sentence Simplification},
author={Zhao, Sanqiang and Meng, Rui and He, Daqing and Andi, Saptono and Bambang, Parmanto},
journal={arXiv preprint arXiv:1810.11193},
year={2018}
}