Sentence-Permuted Paragraph Generation
This repository contains the code package for the EMNLP'2021 paper:
Sentence-Permuted Paragraph Generation [arXiv] [slides] [video]
Wenhao Yu (ND), Chenguang Zhu (MSR), Tong Zhao (ND), Zhichun Guo (ND), Meng Jiang (ND).
In this paper, we propose a novel framework PermGen whose objective is to maximize the expected log-likelihood of output paragraph distributions with respect to all possible sentence orders. PermGen uses hierarchical positional embedding and designs new procedures for training, and decoding. Experiments on three generation benchmarks showed PermGen generates more diverse outputs with a higher quality than existing models.
Model Usage
Step 1: Download datasets
We conducted experiments on three paragraph generation tasks: story generation (ROCStory), news generation (DailyMail), paper abstract generation (AGENDA). For the ROCStory and AGENDA datasets, we directly download them from their official repos. For the DailyMail dataset, We use randomly sampled 53,102 news articles from the original corpus and extract keyphrases from each sentence using RAKE.
| Dataset Name | Original Link | Paper Link | Our Pre-processed |
|---|---|---|---|
| ROCStory | OL-ROC | PL-ROC | OP-ROC |
| AGENDA | OL-AG | PL-AG | OP-AG |
| DailyMail | OL-DM | PL-DM | OP-DM |
After downloading the pre-processed datasets, please put them in the dataset folder.
Step 2: Install packages
The python version should be at least 3.6.0.
conda create -n permgen python=3.6
conda activate permgen
pip install transformers==3.3.1
pip install torch==1.7.0
Step 3: Randomly permute sentences
Add/delete --dataset to choose the dataset.
python dataset/preprocessing.py --agenda True --dailymail True --rocstory True
Step 4: Train the model
bash scripts/train_agenda.sh
bash scripts/train_dailymail.sh
bash scripts/train_rocstory.sh
Step 5: Test with saved checkpoints
Do not forget to specify the path for saved checkpoints!
bash scripts/test_agenda.sh
bash scripts/test_dailymail.sh
bash scripts/test_rocstory.sh
Easy-to-use baseline implementation
The baseline BART implementation can be found at here. The repository contains the code to reproduce the baseline performance reported in our paper. All hyperparameters and evaluations are the same as in this repository.
Output examples
Please find our output examples in the examples folder.
Reference
If you find this repository useful in your research, please consider to cite our paper:
@inproceedings{yu2021sentence,
title={Sentence-Permuted Paragraph Generation},
author={Yu, Wenhao and Zhu, Chenguang and Zhao, Tong and Guo, Zhichun and Jiang, Meng},
booktitle={Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
year={2021}
}
Contact
If you have any questions, please contact Wenhao Yu ([email protected])