PSPE
Source code of EMNLP2020 paper "Pre-training Entity Relation Encoder with Intra-span and Inter-spanInformation".
It is based on our NERE toolkit.
Requirements
python: 3.7.6pytorch: 1.4.0transformers: 2.8.0configargparse: 1.1bidict: 0.18.0fire: 0.2.1
Pre-training
Before pre-training, please prepare a pre-training corpus (e.g. Wikipedia), the format of the pre-training corpus must be the same as the file data/wiki/wikipedia_sentences.txt.
Then preprocess the pre-training corpus for convenience:
$ python inputs/preprocess.py contrastive_loss_preprocess \
data/wiki/wikipedia_sentences.txt \
data/wiki/wikipedia_pretrain.json \
data/bert_base_cased_vocab.jsonPre-training:
$ PYTHONPATH=$(pwd) python examples/entity_relation_pretrain_nce/entity_relation_extractor_pretrain_nce.py \
--config_file examples/entity_relation_pretrain_nce/config.yml \
--device 0 \
--fine_tuneFine-tuning
$ mkdir pretrained_models
$ cd pretrained_modelsBefore fine-tuning, please download the pre-trained model SPE(password: dct8), and place the pre-trained model in the folder pretrained_models. And make sure that the format of the dataset must be the same as data/demo/train.json.
PYTHONPATH=$(pwd) python examples/attention_entity_relation/att_entity_relation_extractor.py \
--config_file examples/attention_entity_relation/config.yml \
--device 0 \
--fine_tuneCite
If you find our code is useful, please cite:
@inproceedings{wang-etal-2020-pre,
title = "Pre-training Entity Relation Encoder with Intra-span and Inter-span Information",
author = "Wang, Yijun and
Sun, Changzhi and
Wu, Yuanbin and
Yan, Junchi and
Gao, Peng and
Xie, Guotong",
booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
month = nov,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.emnlp-main.132",
doi = "10.18653/v1/2020.emnlp-main.132",
pages = "1692--1705",
abstract = "In this paper, we integrate span-related information into pre-trained encoder for entity relation extraction task. Instead of using general-purpose sentence encoder (e.g., existing universal pre-trained models), we introduce a span encoder and a span pair encoder to the pre-training network, which makes it easier to import intra-span and inter-span information into the pre-trained model. To learn the encoders, we devise three customized pre-training objectives from different perspectives, which target on tokens, spans, and span pairs. In particular, a span encoder is trained to recover a random shuffling of tokens in a span, and a span pair encoder is trained to predict positive pairs that are from the same sentences and negative pairs that are from different sentences using contrastive loss. Experimental results show that the proposed pre-training method outperforms distantly supervised pre-training, and achieves promising performance on two entity relation extraction benchmark datasets (ACE05, SciERC).",
}