DFKI-NLP / Distre
Programming Languages
Projects that are alternatives of or similar to Distre
Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction
This repository contains the code of our paper:
Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction
Christoph Alt, Marc Hübner, Leonhard Hennig
Our code depends on huggingface's PyTorch reimplementation of the OpenAI GPT, and AllenNLP - so thanks to them.
The code is tested with:
- Python 3.6.6
- PyTorch 1.0.1
- AllenNLP 0.7.1
Installation
First, clone the repository to your machine and install the requirements with the following command:
pip install -r requirements.txt
Second, download the OpenAI GPT archive (containing all model related files):
wget --content-disposition https://cloud.dfki.de/owncloud/index.php/s/kKdpoaGikWnL4tn/download
Prepare the data
We evaluate our model on the NYT dataset and use the version provided by OpenNRE.
Follow the OpenNRE instructions for creating the NYT dataset in JSON format:
- download the nyt.tar file.
- extract the archive with:
tar -xvf nyt.tar
- create the protobuf files:
protoc --proto_path=. --python_out=. Document.proto
- convert the protobuf files to json:
python protobuf2json.py .
- move
train.json
andtest.json
todata/open_nre_nyt/
Training
E.g. for training on the NYT dataset, run the following command:
CUDA_VISIBLE_DEVICES=0 allennlp train \
experiments/configs/model_paper.json \
-s <MODEL AND METRICS DIR> \
--include-package tre
Evaluation
CUDA_VISIBLE_DEVICES=0 python ./experiments/utils/pr_curve_and_predictions.py \
<MODEL AND METRICS DIR> \
./data/open_nre_nyt/test.json \
--output-dir <RESULTS DIR> \
--archive-filename <MODEL ARCHIVE FILENAME>
Trained Models
The model(s) we trained on NYT to produce our paper results can be found here:
Dataset | Masking Mode | AUC | Download |
---|---|---|---|
NYT | None | 0.422 | Link |
Download and extract model files
Download the archive corresponding to the model you want to evaluate (links in the table above).
wget --content-disposition <DOWNLOAD URL>
Run evaluation
For example, to evaluate the NYT model used in the paper, run the following command:
CUDA_VISIBLE_DEVICES=0 python ./experiments/utils/pr_curve_and_predictions.py \
<DIR CONTAINING THE MODEL ARCHIVE> \
./data/open_nre_nyt/test.json \
--output-dir ./results/ \
--archive-filename model_lm05_wu2_do2_bs16_att.tar.gz
Citations
If you use our code in your research or find our repository useful, please consider citing our work.
@inproceedings{alt-etal-2019-fine,
title = "Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction",
author = {Alt, Christoph and
H{\"u}bner, Marc and
Hennig, Leonhard},
booktitle = "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics",
month = jul,
year = "2019",
address = "Florence, Italy",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/P19-1134",
pages = "1388--1398",
}
License
DISTRE is released under the Apache 2.0 license. See LICENSE for additional details.