Relation Extraction with Question-Answer Pairs (ReQuest)
Source code and data for WSDM'18 paper Indirect Supervision for Relation Extraction Using Question-Answer Pairs.
Performance
Performance comparison with several relation extraction systems over KBP 2013 dataset (sentence-level extraction).
| Method | Precision | Recall | F1 |
|---|---|---|---|
| Mintz (our implementation, Mintz et al., 2009) | 0.296 | 0.387 | 0.335 |
| LINE + Dist Sup (Tang et al., 2015) | 0.360 | 0.257 | 0.299 |
| MultiR (Hoffmann et al., 2011) | 0.325 | 0.278 | 0.301 |
| FCM + Dist Sup (Gormley et al., 2015) | 0.151 | 0.498 | 0.300 |
| CoType-RM (Ren et al., 2017) | 0.342 | 0.339 | 0.340 |
| ReQuest (our model, [Wu et al., 2018]) | 0.386 | 0.410 | 0.397 |
Dependencies
We will take Ubuntu for example.
- python 2.7
- Python library dependencies
$ pip install pexpect ujson tqdm
- stanford coreNLP 3.7.0 and its python wrapper. Please put the library under `ReQuest/code/DataProcessor/'.
$ cd code/DataProcessor/
$ git clone [email protected]:stanfordnlp/stanza.git
$ cd stanza
$ pip install -e .
$ wget http://nlp.stanford.edu/software/stanford-corenlp-full-2016-10-31.zip
$ unzip stanford-corenlp-full-2016-10-31.zip
- eigen 3.2.5 (already included).
Data
We process (using our data pipeline) two public RE datasets to our JSON format. We ran Stanford NER on training set to detect entity mentions, and performed distant supervision using DBpediaSpotlight to assign type labels:
- NYT (Riedel et al., 2011): 1.18M sentences sampled from 294K New York Times news articles. 395 sentences are manually annotated with 24 relation types and 47 entity types. (Download JSON)
- Wiki-KBP: the training corpus contains 1.5M sentences sampled from 780k Wikipedia articles (Ling & Weld, 2012) plus ~7,000 sentences from 2013 KBP corpus. Test data consists of 14k mannually labeled sentences from 2013 KBP slot filling assessment results. It has 13 relation types and 126 entity types after filtering of numeric value-related relations. (Download JSON)
Please put the data files in corresponding subdirectories under ReQuest/data/source
We use the answer sentence selection dataset from TREC QA as our source of indirect supervision. We ran Stanford NER to extract entity mentions on both question and answer sentences and process the dataset into JSON format containing QA-pairs. Details of how we construct QA-pairs can be found in our paper.
We provide the processed qa.json file and it should be put into each data folder under ReQuest/data/source.
Makefile
To compile request.cpp under your own g++ environment
$ cd ReQuest/code/Model/request; make
Default Run & Parameters
Run ReQuest for the task of Relation Extraction on the Wiki-KBP dataset
Start the Stanford corenlp server for the python wrapper.
$ java -mx4g -cp "code/DataProcessor/stanford-corenlp-full-2016-10-31/*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer
Feature extraction, embedding learning on training data, and evaluation on test data.
$ ./run_kbp.sh
The hyperparamters for embedding learning are included in the run_{dataname}.sh script.
Evaluation
Evaluates relation extraction performance (precision, recall, F1): produce predictions along with their confidence score; filter the predicted instances by tuning the thresholds.
$ python code/Evaluation/emb_test.py extract KBP request cosine 0.0
$ python code/Evaluation/tune_threshold.py extract KBP emb request cosine