Heterogeneous Partial-Label Embedding
Source code and data for SIGKDD'16 paper Label Noise Reduction in Entity Typing by Heterogeneous Partial-Label Embedding.
Given a text corpus with entity mentions detected and heuristically labeled by distant supervision, this code performs (1) label noise reduction over distant supervision, and (2) learning type classifiers over de-noised training data. For example, check out PLE's output on Tech news.
An end-to-end tool (corpus to typed entities) is under development. Please keep track of our updates.
Performance
Performance of fine-grained entity type classification over Wiki (Ling & Weld, 2012) dataset. We applied PLE to clean training data and ran FIGER (Ling & Weld, 2012) and over the de-noised labeled data to train type classifiers (thus the FIGER + PLE is the name of our final system).
| Method | Accuray | Macro-F1 | Micro-F1 |
|---|---|---|---|
| HYENA (Yosef et al., 2012) | 0.288 | 0.528 | 0.506 |
| WSABIE (Yogatama et al,., 2015) | 0.480 | 0.679 | 0.657 |
| FIGER (Ling & Weld, 2012) | 0.474 | 0.692 | 0.655 |
| FIGER + All Filter (Gillick et al., 2014) | 0.453 | 0.648 | 0.582 |
| FIGER + PLE (Ren et al., 2016) | 0.599 | 0.763 | 0.749 |
System Output
The output on BBN dataset can be found here. Each line is a sentence in the test data of BBN, with entity mentions and their fine-grained entity typed identified.
Dependencies
- python 2.7, g++
- Python library dependencies
$ pip install pexpect unidecode six requests protobuf
- Setup stanford coreNLP and its python wrapper.
$ cd DataProcessor/
$ git clone [email protected]:stanfordnlp/stanza.git
$ cd stanza
$ pip install -e .
$ wget http://nlp.stanford.edu/software/stanford-corenlp-full-2016-10-31.zip
$ unzip stanford-corenlp-full-2016-10-31.zip
$ rm stanford-corenlp-full-2016-10-31.zip
- eigen 3.2.5 (already included).
Data
We processed (using our data pipeline) three public datasets to our JSON format. We ran Stanford NER on training set to detect entity mentions, and performed distant supervision using DBpediaSpotlight to assign type labels:
- Wiki (Ling & Weld, 2012): 1.5M sentences sampled from 780k Wikipedia articles. 434 news sentences are manually annotated for evaluation. 113 entity types are organized into a 2-level hierarchy (download JSON)
- OntoNotes (Weischedel et al., 2011): 13k news articles with 77 of them are manually labeled for evaluation. 89 entity types are organized into a 3-level hierarchy. (download JSON)
- BBN (Weischedel et al., 2005): 2,311 WSJ articles that are manually annotated using 93 types in a 2-level hierarchy. (download JSON)
Type hierarchesfor each dataset are included.- Please put the data files in the corresponding subdirectories under
PLE/Data/.
Makefile
We have included compilied binaries. If you need to re-compile hple.cpp under your own g++ environment
$ cd PLE/Model/ple/; make
Default Run
Run PLE for the task of Reduce Label Noise on the BBN dataset
$ java -mx4g -cp "DataProcessor/stanford-corenlp-full-2016-10-31/*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer
$ ./run.sh
- The run.sh contains parameters for running on three datasets.
Parameters - run.sh
Dataset to run on.
Data="BBN"
Evaluation
Evaluate prediction results (by classifier trained on de-noised data) over test data
python Evaluation/evaluation.py BBN hple hete_feature
- python Evaluation/evaluation.py -DATA(BBN/ontonotes/FIGER) -METHOD(hple/...) -EMB_MODE(hete_feature)
Reference
Please cite the following paper if you found the codes/datasets useful:
@inproceedings{ren2016label,
title={Label Noise Reduction in Entity Typing by Heterogeneous Partial-Label Embedding},
author={Ren, Xiang and He, Wenqi and Qu, Meng and Voss, Clare R and Ji, Heng and Han, Jiawei},
booktitle={Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining},
pages={1825--1834},
year={2016},
organization={ACM}
}