DAC Entity Linker
Entity linker for the Dutch historical newspaper collection of the Koninklijke Bibliotheek, National Library of the Netherlands. The linker links named entity mentions in newspaper articles to relevant DBpedia descriptions using either a binary SVM classifier or a neural net. For background information, please see the project description on the Koninklijke Bibliotheek website.
Usage
Basic command line execution with the default values for all options:
$ cd dac
$ ./dac.py
This will link all recognized entities in a sample article using a neural network:
{'linkedNEs': [{'label': u'Winston Churchill',
'link': u'http://nl.dbpedia.org/resource/Winston_Churchill',
'prob': '0.9997673631',
'reason': 'Predicted link',
'text': u'Churchill'},
{'label': u'Willem Drees',
'link': u'http://nl.dbpedia.org/resource/Willem_Drees',
'prob': '0.9968996048',
'reason': 'Predicted link',
'text': u'Drees'},
...
Command line interface
Additional options when using the command line interface:
usage: dac.py [-h] [--url URL] [--ne NE] [-m MODEL] [-d] [-f] [-c] [-e]
optional arguments:
-h, --help show this help message and exit
--url URL resolver link of the article to be processed
--ne NE specific named entity to be linked
-m MODEL, --model MODEL model used for link prediction (svm, nn or bnn)
-d, --debug include unlinked entities in response
-f, --features return feature values
-c, --candidates return candidate list
-e, --errh turn on error handling
Web interface
The DAC Entity Linker can be started as a web application by running:
$ ./web.py
This starts a Bottle web server listening on http://localhost:5002. The URL parameters are similar to the command line options:
required arguments:
- url resolver link of the article to be processed
optional arguments:
- ne specific named entity to be linked
- model model used for link prediction (svm, nn or bnn)
- debug include unlinked entities in response
- features include feature values for predicted links
- candidates include the list of candidates for each entity
- callback name of a JavaScript callback function
Training new models
Given the availability of training set in the format created by the DAC Web Interface, new models can be trained in two simple steps. First, the web interface training set is extended with the features values for each training example:
$ cd training
$ ./generate.py
The default input file used here is ../../../dac-web/users/tve/art.json and the output is written to a training.csv file. These locations can be adjusted, however, using the --input and --output options of the generate.py script. The features calculated are listed in features/features.json.
The resulting training.csv file can now be used to train new models. Note that existing models in the models directory will be replaced, so these need to be backed up manually if they are to be preserved. To train, for example, a new Support Vector Machine, run:
$ ./models.py -t -m svm
This will create a models/svm.pkl file, using the feature set of features/svm.json, that can now be applied to new named entity examples.
Full command line options for training and cross-validation:
usage: models.py [-h] [-w] [-t] [-v] [-m MODEL]
optional arguments:
-h, --help show this help message and exit
-w, --weights show the feature weights of the current model
-t, --train train and save new model
-v, --validate cross-validate new model
-m MODEL, --model MODEL model type (svm, nn or bnn)
Evaluation
Once one or more models have been trained, the linker performance can be evaluated on a separate training set in the format created by the DAC Web Interface. To test the performance of, e.g., a first version of a neural net, run:
$ cd training
$ ./test.py -m nn -v 1
This will evaluate the current neural network model on the ../../../dac-web/users/test-clean/art.json file, but a different test set can be specified with the --input option.
A summary of the results will be printed out:
Number of instances: 500
Number of correct predictions: 467
Prediction accuracy: 0.934
---
Number of correct link predictions: 347
(Min) number of link instances: 362
(Max) number of link instances: 382
(Min) link recall: 0.908376963351
(Mean) link recall: 0.933470249631
(Max) link recall: 0.958563535912
---
Number of correct link predictions: 347
Number of link predictions: 358
Link precision: 0.969273743017
---
(Mean) link F1-measure: 0.951035143299
(Max) link F1-measure: 0.963888888889
The version number specified will be used to name a file containing the full results of the test run, e.g. training/results-nn-1.csv.
Further command line options for the test script:
usage: test.py [-h] -m MODEL -v VERSION [-i INPUT]
required arguments:
-m MODEL, --model MODEL model name (svm, nn or bnn)
-v VERSION version number
optional arguments:
-h, --help show this help message and exit
-i INPUT path to test set