Named Entity Recognition Tool for
Europeana Newspapers 
This tool takes container documents (MPEG21-DIDL, METS), parses all references to ALTO files and tries to find named entities in the pages (with most models: Location, Person, Organisation, Misc). The aim is to keep the physical location on the page available through the whole process to be able to highlight the results in a viewer.
Read more about it on the KBNLresearch blog.
Stanford NER is used for tagging. The goal during development was to use 'loose coupling', this enables us to quickly inherit/benefit from upstream development. Most of the development is done at the research department of the KB, national library of the Netherlands. If you are looking for a project which does more interaction with the core of Stanford-NER, take a peek at the project from our colleagues INL, Institute for Dutch Lexicology INL-NERT, although they are separate branches now, there is a desire to integrate both in the future.
This version is not longer maintained, for a maintained version go here: https://github.com/EuropeanaNewspapers/ner-app
Input formats
The following input formats are implemented:
- ALTO 1.0
- HTML
- Mets
- MPEG21 DIDL
- Text
Output formats
The following output formats are implemented:
- ALTO 2.1 (soon to be replaced with 3.0)
- ALTO-with-Alternatives (aka. inline ALTO)
- BIO
- CSV
- HTML
- SQL(db)
Building
Building from source:
Install Maven, Java (v1.7 and up). Clone the source from github, and in the toplevel directory run:
mvn package
This command will generate a JAR and a WAR of the NER located in the target/ directory.
To deploy the WAR, just copy it into the Tomcat webapp directory, or use Tomcat
manager to do it for you.
Or move quickly and run (on *nix systems):
git clone https://github.com/KBNLresearch/europeananp-ner.git
cd europeananp-ner/
./go.sh
Usage command-line-interface
Invoking help:
java -jar NerAnnotator.jar --help
usage: java -jar NerAnnotator.jar [OPTIONS] [INPUTFILES..]
-c,--container <FORMAT> Input type: mets (Default), didl,
alto, text, html
-d,--output-directory <DIRECTORY> output DIRECTORY for result files.
Default ./output
-f,--export <FORMAT> Output type: log (Default), csv,
html, db, alto, alto2_1, alto3, bio.
Multiple formats:" -f html -f csv"
-l,--language <ISO-CODE> use two-letter ISO-CODE for language
selection: en, de, nl ....
-m,--models <language=filename> models for languages. Ex. -m
de=/path/to/file/model_de.gz -m
nl=/path/to/file/model_nl.gz
-n,--nthreads <THREADS> maximum number of threads to be used
for processing. Default 8
If there are no input files specified, a list of file names is read from stdin.
Example invocation for classification of german_alto.xml:
java -Xmx800m -jar NerAnnotator.jar -c mets -f alto -l de -m de=./test-files/german.ser.gz -n 2 ./test-files/german_alto.xml
The given example takes the language model called 'german.ser.gz' and applies it to 'german_alo.xml' using 2 threads, and container type METS.
Usage web-interface
Webinterface standalone:
mvn jetty:run
This will try to bind to port 8080, using Jetty.
Once deployed to Tomcat the following applies. The default configuration (as well as test-classifiers)
reside in src/main/resources/config.ini, this file references the available classifiers.
See the provided sample for some default settings. The landing page of the application
will show the available options once invoked with the browser. The config.ini and the
classifiers will end up in WEB-INF/classes/, after deployment.
Working with classifiers and binary model generation
To be able to compare your results with a baseline we provide
some test files located in the test-files directory.
To run a back-to-front test try:
cd test-files;./test_europeana_ner.sh
The output should look something like:
Generating new classification model. (de)
-rw-rw-r-- 1 aloha aloha 1.4M Sep 11 15:55 ./eunews_german.crf.gz
real 0m3.984s
user 0m5.452s
sys 0m0.235s
Applying generated model (de).
Results:
Locations: 4
Organizations: 0
Persons: 1071
real 0m13.512s
user 0m17.771s
sys 0m0.336s
Generating new classification model. (nl)
-rw-rw-r-- 1 aloha aloha 1.7M Sep 11 15:56 ./eunews_dutch.crf.gz
real 0m8.816s
user 0m10.437s
sys 0m0.371s
Applying generated model (nl).
Results:
Locations: 1
Organizations: 8
Persons: 0
real 0m5.048s
user 0m9.278s
sys 0m0.233s
To generate a binary classification model, use the following command:
cd test-files; java -Xmx5G -cp ../target/NerAnnotator-0.0.2-SNAPSHOT-jar-with-dependencies.jar edu.stanford.nlp.ie.crf.CRFClassifier -prop austen_dutch.prop
This should result in a file called eunews_dutch.crf.gz, with a file-size of +/- 1MB.
To verify the NER software, use the created classifier to process the provided example file.
cd test-files; java -jar ../target/NerAnnotator-0.0.2-SNAPSHOT-jar-with-dependencies.jar -c alto -d out -f alto -l nl -m nl=./eunews_dutch.crf.gz -n 8 ./dutch_alto.xml
Resulting in a directory called out containing ALTO files with inline annotation.
General remarks on binary classification model generation
The process of generating a binary classification model is a delicate one. The input
.bio file needs be as clean as possible to prevent the garbage in-out rule from
happening. Thus, use noise filters while creating .bio files.
Gazette's greatly improve the quality of your classification process, but a big model in memory may slow down processing speed. Overall there is a strong correlation in model size and performance.
The Stanford NER package offers a lot of settings that can influence the
binary model generation process. These settings can be configured using
austen.prop, For more information on the Stanford settings see
Stanford NER FAQ.
Binary classification models generated with this tool are fully compatible with the upstream version of the Stanford NER.