crf4j: CRF model training and testing for Java
This is a pure Java port of taku's crfpp(also known as crf++), which is based on codes of crfpp-0.58.
Credits to komiya's for his Java double array trie implementation.
Features
- pure Java, with least dependencies(only commons-cli as runtime deps)
- compatible commandline options and template/input format with crfpp
- load model from classpath
- compatible text model format with crfpp
- convert text model to (our)binary model and (our)binary model to text model
- multi-threading support
- CRF-L1/CRF-L2/MIRA algorithms supports
- n-best outputs
- CRF Model wrapper for API call
- Tests and demo for usage demonstration
Usage
Building
mvn clean package
Run tests:
mvn test
Training
java -cp crf4j-<version>-jar-with-dependencies.jar com.github.zhifac.crf4j.CrfLearn <template file> <train datafile> <model path>
For more options, please run
java -cp crf4j-<version>-jar-with-dependencies.jar com.github.zhifac.crf4j.CrfLearn -h
For details on format of template file and train file, please refer to original page of crfpp.
Testing
to print output to console:
java -cp crf4j-<version>-jar-with-dependencies.jar com.github.zhifac.crf4j.CrfTest -m <model path> <test datafile>
to print output to file:
java -cp crf4j-<version>-jar-with-dependencies.jar com.github.zhifac.crf4j.CrfTest -m <model path> <test datafile> -o <outputfile>
API call
please refer to CrfDemo.java.
Performance
Concurrent Access
In an example of using crf4j model to recognize name entity, we used jmeter to test 400 concurrent access to the same Http interface, and here is the result.
| #Samples | Average | Median | 90% Line | Min | Max | Throughput |
|---|---|---|---|---|---|---|
| 4000 | 41 | 4 | 60 | 0 | 746 | 1250/sec |
The test environment is:
| OS | CPU | MEM |
|---|---|---|
| Windows 7x64 | Intel Core [email protected] | 8GB |
Notes
The binary model generated by CrfLearn is incompatible with crfpp, but the text model is. If you somehow want to reuse a crfpp model with crf4j, please generate a text model when you train with crfpp(add -t option), and then run java -cp crf4j.jar com.github.zhifac.crf4j.EncoderFeatureIndex <crfpp_text_model> <output_crf4j_binarymodel> to convert the crfpp text model to crf4j binary model. Or if you somehow can not retrain the same text model(e.g. missing train data), you can still convert an existing crfpp binary model to text model with modified version of crfpp from here.
TODO
- Optimize memory usage when training(it currently consumes about 8GB heap memory for 24224128 features, whereas crfpp uses 2GB)
License
LGPL & Modified BSD
Chinese version:
crf4j: crfpp(crf++)的Java实现
（基于crfpp 0.58）