pltrdy / Files2rouge
Programming Languages
Projects that are alternatives of or similar to Files2rouge
Files2ROUGE
Motivations
Given two files with the same number of lines, files2rouge
calculates the average ROUGE scores of each sequence (=line). Each sequence may contain multiple sentences. In this case, the end of sentence string must be passed using the --eos
flag (default: "."). Running files2rouge
with a wrong eos delimiter may lead to incorrect ROUGE-L score.
You may also be interested in a Python implementation (instead of a wrapper): https://github.com/pltrdy/rouge.
$ files2rouge --help
usage: files2rouge [-h] [-v] [-a ARGS] [-s SAVETO] [-e EOS] [-m] [-i]
reference summary
Calculating ROUGE score between two files (line-by-line)
positional arguments:
reference Path of references file
summary Path of summary file
optional arguments:
-h, --help show this help message and exit
-v, --verbose Prints ROUGE logs
-a ARGS, --args ARGS ROUGE Arguments
-s SAVETO, --saveto SAVETO
File to save scores
-e EOS, --eos EOS End of sentence separator (for multisentence).
Default: "."
-m, --stemming DEPRECATED: stemming is now default behavior
-nm, --no_stemming Switch off stemming
-i, --ignore_empty
Getting Started
0) Install prerequisites
pip install -U git+https://github.com/pltrdy/pyrouge
(NOTE: running pip install pyrouge
would not work as the package is out of date on PyPI)
1) Clone the repo, setup the module and ROUGE
git clone https://github.com/pltrdy/files2rouge.git
cd files2rouge
python setup_rouge.py
python setup.py install
Do not forget to run setup_rouge
2) Run files2rouge.py
files2rouge references.txt summaries.txt
Outputs:
When --verbose
is set, the script prints progress and remaining time on stderr
. This can be changed using --verbose
in order to outputs ROUGE execution logs.
Default output example:
Preparing documents...
Running ROUGE...
---------------------------------------------
1 ROUGE-1 Average_R: 0.28242 (95%-conf.int. 0.25721 - 0.30877)
1 ROUGE-1 Average_P: 0.30157 (95%-conf.int. 0.27114 - 0.33506)
1 ROUGE-1 Average_F: 0.28196 (95%-conf.int. 0.25704 - 0.30722)
---------------------------------------------
1 ROUGE-2 Average_R: 0.10395 (95%-conf.int. 0.08298 - 0.12600)
1 ROUGE-2 Average_P: 0.11458 (95%-conf.int. 0.08873 - 0.14023)
1 ROUGE-2 Average_F: 0.10489 (95%-conf.int. 0.08303 - 0.12741)
---------------------------------------------
1 ROUGE-L Average_R: 0.25231 (95%-conf.int. 0.22709 - 0.27771)
1 ROUGE-L Average_P: 0.26830 (95%-conf.int. 0.23834 - 0.29818)
1 ROUGE-L Average_F: 0.25142 (95%-conf.int. 0.22741 - 0.27533)
Elapsed time: 0.458 secondes
files2rouge
from Python
Call import files2rouge
files2rouge.run(hyp_path, ref_path)
ROUGE Args
One can specify which ROUGE args to use using the flag --args
(or -a
).
The default behavior is equivalent to:
files2rouge reference.txt summary.txt -a "-c 95 -r 1000 -n 2 -a" # be sure to write args betwen double-quotes
You can find more informations about these arguments here
Known issues
-
ROUGE-1.5.5.pl - XML::Parser dependency error
: see issue #9.
More informations
- ROUGE Original Paper (Lin 2004)
- ROUGE-1.5.5/README.txt
- Use cases:
- About
files2rouge.py
: runfiles2rouge.py --help