rougescore

This is a Python re-implementation of the ROUGE evaluation package.

ROUGE is a suite of evaluation metrics for automatic text summarization. In ROUGE, a "peer" summary produced by a machine summarization system is compared against one or more hand-written "model" summaries and then assigned a score from 0 to 1. This score is the F-measure of recall vs. precision, and the evaluator can adjust a parameter α to control whether this score favors recall (does the peer summary contain all of the information in the model summaries?) or precision (does the peer summary contain only information in the model summaries?). When α ≈ 0, this score favors recall; when α ≈ 1, it favors precision. In the DUC conferences, α was set to 0, and a hard length limit was imposed on generated summaries. The original ROUGE implementation uses α = 0.5 by default.

This package is meant to serve as a faithful replacement for parts of the original Perl implementation of ROUGE, providing pure Python implementations of ROUGE-N and ROUGE-L. The standard Perl script requires that all inputs be passed through files on disk, which makes it cumbersome to fit into a Python workflow and limits performance. In many cases it is more convenient to be able to call a Python function which operates on native Python data types. This version also does not perform any pre-processing of words or characters, admitting lists of arbitrary objects that support equality and hashing.

This package deliberately does not implement stemming or stopword removal. It is up to you to pre-process the peer and model summaries as you see fit before comparing them.

One difference in the implementation of ROUGE-L from the original is that separation of the peer and model summaries into "units" (i.e. sentences) is not supported. That is, peer and model summaries are always treated as single, continuous strings of tokens, whereas the original ROUGE script allows the user to split them into sub-units, which affects the matching of common sub-sequences.

Installation

To install this package (possibly in a virtualenv), run:

git clone https://github.com/bdusell/rougescore.git
cd rougescore
python setup.py install

or simply

pip install git+git://github.com/bdusell/rougescore.git