Title: RankDCG: Rank-Ordering Evaluation Measure,
Abstract: Ranking is used for a wide array of problems, most notably information
retrieval (search). There are a number of popular approaches to the evaluation
of ranking such as Kendall's $\tau$, Average Precision, and nDCG. When dealing
with problems such as user ranking or recommendation systems, all these
measures suffer from various problems, including an inability to deal with
elements of the same rank, inconsistent and ambiguous lower bound scores, and
an inappropriate cost function. We propose a new measure, rankDCG, that
addresses these problems. This is a modification of the popular nDCG algorithm.
We provide a number of criteria for any effective ranking algorithm and show
that only rankDCG satisfies all of them. Results are presented on constructed
and real data sets. We release a publicly available rankDCG evaluation package.