Title: Consensus measure of rankings,
Abstract: A ranking is an ordered sequence of items, in which an item with higher
ranking score is more preferred than the items with lower ranking scores. In
many information systems, rankings are widely used to represent the preferences
over a set of items or candidates. The consensus measure of rankings is the
problem of how to evaluate the degree to which the rankings agree. The
consensus measure can be used to evaluate rankings in many information systems,
as quite often there is not ground truth available for evaluation.
This paper introduces a novel approach for consensus measure of rankings by
using graph representation, in which the vertices or nodes are the items and
the edges are the relationship of items in the rankings. Such representation
leads to various algorithms for consensus measure in terms of different aspects
of rankings, including the number of common patterns, the number of common
patterns with fixed length and the length of the longest common patterns. The
proposed measure can be adopted for various types of rankings, such as full
rankings, partial rankings and rankings with ties. This paper demonstrates how
the proposed approaches can be used to evaluate the quality of rank aggregation
and the quality of top-$k$ rankings from Google and Bing search engines.