Title: Approximate Ranking from Pairwise Comparisons,
Abstract: A common problem in machine learning is to rank a set of n items based on
pairwise comparisons. Here ranking refers to partitioning the items into sets
of pre-specified sizes according to their scores, which includes identification
of the top-k items as the most prominent special case. The score of a given
item is defined as the probability that it beats a randomly chosen other item.
Finding an exact ranking typically requires a prohibitively large number of
comparisons, but in practice, approximate rankings are often adequate.
Accordingly, we study the problem of finding approximate rankings from pairwise
comparisons. We analyze an active ranking algorithm that counts the number of
comparisons won, and decides whether to stop or which pair of items to compare
next, based on confidence intervals computed from the data collected in
previous steps. We show that this algorithm succeeds in recovering approximate
rankings using a number of comparisons that is close to optimal up to
logarithmic factors. We also present numerical results, showing that in
practice, approximation can drastically reduce the number of comparisons
required to estimate a ranking.