Peformance evaluation of ranking algorithms
Kaggle's famous competition Chess ratings - Elo versus the Rest of the World, that aimed "to discover whether other approaches can predict the outcome of chess games more accurately than the workhorse Elo rating system", used this structure


  Competitors train their rating systems using a training dataset of over 65,000 recent results for 8,631 top players. Participants then use their method to predict the outcome of a further 7,809 games


A similar structure - starting from a complete dataset, using first part 
 for training and last part to check the outcome - could be useful to measure the performance of the ranking algorithms.