Database of chess games for evaluating ranking systems?
There was a Kaggle competition sponsored by the FIDE looking for an alternative to the current ELO system. The link is:

http://www.kaggle.com/c/ChessRatings2

The dataset has over a million actual games covering ~50k players. There's also discussions of the alternative approaches used (the competition itself is now over) and scoreboard of metrics to see how well your ranking system compares.