A Continuous Information Gain Measure to Find the Most Discriminatory
  Problems for AI Benchmarking
  This paper introduces an information-theoretic method for selecting a subset
of problems which gives the most information about a group of problem-solving
algorithms. This method was tested on the games in the General Video Game AI
(GVGAI) framework, allowing us to identify a smaller set of games that still
gives a large amount of information about the abilities of different
game-playing agents. This approach can be used to make agent testing more
efficient. We can achieve almost as good discriminatory accuracy when testing
on only a handful of games as when testing on more than a hundred games,
something which is often computationally infeasible. Furthermore, this method
can be extended to study the dimensions of the effective variance in game
design between these games, allowing us to identify which games differentiate
between agents in the most complementary ways.
