How can AI researchers avoid "overfitting" to commonly-used benchmarks as a community? In fields such as Machine Learning, we typically (somewhat informally) say that we are overfitting if improve our performance on a training set at the cost of reduced performance on a test set / the true population from which data is sampled.

More generally, in AI research, we often end up testing performance of newly proposed algorithms / ideas on the same benchmarks over and over again. For example:


For over a decade, researchers kept trying thousands of ideas on the game of Go.
The ImageNet dataset has been used for huge amounts of different publications
The Arcade Learning Environment (Atari games) has been used for thousands of Reinforcement Learning papers, having become especially popular since the DQN paper in 2015.


Of course, there are very good reasons for this phenomenon where the same benchmarks keep getting used:


Reduced likelihood of researchers "creating" a benchmark themselves for which their proposed algorithm "happens" to perform well
Easy comparison of results to other publications (previous as well as future publications) if they're all consistently evaluated in the same manner.


However, there is also a risk that the research community as a whole is in some sense "overfitting" to these commonly-used benchmarks. If thousands of researchers are generating new ideas for new algorithms, and evaluate them all on these same benchmarks, and there is a large bias towards primarily submitting/accepting publications that perform well on these benchmarks, the research output that gets published does not necessarily describe the algorithms that perform well across all interesting problems in the world; there may be a bias towards the set of commonly-used benchmarks.



Question: to what extent is what I described above a problem, and in what ways could it be reduced, mitigated or avoided?
Great question Dennis!

This is a perennial topic at AI conferences, and sometimes even in special issues of journals. The most recent one I recall was Moving Beyond the Turing Test in 2015, which ended up leading to a collection of articles in AI magazine later that year. 

Usually these discussions cover a number of themes:


"Existing benchmarks suck". This is usually the topic that opens discussion. In the 2015/2016 discussion, which focused on the Turing Test as a benchmark specifically, criticisms ranged from "it doesn't incentivize AI research on the right things", to claims that it was poorly defined, too hard, too easy, or not realistic.
General concensus that we need new benchmarks.
Suggestions of benchmarks based on various current research directions. In the latest discussion this included answering standardized tests for human students (well defined success, clear format, requires linking and understanding many ares), playing video games (well defined success, requires visual/auditory processing, planning, coping with uncertainty), and switching focus to robotics competitions.


I remember attending very similar discussions at machine learning conferences in the late 2000's, but I'm not sure anything was published out of it.

Despite these discussions, AI researchers seem to incorporate the new benchmarks, rather than displacing the older ones entirely. The Turing Test is still going strong for instance. I think there are a few reasons for this.

First, benchmarks are useful, particularly to provide context for research. Machine learning is a good example. If the author sets up an experiment on totally new data, then even if they apply a competing method, I have to trust that they did so faithfully, including things like optimizing the parameters as much as with their own methods. Very often they do not do this (it requires some expertise with competing methods), which inflates the reported advantage of their own techniques. If they also run their algorithm on a benchmark, then I can easily compare it to the benchmark performances reported by other authors, for their own methods. This makes it easier to spot a technique that's not really effective.

Second, even if new benchmarks or new problems are more useful, nobody knows about them! Beating the current record for performance on ImageNet can slingshot someone's career in a way that top performance on a new problem simply cannot. 

Third, benchmarks tend to be things that AI researchers think can actually be accomplished with current tools (whether or not they are correct!). Usually iterative improvement on them is fairly easy (e.g. extend an existing technique). In a "publish-or-perish" world, I'd rather publish a small improvement on an existing benchmark than attempt a risker problem, at least pre-tenure. 

So, I guess my view is fixing the dependence on benchmarks involves fixing the things that make people want to use them:


Have some standard way to compare techniques, but require researchers to also apply new techniques to a real world problem.
Remove the career and prestige rewards for working on benchmark problems, perhaps by explicitly tagging them as artificial.
Remove the incentives for publishing often.