How can AI researchers avoid "overfitting" to commonly-used benchmarks as a community?
Great question Dennis!

This is a perennial topic at AI conferences, and sometimes even in special issues of journals. The most recent one I recall was Moving Beyond the Turing Test in 2015, which ended up leading to a collection of articles in AI magazine later that year. 

Usually these discussions cover a number of themes:


"Existing benchmarks suck". This is usually the topic that opens discussion. In the 2015/2016 discussion, which focused on the Turing Test as a benchmark specifically, criticisms ranged from "it doesn't incentivize AI research on the right things", to claims that it was poorly defined, too hard, too easy, or not realistic.
General concensus that we need new benchmarks.
Suggestions of benchmarks based on various current research directions. In the latest discussion this included answering standardized tests for human students (well defined success, clear format, requires linking and understanding many ares), playing video games (well defined success, requires visual/auditory processing, planning, coping with uncertainty), and switching focus to robotics competitions.


I remember attending very similar discussions at machine learning conferences in the late 2000's, but I'm not sure anything was published out of it.

Despite these discussions, AI researchers seem to incorporate the new benchmarks, rather than displacing the older ones entirely. The Turing Test is still going strong for instance. I think there are a few reasons for this.

First, benchmarks are useful, particularly to provide context for research. Machine learning is a good example. If the author sets up an experiment on totally new data, then even if they apply a competing method, I have to trust that they did so faithfully, including things like optimizing the parameters as much as with their own methods. Very often they do not do this (it requires some expertise with competing methods), which inflates the reported advantage of their own techniques. If they also run their algorithm on a benchmark, then I can easily compare it to the benchmark performances reported by other authors, for their own methods. This makes it easier to spot a technique that's not really effective.

Second, even if new benchmarks or new problems are more useful, nobody knows about them! Beating the current record for performance on ImageNet can slingshot someone's career in a way that top performance on a new problem simply cannot. 

Third, benchmarks tend to be things that AI researchers think can actually be accomplished with current tools (whether or not they are correct!). Usually iterative improvement on them is fairly easy (e.g. extend an existing technique). In a "publish-or-perish" world, I'd rather publish a small improvement on an existing benchmark than attempt a risker problem, at least pre-tenure. 

So, I guess my view is fixing the dependence on benchmarks involves fixing the things that make people want to use them:


Have some standard way to compare techniques, but require researchers to also apply new techniques to a real world problem.
Remove the career and prestige rewards for working on benchmark problems, perhaps by explicitly tagging them as artificial.
Remove the incentives for publishing often.