The idea of *adaptive data analysis* is that you alter your plan for analyzing the data as you learn more about it. In the case of exploratory data analysis (EDA), this is generally a good idea (you are often looking for unforeseen patterns in the data), but for a confirmatory study, this is widely accepted as a very flawed method of analysis (unless all the steps are clearly defined and properly planned out in advanced).

That being said, adaptive data analysis **is** typically how many researchers actually conduct their analyses, much to the dismay of statisticians. As such, if one could do this in a statistical valid manner, it would revolutionize statistical practice.

The following *Science* article claims to have found a method for doing such (I apologize for the paywall, but if you are at a university, you likely have access): [Dwork et al, 2015, The reusable holdout: Preserving validity in adaptive data analysis](http://science.sciencemag.org/content/349/6248/636).

Personally, I've always been skeptical of statistics articles published in *Science*, and this one is no different. In fact, after reading through the article twice, including the supplementary material, I cannot understand (at all) why the authors claim that their method prevents over-fitting. 

My understanding is that they have a holdout dataset, which they will reuse. They seem to claim by "fuzzing" the output of the confirmatory analysis on the holdout dataset, over-fitting will be prevented (it is worth noting that the fuzzing seems to be just adding noise *if the calculated statistic on the training data is sufficiently far from the calculated statistic on the holdout data*). As far as I can tell, there is no real reason this should prevent over-fitting. 

Am I mistaken on what the authors are proposing? Is there some subtle effect that I'm overlooking? Or has *Science* endorsed the worst statistical practice to date?