Today, at the Cross Validated Journal Club (why weren't you there?), @mbq asked:

> 
> Do you think we (modern data scientists) know what significance means? And how it relates to our confidence in our results?
> 
> 
> 

@Michelle replied as some (including me) usually do:

> 
> I'm finding the concept of significance (based on p-values) less and less helpful as I continue in my career. For example, I can be using extremely large datasets so everything is statistically significant ($p<.01$)
> 
> 
> 

This is probably a stupid question, but isn't the problem the hypothesis being tested? If you test the null hypothesis "A is equal to B" then you know the answer is "No". Bigger data sets will only bring you closer to this inevitably true conclusion. I believe it was Deming who once gave an example with the hypothesis "the number of hairs on the right side of a lamb is equal to the number of hairs on its left side." Well, of course it isn't.

A better hypothesis would be "A does not differ from B by more than so much." Or, in the lamb example, "the number of hairs on the sides of a lamb does not differ by more than X%".

Does this make sense?