Title: Second-generation p-values: improved rigor, reproducibility, & transparency in statistical analyses,
Abstract: Verifying that a statistically significant result is scientifically
meaningful is not only good scientific practice, it is a natural way to control
the Type I error rate. Here we introduce a novel extension of the p-value - a
second-generation p-value - that formally accounts for scientific relevance and
leverages this natural Type I Error control. The approach relies on a
pre-specified interval null hypothesis that represents the collection of effect
sizes that are scientifically uninteresting or are practically null. The
second-generation p-value is the proportion of data-supported hypotheses that
are also null hypotheses. As such, second-generation p-values indicate when the
data are compatible with null hypotheses, or with alternative hypotheses, or
when the data are inconclusive. Moreover, second-generation p-values provide a
proper scientific adjustment for multiple comparisons and reduce false
discovery rates. This is an advance for environments rich in data, where
traditional p-value adjustments are needlessly punitive. Second-generation
p-values promote transparency, rigor and reproducibility of scientific results
by a priori specifying which candidate hypotheses are practically meaningful
and by providing a more reliable statistical summary of when the data are
compatible with alternative or null hypotheses.