The Assessment of Intrinsic Credibility and a New Argument for p<0.005
Leonhard Held














The Assessment of Intrinsic Credibility and a New Argument for p<0.005
    Leonhard
  Held
 
  Epidemiology, Biostatistics
  and Prevention Institute (EBPI)
 
  and Center for Reproducible Science (CRS) 

  University of Zurich
 Hirschengraben 84,
  8001 Zurich, Switzerland
 Email: 
    Accepted 23/03/2018
====================================================================================================================================================================================================================





12cm
Abstract: The concept of intrinsic credibility has been
recently introduced to check the credibility of “out of the blue”
findings without any prior support.  A significant result is deemed
intrinsically credible if it is in conflict with a sceptical prior
derived from the very same data that would make the effect
non-significant.  In this paper I propose to use Bayesian
prior-predictive tail probabilities to assess intrinsic credibility.
For the standard 5% significance level, this leads to a new p-value
threshold that is remarkably close to the recently proposed p<0.005
standard.  I also introduce the credibility ratio, the ratio of the
upper to the lower limit of a standard confidence interval for the
corresponding effect size.  I show that the credibility ratio has to
be smaller than 5.8 such that a significant finding is also
intrinsically credible.  Finally, a p-value for intrinsic
credibility is proposed that is a simple function of the ordinary
p-value and has a direct frequentist interpretation in terms of the
probability of replicating an effect. 

Key Words: Confidence Interval; Credibility Ratio; Intrinsic Credibility; 
    Prior-Data Conflict; P-value; Replication; Significance Test








§ INTRODUCTION



The so-called replication crisis of science has been discussed
  extensively within the scientific community <cit.>. One aspect of the 
problem is the widespread misunderstanding and misinterpretation of
basic statistical concepts, such as the p-value <cit.>. This has lead to a major rethinking and
new proposals for statistical inference, such as
to lower the threshold for statistical significance
from the traditional 0.05 level to 0.005
<cit.>. The proposal
has created a lot of discussion in the scientific community and the
shortcut “p<0.005” has been even shortlisted and highly commended
in the Statistic of the Year competition by the Royal Statistical
Society, see <http://bit.ly/2yWFPbD>.

Two arguments for this step are provided in
<cit.>: The first is based on the Bayes factor,
the second is based on the false discovery rate. Both arguments are
actually not new, <cit.> have already emphasized that
the evidence of p-values around 0.05 against a point null
hypothesis, as quantified by the Bayes factor, is much smaller than
one would naively expect: “Even the utmost generosity to the
alternative hypothesis cannot make the evidence in favor of it as
strong as classical significance levels might suggest.” Likewise,
<cit.> have already argued that the false positive
rate “could be considerably reduced by increasing the sample sizes
and by restricting the allowance made for the α error,
which should be set to a 1% level as a minimum requirement.”
<cit.> therefore propose to lower the threshold
for statistical significance to 0.005 and to declare results with
0.05>p>0.005 as “suggestive”, emphasizing the need for replication.


In this paper I provide a new argument for this categorization
  into three levels of evidence. The approach is based on the concept
  of intrinsic credibility <cit.>, a specific
  reverse-Bayes method to assess the credibility of claims of new
  discoveries.  I review and refine the approach and show that, if you
  dichotomize p-values into “significant” and “non-significant”,
  the proposed method naturally leads to a more stringent threshold
  for intrinsic credibility. For the standard 5% significance level,
  the new p-value thres­hold is 0.0056, remarkably close to
  the recently proposed p<0.005 standard.  
  
  To assess intrinsic credibility based on a confidence interval
  rather than a p-value, I propose the credibility ratio, the ratio
  of the upper to the lower limit of a standard confidence interval
  for the corresponding effect size.  
  I show that the credibility
  ratio has to be smaller than 5.8 to ensure that a significant
    finding is also intrinsically credible.  In Section
  <ref> I provide a brief summary of the Analysis of
  Credibility and the specific concept of intrinsic credibility. The
  latter is central for the derivation of a threshold for intrinsic
  credibility, as outlined in Section <ref>.  

 Lowering the threshold of statistical significance is only a
  temporary measure to the replication crisis <cit.>. A
  more radical step would be to abandon significance thresholds
  altogether <cit.>, leaving p-values as a purely
  quantitative measure of the evidence against a point null
  hypothesis. In this spirit I extend the concept of intrinsic
  credibility and propose in Section <ref> the p-value for
  intrinsic credibility, p_IC.  This new measure can be used to
  quantify the evidence for intrinsic credibility – without any need
  for thresholding – and has a direct and useful interpretation in
  terms of the probability of replicating an effect
  <cit.>. Intrinsic credibility is thus directly linked
  to replication, a topic of central importance in the current debate on research
  reproducibility <cit.>.  I close with some discussion
  in Section <ref>.  



§ ANALYSIS OF CREDIBILITY




Reverse-Bayes approaches allow the extraction of the properties of the
prior distribution needed to achieve a certain posterior statement for
the data at hand.  The idea to use Bayes's theorem in reverse
originates in the work by IJ Good <cit.> and is
increasingly used to assess the plausibility of scientific claims and
findings
<cit.>.
<cit.> has proposed the Analysis of
Credibility, a specific reverse-Bayes method to challenge claims of “significance”, 
  see <cit.> for more recent developments. 

Analysis of Credibility is based on a conventional confidence interval
of level γ, say, for an unknown effect size θ with
lower limit L and upper limit U, say.  In the following I assume
that both L and U are symmetric around the effect estimate θ̂ (assumed to be normally distributed) and that both are either
positive or negative, the effect is significant at significance
level α =1-γ.  <cit.> proposed
assessing the credibility of a statistically significant finding by
computing from the data a sceptical prior distribution for the effect
size θ, normal with mean zero, that - combined with the
information given in the confidence interval for θ - results in
a posterior distribution which is just non-significant at level
α, either the α/2 or the 1-α/2 posterior
quantile is zero.  It can be shown that the limits ± S of the
corresponding equi-tailed prior credible interval at level
γ are given by

    S = (U-L)^2/4 √(UL),

where S is called the scepticism limit and the interval
[-S,S] is called the critical prior interval.  Note that
(<ref>) holds for any level γ, not just for
the traditional 95% level.



It is convenient to express 
the variance τ^2 of the sceptical prior 
as a function of the variance σ^2 (the squared
standard error, assumed to be known) of the estimate θ̂,
the corresponding test statistic t=θ̂/σ, and
z_α/2, the 1-α/2 quantile of the standard normal
distribution:

    τ^2 = σ^2/t^2/z_α/2^2 - 1,

where t^2>z_α/2^2 is required for significance at level
α.  Equation (<ref>) shows that the prior variance
τ^2 can be both smaller or larger than σ^2, depending on
the value of t^2. If t^2 is substantially larger than
z_α/2^2, then the sceptical prior variance will be relatively
small, a relatively tight prior is needed to make the significant result non-significant. 
If t^2 is close to z_α/2^2 (the effect is
“borderline significant”), then the sceptical prior variance will be
relatively large, a relatively vague prior is sufficient to make
the significant result non-significant. 


Two applications of the Analysis of Credibility are shown in
  Figure <ref>. Both are based on a confidence interval of
  width 3, but with different location (θ̂= 2.5 and 11/6=1.83,
  respectively).
  Each Figure has to be read from right to left: To
  obtain a 95% posterior credible interval with lower limit 0 (shown in green), the
  95% confidence interval for the unknown effect size θ (shown in red) has to be combined with a sceptical
  prior with variance (<ref>) (shown in blue). 


































 In this paper I focus on claims of new discoveries without any prior support. 
To assess the credibility of such “out of the blue”  findings, 
  <cit.> suggested the concept of intrinsic
  credibility, declaring an effect as intrinsically credible if it is in conflict with the sceptical
  prior (with mean zero and variance (<ref>))
  that would make the effect
  non-significant.  This can be thought of as an additional check to ensure
  that a significant effect is not spurious.  Specifically,
  <cit.> declares a result as intrinsically credible at
  level γ, if the effect estimate θ̂ is outside the
  sceptical prior interval, θ̂ > S.  





He shows
that, for confidence intervals at level γ=0.95, this is
equivalent to the conventional two-sided p-value being smaller than
0.0127.  I refine the definition of intrinsic credibility in the following Section
  <ref> based on the <cit.> prior-predictive approach, 
    leading to the more stringent p-value threshold 0.0056 for intrinsic credibility at the 95% level.




§ A NEW THRESHOLD FOR INTRINSIC CREDIBILITY




Matthews' check for intrinsic credibility compares the size of θ̂ with the scepticism limit (<ref>), so does not take the uncertainty
of θ̂ into account. He compares the estimate θ̂  with the
(sceptical) prior distribution, not with the corresponding prior-predictive
distribution.  However, use of the latter is the established way to check the compatibility of the data and the prior <cit.>.
In
what follows I will 
therefore apply the 
approach by <cit.> for the assessment of prior-data conflict based on the prior-predictive distribution,
with the perhaps slightly unusual feature that the prior has been
derived from the data. I argue that there is nothing intrinsically
inconsistent in investigating the compatibility of a prior, defined
through the data, and the data itself, extending an argument by
<cit.> to the reverse-Bayes setting.

The <cit.> check for prior-data conflict
is based on the prior-predictive distribution, which is in our case
normal with mean zero and variance τ^2 + σ^2 <cit.>.  The procedure
is based on the test statistic

t_ = θ̂/√(τ^2+σ^2)

and the (two-sided) tail probability 

p_ =(χ^2(1) ≥ t_^2)

as the corresponding upper tail of a
χ^2-distribution with one degree of freedom. 
Small values of p_ indicate a conflict between the sceptical prior
and the data. 



Now suppose we fix the confidence level at the conventional 95% level, γ=0.95.
 Intrinsic credibility at the 95% level (p_ < 0.05) can
then be shown to be equivalent to the requirement p<0.0056 for the conventional two-sided
p-value.  
To derive this result, note that with (<ref>) we have

τ^2 + σ^2 = σ^2 / (1- z_α/2^2 / t^2 )

and so 

 t_^2  
  =  t^2 - z_α/2^2. 

The requirement t_^2 > z_α/2^2 for intrinsic
credibility at level γ=1-α then translates to

    t^2    ≥     2   z_α/2^2.

This criterion is to be compared with the traditional check for
significance, which requires only t^2 ≥ z_α/2^2.


It follows directly that
the threshold

    α_IC   =    2 {1 - Φ(t=√(2)  z_α/2) },

here Φ(.) denotes the cumulative standard normal distribution
  function,  can be used to assess intrinsic credibility based on
  the conventional two-sided p-value p: If p is smaller than
α_IC, then the result is intrinsically credible at level
γ=1-α.  For α=0.05 we have t=√(2)· 1.96=
2.77 and the threshold (<ref>) turns
out to be α_IC = 0.0056, as claimed
above. For other confidence levels we will obtain other intrinsic
  credibility thresholds. For example, <cit.> prefer to
  use 90% confidence intervals “on the grounds that they give a better
  impression of the range of plausible values”. Then γ=0.9 and
  we obtain the intrinsic credibility threshold α_IC =
  0.02.



Figure <ref> compares the new threshold with the one obtained
by <cit.> (using
t=1.272   z_α/2) for values of
α below 10%.  
The Matthews threshold for intrinsic credibility
is larger than the proposed new threshold (<ref>), because it compares 
the effect estimate θ̂ with the prior distribution (with variance τ^2) 
and 
not the prior-predictive distribution (with variance τ^2 + σ^2).














Intrinsic credibility can also be assessed based on the
  confidence interval [L, U], rather than the conventional p-value p.
To see this, note that t_^2
can be written in terms of L and U, 

    t_^2 = z_α/2^2 4   U L/(U-L)^2,

and the requirement t_^2 ≥ z_α/2^2 for 
intrinsic credibility is then equivalent to require
that the credibility ratio U/L (or L/U if both L and U are negative) fulfills

    U/L≤ d = 3 + 2  √(2)≈ 5.8.

To
derive the cut-point d in (<ref>), set U=L   d. The
requirement t_^2 = z_α/2^2 then reduces to 

    1 = 4   U L/(U-L)^2 = 4   d/(d-1)^2,
 a quadratic
equation in d with d=3 + 2  √(2) 
as solution. 

Thus, there is a second way to assess intrinsic credibility based
on the ratio of the limits of a confidence interval at any level
γ: if the credibility ratio is smaller than
5.8 than the result is credible at level
γ. 
For example, in Figure <ref> the credibility
ratio is 4 in the top and 10 in the bottom panel, so the result
shown in the top panel is intrinsically credible at level 95%, but the one in the bottom is not.




If the sceptical prior distribution is available, then a third
way to assess intrinsic credibility is to compare the prior variance
τ^2 to the data variance σ^2. Comparing (<ref>)
with (<ref>) it is easy to see that intrinsic credibility 
is achieved if and only if the sceptical prior variance τ^2 is not larger
than the variance σ^2 of the effect estimate θ̂. With
this in mind we see immediately from Figure <ref> that the first
result shown in the top panel is intrinsically credible (τ^2 < σ^2), whereas the
second isn't (τ^2 > σ^2).










§ A P-VALUE FOR INTRINSIC CREDIBILITY



 A disadvantage of the dichotomous assessment of intrinsic
  credibility described in the previous section is the dependence on
  the confidence level γ of the underlying confidence
  interval, or, equivalently, the significance level
  α=1-γ.  However, there is a way to free ourselves from
  this dependence.  In analogy to the well-known duality of confidence
  intervals and standard p-values, I propose to derive the value
  α^⋆, say, that just achieves intrinsic credibility, where equality holds in (<ref>).  This defines the p-value for intrinsic credibility p_IC=α^⋆, which
  provides a quantitative assessment of the evidence for intrinsic
  credibility. Of course, the p-value for intrinsic credibility
  p_IC can also be used to assess intrinsic credibility as
  described in Section <ref>: if p_IC≤α, then
  the result is intrinsically credible at level γ=1-α.


The p-value p_IC for intrinsic
credibility can be derived by replacing α_IC
with p and α with p_IC in equation (<ref>) and then solving for p_IC:

    p_IC=2[1-Φ(t/√(2))].

Here t= Φ^-1(1-p/2) is the standard test statistic for
significance where p is the conventional two-sided
p-value. Note that the test statistic t_I = t /
√(2) for intrinsic credibility in (<ref>) is a root-2 shrunken version of the
test statistic t for significance.




 Figure <ref> shows that the p-value p_IC for
  intrinsic credibility is considerably larger than the conventional p-value p, 
  particularly for small values of p.  For example, the two confidence intervals shown 
  in Figure  <ref> have 
  conventional p-values p=0.0011 (top) and p=0.017 (bottom),  
  while the corresponding
  p-values for intrinsic credibility are p_IC=0.021
  and p_IC=0.09,
  respectively. If we are prepared to adapt the “rough and ready” p-value guide
  by <cit.> to p_IC, then
  
  
  

  p_IC=0.021 provides moderate
  evidence and p_IC=0.09 only weak
  evidence for intrinsic credibility. 











  
  There is a direct and useful interpretation of p_IC  
  in terms of the probability of replicating an effect <cit.>, the probability 
  that an identically designed but independent replication study 
will give an estimated effect θ̂_2 in the
same direction as the estimate θ̂_1 = θ̂ from the current
(first) study. To see this, note that under an initial uniform prior
the posterior for θ is θθ̂_1 ∼(θ̂_1,
σ^2). This posterior now serves as the prior for the mean of the (unobserved) estimate
θ̂_2 θ∼(θ, σ^2) from the second (hypothetical) study,
where we assumed the two studies to be identically designed, having equal
variances σ^2.  This leads to the prior-predictive distribution
θ̂_2  θ̂_1 ∼(θ̂_1, 2  σ^2) and the p-value 
for intrinsic credibility (<ref>) can be seen to be
twice the probability that the second study will give an 
estimate θ̂_2 in the opposite direction as the estimate θ̂_1 of the 
first study:

    p_IC   =    2 [1-Φ(t/√(2)) ] 
       =    2 Φ(-t/√(2)) 
       =    2 Φ(0 - θ̂_1/√(2)σ) 
       =     2 (θ̂_2 ≤ 0 θ̂_1 > 0).

If θ̂_1 < 0, then p_IC = 2 (θ̂_2 ≥ 0 θ̂_1 < 0).

 
  The probability (θ̂_2 ≤ 0 θ̂_1 >
  0) is one of the three replication probabilities that have been
  considered by <cit.> in response to 
  <cit.>. The complementary probability (θ̂_2 > 0 θ̂_1 > 0) = 1 - p_IC/2 can be
  identified as the probability of replicating an effect, ,  
  advocated by <cit.>
  as an alternative to traditional p-values, see
<cit.> for further discussion and additional references. 
  Of course,  can only be correct under the assumption that the null hypothesis is false. 
  
  
  Nevertheless,
  <cit.> argues that  is a useful alternative to traditional p-values.


In practice, we can thus use p_IC to assess the probability of
  replicating an effect, assuming that the null hypothesis is false: = 1 - p_IC/2. An intrinsically
  credible result with p_IC≤γ therefore has ≥ (1+γ)/2. 
  For example, for γ=95% we have ≥ 97.5%. 
For numerical illustration, recall that the p-values for
intrinsic credibility in Figure <ref> are
p_IC=0.021 (top) and
p_IC=0.09
(bottom). The corresponding replication probabilities are thus
=99.0% and
=95.5%.  In
the second example, there is thus a
=4.5%
chance that an identically designed replication study will give a negative effect estimate.





§ DISCUSSION



Based on the Analysis of Credibility, I have shown that, if you
  dichotomize p-values into “significant” and “non-significant” at
  some pre-specified threshold α, the Analysis of Credibility
  directly leads to a more stringent threshold α_IC for
  intrinsic credibility. If you prefer to avoid any thresholding of conventional p-values,
  a new p-value for intrinsic credibility, p_IC, has been proposed.    
  p_IC is a quantitative measure of the evidence for intrinsic credibility with a
  direct connection to , the probability of replicating an effect
<cit.>.


The assessment of intrinsic credibility can
be thought of as an additional challenge,  
ensuring that claims of new
  discoveries are not spurious. Conventionally significant results
with 0.05 > p > 0.0056 
lack intrinsic credibility, they are not in conflict with a
sceptical prior that would make the effect non-significant. This
matches the classification as “suggestive” by
<cit.>. Specifically, p > 0.0056 implies 
p_IC > 0.05 and thus < 97.5%, 
emphasizing the need for replication.
If p<0.0056, then the result
is both significant and intrinsically credible at the 95% level, 
so p_IC≤ 0.05 and ≥ 97.5%.

The credibility ratio provides a simple and convenient tool to check
whether a “significant” confidence interval at any level γ
is also intrinsically credible. If
the credibility ratio is smaller 5.8, the result can be considered
as intrinsically credible at level γ.
It is noteworthy that the concept of intrinsic credibility does not require to change the
original confidence level γ. Indeed, the check for credibility is done
at the same level as the original confidence level. I have used
γ=0.95 by convention, where it follows that the check for
intrinsic credibility is equivalent to the requirement p<0.0056.
This implies that in standard statistical reporting there is no need to
replace 95% confidence intervals with 99.5% confidence intervals, say. 
However, I suggest to complement or to replace the ordinary p-value
with the proposed p-value for intrinsic credibility, p_IC.







Although derived using a Bayesian approach, the proposed check for
intrinsic credibility is based on a standard confidence interval and
thus constitutes a Bayes/non-Bayes compromise <cit.>.
Specifically, it does not require the specification of a prior
probability of the null hypothesis of no effect. In fact, this
  prior probability is always zero.  This is in contrast to the
calibration of p-values to lower bounds on the posterior probability
of the null, which requires specification of a prior probability.
Minimum Bayes factors have also been proposed to calibrate p-values,
see <cit.> for a recent review.  They have the
  advantage that they do not require specification of a prior
  probability of the null hypothesis and provide a direct
  “forward-Bayes” assessment of the evidence of p-values. However,
  the underlying rationale is still based on a point null hypothesis with
  positive prior probability, fundamentally different from the
  approach proposed here.  














The Analysis of Credibility assumes a simple mathematical framework,
where likelihood, prior and posterior are all normally
distributed. This can be justified because Gaussian
  approximations are commonly used in the calculation of confidence
  intervals and statistical hypothesis tests, if the sample size is fairly large <cit.>. Of course, suitable transformations of the
  parameter of interest may be needed to achieve normality, for
  example, confidence intervals for odds ratios and hazard ratios should be transformed to the
  log scale. For small studies, however, the normal assumption for the likelihood may be questionable and the 
  assessment of intrinsic credibility would need appropriate refinement, for example based on the t-distribution.







§ ACKNOWLEDGMENTS



I am grateful to Robert Matthews, Stefanie Muff, Manuela Ott and Kelly Reeve 
for helpful comments on earlier drafts of this manuscript. 
 


apalike



