Two sources of poor coverage of confidence
intervals after model selection
Paul Kabaila∗ and Rheanna Mainzer

arXiv:1711.01739v1 [math.ST] 6 Nov 2017

Department of Mathematics and Statistics, La Trobe University, Australia

ABSTRACT
We compare the following two sources of poor coverage of post-model-selection confidence intervals: the preliminary data-based model selection sometimes chooses the
wrong model and the data used to choose the model is re-used for the construction
of the confidence interval.

Keywords: Coverage probability, confidence interval, model selection

* Corresponding author. Department of Mathematics and Statistics, La Trobe University, Victoria 3086, Australia. Tel.: +61 3 9479 2594; fax +61 3 9479 2466.
E-mail address: P.Kabaila@latrobe.edu.au.

1

1. Introduction
In applied statistics a model is often chosen using a data-based model selection
procedure. Such procedures include hypothesis testing, minimizing a criteron such
2
(Sheather, 2008). Inferences
as AIC or BIC, and maximizing a criterion such as Radj

made using a model that has been selected using data-based model selection can
be misleading (see e.g. Chatfield 1995). Suppose that the inference of interest is a
confidence interval for a parameter θ, with minimum coverage 1 − α. Typically, a
confidence interval for θ that is constructed using a model chosen by some form of
data-based model selection procedure will have poor coverage properties (Kabaila,
2009, Leeb and Pötscher, 2005). This poor coverage probability can be attributed
to two different sources. The first source is that the preliminary model selection
procedure sometimes chooses the wrong model. When this happens, the incorrect
model is treated as if it were the true model during the construction of the confidence
interval. The second source is that the data used to choose the model is re-used
for the construction of the confidence interval, without due acknowledgment. It is
important to know the relative importance of these two sources of poor coverage.
Miller (2002, Ch.6) considers the problems that arise for parameter estimation
due to the re-use of the data used to choose a model. Hurvich and Tsai (1990) examine the coverage probability of confidence regions for the regression parameter vector
in linear regression models after data-based model selection. They acknowledge the
problem of re-using the same data and propose a possible solution:
One possible solution ... is to actually perform model selection and
inference on separate parts of the data. ... This procedure may seem
wasteful, but we note that linear regression model selection can now be
performed on quite small samples. ... Thus the available data set need
not be split in half; instead, a small part may be used for the model
selection stage and the remainder may be used for inference.
Thus if the second source of poor coverage (re-use of the data) dominates the first
source (possible choice of the wrong model) we could split the data into two parts:
the first part could be used to choose the model and the second to construct a
confidence interval. By allocating most of the data to the second part, we might
hope to avoid constructing confidence intervals that are too wide.
If, on the other hand, the first source of poor coverage dominates the second
source, this data splitting strategy would not correct the defect of poor coverage
2

probability. Instead we would need to consider other methods to improve the coverage probability. We expect that problems caused by possibly choosing a wrong
model would be ameliorated by using an appropriate data-weighted average of models in place of putting all of the weight on a model that may be wrong. In other
words, if the first source of poor coverage dominates the second source then some
form of frequentist model averaged confidence interval (Buckland, Burnham and
Augustin, 1997, Hjort and Claeskens 2003, Fletcher and Turek, 2011, Turek and
Fletcher, 2012, Efron, 2014) would be expected to greatly improve the coverage
probability.
In this paper we examine the extent to which each of these two sources contributes to the difference between true and nominal coverage probability. The basic
argument presented in Section 2 gives us the means to identify Dwm , the deficit in
coverage probability due to the possible choice of the wrong model, and Drd , the
deficit in coverage probability due to re-use of the data. The coverage probability of
the confidence interval, constructed after a preliminary data-based model selection,
is given by 1 − α − Dwm − Drd .
In Section 3, we apply this basic argument to the simple scenario that preliminary
data-based choice is between two nested linear regression models. The simpler model
is obtained from the full model by setting τ , a given linear combination of the
regression parameters, to 0. The parameter of interest θ is a distinct given linear
combination of the regression parameters. Let θb and τb denote the least squares
estimators of θ and τ , respectively. We consider preliminary model selection using
a t test of the null hypothesis τ = 0 against the alternative hypothesis τ 6= 0. Let
n denote the dimension of the response vector and p denote the dimension of the
regression parameter vector. Define the unknown parameter γ = τ /(var(b
τ ))1/2 and
b τb).
the known correlation ρ = corr(θ,
Figure 1 is for the confidence interval, with nominal coverage 0.95, constructed
after a preliminary t test with size 0.1, when n−p = 40 and |ρ| ∈ {0.3, 0.6, 0.8}. The
top panel shows graphs of Dwm , the deficit in coverage due to the choice of the wrong
model, as a function of |γ|. The middle panel shows graphs of Drd , the deficit in
coverage due to the re-use of the data, as a function of |γ|. The bottom panel shows
graphs of the coverage probability of this post-model-selection confidence interval,
given by 1 − α − Dwm − Drd , as a function of |γ|. We do not include the value ρ = 0
in these graphs since, to a very good approximation, Dwm = 0, Drd = 0 and the

3

Dwm for m = 40
0.30

| ρ | = 0.8
| ρ | = 0.6
| ρ | = 0.3

0.25
0.20
0.15
0.10
0.05
0.00
0

1

2

3

4

5

|γ|

Drd for m = 40
0.30

| ρ | = 0.8
| ρ | = 0.6
| ρ | = 0.3

0.25
0.20
0.15
0.10
0.05
0.00
0

1

2

3

4

5

|γ|

Coverage Probability (1 − α − Dwm − Drd) for m = 40
0.95
0.90
0.85
0.80
0.75

| ρ | = 0.8
| ρ | = 0.6
| ρ | = 0.3

0.70
0.65
0

1

2

3

4

5

|γ|

Figure 1: This figure is for the confidence interval, with nominal coverage 0.95,
constructed after a preliminary t test with size 0.1, when n − p = 40 and |ρ| ∈
{0.3, 0.6, 0.8}. The top panel shows graphs of Dwm , the deficit in coverage due to
the choice of the wrong model, as a function of |γ|. The middle panel shows graphs
of Drd , the deficit in coverage due to the re-use of the data, as a function of |γ|. The
bottom panel shows graphs of the coverage probability of this confidence interval,
given by 1 − α − Dwm − Drd , as a function of |γ|.
4

coverage probability is 0.95 for all |γ|. This figure shows that, for size 0.1 of the
preliminary t test, n − p = 40 and |ρ| ∈ {0.6, 0.8}, Dwm , the deficit in coverage due
to the choice of the wrong model, dominates Drd , the deficit in coverage due to the
re-use of the data. An extensive numerical investigation of Dwm and Drd , covering a
wide range of values of n − p, ρ, 1 − α and size of the preliminary t test, is reported
in the Supplementary Material and summarised in Section 4.
2. The basic argument
Suppose that we observe a random n-vector of responses Y for a general regression model, with given values of the explanatory variables. We denote this “full
model” by M. Suppose that the parameter of interest is the scalar θ. Also suppose
that a given preliminary data-based model selection is used to select between the full
model and a specified set of submodels. Assuming that the selected model had been
given to us a priori as the true model, we construct the usual confidence interval
for θ, with nominal coverage 1 − α. Let K denote the resulting post-model-selection
confidence interval.
To identify the deficit in coverage probability due to the possible choice of the
wrong model and the deficit in coverage probability due to re-use of the data, we
consider the following hypothetical situation. Suppose that a genie provides an observation of Y ∗ the random n-vector of responses for a replicate (i.e. an independent
realization) of the original experiment that gave rise to Y , with the same values of
the explanatory variables. Also suppose that the same preliminary model selection
based on Y is used. Assuming that the selected model had been given to us a priori
as the true model, we construct the usual confidence interval for θ, with nominal
coverage 1 − α using the genie’s data Y ∗ . Let K ∗ denote the resulting confidence
interval.
Observe that K and K ∗ are influenced by exactly the same, possibly wrong,
choice of model based on Y . However, unlike K, the confidence interval K ∗ is not
influenced by the re-use of the data for its construction. Because of this difference
between K and K ∗ , we are able to separate out these two sources of poor coverage
as follows. The coverage probability of K is


 


P θ ∈ K = (1 − α) − (1 − α) − P θ ∈ K ∗ − P θ ∈ K ∗ − P θ ∈ K
= (1 − α) − Dwm − Drd ,

5


where Dwm = (1 − α) − P θ ∈ K ∗ is the deficit in coverage probability due solely


to possibly choosing the wrong model and Drd = P θ ∈ K ∗ − P θ ∈ K is the
deficit in coverage probability due solely to re-using the data.
3. The scenario of two nested linear regression models
Consider the multiple linear regression model
Y = Xβ + ε

(1)

where Y is a random n-vector of responses, X is a known n × p matrix with linearly
independent columns, β is an unknown parameter p-vector and ε ∼ N (0, σ 2 In ),
where σ 2 is an unknown positive parameter. Suppose that n > p. We refer to (1)
as the full model and denote it by M.
Suppose that the quantity of interest is θ = a> β, where a is a known p-vector
(a 6= 0). Also suppose that the inference of interest is a 1 − α confidence interval
for θ. Define the parameter τ = c> β − r, where c is a known p-vector, the number
r is specified and the vectors a and c are linearly independent. Let βb denote the
least squares estimator of β. Then θb = a> βb is the least squares estimator of θ
and τb = c> βb − r is the least squares estimator of τ . Let m = n − p and σ
b2 =

b > (Y − X β)
b m. Also let vθ = var(θ)/σ
b 2 , vτ = var(b
b τb)
(Y − X β)
τ )/σ 2 , ρ = corr(θ,

1/2
and γ = τ (σ vτ )
We consider preliminary model selection using a t test of the null hypothesis
H0 : τ = 0 against the alternative hypothesis HA : τ 6= 0. The test statistic is
τb

T =

1/2

.

σ
b vτ

Define the quantile tm,a by P (T ≤ tm,a ) = a for T ∼ tm . Suppose that we reject H0
if |T | ≥ tm, 1−eα/2 ; otherwise we accept H0 . This test has size α
e. Let Mτ =0 denote
the model M for τ = 0. If we reject H0 then we choose the model Mτ =0 ; otherwise
we choose the model M.
3.1 The confidence interval K
The usual 1 − α confidence interval for θ based on the full model M is

1/2 
I = θb ± tm, 1−α/2 vθ σ
b ,

6

where [a ± b] = [a − b, a + b] (b ≥ 0). Under the model Mτ =0 the usual 1 − α
confidence interval for θ is
"

1/2 #
2
2

m
σ
b
+
(b
τ
/v
)
1/2
τ
1/2
1/2
J = θb − ρ vθ (b
1 − ρ2
.
τ /vτ1/2 ) ± tm+1,1−α/2 vθ
m+1
Let K be the confidence interval, with nominal coverage 1 − α, for θ constructed
as follows. We use the data Y to choose the model. We then construct the confidence
interval for θ using the same data Y based on the assumption that this model had
been given to us a priori as the true model. Therefore K is the post-model-selection
confidence interval, with nominal coverage 1 − α, for θ. In other words,
(
I
if |T | ≥ tm,1−eα/2
K=
J if |T | < tm,1−eα/2 .
3.2 The confidence interval K ∗
Now suppose that a genie provides the following replicate (i.e. an independent
realization) of the original experiment that gave rise to Y , with the same values of
the explanatory variables:
Y ∗ = Xβ + ε∗ ,
where Y ∗ is a random n-vector of responses and ε∗ ∼ N (0, σ 2 In ). Note that ε and ε∗
are independent. Let βb∗ , θb∗ and τb∗ denote the least squares estimator of β, θ and τ ,
respectively, based on the genie’s data Y ∗ . Let (b
σ ∗ )2 = (Y ∗ −X βb∗ )> (Y ∗ −X βb∗ )/m.
The usual 1 − α confidence interval for θ based on M, and using only the genie’s
data Y ∗ , is

1/2 ∗ 
b .
I ∗ = θb∗ ± tm, 1−α/2 vθ σ
The usual 1 − α confidence interval for θ based on Mτ =0 , and using only the genie’s
data Y ∗ , is
"

1/2 #
∗ 2
∗ 2

m
(b
σ
)
+
((b
τ
)
/v
)
1/2
τ
1/2
1/2
1 − ρ2
J ∗ = θb∗ − ρ vθ (b
τ ∗ /vτ1/2 ) ± tm+1,1−α/2 vθ
.
m+1
Let K ∗ be the confidence interval, with nominal coverage 1−α, for θ constructed
as follows. We use the original data Y to choose the model i.e. we use the test
statistic T . We then construct the confidence interval for θ using the genie’s data
Y ∗ based on the assumption that this model had been given to us a priori as the
true model. In other words,
(
I∗
K∗ =
J∗

if |T | ≥ tm, 1−eα/2
if |T | < tm, 1−eα/2 .
7

3.3 Computationally convenient formulas for the deficits Dwm and Drd
By the law of total probability,
P (θ ∈ K) = P (θ ∈ J, |T | < tm, 1−eα/2 ) + P (θ ∈ I, |T | ≥ tm, 1−eα/2 )
= P (θ ∈ J, |T | < tm, 1−eα/2 ) + P (θ ∈ I) − P (θ ∈ I, |T | < tm, 1−eα/2 )
= (1 − α) + P (θ ∈ J, |T | < tm, 1−eα/2 ) − P (θ ∈ I, |T | < tm, 1−eα/2 ),
since P (θ ∈ I) = 1 − α. By the law of total probability, and the independence
between the original data Y and the genie’s data Y ∗ ,
P (θ ∈ K ∗ ) = P (θ ∈ J ∗ , |T | < tm, 1−eα/2 ) + P (θ ∈ I ∗ , |T | ≥ tm, 1−eα/2 )
= P (θ ∈ J ∗ ) P (|T | < tm, 1−eα/2 ) + P (θ ∈ I ∗ ) P (|T | ≥ tm, 1−eα/2 )
= P (θ ∈ J) P (|T | < tm, 1−eα/2 ) + P (θ ∈ I) 1 − P (|T | < tm, 1−eα/2 )




= P (θ ∈ I) + P (|T | < tm, 1−eα/2 ) P (θ ∈ J) − P (θ ∈ I)

= (1 − α) + P (|T | < tm, 1−eα/2 ) P (θ ∈ J) − (1 − α) .
Therefore, the deficit in coverage probability due solely to possibly choosing the
wrong model using the original data Y is, by definition,
Dwm = (1 − α) − P (θ ∈ K ∗ )

= P (|T | < tm, 1−eα/2 ) (1 − α) − P (θ ∈ J) ,
and the deficit in coverage probability due solely to re-using the original data Y is,
by definition,
Drd = P (θ ∈ K ∗ ) − P (θ ∈ K)


= P θ ∈ I, |T | < tm, 1−eα/2 − P θ ∈ J, |T | < tm, 1−eα/2
+ P |T | < tm, 1−eα/2





P θ ∈ J − (1 − α) .

Let W = σ
b/σ and note that W has the same distribution as (R/m)1/2 , where


1/2
1/2
R ∼ χ2m . Also let G = (θb − θ) (σ vθ ) and H = τb (σ vτ ). Note that W and
(G, H) are independent, with


G
H



  

0
1 ρ
∼N
,
.
γ
ρ 1
8

(2)

Thus T = H/W has a noncentral t distribution with degrees of freedom m and

noncentrality parameter γ. Therefore we can easily evaluate P |T | < tm, 1−eα/2 . We
now show how the other probabilities that appear in the expressions for Dwm and
Drd are readily evaluated. Let φ and Φ denote the N (0, 1) pdf and cdf, respectively.


Also let fW denote the pdf of W . Define Ψ(x, u) = Φ g2 (x, u) − Φ g1 (x, u) , where

g1 (x, u) = −tm+1,1−α/2 x

m + u2
m+1

1/2
+

ργ
(1 − ρ2 )1/2

and

g2 (x, u) = tm+1,1−α/2 x

m + u2
m+1

1/2
+

ργ
(1 − ρ2 )1/2

.

The following theorem gives computationally convenient formulas for the probabilities that we need to evaluate to find Dwm and Drd .
Theorem 1.
Z

∞

Z

∞

P (θ ∈ J) =

Ψ(w, h/w) φ(h − γ) dh fW (w) dw,
0

−∞


P θ ∈ J, |T | < tm, 1−eα/2 is equal to
Z

∞

Z

1

tm,1−eα/2
0


Ψ w, tm,1−eα/2 y φ(tm,1−eα/2 w y − γ) dy w fW (w) dw,

−1


and P θ ∈ I, |T | < tm, 1−eα/2 is equal to


G
H
P −tm, 1−α/2 ≤
≤ tm, 1−α/2 , −tm, 1−eα/2 ≤
≤ tm, 1−eα/2 .
W
W

This probability is readily evaluated using the fact that G/W, H/W has a bivariate
noncentral t distribution (Kshirsagar definition).
Theorem 1 is proved in the appendix. It follows from Theorem 1 that the two
deficits Dwm and Drd , and the coverage probability P (θ ∈ K) depend only on one
unknown parameter γ, and four known quantities 1−α, α
e, ρ and m. Using Theorem
1, it is simple to calculate Dwm , Drd and P (θ ∈ K). Our computer programs are
written in R. We use the pmvt function in the R package mvtnorm to evaluate the
bivariate noncentral t distribution (Kshirsagar definition).
The following theorem reduces the number of combinations of ρ and γ that we
need to consider in order to understand the properties of the Dwm , Drd and P (θ ∈ K)
functions.
9

Theorem 2. For any given ρ, the deficits Dwm , Drd and the coverage probability
P (θ ∈ K) are even functions of γ. For any given γ, the deficits Dwm , Drd and the
coverage probability P (θ ∈ K) are even functions of ρ.
Theorem 2 will be true if each of the probabilities appearing in the expressions
for Dwm and Drd are, for any given ρ, even functions of γ and, for any given γ,
even functions of ρ. Appendix C of Kabaila and Giri (2009) provides a proof that
the coverage probability P (θ ∈ K) is, for any given ρ, an even function of γ and,
for any given γ, an even function of ρ. It follows directly from this proof that
P (θ ∈ J, |T | < tm,1−eα/2 ) and P (θ ∈ I, |T | < tm,1−eα/2 ) are, for any given ρ, even
functions of γ and, for any given γ, even functions of ρ. Using the same arguments
as those given in Appendix C of Kabaila and Giri (2009), it is straightfoward to
show that P (θ ∈ J) and P (|T | < tm,1−eα/2 ) are, for any given ρ, even functions of γ
and, for any given γ, even functions of ρ.
4. Extensive numerical investigation of the deficits Dwm and Drd
In this section we summarise the extensive numerical investigation of Dwm , the
deficit in coverage due to the choice of the wrong model, and Drd , the deficit in coverage due to the re-use of the data, reported in the Supplementary Material. Primarily
our interest lies in the cases where the coverage probability of K, minimized over
γ, i.e. minγ P (θ ∈ K), is substantially below 1 − α. In the Supplementary Material we compute minγ P (θ ∈ K) for m ∈ {1, 2, 5, 10, 40, 100}, |ρ| ∈ {0, 0.3, 0.6, 0.8},
1 − α ∈ {0.9, 0.95, 0.98} and α
e ∈ {0.02, 0.05, 0.1}. We find that Dwm dominates Drd
when minγ P (θ ∈ K) is substantially below 1 − α. In other words, the main cause
for minγ P (θ ∈ K) being substantially below 1 − α is that the preliminary t test
sometimes chooses the wrong model.
5. Discussion
We have defined and shown how to assess Dwm , the deficit in coverage probability
due to the possible choice of the wrong model, and Drd , the deficit in coverage
probability due to re-use of the data. The numerical results presented in Figure 1 and
the Supplementary Material show that the main cause for the coverage probability,
minimized over γ, being substantially below nominal is that the preliminary t test
sometimes chooses the wrong model. This suggests that data-splitting will not help
improve the coverage probability, but that some form of frequentist model averaging
10

will help. The model averaged tail area confidence interval of Turek and Fletcher
(2012) is a promising method (Kabaila, Welsh and Abeysekera, 2016, and Kabaila,
Welsh and Mainzer, 2016 and Kabaila, 2017).
Acknowledgement
This work was supported by an Australian Government Research Training Program Scholarship.
Appendix: Proof of Theorem 1
Let
Z=

G − ρ(H − γ)
.
(1 − ρ2 )1/2

It follows from (2) that Z, H and W are independent random variables and that
Z ∼ N (0, 1).

The event {θ ∈ J} = g1 (W, H/W ) ≤ Z ≤ g2 (W, H/W ) . Thus


P (θ ∈ J) = P g1 (W, H/W ) ≤ Z ≤ g2 (W, H/W )
Z

∞

Z

∞

Z

∞

=
−∞

0

Z



P g1 (w, h/w) ≤ Z ≤ g2 (w, h/w) φ(h − γ) dh fW (w) dw

∞


Ψ w, h/w φ(h − γ) dh fW (w) dw.

=
−∞

0



|T | < tm, 1−eα/2 =

P θ ∈ J, |T | < tm, 1−eα/2 is equal to
The event



− tm, 1−eα/2 W ≤ H ≤ tm, 1−eα/2 W . Hence



P g1 (W, H/W ) ≤ Z ≤ g2 (W, H/W ), −tm, 1−eα/2 W ≤ H ≤ tm, 1−eα/2 W
Z
=
0

∞

Z

tm,1−α/2
w
e




P g1 (w, h/w) ≤ Z ≤ g2 (w, h/w) φ h − γ dh fW (w) dw.

−tm,1−α/2
w
e


Changing the variable of integration of the inner integral to y = h (tm,1−eα/2 w), this
expression becomes
Z ∞Z
tm, 1−eα/2
0

1



Ψ w, tm, 1−eα/2 y φ tm, 1−eα/2 w y − γ dy w fW (w) dw.

−1

The event {θ ∈ I} =



− tm, 1−eα/2 W ≤ G ≤ tm, 1−eα/2 W . Therefore P (θ ∈

I, |T | < tm, 1−eα/2 ) is equal to


H
G
P −tm, 1−α/2 ≤
≤ tm, 1−α/2 , −tm, 1−eα/2 ≤
≤ tm, 1−eα/2 .
W
W
11

References
Buckland, S.T., Burnham, K.P. and Augustin, N.H., 1997. Model selection: an
integral part of inference. Biometrics, 53, 603–618.
Chatfield, D., 1995. Model uncertainty, data mining and statistical inference. Journal of the Royal Statistical Society, Series A, 158, 419–466.
Efron, B., 2014. Estimation and accuracy after model selection. Journal of the
American Statistical Association, 109, 991–1022.
Fletcher, D. and Turek, D., (2011). Model-averaged profile likelihood confidence
intervals. Journal of Agricultural, Biological and Environmental Statistics, 17,
38–51.
Hjort, N.L. and Claeskens, G., 2003. Frequentist model average estimators. Journal
of the American Statistical Association, 98, 879–899.
Hurvich, C. M. and Tsai, C. L., 1990. The impact of model selection on inference
in linear regression. The American Statistician, 44, 214–217.
Kabaila, P., 2009. The coverage properties of confidence regions after model selection. International Statistical Review, 77, 405 – 414.
Kabaila, P., 2017. On the minimum coverage probability of model averaged tail area
confidence intervals. Accepted for publication in Canadian Journal of Statistics.
Kabaila, P. and Giri, K., 2009. Upper bounds on the minimum coverage probability
of confidence intervals in regression after model selection. Australian & New
Zealand Journal of Statistics, 51, 271–287.
Kabaila, P., Welsh, A.H. and Abeysekera, W. 2016. Model-averaged confidence
intervals. Scandinavian Journal of Statistics, 43, 35–48.
Kabaila, P., Welsh, A.H. and Mainzer, R., 2016. The performance of model averaged tail area confidence intervals. Communications in Statistics - Theory and
Methods, DOI: 10.1080/03610926.2016.1242741
Leeb, H. and Pötscher, B., 2005. Model selection and inference: facts and fiction.
Econometric Theory 21, 21–59.
Miller, A., 2002. Subset Selection in Regression, 2nd ed. Chapman & Hall/CRC,
Boca Raton FL.
Sheather, S. J., 2009. A Modern Approach to Regression with R. Springer, New
York.
Turek, D. and Fletcher, D., 2012. Model-averaged Wald intervals. Computational
Statistics and Data Analysis, 56, 2809–2815.
12

