On Hodges’ Superefficiency and Merits of Oracle
Property in Model Selection∗
Xianyi Wu and Xian Zhou
East China Normal University, Shanghai, P.R. China
Macquarie University, Sydney, Australia.

Abstract
The oracle property of model selection procedures has attracted a large volume of favorable
publications in the literature, but also faced criticisms of being ineffective and misleading in
applications. In this paper, we introduce a class of estimators that can easily produce model
selection procedures possessing the oracle property and discuss the merits of the oracle property
by analyzing the performance of such estimators in finite sample size theoretically.
Specifically, we propose a new type of Hodges’ estimators capable of reducing the asymptotic
variance of any given estimator over a multi-dimensional subspace of the parameter space, which
can easily produce model selection procedures with the oracle and some other desired properties.
This new type of oracle estimators, however, perform poorly at some values of the parameters for
estimation, and there is no convincing reason to declare that oracle estimators are better than
the traditional estimators such as the MLE and LSE. Consequently, the merits of the oracle
property for model selection as claimed in the literature are probably grossly overstated, and the
criticisms of the oracle property are justifiable.

∗

This project is supported by NSFC under grant No. 71371074

1

1

Introduction

Basically, two philosiphies dominate the statistical data analysis on practical problems. One is to
consider a procedure of statistical inference as comprised of two steps, reflecting utilization of two
sources of information:
• The first step uses prior knowledge regarding the background and the mechanism of the prob-

lem to formulate it as a population with certain unknown components, in any of parametric,
nonparametric or semiparametric forms.

• The second step draws inference on certain assertions involving unknown components of the
population based on the information from collected data.

On the contrary, the second philosophy establishes models based on the information from the data
and then make inference according to both the data and the established models – referred to as the
“data-driven” approach. Remarkable efforts have been contributed to a type of statistical activities
known as modelling or model building, in which the techniques of model/variable selections stand
out. To motivate model selection, a popular statement is: when modeling data, a researcher has a
menu of a priori specified competing candidate models available to describe the data (cf. Leeb &
Pötscher, 2008a). In regression analysis, the usual motivation for variable selection is the suspicion
that some regressors are superfluous for explaining the dependent variable in the sense that the true
value of the coefficients of these regressors are zero, but the researcher does not know which of the
regressors are superfluous.
With the former philosophy, although statisticians are usually involved, the first step seems
little to do with statistical inference since the formulated model (population) reflects only people’s
knowledge on the background and mechanisms of the problem. For the data-driven procedure,
statistical inferences exist in the whole process of data analysis.
Even if model selection may have not taken a central role of statistical research activities in the
past decades, it is ubiquitous partly due to increasing computational power. Almost all textbooks
on linear regressions (including those at undergraduate level) may contain a separate chapter on,
or contribute some sections to, variable selections. A large volume of literature on model selection
have been published, and widely used statistical software packages contain more or less routines for
model selection.
Arguably, model selection are generally carried out with two goals:
• To select a model that is the best to describe the data and the model selection itself is the
final goal of the statistical activity.

• To act only as an intermediate step. The analyst then performs further statistical analyses
2

based on the selected “good” model, pretending it is the “true”. This amounts to a two-step
statistical inference (Berk et. al. 2010, 2013).
The latter routine, referred frequently to as data-driven model selection in the literature, has been
followed by many researchers, see e.g., Berk et al. (2010) for an exemplified list of such research
works. To be specific, let M denote the collection of competing candidate models and M̂ ∈ M the

outcome of the data-driven model selection (usually stochastically, depending again on the same
data). Let P̂M be the estimate of the true distribution P of data based on model M ∈ M. Then this
P
two-step estimate of P , denoted by P̃ , has a compound nature: P̃ = P̂M̂ = M ∈M P̂M I(M̂ = M ),

where I(A) denotes the indicator of an event A. Estimators/inference resulted from such a two-step

procedure are referred to as “post-selection estimators/inference”. Even for those model selection
procedures declared as a one-step inference, i.e., estimating the unselected parameters by zero and
others by nonzero statistics, such as the contemporarily proposed LASSO and SCAD, the estimate
can have similar features of the post-selection estimation.
Data analysts, however, rarely terminate their work after selecting “good” models and then
benefit from them without any futher activities. In fact almost all activities in statistics (especially
in real-data analysis) involving model selection follow the two-step procedures. Many textbooks
on linear regression oriented in practical data analysis taught students how to use well-known
techniques such as AIC (Akaike’s information criterion, Akaike, 1973), BIC (Bayes information
criterion, Schwartz, 1978), Adjusted R-squares, Mallows’ Cp (Mallows, 1965) and so on to select
“good” models and then perform further data analyses based on the selected models, pretending
they are the true ones. Typical softwares, such as SAS and R, generally select a model by some
criterion (such as those just mentioned) and then output the estimates of the coefficients of the
selected explanatory variables as well as their corresponding p-values computed by pretending that
the selected models are true.
In general situations, one has an overall model and the candidate models are obtained by setting
some parameters to zero. Thus selecting a model corresponds to estimating some parameters by
zero and then use the selected model for further statistical analyses. More recently, based on the
idea of penalized maximum likelihood/least squares inherited from AIC and BIC, a few researchers
found a way to integrate this two-step analysis into a single-step procedure: simply associate some
cleverly designed penalties to the likelihood function (or squared errors in regression models) to
estimate some parameters by zero and then select the models with nonzero estimated parameters.
This method began with the famous LASSO algorithm (Least Absolute Selection and Shrinkage
Operator) proposed by Tibshirani (1996), the first estimation procedure that can perform variable
selection at the same time. To be specific, let l(Y, θ) denote the log-likelihood of the parameters
θ = (θ1 , θ2 , . . . , θp)0 . Then the estimator of θ is obtained by maximizing the penalized log-likelihood

3

(PL)
P L(θ) = l(Y, θ) +

p
X

fi (θi , λn ),

i=1

where Y is the sample, fi (θi , λn) the penalty associated with parameter θi , n the sample size and λn
a known tuning parameter. In an ordinary linear regression model Y = Xn×p θ + ε, the estimators
are obtained by minimizing penalized squares
0

P S(θ) = (Y − Xθ) (Y − Xθ) + 2

p
X

fi (θi , λn).

i=1

For particularly assigned penalty function fi , maximizing P L(θ) produces such estimators that
automatically estimates some θi by zero, so that one can simultaneously select model and estimate the parameters of the selected model. Different designs of the penalties generate different
selection-estimation algorithms, such as LASSO (Tibshirani, 1996), adaptive LASSO (Zou, 2006),
hard thresholding estimators, soft thresholding estimators, Bridge regression estimators (Frank and
Friedman, 1993), SCAD (smoothly clipped absolute deviation penalty, Fan and Li, 2001), Elastic
net method (Zou and Hastie, 2005) and MCP (minimax concave penalty, Zhang, 2010). Unlike
AIC that may be supported by certain theoretical derivations (Akaike, 1973), the other penalized
maximum likelihood/least squares were proposed without any theoretical motivation.
The first theoretical justification of the penalized maximum likelihood method is provided by
Fan and Li (2001), who proved the following appealing property of their SCAD estimators, referred
to as the oracle property. To introduce it, let
b(θ) = {i : θi 6= 0, i = 1, 2, . . ., p} and b̄(θ) = {1, 2, . . ., p} − b(θ).

(1.1)

0
0
Rearrange the parameter vector as θ = (θb(θ)
, θb̄(θ)
) without loss of generality.

Definition 1.1 An estimator θ̂n is said to possess the oracle property if
(1) lim Prθ (θ̂n,b̄(θ) = 0) = 1; and
n→∞

(2)

√

d

d

−1
n(θ̂n,b(θ) − θb(θ) ) → N (0, Fb(θ)
), where → indicates convergence in distribution and Fb(θ) is

the Fisher’s information matrix of θb(θ) knowing θb̄(θ) = 0.

We will refer to an estimator possessing the oracle property as an oracle estimator. The oracle property defined above is described in limit sense when the sample size tends to infinity. It states that an
oracle estimator is asymptotically equivalent to the ideal estimator with the active parameters only.
This gives a justification to the use of the single-step procedure like SCAD estimation. After Fan
and Li (2001), many researchers on model selection focused on finding model selection procedures
producing oracle estimators and performing better in finite sample size. Significant works in this line
4

of research include adaptive LASSO, elastic net method and MCP. What is worth special mention
is the explosion of the literature in model selection under a huge number of statistical models that
aimed at finding procedures producing oracle estimators.
Such a popular property, however, is not universally accepted without criticisms. The Vienna
school, led by H. Leeb and B. M. Pötscher, argued that the oracle property is “too good to be true”
and seriously questioned the advertised merits of oracle model selection procedures in a series of
papers, including Leeb and Pötscher (2008a, 2008b), Pötscher, (2009), Pötscher and Leeb (2009),
Pötscher and Schneider (2009) etc. and the references therein. They considered oracle estimators
as a return of Hodges’ estimators (Le Cam, 1953), which provided a typical counterexample to
a conjecture by R. A. Fisher that the MLEs have minimum asymptotic variance (referred to as
asymptotic efficient), and are known to perform poorly at some parameter values in finite sample
size. Their arguments are summarized below.
(1) Theoretically, the overall risk of a sparse estimator can be unbounded as n → ∞.
(2) Numerous Monte Carlo simulations were conducted to demonstrate that, in finite sample size,
a number of oracle estimators perform quite poorly when the parameters take values in a
vicinity of zero.
In particular, by replicating and extending Monte Carlo simulations of the performance of the SCAD
estimator in Example 4.1 of Fan and Li (2001), Leeb and Pötscher (2008b) demonstrated that this
estimator, when tuned to enjoy sparsity and oracle properties, can perform poorly in finite samples.
Even if not tuned to sparsity, the SCAD estimator perform worse than the least squares estimator
in parts of the parameter space. It is interesting to note that the simulation study in Fan and Li
(2001) was conducted only at some points that avoided the parts of the parameter space examined
by Leeb and Pötscher (2008b). This highlights the fact that simulations are not reliable to judge
the goodness of an estimator, which is generally recognized in the statistical community but often
overlooked. Consequently, Leeb and Pötscher (2008b) argued that the oracle property is highly
misleading and cannot be relied on to justify an estimator.
This controversy about the merits of the oracle property is highly significant in statistical theory
and methodologies, as it goes to the heart of such vital questions as: Are oracle estimators and
model selection procedures really useful in data analyses? Is the huge literature on finding oracle
estimators and procedures worth so many efforts? And is this modern research direction along the
line of the oracle property on the right track?
In this paper, we attempt to address this controversy by revisiting the issue of Hodges’ superefficiency and theoretically analyzing the performance of oracle estimators in a class of generalized
Hodges’ estimators without relying on numerical simulations. As simulations are subject to large
5

variations and may produce different results in either way of a controversy, they are unreliable
in resolving the controversy. The theoretical analysis we pursue can avoid such drawbacks of the
simulation approach and thus provide more convincing conclusions.
Our arguments proceed as follows.
(1) Generally, asymptotic bias and variance do not reflect their versions in finite sample size.
Thus the asymptotic superefficiency and the oracle property do not necessarily lead to good
performance of the estimators in any fixed sample size.
(2) We introduce a new type of Hodges’ estimators, referred to as oracle Hodges’ estimators, which
can reduce the asymptotic variance of any given estimator over certain multi-dimensional
subsets of the parameter space and provide an easy way to obtain oracle model selection
procedures.
(3) By analyzing the performance of oracle Hodges’ estimators in finite sample size, we theoretically demonstrate that such estimators perform poorly at some values of the parameters
despite possessing the oracle property.
(4) The performance of oracle Hodges’ estimators indicates that the oracle property does not
justify the use of the oracle model selection procedures, and the merits of the oracle property
as claimed in the literature are doubtful and questionable. In particular, there is no convincing
reason that an oracle model selection procedure produces better estimates than the traditional
MLE/LSE methods in finite samples.
(5) Our results provide theoretical justification to confirm the view of Leeb and Pötscher (2008b)
against the oracle model selection approach. In addition, we further clarified that the oracle
property is not a simple return of classical Hodges’ superefficiency as thought of by Leeb and
Pötscher in a series of papers. Instead, the newly defined oracle Hodges’ estimators, rather
than their classical versions, have the same asymptotic behavior as oracle model selection
procedures.
To conclude this introduction section, we would like to point out that the classical Hodges’
estimators are a special type of preliminary-test estimators. It appears, however, that the finite
sample behaviors of preliminary-test estimators have not been adequately addressed in the literature
either, despite the importance of this problem in statistics. Some exceptions can be found in, e.g.,
the book by Judge and Bock (1978) and the review paper of Giles and Giles (1993).
In Section 2 next, the classical and the new oracle versions of Hodges’ estimators are defined and
their asymptotic behaviors (superefficiency) are discussed. Section 3 introduces the model selection
methods derived from Hodges’ estimators and discusses their connections to and differences from
6

penalized maximum likelihood estimators or penalized least squares estimators. The performance
of oracle Hodges’ estimators is theoretically analyzed in Section 4 after a brief discussion on the
connection of asymptotic biases and variances to their finite sample size versions. Some concluding
remarks are discussed in Section 5. The proofs are collected in the Appendix.

2

Asymptotic efficiency and Hodges’ estimators

The concept of asymptotic efficiency was introduced by Sir R. A. Fisher to justify the goodness
of MLEs and has great impacts on statistical inference in large samples. For a d-dimensional
parameter θ = (θ1 , θ2 , . . ., θd )0 , let θ̂n = (θ̂n1 , θ̂n2 , . . . , θ̂nd )0 be any sequence of its estimators such
d

that rn (θ̂n − θ) → X for some sequence of scalars rn → ∞ and a d-dimensional random vector X

with mean zero and variance (covariance matrix) V −1 for some matrix V = (vij )d×d . A sequence
√
d
of estimators {θ̂n } of θ such that n(θ̂n − θ) → N (0, Σ(θ)), whatever is the true value of θ, is

said to be asymptotically efficient (or asymptotically optimal in some literature) if Σ(θ) = F−1 (θ),

where F(θ) is the Fisher’s information matrix of the distribution. Namely, θ̂n is efficient if it is
asymptotically unbiased and “optimal” (in a certain sense based on the asymptotic variance) in the
class of asymptotically unbiased estimators with order n−1/2 .
The general results state that, under certain regularity conditions, a sequence of roots of likelihood equations is asymptotically efficient. There have been a huge number of research efforts
aimed at seeking asymptotically efficient estimators. The idea of measuring efficiency by asymptotic variance, however, appears not as successful as its counterpart in finite sample size, and the
lower bounds defined by F−1 (θ) are not sufficiently low in the class of asymptotically unbiased estimators. The example in Subsection 2.1 below, whose prototype was made widely known by Le
Cam (1953) under the name Hodges’ estimator, shows that, given any sequence of estimators θ̂n
d

with rn (θ̂n − θ) → X, one can construct another estimator sequence with asymptotic variance no

more than that of θ̂n at any value of θ, and strictly less at certain values of θ. This phenomenon is

referred to as (asymptotic) superefficiency. In spirit of this idea, we introduce variants of Hodges’
estimators in Subsection 2.2, which possess the oracle property (Definition 1.1) and can easily serve
the purpose of finding
oracle model selection procedures.
q
Pd
d
2
Let kxk =
i=1 xi denote the Euclidean norm of x = (x1 , x2 , . . . , xd ) ∈ R , d(x, A) =

inf y∈A kx − yk the distance between a point x and a subset A and d(A, B) = inf x∈A,y∈B kx − yk the

distance between two subsets of Rd under the Euclidean norm.

2.1

Classical Hodges’ example of superefficiency
d

Let θ̂n be any sequence of estimators such that rn (θ̂n − θ) → X for a sequence of deterministic

scalars rn → ∞, where the distribution of X may depend on θ, whatever is the true value of θ.
7

Definition 2.1 (Hodges’ estimator) Let {an } be a sequence of scalars and c any fixed point in
the parameter space Θ. The Hodges’ estimator of θ is defined by

θ̆n (c) = cI(kθ̂n − ck ≤ an ) + θ̂n I(kθ̂n − ck > an ).
This estimator was initiated by Jr. Hodges with rn =

√

(2.1)

n, c = 0 and an = n−1/4 for one-dimensional

θ (reported by Le Cam, 1953; see also Lehmann and Casella, 1998, p. 420, Example 2.5) and has
been revisited many times by, e.g., Leeb and Pötscher (2005, 2008a, b) when they discussed the
implications of consistent model selections. While (2.1) may be slightly generalized to
θ̆n (c) = ((1 − α)c + αθ̂n )I(|θ̂n − c| ≤ an ) + θ̂n I(|θ̂n − c| > an ),
where α ∈ [0, 1] (cf., e.g., van der Vaart, 1998 for the case with rn =

√

n and c = 0), we here take

α = 0 as this is sufficient for our purpose. The asymptotic distribution of θ̆n (c) in Definition 2.1 is
given in the following theorem.
Theorem 2.1 For any sequence {an } such that an = o(1) and rn an → ∞ as n → ∞, we have
d

rn (θ̆n (c) − θ) → XI(θ 6= c).
It is clear that, in terms of asymptotic variances, θ̆n (c) is no worse than θ̂n at any θ and strictly
better than θ̂n at θ = c because the asymptotic variance of θ̆n (c) is zero at θ = c. This example
revealed an interesting phenomenon that, in terms of the asymptotic variance, any estimate can be
improved at any arbitrarily prefixed point in the parameter space.

2.2

Oracle Hodges’ estimators

We next introduce a few closely linked variants of Hodges’ estimators, which differ from the classical
version in Definition 2.1, but we have kept Hodges’ name for the new estimators because they
retain the feature of superefficiency. We will refer to the new type of Hodges’ estimators as oracle
Hodges’ estimators because they possess the oracle property in Definition 1.1, as will be shown via
Theorems 2.2 to 2.5 below. The definition of oracle Hodges’ estimators is presented in four versions:
general version, continuous/smoothing version, MLE version and LSE version, where the first one
is fundamental, the second is a refinement, and the other two are special cases with origins θ̂n being
MLE and LSE, respectively.
1. General version.
For any subset b ⊂ {1, 2, . . ., d} and its complement b̄ = {1, 2, . . ., d} − b, rearrange the components
d

0 , θ̂0 ), respectively, such that r (θ̂ − θ) → X, where the
of θ and θ̂n as θ0 = (θb0 , θb̄0 ) and θ̂n0 = (θ̂n,b
n n
n,b̄

8

current X = (Xb0 , Xb̄0 )0 is also a rearrangement of the original X in the same way as θ. Clearly, the
mean of X is still a zero vector but the covariance matrix changes to V −1 with

 

Vbb Vbb̄
(vij )i,j∈b (vij )i∈b,j∈b̄
=
.
V =
Vb̄b Vb̄b̄
(vij )i∈b̄,j∈b (vij )i,j∈b̄
It is also easy to see that, if b 6= ∅, then the marginal vector θ̂n,b has an asymptotic distribution
d

given by rn (θ̂n,b − θb ) → Xb with mean zero and covariance matrix
∆b = (Vbb − Vbb̄ Vb̄−1
Vb̄b )−1 .
b̄

(2.2)

Note that the distribution of X, and hence the variance matrix V −1 , may be related to parameters
θ. The only requirement is that V −1 is a continuous function of θ. We use V̂ −1 to denote any of
consistent estimators of V −1 , e.g., obtained by substituting θ̂n for θ, so that the symbols V̂bb, V̂bb̄
and so on are self-explained.
Let c = (c1 , c2 , . . . , cd)0 be a known d-vector. For every nonempty and proper subset b of
{1, 2, . . ., d} (i.e., ∅ =
6 b 6= {1, 2, . . ., d}), denote
θ̌n,b = θ̂n,b + V̂bb−1 V̂bb̄ (θ̂n,b̄ − cb̄) and

0
θ̌n (b) = (θ̌n,b
, c0b̄)0

(2.3)

with the convention θ̌n,{1,2,...,d} = θ̌n ({1, 2, . . ., d}) = θ̂n . Moreover, analogy to (1.1), we redefine
b(θ) and b̄(θ) by
b(θ) = {j : j ∈ {1, 2, . . ., d}, θj 6= cj } and b̄(θ) = {1, 2, . . ., d} − b(θ).

(2.4)

The following definition introduces a sequence θ̃n (c) of oracle Hodges’ estimators in multi-dimensional
case derived from θ̂n .
Definition 2.2 (Oracle Hodges’ estimators) Let (anj ) = (an1 , . . . , and ), n = 1, 2, . . ., denote a
sequence of d-vectors with positive components. For every n = 1, 2, . . ., define two complementary
random sets by
bn (c) = {j : |θ̂nj − cj | > anj } and b̄n (c) = {j : |θ̂nj − cj | ≤ anj }
and the corresponding oracle Hodges’ estimator by


if bn (c) = {1, 2, . . ., d},

 θ̂n
0
θ̃n (c) = θ̌n (bn (c)) =
, c0b̄ (c) )0 if bn (c) 6= ∅, bn(c) 6= {1, 2, . . ., d},
(θ̌n,b
n (c)
n



c
if bn (c) = ∅,

(2.5)

(2.6)

where θ̌n (bn (c)) and θ̌n,bn (c) are obtained from the two equations in (2.3) by substituting bn (c) for

b, and bn (c) is defined in (2.5).
9

For later reference, denote

 V −1 (V
bb
bb
X̌b =
 0,

Vbb̄ )X,

if b 6= ∅,

(2.7)

if b = ∅.

The asymptotic properties of θ̃n (c) defined in (2.6) are provided in the next theorem.
Theorem 2.2 If the sequence of d-vectors {(anj )} satisfies
max anj → 0

rn min anj → ∞ as n → ∞,

and

1≤j≤d

(2.8)

1≤j≤d

then for any b ⊂ {1, 2, . . ., d}, θb̄ = cb̄ implies lim Pr(θ̃n,b̄ (c) = θb̄ ) → 1 and
n→∞

d



rn (θ̃n (c) − θ) → 



X̌b(θ)

 under Prθ ,

0

(2.9)

whatever is the true value of θ, where X̌b(θ) is defined as in (2.7) with b replaced by b(θ) in (2.4).
2. Continuous/smoothing version.
As a function of θ̂n , the estimator θ̃n (c) is not continuous at any point θ̂n = (θ̂n1 , θ̂n2 , . . . , θ̂nd ) such
that |θ̂nj −cj | = anj for some j ∈ {1, 2, . . ., d}. Some authors think of the continuity as an important
property (see, e.g., Fan and Li, 2001) but others may disagree. If preferred, a continuous version of
θ̃n (c) can be easily achieved by the following procedure.
(1)

(2)

Let (anj ) and (anj ) be two sequences of d-vectors both satisfying conditions (2.8) on (anj ) and
(1)

(2)

(1)

−1/2

anj < anj , j = 1, 2, . . ., d. A possible choice is anj = rn
two sequences of oracle Hodges’ estimators

(1)
θ̃n (c)

and

(2)

−1/2

and anj = 2rn

(2)
θ̃n (c)

by

(1)
(anj )

, j = 1, 2, . . . , d. Define
(2)

and (anj ) respectively as in

Definition 2.2. Let f1 (x), . . ., fd (x) be any d continuous and increasing functions on x ∈ R+ such
(1)

(2)

(2)

that fj (cnj ± anj ) = cnj and fj (cnj ± anj ) = cnj ± anj . Define



 cnj
θ̃nj (c; f ) =
f (θ̂nj )


 (2)
θ̃nj (c)

(1)

if |θ̂nj − cj | ≤ anj ,
(1)

(2)

if anj ≤ |θnj − cj | ≤ anj ,

(2.10)

otherwise.

Then θ̃nj (c; f ) is a continuous version of θ̃n (c) such that
(2)

(1)

|θ̃nj (c) − cnj | ≤ |θ̃nj (c; f ) − cnj | < |θ̃nj (c) − cnj |.
These inequalities ensure the following result.
Theorem 2.3 The estimators θ̃n (c; f ) defined in (2.10) have the same asymptotic properties of
(i)

θ̃n (c), i = 1, 2, as presented in Theorem 2.2.
10

3. MLE version.
We next discuss the maximum likelihood estimation with the simplest i.i.d. case as an example. It
is not difficult to extend the results to general situations.
The log-likelihood function of θ from i.i.d. X1 , . . . , Xn with a common density f (x; θ) is
l(θ) = l(θ; X1, . . . , Xn) =

n
Y

log f (Xi ; θ).

i=1

It is well known that under certain regularity conditions, there exists a sequence of asymptotically
√
d
efficient MLE θ̂n , i.e., n(θ̂ − θ) → N (0, F−1(θ)), where
 2

∂
log f (X1 ; θ)
F(θ) = −Eθ
∂θ∂θ0
is the Fisher’s information matrix. For any b ⊂ {1, 2, . . ., d} and the corresponding rearrangement

of θ = (θb , θb̄ ), F(θ) can be rewritten as
h 2
i
h 2
i  


f (X1;θ)
∂ log f (X1 ;θ)
Eθ ∂ log
E
0
0
θ
F 0 (θ) Fbb̄0 (θ)
,
h 2 ∂θb ∂θb i
h 2 ∂θb ∂θb̄ i  =  bb
F(θ) = − 
∂ log f (X1;θ)
∂ log f (X1 ;θ)
Eθ
E
F
(θ)
F
(θ)
0
0
θ
b̄b
b̄b̄
∂θ ∂θ0
∂θ ∂θ0
b̄

b̄

b

say.

b̄

For any constant vector c = (c1 , . . ., cd ), Theorem 2.2 yields the following immediate results.
Theorem 2.4 If {θ̂n } is an efficient sequence of maximum likelihood estimators, then the oracle

Hodges’ estimators θ̃(c) in Definition 2.2 have the following properties: For any b ⊂ {1, 2, . . ., d},
θb̄ = cb̄ implies lim Pr(θ̃n,b̄ (c) = θb̄ ) = 1 and
n→∞

√

d

 

n(θ̃(c) − θ) → N 0, 

F−1
b(θ)b0 (θ) (θ) 0
0

0



 .

Clearly, Fbb0 (θ) is the Fisher’s information matrix for parameter θb depending on the unknown
√
θb̄ . The asymptotic variance of n(θ̂n,b − θn,b ) is
(F−1 (θ))bb0 = (Fbb0 (θ) − Fbb̄0 (θ)F−1
(θ)Fb̄b0 (θ))−1 ≥ F−1
bb0 (θ)
b̄b̄0

(2.11)

with strict inequality if b 6= {1, 2, . . ., d}.

Note that Fb(θ)b0 (θ) is the Fisher’s information matrix of the marginal vector θb(θ) knowing that

θb̄(θ) = cb̄(θ) . Taking b = b(θ), Theorem 2.4 shows the superefficiency of θ̃n (c) over the MLE θ̂n at
any θ in the parameter space such that b(θ) 6= {1, 2, . . ., d} (i.e., θj = cj for some j ∈ {1, 2, . . ., d}).
This result covers such parametric models as linear regression with normally distributed errors and
generalized linear regression.

11

4. LSE version.
For a linear regression model Y = Xβ + ε with E[ε] = 0 and Var(ε) = σ 2 In , we can generate the
oracle Hodges’ estimator β̃(c) from the least square estimate β̂LS = (X0 X)−1 X0 Y. Under certain
regularity conditions (see e.g., van de Vaart, 2000, Example 2.28), β̂LS is asymptotically distributed
√
d
−1 0
as n(β̂LS − β) → N (0, σ 2Σ−1
X ), where ΣX = lim n X X. By Theorem 2.2 again, the following
n→∞

result is obvious.

Theorem 2.5 Given any fixed d-vector c and subset b ⊂ {1, 2, . . ., d}, βb̄ = cb̄ implies that lim Pr(β̃n,b̄ (c) =
n→∞

βb̄ ) = 1 and
√

d

n(β̃(c) − β) → N 0, 

This shows that
o
β̂b(β)

 

=

√

σ 2 Σ−1
0
b(β)b(β)
0



 , where Σb(β)b(β) = lim 1 X0 Xb(β).
n→∞ n b(β)
0

(2.12)

n(β̃b(β)(c) − βb(β)) has the same asymptotic distribution as the oracle estimator

(X0b(β)Xb(β))−1 X0b(β)(Y

0
− Xb̄(β)cb̄(β)) if the true value of β is (βb(β)
, c0b̄(β))0 .

We conclude this section with the following two remarks.
Remark 2.1 By Theorems 2.1, the classical version θ̆n (c) has an asymptotic distribution given by
d

rn (θ̆n (c) − θ) → XI(θ 6= c). That is, θ̆n (c) can only improve the asymptotic variance of θ̂n at θ = c

in the parameter space Θ, which is much more restrictive than the improvement achieved by the
limit in (2.9) for θ̃n (c) defined by (2.6). To see this, note that the variance of X̌b(θ) is


Vb(θ),b(θ)
−1
−1
 V −1
(Vb(θ),b(θ) Vb(θ),b̄(θ) )V −1 
= Vb(θ),b(θ)
≤ ∆b(θ) ,
Ṽb(θ) = Vb(θ),b(θ)
b(θ),b(θ)
Vb(θ),b̄(θ)

where ∆b(θ) is the asymptotic variance of rn (θ̂n,b(θ) − θb(θ) ) by (2.2), and the equality holds only
when θ̂n,b(θ) and θ̂n,b̄(θ) are asymptotically independent. Therefore, θ̃n (c) can improve the asymptotic

variance of θ̂n at any θ with b̄(θ) 6= ∅. Note also that rn (θ̃n (c) − θ) → 0 in probability at θ = c.

In terms of asymptotic variances, θ̆(c) improves θ̂n and θ̃n (c) further improves θ̆(c). A further
important feature of θ̃n (c) is its ability to produce oracle model selection procedures due to its form

of asymptotic variance, as we will show in the next subsection. In contrast, neither θ̆n (c) nor θ̂n has
such a capacity. These together highlight the significant differences between the new oracle Hodges’
estimator θ̃n (c) and the classical version θ̆(c).
Remark 2.2 Note that MLE and LSE and their versions of oracle Hodges’ estimators are of root-n
consistency under relevant regularity conditions. But this is not necessary for Definition 2.2. The
general version of θ̃n (c) and its continuous version do not require those regularity conditions and
they are not necessarily of root-n consistency. For example, let Yi = (Yi1 , . . . , Yid )0 , i = 1, 2, . . .,
Q
be independent with identical uniform distributions over dk=1 [−θk , θk ], θk > 0, so that the MLE
12

of θ = (θ1 , . . ., θd ) is θ̂n = (θ̂n1 , . . . , θ̂nd ) with θ̂nk = max(|Y1k |, . . ., |Ynk |), k = 1, 2, . . ., d. Then
d

n(θ̂n − θ) → X = (X1 , . . . , Xd) with mutually independent components X1 , . . ., Xd such that
Prθ (Xk ≤ xk ) =

(

exk /θk

if xk < 0

1

if xk ≥ 0

,

k = 1, 2, . . ., d.

In this case, the general version θ̃n (c) in (2.6) and its continuous version θ̃n (c, f ) in (2.10) are still
valid even though the regularity conditions of the likelihood function are not satisfied, but they are
not of root-n consistency.

3

Model selection function and oracle property of θ̃n (c)

Definition 2.2 and Theorem 2.2 clearly indicate the following properties of θ̃n (c):
(1) θ̃n (c) is a sparse estimate in the sense that some components of θ, say θj , may be estimated
by component cj of c.
(2) lim Pr(θ̃n,b̄(θ) (c) = cb̄(θ) ) = 1 for whatever true value of the parameter θ.
n→∞

(3) For any sequence of estimators θ̂n , it is possible to define a new sequence θ̃n (c) such that its
asymptotic covariance matrix (with the same convergence rate as θ̂n ) is
− equal to that of θ̂n if θj 6= cj for all j ∈ {1, 2, . . ., d}, i.e., b(θ) = {1, 2, . . . , d},
− positive definite and strictly less than that of θ̂n if ∅ =
6 b(θ) 6= {1, 2, . . ., d}; in this case
θ̃n (c) is asymptotically more efficient than θ̂n because

 

−1
0
Vb(θ)b
X̌b(θ)
0 (θ)
 < V −1 ,
=
Cov 
0
0
0

− zero at θ = c, i.e., b = ∅.
Taking the center parameter c = 0, the estimator θ̃n (0) obtained from any θ̂n provides a model
selection procedure that removes any parameter θj estimated by cj = 0 from θ = (θ1 , . . . , θd ). This
allows any model that omits some or all θj to be selected. In contrast, the classical Hodges’ estimator
θ̆(0) can only choose between two extreme models: the full model (corresponding to θ̆(0) 6= 0) or

the null model (corresponding to θ̆(0) = 0).

The model selection methods derived from θ̆(0), θ̃(0), penalized maximum likelihood estimation
(PMLE) and penalized least square estimation (PLSE) are discussed in more details below:
(1) While both Hodges’ estimators θ̆(0) and θ̃(0) can be applied as long as a good estimator θ̂n
is available (rn consistent for some constant sequence rn → ∞), PMLE can only be applied
13

when likelihood functions are available and PLSE is limited to regression models, both under
certain regularity conditions (see, e.g., Fan and Li for a set of regularity conditions) to produce
root-n consistent estimators. This is demonstrated by the example discussed in Remark 2.2,
where both θ̆(0) and θ̃(0) can be applied, but neither PMLE nor PLSE because the regularity
conditions fail to hold.
(2) If θ̆n (0) is used to select model by removing the parameters estimated by zero, then either all
parameters are selected, or all excluded (cf. Remark 2.1), so that the resulting model selection
does not possess oracle property. This highlights a major difference between the new type of
Hodges’ estimators θ̃n (c) and the classical version θ̆n (c).
(3) Due to point (2) above, the model selection method based on θ̆(0) is limited to two candidate
models only: the null model and the full model. In contrast, the other methods allow all
submodels of the full model to be candidates with certain θi set to zero.
(4) The classical Hodges’ estimation is in fact a preliminary-test estimation. This is however not
the case for PMLE/PLSE and oracle Hodges’ estimators (although they might look like so),
as demonstrated by the following fundamental differences:
• Existing preliminary-test procedures are essentially based on a single hypothesis that is
either accepted or rejected as a whole.

• The model selection procedures derived by PMLE/PLSE and oracle Hodges’ estimation
identify every single parameter in the multi-dimensional vector of parameters and judge

if it is estimated by zero or not. These are similar to estimation after multiple tests (also
known as multiple comparisons, see., e.g., Hsu, 1996) for a family of hypotheses; it would
be appropriate to call it preliminary-multiple-test estimation. In particular, the oracle
Hodges’ estimators provide an instance of such an estimation.
(5) Properties (1)–(3) above ensure θ̃(0) model selection to possess the oracle property, provided
the original estimator θ̂n is root-n consistent and efficient – which is the typical case where
PMLE/PLSE can be employed to produce oracle model selection procedures and θ̂n is taken to
be the efficient MLE/LSE. See, e.g., Fan and Li (2001) for a general discussion of the penalties
producing oracle model selection procedures. As a result, properties (1)–(3) are more general
than oracle properties, and both θ̃(0) and PMLE/PLSE methods are sparse and have the
same asymptotic behavior in the case of regular likelihood functions or regression models.
(6) Note that, in particular, when V is a diagonal matrix, the model selection driven by θ̃n (0)
is a hard-threshold model selection mentioned in Fan and Li (2001). If V is not diagonal,
then by Definition 2.2, θ̃j (0) = 0 for j ∈ b̄(0) and θ̃j (0) is obtained from θ̂nj for j ∈
/ b̄(0) with
14

adjustments by all estimators θ̂n1 , . . ., θ̂nd , rather than solely from θ̂nj . This also accounts for
why the raw hard-thresholding model selection does not have the oracle property.
(7) Leeb and Pötscher (2005, 2008b) argued that SCAD and other sparse estimators are a return
of Hodges’ estimators by examining their finite sample performance in the neighborhoods of
θ = 0. For any sparse estimator θ̆ satisfying lim Pr0 (θ̆ = 0) = 1, where Pr0 is the probability
n→∞

computed at θ = 0, the maximum risk over a neighborhood of θ = 0 tends to the maximum
of the employed risk function as n → ∞, and to infinity if the square loss is used. However,
this section shows that SCAD or other oracle estimators are of the same asymptotic property

with the new type of Hodges’ estimator θ̃n (0), rather than a simple return of the classical
form θ̆n (0) because θ̆n (0) does not have the oracle property.
(8) In the special cases of PMLE/PLSE, if one takes θn to be the MLE θ̂M L that maximizes the
penalized likelihood functions, or the LSE θ̂LS that minimizes the penalized squares in linear
regression, then both θ̆(c) and θ̃(c) can be expressed explicitly by θ̂M L or θ̂LS as in Definitions
2.1 and 2.2. These explicit forms of θ̆(c) and θ̃(c) enable us to theoretically derive lower bounds
for the performance of model selectors driven by θ̆(c) or θ̃(c) in finite sample size in the next
section. In contrast, it is generally unknown whether a PMLE can be expressed as a function
of MLE θ̂M L , which is solved by direct maximization using numerical algorithms. While
PLSE is clearly a function of the LSE θ̂LS , its analytical form is also generally unavailable
and numerical methods are again needed to solve it.

4

Performance of Hodges’ estimators

Hodges’ example gives a counterexample to the conjecture of R. A. Fisher that the MLE is asymp√
totically efficient with the smallest asymptotic variance (at rate n). To overcome the difficulty
thus caused, Le Cam (1953) and other researchers proved that the improvement on MLE can only
occur in a subset of the parameter space with zero Lebesgue measure. This argument, however, did
not provide any reason to rule out the use of Hodges’ estimators. An obvious question remains: if
a Hodges’ estimator can outperform the MLE at even one point without paying any price, why not
use it as a preferred one?
We now attempt to answer this question with the following two arguments:
(1) Generally, the asymptotic bias and variance are not necessarily connected to their finite sample
size versions, hence a small asymptotic variance does not imply a small variance of an estimator
even if the sample size is very large. To link the asymptotics to the finite sample size, a further
condition of uniform integrability is required.

15

(2) Even if the required uniform integrability is attached, the performance of Hodges’ estimators
θ̃n (c) is still poor at the vicinity of θ = c due to a lack of uniformity in convergence over θ ∈ Θ.

4.1

Uniform integrability and asymptotics

First we look at the asymptotic mean-squared error (MSE) of an estimator. As a matter of fact,
d

given a normalized sequence rn (θ̂n − θ) → X, the asymptotic mean and variance (hence MSE) are
only the mean and variance of X, but not the limit of rn E[θ̂n − θ] and rn2 Var(θ̂n − θ) in general if
no further conditions on θ̂n are imposed. This is highlighted in a simple example below.

Example 4.1 Let {Xn } be an i.i.d. sequence with mean µ and variance σ 2 and {Zn } another se√
√
quence satisfying Pr(Zn = An ) = Pr(Zn = − Bn ) = 0.5pn and Pr(Zn = 0) = 1 − pn , independent

of {Xn }, where An > 0, Bn > 0 are positive numbers and pn is a sequence of probabilities tending

to zero. Consider two sequences of estimates of µ:
n

µ̂n =

1X
Xi
n

2n

and

µ̃n =

i=1

1 X
Xi + Zn .
2n
i=1

√ √

√
√
√
d
d
It is clear that n(µ̂n −µ) → N (0, σ 2) and n(µ̃n −µ) = 0.5 2n(µ̂2n −µ)+ nZn → N 0, 0.5σ 2
√
p
because n|Zn | → 0, regardless of the values of An and Bn . Therefore, asymptotically, both µ̃n and
√
µ̂n are unbiased and µ̃n is more efficient than µ̂n ( nµ̃n has a smaller asymptotic variance 0.5σ 2
√
than σ 2 for nµ̂n ), as long as pn → 0. The following facts, however, show that µ̃n can be far worse

than µ̂n in finite sample size:

√
√
(1) For every fixed n, µ̂n is unbiased, whereas E[µ̃n − µ] = 0.5pn( An − Bn ), which can take
any values and be arbitrarily large by choosing the values of An and Bn . In particular, pn ,
√
√
An and Bn can be selected such that pn ( An − Bn ) → ±∞ as n → ∞, so that the bias

E[µ̃n − µ] → ±∞.

(2) If An = Bn so that µ̃n is unbiased, then nVar(µ̂n ) = σ 2 and nVar(µ̃n ) = 0.5σ 2 + npn An ,
hence Var(µ̂n ) < Var(µ̃n ) ⇐⇒ 2npn An > σ 2 . When {an } is so selected that An pn → ∞,

Var(µ̃n ) − Var(µ̂n ) = An pn − 0.5σ 2/n → ∞. If σ 2 has a known upper bound σ02 , then An and
pn can be selected to ensure that µ̂n is uniformly better than µ̃n to an arbitrarily expected
extent under the MSE or variance criterion, for every n.

(3) If Zn is distributed with such a heavy tail that its mean does not exist, e.g., with the cumulative
distribution function


pn
2x + 1
I(x < 0) + (1 − pn ) + pn
I(x ≥ 0),
2(1 − x)
2x + 2
then the mean of µ̃n does even not exist, let alone its variance.
16

This artificially designed example indicates that a sequence of unbiased (biased) estimates might
be asymptotically biased (unbiased), and even in the class of unbiased estimators, a sequence of
estimates with smaller variances for every sample size n might have larger asymptotic variance, and
vice versa. Consequently, the concepts of asymptotically unbiased estimate and asymptotic variance
could be highly misleading when they are considered as analogies to unbiased estimate and variance
in finite samples. Unfortunately, this lesson appears to have been overlooked by many statisticians
for a long time.
d

Let θ̂n be a sequence of estimates such that rn (θ̂n − θ) → X, where rn → ∞ as n → ∞ and X

is a random variable with mean zero and finite variance σ 2 . For Xn = θ̂n − θ and Yn = rn (θ̂n − θ),

we have the following two facts by Theorem 2.20 of van der Vaart (1998).

(i) E[Xn] → 0 (E[Yn ] → 0) if and only if {Xn : n ≥ 1} ({Yn : n ≥ 1}) is uniformly integrable.
(ii) E[Xn2] → 0 (E[Yn2 ] → σ 2 ) if and only if {Xn2 : n ≥ 1} ({Yn2 : n ≥ 1}) is uniformly integrable.

4.2

Performance of Hodges’ estimators

To ensure the finite sample size quantities to approach their asymptotic versions as the sample
size increases, additional conditions are required. In the remainder of this section, we assume that
the required uniform integrability described above is satisfied, so that the bias and variance in
finite sample size approach their asymptotic versions when the sample size is large. Even in such a
case, however, both numerical and theoretical analyses below show that the overall performance of
Hodges’ estimators in finite sample does not match its limit as analyzed in this subsection below.
Due to the difficulty to obtain the exact MSE of Hodges’ estimator in closed form, earlier
arguments against the use of Hodges’ estimators were largely based on numerical results. For
example, a result for Hodges’ estimate of the mean θ in the normal distribution N (θ, 1) is wellknown in the literature, see e.g., van der Vaart (1998) and Lehmann and Casella (1998), which
is recalled here. Let θ̂n = X̄ ∼ N (θ, 1/n). Taking c = 0 and an = n−1/4 yields the original
√
version of Hodges’ estimator θ̃n (0) = X̄I(|X̄| > n−1/4 ). Note that the sequences n(θ̂n − θ),
√
n(θ̃n (0) − θ), n(θ̂n − θ)2 and n(θ̃n (0) − θ)2 are all uniformly integrable so that the asymptotic
means and variances are equal to the respective limits of the means and variances in finite sample

size. While the MSE (scaled by n) of X̄ is constant 1 for all θ’s, that of the Hodges’ estimator θ̃n (0)
can only be numerically computed. Figure 1 shows the curves of the MSE of θ̃n (0) in θ for sample
sizes n = 5, 50 and 500, which behave poorly in the vicinity of zero, particularly at large sample
size (n = 500). This illustrates a much worse performance of the Hodges’ estimator θ̃n (0) than the
MLE θ̂n near the center c = 0.
Although finding the exact closed form of the MSE (or other performance measures) of a Hodges’
estimator is difficult, even in the simple case discussed just now, it turns out that some useful lower
17

bounds of regular losses of Hodges’ estimators θ̃n (c) can be obtained to see the rationale behind
their poor performance in the vicinity of the center point c as well as of the subsets of the parameter
space with θj = cj for some j ∈ {1, 2, . . . , d}. This seems to have been overlooked by the community.

The results presented below are from theoretical analyses on a general ground and distribution-

free – they are valid regardless of the underlying distributions of the population.
1. Performance of classical Hodges’ estimators.
We first present the results for classical Hodges’ estimators, which are in fact almost sure results.
Theorem 4.1 Under the conditions of Theorem 2.1, given any k > 0, there exists a (deterministic)
N > 0 such that for all n > N ,
rn kθ̃n (c) − θk ≥ k

for all θ satisfying k ≤ rn kθ − ck ≤ an rn − k.

(4.1)

Furthermore, for θn such that kθn − ck = an /2,
rn kθ̃n (c) − θn k ≥

1
rn an → ∞
2

as n → ∞.

(4.2)

The results of Theorem 4.1 have the following easy implications:
(1) Formula (4.2) proves that even if the MSE of θ̃n (c) (scaled by rn ) converges to that of the
asymptotic distribution, the convergence is not generally uniform because



 1
rn2 max Eθ kθ̃n (c) − θk2 ≥ rn2 Eθn kθ̃n (c) − θn k2 ≥ rn2 a2n → ∞.
4
θ∈Θ
(2) The same arguments also hold for more general loss functions l(θ̂n ; θ) = l(kθ̂n − θk) with
some nondecreasing function l(u) in u > 0, so that the risk function scaled by 1/l(1/rn) is

Rn (θ̂n ; θ) = Eθ [l(kθ̂n − θk)/l(1/rn)]. Let p = min{i ≥ 1 : l (i)(0+) 6= 0}. Then under the

conditions of Theorem 2.1, for any k > 0, there exists N > 0 such that for all n > N ,
l(θ̃n (c); θ)
l(k/rn)
≥
→ kp as n → ∞
l(1/rn)
l(1/rn)

for all θ satisfying k ≤ rn kθ −ck ≤ an rn −k. Furthermore, for all θn such that kθn −ck = an /2,
l(θ̃n(c); θn )
l(an/2)
≥
→ ∞ as n → ∞.
l(1/rn)
l(1/rn)
The last formula also implies
max Rn (θ̃n (c); θ) ≥ Rn (θ̃n (c); θn) ≥
θ∈Θ

18

l(an /2)
→ ∞ as n → ∞.
l(1/rn)

(3) Another way is to analyze a loss function L(θ̂n ; θ) = l(rnkθ̂n − θk) with a nondecreasing

function l(u) in u > 0 (cf. Leeb and Pötscher (2008b)), which corresponds to a sequence of
√
loss functions ln (u) = l(rnu) (rn = n in their paper), so that Rn (θ̂n ; θ) = Eθ [L(θ̂n ; θ)]. Then
similarly under the conditions of Theorem 2.1, given any k > 0, there exists N > 0 such that
L(θ̃n (c); θ) ≥ l(k) for all n > N and θ satisfying k ≤ rn kθ − ck ≤ an rn − k. Moreover, for all

θn such that kθn − ck = an /2, L(θ̃n (c); θn) ≥ l(rnan /2) → l(∞) as n → ∞. In particular, the
last property indicates that

max Rn (θ̃n (c); θ) ≥ Eθn [L(θ̃n (c); θn)] → l(∞).
θ∈Θ

(4.3)

This shows that even if lim Rn (θ̃n (c); θ) = R(X; θ) = Eθ [l(X; θ)] for every θ pointwise, the
n→∞

maximum risk over θ may increasingly tend to l(∞). An example is l(u) = I(x,∞) (u) for any
fixed continuity point x ∈ R+ of the distribution of X. The risk function of an estimator θ̂n
with this l(u) is Rn (θ̂n ; θ) = Prθ (rn kθ̂n − θk > x). Thus

lim Rn (θ̃n (c); θ) = Prθ (kXk > x)I(θ 6= c).

n→∞

In contrast, Prθ (rn kθ̃n (c) − θk > x) = 1 for all θ with k ≤ rn kθ − ck ≤ an rn − k and x ∈ [0, k],

regardless how large is n. Moreover, for any x > 0, if θn = ±an /2 and n is sufficiently large
such that an rn > x, then Prθn (rnkθ̃n (c) − θn k > x) = 1. Consequently,

max Rn (θ̂n ; θ) = max Prθ (rnkθ̃n (c) − θk > x) = 1.
θ∈Θ

θ∈Θ

2. Performance of oracle Hodges’ estimators.
Now we turn to analyze the performance of the Hodges’s estimators defined in Section 2.2. To
simplify the exposition, we assume without loss of generality that the parameter space is Θ = Rd
and define the following subsets of Θ:
Θn1 = {θ : min |θnj − cj | > anj },

Θ̄kn1 = {θ : d(θ, Θn1 ) ≤ k/rn },

Θn2 = {θ : min |θnj − cj | = 0},

Θ̄kn2 = {θ : d(θ, Θn2 ) ≤ k/rn },

Θn3 = Θ − Θn1 (c) ∪ Θn2 (c),

Θ̄kn3 = Θ − Θ̄kn1 (c) ∪ Θ̄kn2 (c),

1≤j≤d

1≤j≤d

which are all dependent on the center point c. Under the uniform integrability of θ̂n , Theorem 2.2
indicates that
lim r 2 E
n→∞ n





kθ̃n (c) − θk2 ≤ lim rn2 E kθ̂n − θk2
n→∞

with the strict inequality at certain values of θ. For fixed sample size n, however, we have the
following contrary results, which are extensions of Theorem 4.1.
19

Theorem 4.2 Under the conditions of Theorem 2.2, for any k > 0, there exists N > 0 such that
rn kθ̃n (c) − θk ≥ k for all θ ∈ Θ̄kn3
(n)

and for any θ(n) ∈ Θ̄kn3 satisfying θj

∀n > N,

(4.4)

as n → ∞.

(4.5)

= cj ± anj /2,

rn kθ̃n (c) − θ(n) k ≥

1
rn anj → ∞
2

The following extensions of Theorem 4.4 are minor modifications of the points presented earlier
for classical Hodges’ estimators.
(1) Formulas (4.4) and (4.5) prove that, under the conditions in Theorem 2.2, for any k > 0, there
exists N > 0 such that


rn2 Eθ kθ̃n (c) − θk2 ≥ k2

and

for any θ ∈ Θ̄kn3 if n > N


 1
max rn2 Eθ kθ̃n (c) − θk2 ≥ rn2 a2nj → ∞
θ∈Θ
4

as n → ∞.

(2) If we use the loss function L(θ̂n ; θ) = l(kθ̂n − θk) with a nondecreasing function l(u) in u > 0,

so that the risk function scaled by 1/l(1/rn) is Rn (θ̂n ; θ) = Eθ [l(kθ̂n − θk)/l(1/rn)], then for

sufficiently large n,

l(θ̃n (c); θ)
l(k/rn)
≥
→ kp
l(1/rn)
l(1/rn)
(n)

and for θ(n) with θj

as n → ∞

for all θ ∈ Θ̄kn3 ;

= c ± anj /2 for some j,
l(anj /2)
l(θ̃n (c); θ(n))
≥
→ ∞ as n → ∞.
l(1/rn)
l(1/rn)

The last formula also implies that
max Rn (θ̃n (c); θ) ≥ Rn (θ̃n (c); θ(n)) ≥
θ∈Θ

l(anj /2)
→∞
l(1/rn)

as n → ∞.

(3) If we analyze a loss function L(θ̂n ; θ) = l(rnkθ̂n − θk) with a nondecreasing function l(u) in
u > 0, as in Leeb and Pötscher (2008b), so that Rn (θ̂n ; θ) = Eθ [L(θ̂n ; θ)], then under the

conditions of Theorem 2.2, for any given k > 0, there exists N > 0 such that for all n > N ,
L(θ̃n (c); θ) ≥ l(k) for all θ ∈ Θ̄kn3 ; and L(θ̃n (c); θ(n)) ≥ l(rn anj /2) → l(∞) as n → ∞ for θ(n)
(n)

with θj

= c ± anj /2 for some j. The last property implies

max Rn (θ̃n (c); θ) ≥ Eθn [L(θ̃n (c); θn)] → l(∞).
θ∈Θ

(4.6)

Because of this fact, even if lim Rn (θ̃n (c); θ) = R(X; θ) = Eθ [l(X; θ)] for every θ pointwise,
n→∞

the maximum risk over θ may increasingly tend to l(∞).
20

Remark 4.1 We conclude this section by the following two points that highlight the difference
between our work and those of others, such as Leeb and Pötscher (2008b).
(1) Leeb and Pötscher (2008b) proved a result similar to equation (4.6), which is more general
√
with an arbitrary sparse estimator (say, θ̃n ), but restricted to the regular case of rn = n,
c = 0 and normally distributed X. It also requires the condition that Pn,k/√n is contiguous
with respect to Pn,0 , where Pn,θ is the distribution of θ̃n , or a stronger condition that Pn,θ is
locally asymptotically normal. In comparison, we obtained stronger results expressed in (4.4)
and (4.5) for classical and oracle Hodges’ estimators, which hold almost surely without such
√
conditions as n-consistency and contiguity.
(2) Leeb and Pötscher revealed the erratic behavior of a sparse estimator only in the vicinity of
c = 0 with θj = 0 for every component θj of θ, whereas we here showed that the erratic behavior
of the oracle model selection procedure derived by a Hodges’ estimator occurs not only in the
vicinity of the point c but also in the vicinity of every subset of Θ with θj = cj for some
component θj and arbitrary for other components.

5

Further discussions

To sum up, we have in this paper demonstrated that:
(1) the oracle model selection procedures are not a simple return of classical Hodges’ estimators
but more like the oracle Hodges’ estimator we have developed in asymptotic sense;
(2) the oracle Hodges’ estimator can easily generate an oracle model selection procedure that
satisfies the requirements of continuity or smoothing;
(3) under the MSE criterion, the oracle Hodges’ estimator θ̃n (c) does not outperform its origin
θ̂n in finite sample size, despite having a smaller asymptotic variance than θ̂n ; and
(4) Hodges’ estimators possessing the oracle property can perform much worse than their origins
under the minimax criterion.
Points (3) and (4) above also provide an answer to the question why Hodges’ estimators are not
preferable to use even if they can improve the asymptotic efficiency at no cost. The key reasons
behind this answer, as discussed earlier, are the disconnection between the finite sample statistics
and their asymptotics, and the lack of uniformity in convergence.
In addition, a few more points worth for further discussions are listed below.

21

1. Uniformity in integrability and the convergence.
By ignoring the uniformity in integrability and convergence, the widely adopted concept of asymptotically efficient estimation tends to place too much weight on the asymptotic distribution, leading
to inappropriate use of asymptotic variance to measure the goodness of estimators. This creates
the concept of superefficiency and supports Hodges estimators as superior, despite their poor performance in finite sample size. Because Hodges’ estimators can be easily applied to generate model
selection procedures possessing the oracle property, as demonstrated in point (2) above, our results
also suggest that the oracle property itself is not a convincing reason to recommend the application
of such oracle model selection procedures in theoretical studies or real-life applications.
This phenomenon was in fact recognized by statisticians decades ago. See for example Hájek
(1971, p. 153)’s warning quoted below (which can also be found in Hušková, Beran and Dupac,
1998, pp. 613-614):
“Having obtained an asymptotic result we are not usually able to tell how far it applies
to particular cases with finite n. . . . . Consequently, in applications we are guided by two
epistemologically very different knowledge: (i) we have limit theorems giving some hope,
but not assurance, of practical sample sizes; (ii) we work with some numerical experience,
which we extend to cases that seem to us to be similar. . . . . Especially misinformative
can be those limit results that are not uniform. Then the limit may exhibit some features
that are not even approximately true for any finite n. . . . . Superefficient estimates
produced by L.J. Hodges (see Le Cam (1953)) have their amazing properties only in
the limit. For any finite n they behave quite poorly for some parameter values. These
values, however, depend on n and disappear in the limit.”
Therefore, in the cases relying on asymptotic distribution of a statistic, we believe it is important
to check the uniformity in integrability and the convergence of the statistic, even if it might be
a difficult task, especially when the statistic does not have an analytical form and can only be
obtained by numerical computations.
2. The power of simulations
Undoubtedly, with the development of contemporary computing facilities, Monte Carlo simulations
provide more and more accessible and powerful tools in exploring properties of statistical inferences. This powerful tool, however, appears being applied excessively to justify certain theoretical
properties that are difficult to prove mathematically. It is even a common practice that, instead
of providing theoretical results on finite sample behaviors, many statisticians justify their statistical methods based on the asymptotic properties of their methods and then supporting their finite
sample properties by Monte Carlo simulations.
22

Therefore, another point worth to mention is the need to avoid the pitfall of relying too much on
simulations to judge the merits of a property or a method, particularly in the era of fast advancing
computer technology and capacity, as demonstrated by Leeb and Pötscher (2008b) on the SCAD
estimator in Example 4.1 of Fan and Li (2001). This pitfall is generally recognized in the statistical
community, but appears often overlooked or ignored in a large volume of publications in the pursuit
of finding new and exciting ideas and methods. Logically, simulation results can be convincing
to counter a claim or conjecture made on a general ground, but not to support such a claim or
conjecture, because it is impossible to exhaust all possible scenarios by simulations. An assertion
on property or goodness of a statistical procedure should be theoretically examined if it is to be
claimed on a general ground, rather than rely on limited numerical simulations only.
3. Generality of the center point
The generality of the center point c = (c1 , . . . , cd) implies that, mathematically, one can make model
selection not only on pre-specified θi but also on more general θi − ci , 1, . . ., d. Take oracle Hodges’

estimators as an example, for any given constant c = (c1 , . . . , cd), one can make a model selection

mathematically with the collection of candidate models defined by setting θi = 0 for one or more
i ∈ {1, . . ., d}. Similarly for PMLE/PLSE, the common practice is to use penalty fi (|θi |, λn) to

select the “best” with certain θi = 0 from the candidates. By the same method and logic, for
any fixed c = (c1 , . . . , cd), it is equally feasible to use penalty fi (|θi − ci |, λn) to select the “best”

model with θi = ci for certain i. This raises an obvious question in practice: why does everyone
only take c = 0 for model selection, not any other c? As an example, if m analysts under a
common supervisor independently analyze a same data set with parameter θ of interest using oracle
Hodges’ estimation, PMLE or PLSE with different points c(1), . . . , c(m), including zero or not, then
they obtain m different estimates θ̂(j) , j = 1, . . ., m. Clearly, all these estimators are of similar
statistical properties and rationales, but there is no statistically sensible guidance for the supervisor
to decide which one should be used. This question appears difficult to answer both logically and
philosophically, and it raises a sobering question on the validity of the commonly adopted model
selection procedures.
4. Open questions
Back to the controversy on the merits of the oracle property and the efforts to find oracle model
selection procedures, the poor performance of Hodges’ estimators and their oracle property appear
to support the criticisms of the oracle estimators and cast serious doubts on the usefulness of oracle
procedures. This further casts doubts in the validity and usefulness of prevailing model selection
methods. We believe that the following open and challenging questions need to be convincingly
answered before a consensus can be reached one way or the other:
23

(1) While oracle property is only an asymptotic property and oracle Hodges’ estimators have poor
finite sample behavior in some parts of the parameter space (under certain loss functions), are
all or most oracle estimators for model selection proposed in the literature essentially of the
nature of oracle Hodges’ estimators? In other words, are there oracle estimators fundamentally
different from Hodges’ estimators; what are they and how they differ?
(2) Without counting on the oracle property, are there any theoretical properties that ensure
good performance of the popular model selection procedures, such as LASSO and SCAD? For
example, is there any reasonable loss function beyond the regular ones discussed in this paper
such that some of the popular model selection procedures can perform well?
(3) It appears that so far none of the existing model selection procedures (including AIC and
BIC) has been proved to give better estimates/inference than the ones obtained directly from
maximum likelihood or least squares. This points to an unpalatable but unavoidable question:
for the purpose of post-selection statistical inference, is the modern research direction on model
selection on the right track?
These open questions call for further research efforts to investigate. Their answers will help resolve
the controversy on the oracle property and thus point to the right direction of research on model
selection.

References
[1] Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In B. N.
Petrov and F. Csaki (eds.) Second Internetional Aymposium on Information Theory. Akadémiai Kiadó,
Budapest. Also in Akaike, H. (1998). Selected Papers of Hirotugu Akaike (pp. 199-213). Springer New
York.
[2] Berk, R., Brown, L. and Zhao, L. (2010). Statistical inference after model selection. Journal of Quantitative Criminology 26 (2): 217-236.
[3] Berk, R. Brown, L., Buja, A. Zhang, K. and Zhao, L. (2013). Valid post-selection inference, The Annals
of Statistics 41 (2): 802-837, (with an electronic supplement.)
[4] Fan, J. and Li, R. (2001) Variable selection via nonconcave penalized likelihood and its oracle properties,
Journal of the American Statistical Association 96: 1348–1360.
[5] Frank, I.E. and Friedman, J.H. (1993). A statistical view of some chemometrics Regression tools. Technometrics, 35: 109-148.
[6] Giles, J. A. and Giles, D. E. A. (1993). Pre-test estimation and testing in econometrics: Recent developments. Journal of Economic Surveys 7: 145197.

24

[7] Hajek, J. (1971). Limiting properties of likelihoods and inference. In: Godambe, V.P., Sprott, D.A.
(Eds.), Foundations of Statistical Inference: Proceedings of the Symposium on the Foundations of Statistical Inference, University of Waterloo, Ontario, March 31-April 9, 1970. Holt, Rinehart & Winston,
Toronto, pp. 142-159.
[8] Hsu, J. (1996). Multiple Comparisons: Theory and Methods. Chapman and Hall/CRC.
[9] Hušková, M., Beran, R. and Dupac, V. (1998). Collected Works of Jaroslav Hàjek: With Commentary.
John Wiley & Sons, Inc. Judge, G.G. and Bock, M.E. (1978). The Statistical Implications of Pre-test
and Stein-Rule Estimators in Econometrics. North-Holland.
[10] Le Cam, L. (1953). On some asymptotic properties of maximum likelihood estimates and related Bayes
estimates. University of California Publications in Statistics 1, 277-330.
[11] Le Cam, L. (1960). Locally asymptotically normal families of distributions. University of California
Publications in Statistics 3, 37-98.
[12] Leeb, H. and Pötscher, B. M. (2005). Model selection and inference: facts and fiction. Econometric
Theory 21: 21-59.
[13] Leeb, H. and Pötscher, B.M. (2008a). Model Selection, in The Handbook of Financial Time Series (T.
G. Anderson, R. A. Davis, J. P. Kreiss and T. Mikosch, eds.) 785–821. Springer, New York.
[14] Leeb, H. and Pötscher, B.M. (2008b). Sparse estimators and oracle property, or the return of Hodges’
estimator, Journal of Econometrics 142: 201-211.
[15] Lehmann, E. L. and Casella, G. (1998). Theory of Point Estimation. Springer.
[16] Mallows, C. L. (1965): Some approaches to regression problems. Unpublished manuscript.
[17] Pötscher, B. M. (2009). Confidence sets based on sparse estimators are necessarily large. Sankhyā: The
Indian Journal of Statistics, Series A 71(1): 1-18.
[18] Pötscher, B.M. and Leeb, H. (2009). On the distribution of penalized maximum likelihood estimators:
The LASSO, SCAD, and thresholding, Journal of Multivariate Analysis 100:2065-2082 .
[19] Pötscher, B.M. and Schneider, U. (2009). On the distribution of the adaptive Lasso estimator, Journal
of Statistical Planning and Inference 139 (8): 2775-2790
[20] Schwarz, G.E. (1978). Estimating the dimension of a model, Annals of Statistics 6 (2): 461464.
[21] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical
Society. Series B (Methodological) 267-288.
[22] van der Vaart, A.W. (1998). Asymptotic statistics. Cambridge University Press. ISBN 978-0-521-78450-4.
[23] Zhang, C. (2010). Nearly unbiased variable selection under minimax concave penalty. The Annals of
Statistics, 38: 894-942.
[24] Zou, H. (2006). The adaptive lasso and its oracle properties, Journal of the American Statistical Association 101 (476): 1418-1429.
[25] Zou, H. and Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the
Royal Statistical Society, Ser. B, 67: 301-320.

25

A

Appendix: Proofs of the theorems

A.1

Proof of Theorem 2.1
p

Proof. Note first that θ̂n → θ. For any θ 6= c, the condition an = o(1) implies that, for any ε > 0,
Prθ (rn kθ̆n (c) − θ̂n k > ε) ≤ Prθ (kθ̂n − ck ≤ an ) ≤ Prθ (kθ − ck − kθ̂n − θk ≤ an )
= Prθ (kθ̂n − θk ≥ kθ − ck − an ) → 0

as n → ∞.

Thus
d

rn (θ̆n (c) − θ) = rn (θ̆n (c) − θ̂n ) + rn (θ̂n − θ) → X.
For θ = c, thanks to rn an → ∞,
Prc (rn kθ̆n (c) − ck > ε) ≤ Prc (θ̆n 6= c) = Prθ0 (rn kθ̂n − ck > rn an ) → 0.

(A.1)

p

This shows rn (θ̆n (c) − c) → 0.

A.2

Proof of Theorem 2.2

Proof. Because we are concerned with the asymptotic distribution of θ̌n,b in this section, without
loss of generality we can treat the easy case where V is known and θ̌n,b is defined by
θ̌n,b = θ̂n,b + Vbb−1 Vbb̄ (θ̂n,b̄ − cb̄) and

0
θ̌n (b) = (θ̌n,b
, c0b̄)0

(A.2)

with the convention θ̌n,{1,2,...,d} = θ̌n ({1, 2, . . . , d}) = θ̂n .
The first assertion is obvious, so we here only prove (2.9). With the definition of b(θ) in (2.4),
it is clear that θb(θ) is the sub-vector (θj : θj 6= cj ) of θ and θb̄(θ) = (cj : j ∈ b̄(θ)) = cb̄(θ) . Note

that, by (A.2), θ̌n,b(θ) = θ̂n,b(θ) and hence θ̌n (b(θ)) = (θ̌n,b(θ) , cb̄(θ) ) are only pseudo-estimators that
depend on the unknown parameters θ. However,
−1
rn (θ̌n,b(θ) − θb(θ) ) = Vb(θ),b(θ)
(Vb(θ)b(θ)
d

−1
(Vb(θ)b(θ)
→ Vb(θ),b(θ)



Vb(θ),b̄(θ) )rn 

θ̂n,b(θ) − θb(θ)

θ̂n,b̄(θ) − θb̄(θ)

Vb(θ),b̄(θ) )X = X̌b(θ) ,




(A.3)

where the components of X has been rearranged according to the order in θ and X̌b(θ) is derived
from (2.7) by replacing b with b(θ).
For any θ, by comparing (2.6) and (A.2),




0
Prθ rn kθ̃n (c) − θ̌n (b(θ))k > ε = Prθ rn kθ̃n (c) − (θ̌b(θ)
, c0b̄(θ) )0 k > ε


0
≤ Prθ θ̃n (c) 6= (θ̌b(θ)
, c0b̄(θ) )0
≤ Prθ (bn(c) 6= b(θ)).
26

Since
{bn (c) 6= b(θ)} =

[

j∈b(θ)

{|θ̂nj − cj | ≤ anj }

[

j∈b̄(θ)

{|θ̂nj − cj | > anj },



Prθ rn kθ̃n (c) − θ̌n (b(θ))k > ε


[
[
≤ Prθ 
{|θ̂nj − cj | ≤ anj }
{|θ̂nj − cj | > anj }
j∈b(θ)

≤

≤

j∈b̄(θ)

X

Prθ {|θ̂nj − cj | ≤ anj } +

X

Prθ {|θ̂nj − θj | ≥ |θj − cj | − anj } +

j∈b(θ)

j∈b(θ)

X

j∈b̄(θ)

Prθ {|θ̂nj − cj | > anj }
X

j∈b̄(θ)

Prθ {rn |θ̂nj − cj | > rn anj }

→ 0 as n → ∞
under the conditions on an . Thus rn (θ̃n (c) − θ̌n (b(θ)) = op(1). Combining this with (A.3), we get


X̌b(θ)
d

rn (θ̃n (c) − θ) = rn (θ̃n (c) − θ̌n (b(θ))) + rn (θ̌n (b(θ)) − θ) → 
(A.4)
0
under Prθ . Next examine the case b(θ) = ∅. Analogous to (A.1), under Prc and the condition
rn min anj → ∞,
1≤j≤d

Prc (rnkθ̃n (c) − ck > ε) ≤ Prc (θ̃n 6= c) ≤

d
X
j=1

Prc (rn |θ̂nj − c| > rn anj ) → 0 as n → ∞.

(A.5)

d

Hence rn |θ̃n (c) − c| → 0. The proof is then complete.

A.3

Proof of Theorem 4.1

Proof. Since an → 0 and an rn → ∞ as n → ∞ under the conditions of Theorem 2.1, the set

{θ : k ≤ rn kθ − ck ≤ an rn − k} is a nonempty ring when n is sufficiently large such that an rn > 2k.

For any θ̃n (c) defined in (2.1), it is clear that kθ̃n (c) − θk ≥ k/rn for any values the sample may

take, as long as θ satisfies k ≤ rn kθ − ck ≤ an rn − k. This proves the first assertion in (4.1). The

second assertion (4.2) is obvious.

A.4

Proof of Theorem 4.2

Proof. Again, we assume that N is sufficiently large such that rn max1≤j≤d anj > k for all n > N .
It can be easily shown that d(Θn1 , Θn2 ) = min1≤j≤d anj . Because for any θ1 ∈ Θ̄kn1 and θ2 ∈ Θ̄kn2 ,
d(Θn1 , Θn2 ) ≤ d(Θn1 , θ1 ) + d(θ1 , θ2 ) + d(θ2 , Θn2 ) ≤
27

2k
+ d(θ1 , θ2 ),
rn

the condition lim rn min1≤j≤d anj = ∞ implies
n→∞

d(Θ̄kn1 , Θ̄kn2 ) =

min

θ1 ∈Θ̄kn1 ,θ2 Θ̄kn2

d(θ1 , θ2 ) ≥ min anj −
1≤j≤d

2k
>0
rn

for sufficiently large n. Consequently, Θ̄kn1 ∩ Θ̄kn2 = ∅ and Θ̄kn3 6= ∅.

By Definition 2.2 of θ̃n (c), θ̂n ∈ Θn1 implies θ̃n (c) = θ̂n ∈ Θn1 and θ̂n ∈ Θ − Θn1 implies

b̄n (c) 6= ∅, so that θ̃n (c) ∈ Θn2 because θ̃n,b̄n (c) (c) − cb̄n (c) = 0. That is, θ̃n (c) takes values only in

Θn1 ∪ Θn2 . For any θ ∈ Θ̄kn3 , it is clear that d(θ, Θkn1 ∪ Θkn2 ) ≥ k/rn . As a result, θ ∈ Θ̄kn3 implies
kθ̃n (c) − θk ≥ d(θ, Θkn1 ∪ Θkn2 ) ≥

k
.
rn

Thus the assertion in (4.4) follows. The second assertion in (4.5) is easy to check.

28

Figure 1. The MSE (times n) of Hodges' Estimator

Legend: Blue curve corresponds to n = 5, purple to n = 50, and olive to n = 500.

