Moment convergence in regularized estimation under multiple
and mixed-rates asymptotics∗
Hiroki Masuda†

Yusuke Shimizu‡

arXiv:1406.6751v5 [math.ST] 16 Apr 2017

Abstract
In M -estimation under standard asymptotics, the weak convergence combined with the
polynomial type large deviation estimate of the associated statistical random field Yoshida
(2011) provides us with not only the asymptotic distribution of the associated M -estimator
but also the convergence of its moments, the latter playing an important role in theoretical
statistics. In this paper, we study the above program for statistical random fields of multiple
and also possibly mixed-rates type in the sense of Radchenko (2008) where the associated
statistical random fields may be non-differentiable and may fail to be locally asymptotically
quadratic. Consequently, a very strong mode of convergence of a wide range of regularized
M -estimators is ensured. The results are applied to regularized estimation of an ergodic
diffusion observed at high frequency.

1

Introduction

We are concerned here with moment convergence in regularized estimation under multiple and
mixed-rates asymptotics. Let us begin with some basic background of the present study. Suppose
that we observe data whose distribution is indexed by a finite-dimensional parameter θ ∈ Θ ⊂
Rp . Let θ0 ∈ Θ denote the true value of θ. In order to estimate θ0 by means of the M-estimation
theory, we introduce an appropriate (quasi-)likelihood or contrast function Hn : Ω × Θ → R, and
estimate an optimal parameter value by any point θ̂n ∈ argmin Hn . For assessing asymptotic
performance of θ̂n quantitatively, we look at the statistical random fields
Mn (u; θ0 ) = Hn (θ0 + An (θ0 )u) − Hn (θ0 ),

u ∈ Rp ,

(1.1)

where An (θ0 ) denotes the rate matrix such that |An (θ0 )| → 0 as n → ∞ and that the components
may decrease at different rates, say, of multiple-rates asymptotic; estimation under multiple-rates
asymptotic has appeared in the literature of, for example, econometrics [4]. As is well-known,
the weak convergence of Mn to some M0 over compact sets, the identifiability condition on M0 ,
and the tightness of the scaled estimator ûn := An (θ0 )−1 (θ̂n − θ0 ) make the “argmin” functional
L

for Mn continuous: ûn ∈ argmin Mn −
→ argmin M0 . In many statistical models, this procedure
is enough to completely clarify the asymptotic distribution of all the components of θ̂n . We will
refer this case to “standard asymptotics”. See [30, Chapter 5] for a general yet concise account
of the M-estimation theory.
Beyond the weak convergence, convergence of moments of ûn serves as a fundamental tool
when analyzing asymptotic behavior of the expectations of statistics depending on the estimator
such as asymptotic bias and mean squared prediction error. We refer to [7], [10], [14], [24], [26],
∗

This version: April 5, 2018
Faculty of Mathematics, Kyushu University. 744 Motooka Nishi-ku Fukuoka 819-0395, Japan.
‡
Department of Mathematics, Josai University. 1-1 Keyakidai Sakado Saitama 350-0295, Japan.
Corresponding author:yshimizu@josai.ac.jp
†

1

and [27] for some related results. Also, [1] recently discussed optimal selection of random and
k-fold cross-validation estimators, the theoretical backbone of which involves some moment
bounds of the estimators used; the related paper [2] studied the uniform integrability of the
ordinary least-squares estimator in the linear regression setting. In the standard asymptotics,
the polynomial type large deviation inequality (PLDI) of [33], which estimates the tail of L(ûn )
in such a way that
sup sup rL P (|ûn | ≥ r) < ∞
r>0 n>0

for a given L > 0, provides us with a widely-applicable tool for verifying the convergence of
moments. In [33], it has been shown that the PLDI can be proved under modest conditions
especially when Mn admits a locally asymptotically quadratic (LAQ) structure1
1
Mn (u; θ0 ) = ∆n (θ0 )[u] + Γ0 (θ0 )[u, u] + rn (u; θ0 ),
2

u ∈ Rp ,

where ∆n (θ0 ) and Γ0 (θ0 ) are random
respectively (we use the multiP linear and bilinear forms,
P
linear-from notations: A[u] :=
A
u
and
B[u,
u]
:=
B
ui uj , and so on), and where
i
i
ij
i
i,j
rn (u; θ0 ) is expected to be stochastically ignorable for each u. The LAQ property is known to be
satisfied for many situations including non-ergodic (asymptotically mixed-normal type) models
under multi-scaling. A more detailed description is given in Appendix A.
In principle, any M -estimation procedure may have its “regularized” counterpart. The
logic of the sparse and more generally shrinkage estimation would be most clearly described
by the context of multiple linear regression, with many deep theoretical interpretation such as
geometrical (projection) characterization, variable selection, stabilized prediction performance,
etc.; see e.g. [5], [12, Chapter 3], and [22]. Statistical random fields associated with regularized
estimation may not be LAQ, if the random field is of the “mixed-rates” type in the sense of [23]
where the target statistical random fields may have components converging at different rates.
In that case, it may even happen that the random function Mn (u; θ0 ) diverges in probability for
each u; indeed, the sparse-type estimation, which is a particular case of regularized estimation,
falls into the region of mixed-rates asymptotics. In this case, convergence of moments does not
follow from a direct application of [33], and we are aware of only the following previous studies
in this direction: [29] deduced the convergence of moments of a regularized sparse maximumlikelihood estimator of the generalized linear model, and applied it to verify the AIC type variable
selection; [25] (also [20]) deduced the moment convergence in regularized estimation of a linear
regression model with general regularization term. Nevertheless, the proofs of these results made
particular use of the special structure of the considered models and/or the convexity argument,
and both of them do not tell us much about the case of general regularized M -estimation.
The primary contribution of this paper is to compose a first systematic study of not only
asymptotic distribution but also convergence of moments associated with a class of regularized
LAQ models under multiple and mixed-rates asymptotics. In particular, in Theorem 3.20, we
will show how the PLDI of [33] can carry over to the case of mixed-rates M -estimation.
This paper is organized as follows. Section 2 describes our model setup, based on which a
series of basic asymptotic statements are given in Section 3. In Section 3.1 we will deal with the
sparse asymptotics, and then the case of standard asymptotics is briefly mentioned in Section 3.2.
In Section 3.3.1, we will discuss a simple specific example of component-wise tuning-parameter
choice. In Section 4 we will apply the foregoing results to regularized estimation of an ergodic
diffusion process observed at high frequency. The model is described by the Wiener-driven
stochastic differential equation
dXt = a(Xt , α)dwt + b(Xt , β)dt,
1

The sign in front of the quadratic term (1/2)Γ0 (θ0 )[u, u] is different from the original LAQ of [33] since we
consider minimization of (1.1).

2

and is known to be of multiple-scaling type. In the literature, [8] previously studied an adaptivelasso type regularized estimation of the same model, and deduced the oracle property. In this
study, we will derive the sparse consistency of the zero parameters, the asymptotic normality of
the non-zero parameters, and the PLDI for a regularized estimator without resorting to convexity
at all.
It is important to study associated practical computational algorithms, e.g. tuning parameter
selection by making use of AIC (see Section 3.3.2), and numerical optimization subjects such as
the alternating direction method of multipliers. They are beyond the scope of this paper and
we would like to leave it as one of future works.

2

Setup

Let us begin with description of the basic model setup for Section 3. Throughout we are given an
underlying probability space (Ω, F, P ). For the purpose of accelerating estimation performance,
we consider M -estimation of an additive regularization type. We will focus on the case of
two-scaling, where the target statistical parameter θ ∈ Θ, is divided into two parts:
θ = (α, β).
We set α ∈ Rp and β ∈ Rq , and Θ = Θα × Θβ to be a bounded convex domain in Rp+q .
We are given a function Mn : Ω × Θ → R, and regularization (possibly random) functions
a
b
Rn (α) and Rn (β). We then consider a contrast function Hn : Ω × Θ → R of the form
a

b

Hn (θ) = Hn (α, β) = Mn (α, β) + Rn (α) + Rn (β).

(2.1)

The associated regularized M -estimator is defined to be any element (for brevity, implicitly
assumed to exist)
θ̂n ∈ argmin Hn (θ).
θ∈Θ

We quantitatively distinguish zero parameters from non-zero ones. We denote by θ0 = (α0 , β0 )
the value we want to estimate (typically the true value of θ) and assume that it takes the form
◦ ) 0 , (α∗
◦
∗
◦
∗
α0 = (α0◦ , α0∗ ) = ((α0,k
0 k
0,k00 )k00 ) and β0 = (β0 , β0 ) = ((β0,l0 )l0 , (β0,l00 )l00 ) with
◦

◦

∗
β0,l
00 6= 0.

∗
α0,k
00 6= 0,

◦
β0,l
0 = 0,

◦
α0,k
0 = 0,
∗

∗

We set α0◦ ∈ Rp , β0◦ ∈ Rq , α0∗ ∈ Rp and β0∗ ∈ Rq with p◦ , q ◦ , p∗ , q ∗ ∈ N; then, p = p◦ + p∗ and
q = q ◦ + q ∗ . Correspondingly, we write θ = (θ◦ , θ∗ ) with θ◦ = (α◦ , β ◦ ) and θ∗ = (α∗ , β ∗ ). We
also write θ̂n = (α̂n , β̂n ) = (α̂n◦ , α̂n∗ , β̂n◦ , β̂n∗ ) with θ̂n◦ = (α̂n◦ , β̂n◦ ) and θ̂n∗ = (α̂n∗ , β̂n∗ ). For clarity we
focus on regularization terms of the form
a
Rn (α)

=

p
X

b
Rn (β)

λan,k Ra (αk ),

=

k=1

q
X

λbn,l Rb (βl ).

(2.2)

l=1

The regularization-function form subsumes many existing ones, although not essential for our
basic asymptotic results given in Sections 3.1 and 3.2. For later reference we write more specifically
◦

a
Rn (α)

=

a◦
Rn (α◦ )

+

a∗
Rn (α∗ )

=

p
X

∗

a ◦
λa◦
n,k0 R (αk0 )

k0 =1

=

b◦
Rn (β ◦ )

+

b∗
Rn (β ∗ )

=

q
X

a ∗
λa∗
n,k00 R (αk00 ),

(2.3)

k00 =1

◦

b
Rn (β)

+

p
X

∗

b ◦
λb◦
n,l0 R (βl0 )

l0 =1

where:
3

+

q
X
l00 =1

b ∗
λb∗
n,l00 R (βl00 ),

(2.4)

a∗
b◦
b∗
• λa◦
n,k0 , λn,k00 , λn,l0 and λn,l00 are non-negative random variables;

• Ra (·), Rb (·) are nonrandom non-negative functions on R such that Ra (0) = Rb (0) = 0;
• For all a0 , b0 6= 0 and k > 0, there exists a constant C = C(a0 , b0 , k) > 0 such that
|Ra (a0 ) − Ra (a0 )| + |Rb (b0 ) − Rb (b0 )|
≤ C.
|a0 − a0 | + |b0 − b0 |
(a0 ,b0 ): |a0 |∨|b0 |≤k
sup

(2.5)

The last condition (local Lipschitz continuity) is a technical one. Further conditions on Mn ,
a
b
Rn (α) and Rn (β) will be imposed later on; in Section 3.3.1, we will briefly discuss about how
to set the regularization terms in some specific ways.
We will deal with a situation where the non-zero part of the first component α can be
estimated faster than that of the second component β; more specifically, we will suppose that
the sequence


∗
∗
−1 ∗
∗
s−1
n (α̂n − α0 ), tn (β̂n − β0 )
has a non-trivial asymptotic distribution for some possibly different positive sequence (sn ) and
(tn ), both tending to zero and satisfying that sn = o(tn ). Although not explicitly mentioned, we
presuppose that the “principal” part Mn (θ) reasonably makes sense even without regularization
a
b
terms Rn (α)+Rn (β); most typically, the un-regularized case, where Hn (θ) = Mn (θ), corresponds
to a negative of a (quasi) log-likelihood. We should note that the additive regularization can
be interpreted as incorporating a prior information about the parameter of interest; see Section
3.3.1.
We end this section with noting that an extension to cases of more-than-two scaling is a trivial
matter while making notation messy, and also that we can consider the usual single-scaling case
as well simply by omitting the parameter β.

3

Asymptotics

3.1

Sparse case

Under the setting described in Section 2, we consider regularity conditions under which the
following properties hold without assuming the convexity of Hn .
1. The (weak) consistency of θ̂n = (θ̂n◦ , θ̂n∗ ).
2. The asymptotic distributions:
(a) The sparse consistency of θ̂n◦ , i.e. P (θ̂n◦ = 0) → 1;
(b) The asymptotic distribution of θ̂n∗ at possibly multiple rates of convergence (via a
matrix norming).
3. The uniform tail-probability estimate of θ̂n = (θ̂n◦ , θ̂n∗ ).
Hereafter, we will prove the results in the above order: Theorem 3.4 describes the consistency of
θ̂n . In Theorem 3.8 we derive the auxiliary convergence rate, followed by the sparse consistency
of θ̂n◦ in Theorem 3.12. We also derive the asymptotic distribution of θ̂n∗ in Theorem 3.15.
Finally, we will present the uniform tail-probability estimate of θ̂n = (θ̂n◦ , θ̂n∗ ) in Theorem 3.20.

4

3.1.1

Consistency

We impose the uniform law of large numbers plus identifiability condition, and additionally some
stochastic-order conditions on the regularization terms. Recall that the parameter space Θ is a
bounded convex domain.
Assumption 3.1.
1. (sn ) and (tn ) are positive nonrandom sequences such that max(sn , tn ) → 0 and that sn =
o(tn ).
a

b

2. There exist continuous random functions M 0 : Ω × Θα → R and M 0 : Ω × Θ → R such
that:
a

(a) sup s2n {Mn (α, β0 ) − Mn (α0 , β0 )} − M 0 (α)
α

p

b

+ sup t2n {Mn (α, β) − Mn (α, β0 )} − M 0 (θ) →
− 0;
θ

a

b

(b) argmin M 0 (α) = {α0 } a.s. and argmin M 0 (α0 , β) = {β0 } a.s.
α

β

a

b

p

− 0.
3. sup s2n Rn (α) + sup t2n Rn (β) →
α

β

We will take advantage of a general result concerning mixed-rates asymptotics.
Lemma 3.2. Let Mn (u, v) : Rp × Rq → R be a random function, and (ûn , v̂n ) and (û0 , v̂0 )
random variables taking their values in Rp × Rq . Assume that Mn is of the form
Mn (u, v) = ān fn (u) + b̄n gn (u, v),

(u, v) ∈ Rp × Rq ,

where:
(L1) ān and b̄n are positive constants such that b̄n = o(ān );
L

(L2) (fn (·), gn (·, ·)) → (f0 (·), g0 (·, ·)) in C(Kf × Kg ) for every compact Kf × Kg ⊂ Rp × Rq ;
(L3) Mn (ûn , v̂n ) ≤ inf (u,v) Mn (u, v) + op (b̄n );
(L4) (ûn , v̂n ) = Op (1);
(L5) u 7→ f0 (u) has a.s. unique minimum at u = û0 ;
(L6) v 7→ g0 (û0 , v) has a.s. unique minimum at v = v̂0 .
Then we have the following.
L

1. (ûn , v̂n ) → (û0 , v̂0 ) under the conditions (L1) to (L6).
L

2. ûn → û0 under the conditions (L1) to (L5).
Proof. The first claim is just a simplified version of [23, Theorem 1]. The second one follows on applying the usual argmax theorem to the rescaled function u 7→ ā−1
n Mn (u, v̂n ) =
fn (u) + (b̄n /ān )gn (u, v̂n ): this random function admits an approximate minimizer ûn and weakly
converges to f0 in C(Kf ) for every compact Kf ⊂ Rp .
Remark 3.3. Although we do not use the second claim of Lemma 3.2 in this paper, it may be
useful when considering stepwise estimation. See [28] for an ergodic diffusion model.
5

To apply the first claim of Lemma 3.2 under Assumption 3.1 with (u, v) = (α, β), we set
−2
Mn (α, β) = Hn (α, β) − Hn (α0 , β0 ), ān = s−2
n , b̄n = tn :

a
a
fn (α) = s2n Mn (α, β0 ) − Mn (α0 , β0 ) + Rn (α) − Rn (α0 ) ,


b
b
gn (α, β) = t2n Mn (α, β) − Mn (α, β0 ) + Rn (β) − Rn (β0 ) .
a

b

According to the a.s. continuity of the random functions α 7→ M 0 (α) and θ 7→ M 0 (θ), it
is straightforward to verify all the conditions in Lemma 3.2 under Assumption 3.1, hence the
following claim:
p

Theorem 3.4. We have θ̂n →
− θ0 under Assumption 3.1.
3.1.2

Rates of convergence

Next we prove that (ûn , v̂n ) = Op (1) where
ûn := s−1
n (α̂n − α0 ),

v̂n := t−1
n (β̂n − β0 )

(3.1)

under additional conditions. To this end we introduce the following general result, which is a
simplified version of [23, Lemma 1], to deduce “correct” convergence rate of θ̂n∗ and a “auxiliary”
convergence rate of θ̂n◦ . Here, the auxiliary convergence rate may not be the optimal convergence
rate for θ̂n◦ , for which we will show more precisely the sparse consistency. Nevertheless, the
auxiliary rate should be useful for verification of the conditions for the sparse consistency of θ̂n◦
(see Remark 3.13 below). Let [a]+ := max(a, 0) for a ∈ R.
Lemma 3.5. Let ξ denote either α or β, and assume that the real-valued random function Hn (ξ)
satisfies the following conditions:
1. Hn (ξˆn ) ≤ Hn (ξ0 ) a.s.
2. There exist random functions Un (ξ) and Vn (ξ) such that
Hn (ξ) − Hn (ξ0 ) = Un (ξ) − Vn (ξ)
where, for some random variable U0 > 0 a.s., constants 0 ≤ γ < ρ, and positive nonrandom
sequence (kn ) such that kn → 0, we have:


(a) P Un (ξˆn ) ≥ |ξˆn − ξ0 |ρ U0 → 1;
(b) [Vn (ξˆn )]+ = Op (kn |ξˆn − ξ0 |γ ).
−1/(ρ−γ)

Then kn

(ξˆn − ξ0 ) = Op (1).

To deduce Lemma 3.5, observe that on the set {|ξˆn − ξ0 |ρ ≤ U0−1 Un (ξˆn )} we have |ξˆn −
ξ0 |ρ ≤ U0−1 {Vn (ξˆn ) + Hn (ξˆn ) − Hn (ξ0 )} ≤ U0−1 Vn (ξˆn ) ≤ U0−1 [Vn (ξˆn )]+ = Ln kn |ξˆn − ξ0 |γ , where
Ln = Op (1). By applying [23, Lemma 3], for any δ > 0 we can find a tight sequence (Nn )
ρ/(ρ−γ)
in R for which Ln kn |ξˆn − ξ0 |γ ≤ δ|ξˆn − ξ0 |ρ + Nn kn
. Picking a δ < 1, we conclude that
1/(ρ−γ)
|ξˆn − ξ0 | ≤ kn
Nn , hence the assertion.
In what follows, for any matrix A we write A⊗2 = AA> with > denoting the transpose, and
for any real symmetric matrix A we denote by λmin (A) the smallest eigenvalue of A. Coming
back to our model, we next impose:
Assumption 3.6.
6

1. Mn ∈ C 3 (Θ) a.s., and it holds that:
(a) sup |sn ∂α Mn (α0 , β)| + |tn ∂β Mn (θ0 )| = Op (1);
β

(b) sup |sn tn ∂α ∂β Mn (α, β0 )| = Op (1);
α

(c) sup s2n ∂ζ ∂α2 Mn (θ) + sup t2n ∂ζ ∂β2 Mn (θ) = Op (1) for ζ = α, β;
θ

θ

(d) There exist symmetric random functions Γα0 : Ω × Θα → Rp ⊗ Rp and Γβ0 : Ω × Θ →
Rq ⊗ Rq such that
p

− 0,
s2n ∂α2 Mn (θ0 ) − Γα0 (α0 ) + t2n ∂β2 Mn (θ0 ) − Γβ0 (θ0 ) →


with λmin Γα0 (α0 ) ∧ λmin Γβ0 (θ0 ) > 0 a.s.
b∗
00
00
2. sn λa∗
n,k00 = Op (1) and tn λn,l00 = Op (1) for each k and l .

Remark 3.7. Note that Assumption 3.6.2 is only concerned with the non-zero parameter parts,
which of course are unknown a priori in practice. Hence, as is well recognized as a common
situation in adaptive type sparse estimation, some appropriate data-driven choices of the weights
are desirable. There would be many possibilities for this issue. See Section 3.3.1 for more
discussions.
p

Let Assumptions 3.1 and 3.6 hold, so that θ̂n →
− θ0 by Theorem 3.4. We apply Lemma 3.5
for proving ûn = Op (1) and v̂n = Op (1) separately. To show ûn = Op (1), set
Hn (α) = s2n Hn (α, β̂n ).
For notational convenience, for any random function Fn (θ) we will write Fn (θ) = Op∗ (1) and
Fn (θ) = o∗p (1) if supθ |Fn (θ)| = Op (1) and supθ |Fn (θ)| = op (1), respectively. The first condition in Lemma 3.5 is trivial. To deduce the second one, making use of the third-order Taylor
expansion we derive Hn (α̂n ) − Hn (α0 ) = Un (α̂n ) − Vn (α̂n ) for
◦

p
X


1
a ◦
Un (α̂n ) := s2n ∂α2 Mn (α̃n , β̂n ) (α̂n − α0 )⊗2 + s2n
λa◦
n,k0 R (α̂n,k0 )
2
0
k =1

=

=

=

1
2

s2n ∂α2 Mn (α0 , β̂n )

+

s2n ∂α3 Mn (α̌n , β̂n )[α̃n

p◦

X

⊗2
2
a ◦
− α0 ] (α̂n − α0 )
+ sn
λa◦
n,k0 R (α̂n,k0 )
k0 =1

o

1n α
Γ0 (α0 ) + Op∗ (|α̂n − α0 | ∨ |β̂n − β0 |) (α̂n − α0 )⊗2 + s2n
2


1 α
Γ0 (α0 ) + o∗p (1) (α̂n − α0 )⊗2 + s2n
2

p◦
X

p◦
X

a ◦
λa◦
n,k0 R (α̂n,k0 )

k0 =1

a ◦
λa◦
n,k0 R (α̂n,k0 ),

(3.2)

k0 =1

∗

Vn (α̂n ) := −s2n ∂α Mn (α0 , β̂n )[α̂n − α0 ] − s2n

p
X


a ∗
a ∗
λa∗
n,k00 R (α̂n,k00 ) − R (α0,k00 ) ,

k00 =1

where the points α̃n and α̌n are located on the segments connecting α0 and α̂n , and α0 and α̃n ,
respectively. Note that the non-negativity of Ra enables us to ignore the second term of the righthand side of (3.2) when estimating Un (α̂n ) from below. Also, under the local Lipschitz continuity
(2.5) of Ra , we see that the conditions of Lemma 3.5 are satisfied with γ = 1, ρ = 2, (e.g.)
U0 = λmin (Γα0 (α0 )) /4, and kn = sn : we have Un (α̂n ) ≥ (1/2){op (1) + λmin (Γα0 (α0 ))}|α̂n − α0 |2
7

and |Vn (α̂n )| ≤ Op (sn |α̂n − α0 |). Hence ûn = Op (1) is proved. To deduce v̂n = Op (1), we can
follow exactly the same way as above along with
Hn (β) = t2n {Hn (α̂n , β) − Hn (α̂n , β0 )}
in place of Hn (α) = s2n Hn (α, β̂n ).
Theorem 3.8. We have (ûn , v̂n ) = Op (1) under Assumptions 3.1 and 3.6.
3.1.3

Sparse consistency

The sparse consistency P (θ̂n◦ = 0) → 1 implies that the asymptotic distribution of θ̂n◦ degenerates
at the origin with arbitrarily fast rate of convergence: rn θ̂n◦ = op (1) for arbitrary rn → ∞. The
next general result is a variant of [23, Theorem 2], which is a tailor-made tool to establish the
property.
Lemma 3.9. Let ξ denote either α or β (so ξ = (ξ ◦ , ξ ∗ ) and ξ0 = (0, ξ0∗ )), and assume that the
real-valued random function Hn (ξ) = Hn (ξ ◦ , ξ ∗ ) satisfies the following conditions.
1. Hn (ξˆn◦ , ξˆn∗ ) ≤ Hn (0, ξˆn∗ ) a.s.
2. There exist random functions Un (ξ) and Vn (ξ) such that
Hn (ξ) − Hn (0, ξ ∗ ) = Un (ξ) − Vn (ξ),
where it holds that for some random variable U0 > 0 a.s. and constants ρ > 0,
n
o n
o
P
[Vn (ξˆn )]+ = 0 ∩ Un (ξˆn ) ≥ |ξˆn◦ |ρ U0
→ 1.

Then P (ξˆn◦ = 0) → 1.
Lemma 3.9 follows on observing the following: on the event An := {[Vn (ξˆn )]+ = 0} ∩
{Un (ξˆn◦ , ξˆn∗ ) ≥ |ξˆn◦ |ρ U0 }, we have |ξˆn◦ |ρ ≤ U0−1 {Vn (ξˆn ) + Hn (ξˆn ) − Hn (0, ξˆn∗ )} ≤ U0−1 [Vn (ξˆn )]+ = 0.
Hence P (|ξˆn◦ | = 0) can be bounded below by P (An ) → 1.
Remark 3.10. To conclude P (ξˆn◦ = 0) → 1 we may replace the second condition of Lemma 3.9
by
n
o n
o
◦
ρ
ˆ
ˆ
ˆ
P [Vn (ξn )]+ = 0 ∩ Un (ξn ) ≥ |ξn,m0 | U0
→1
for each m0 running through {1, . . . , p◦ } or {1, . . . , q ◦ } according as ξ = α or β; for each m0 , the
◦
same proof as above leads to P (ξˆn,m
0 = 0) → 1.
Let us go back to our main context. We keep imposing Assumptions 3.1 and 3.6. Denoting
◦
◦
by Γα0 (α) (resp. Γβ0 (θ)) the upper left p◦ × p◦ part of Γα0 (α) (resp. the upper left q ◦ × q ◦ part
of Γβ0 (θ)), we have
◦

◦

p

s2n ∂α2 ◦ Mn (θ0 ) − Γα0 (α0 ) + t2n ∂β2◦ Mn (θ0 ) − Γβ0 (θ0 ) →
− 0

(3.3)



◦
◦
with λmin Γα0 (α0 ) ∧ λmin Γβ0 (θ0 ) > 0 a.s. The next assumption imposes component-wise
“good balances” between the quadratic expansions of Mn and the regularization terms.
8

Assumption 3.11. There exist constants ak0 , bl0 ∈ (0, 1/2) such that

◦
◦
∗
◦
P s2n ∂α◦0 Mn (0, . . . , 0, α̂n,k
0 , . . . , α̂n,p◦ , α̂n , β̂n )α̂n,k 0
k

a ◦
s2n λa◦
n,k0 R (α̂n,k0 )

◦
−ak0 λmin Γα0 (α0 )



◦
2
|α̂n,k
0|

+
≥

◦
◦
∗ ◦
P t2n ∂β ◦0 Mn (α̂n , 0, . . . , 0, β̂n,l
0 , . . . , β̂n,q ◦ , β̂n )β̂n,l0


→ 1,

l

+

b◦
b ◦
t2n λn,l
0 R (β̂n,l0 )

≥

 ◦ 2
◦
−bl0 λmin Γβ0 (θ0 ) |β̂n,l
0|


→1

for each k 0 ∈ {1, . . . , p◦ } and l0 ∈ {1, . . . , q ◦ }.
First, we apply Lemma 3.9 with Hn (α) = s2n Hn (α, β̂n ) to deduce P (α̂n◦ = 0) → 1. To show
the second condition, by representing the Mn -part as a sum of α-component-wise differences
and then applying the second-term Taylor expansion, we have
Hn (α̂n ) − Hn (0, α̂n∗ )
= s2n {Hn (α̂n , β̂n ) − Hn (0, α̂n∗ , β̂n )}
a◦

= s2n {Mn (α̂n , β̂n ) − Mn (0, α̂n∗ , β̂n ) + Rn (α̂n◦ )}
p◦ 
X
◦
◦
∗
◦
s2n ∂α◦0 Mn (0, . . . , 0, α̂n,k
=
0 , . . . , α̂n,p◦ , α̂n , β̂n )α̂n,k 0
k

k0 =1


s2n 2
a ◦
2
2 a◦
◦
∗
◦
◦
◦
+ ∂α◦0 Mn (0, . . . , 0, α̃n,k0 , α̂n,k0 +1 , . . . , α̂n,p◦ , α̂n , β̂n )|α̂n,k0 | + sn λn,k0 R (α̂n,k0 ) ,
2 k

(3.4)

◦
◦
where each point α̃n,k
0 is located on the segment connecting zero and α̂n,k 0 . Set Vn (α) ≡ 0 and
also the rightmost side in (3.4) to be Un (α̂n ). By the assumptions, in particular (3.3), on an
event having probability tending to 1 we can estimate as
◦

Un (α̂n ) ≥

≥

≥

p  2
X
s

2

◦
◦
◦
∗
∂α2 ◦0 Mn (0, . . . , 0, α̃n,k
0 , α̂n,k 0 +1 , . . . , α̂n,p◦ , α̂n , β̂n )
k

2



◦
◦
2
∂α2 ◦0 Mn (θ0 ) − ak0 λmin Γα0 (α0 ) + op (1) |α̂n,k
0|

n

k0 =1
p◦  2
X
sn
k0 =1
p◦ 
X
k0 =1

−

◦
ak0 λmin Γα0 (α0 )




◦
2
|α̂n,k
0|

k




1
α◦
◦
2
− ak0 λmin Γ0 (α0 ) + op (1) |α̂n,k
0| .
2

Hence it follows that for a := max1≤k0 ≤p◦ ak0 ∈ (0, 1/2) we have




p◦
X
 ◦ 2 1 1

1 1
◦
α◦
Un (α̂n ) ≥
− ak0 λmin Γ0 (α0 ) |α̂n,k0 | ≥
− a λmin Γα0 (α0 ) |α̂n◦ |2 ,
2 2
2 2
0
k =1

verifying the second condition in Lemma 3.9 with ρ = 2: we get P (α̂n◦ = 0) → 1. To deduce
P (β̂n◦ = 0) → 1, set
Hn (β) = t2n {Hn (α̂n , β) − Hn (α̂n , 0, β ∗ )}
and follow the same way as above. We have thus obtained the sparse consistency:
9

Theorem 3.12. We have P (θ̂n◦ = 0) → 1 under Assumptions 3.1, 3.6 and 3.11.
Remark 3.13. Under the tightness of (ûn , v̂n ) we have
◦
◦
∗
sn ∂α◦0 Mn (0, . . . , 0, α̂n,k
0 , . . . , α̂n,p◦ , α̂n , β̂n ) = Op (1),
k

◦
◦
∗
tn ∂β ◦0 Mn (α̂n , 0, . . . , 0, β̂n,l
0 , . . . , β̂n,q ◦ , β̂n ) = Op (1),

k 0 = 1, . . . , p◦ ,
l0 = 1, . . . , q ◦ .

l

(3.5)
(3.6)

This facts should be effectively used for verification of Assumption 3.11, as in [23, Section 5].
As a concrete example, let us consider Assumption 3.11 for the bridge-type regularization
◦

a◦
Rn (α̂n◦ )

=

p
X

◦

◦
γa
λa◦
n,k0 |α̂n,k0 | ,

b◦
Rn (β̂n◦ )

=

k0 =1

q
X

◦ γb
λb◦
n,l0 |β̂n,l0 | ,

l0 =1

b◦
where λa◦
n,k0 , λn,l0 > 0 and γa , γb ∈ (0, 1] are nonrandom tuning parameters with
γa
min
(λa◦
n,k0 sn ) → ∞,
0

γb
min
(λb◦
n,l0 tn ) → ∞.
0

k

l

(3.7)

◦ | = O (1), |t−1 β̂ ◦ | = O (1), (3.5), and (3.6), then for any γ ∈ (0, 1] we have
If further |sn−1 α̂n,k
0
p
a
p
n
n,l0
◦
◦
∗
◦
2 a◦
◦
γa
s2n ∂α◦0 Mn (0, . . . , 0, α̂n,k
0 , . . . , α̂n,p◦ , α̂n , β̂n )α̂n,k 0 + sn λn,k 0 |α̂n,k 0 |
k

◦
α̂n,k
0
1
◦
◦
∗
= sn ∂α◦0 Mn (0, . . . , 0, α̂n,k
,
.
.
.
,
α̂
,
α̂
,
β̂
)
◦
0
n
n,p
n
◦
◦
k
|α̂n,k0 | |s−1
n α̂n,k0 |

◦
−2 a◦
◦
γa
◦
2
+ s2n |α̂n,k
λn,k0 |α̂n,k
|α̂n,k
0|
0|
0|


1
1
2
◦
γa
◦
|α̂n,k
|
|α̂
= Op (1) −1 ◦
+ −1 ◦ 2 λa◦
0|
0
0
n,k
n,k
|sn α̂n,k0 | |sn α̂n,k0 |



1
2
◦
γa −1 γa a◦
−1 ◦
=
sn λn,k0 |α̂n,k
Op (1) + |sn α̂n,k0 |
0| .
◦
|s−1
n α̂n,k0 |

(3.8)

Therefore Assumption 3.11 holds. For the β-part, we can follow the same way as above. The
condition (3.7) might turn out to be too much to ask if we look at the coefficient in front of
◦ |2 ” in (3.8) more precisely, whenever available. It should be noted that the above argument
“|α̂n,k
0
may apply for more general regularization terms: see the conditions (4.3) and (4.4) in Section
4.
3.1.4

Asymptotic non-degenerate distribution

We now proceed to derivation of an asymptotic non-trivial distribution of


∗
∗
−1 ∗
∗
(α̂
−
α
),
t
(
β̂
−
β
)
.
ŵn∗ = (û∗n , v̂n∗ ) := s−1
n
n
0
n
n
0
−1
Let Assumptions 3.1, 3.6 and 3.11 hold, so that we have both (s−1
n (α̂n −α0 ), tn (β̂n −β0 )) = Op (1)
and P (θ̂n◦ = 0) → 1 in hand. By the definition, the variable ŵn∗ is a minimizer of the random
function

Mn (u∗ , v ∗ ) := Hn (α̂n◦ , α0∗ + sn u∗ , β̂n◦ , β0∗ + tn v ∗ ) − Hn (α̂n◦ , α0∗ , β̂n◦ , β0∗ )
a∗

b∗

= M0n (u∗ , v ∗ ) + ∆Rn (u∗ ) + ∆Rn (v ∗ ),

10

where we wrote
M0n (u∗ , v ∗ ) = Mn (α̂n◦ , α0∗ + sn u∗ , β̂n◦ , β0∗ + tn v ∗ ) − Mn (α̂n◦ , α0∗ , β̂n◦ , β0∗ ),
a∗

a∗

b∗

b∗

a∗

∆Rn (u∗ ) = Rn (α0∗ + sn u∗ ) − Rn (α0∗ ),
b∗

∆Rn (v ∗ ) = Rn (β0∗ + tn v ∗ ) − Rn (β0∗ ).
Let Dn := diag(sn Ip∗ , tn Iq∗ ) and w∗ := (u∗ , v ∗ ). Concerning M0n , we apply the third-order
Taylor expansion under the present assumptions to obtain, for a suitable random point (α̃n∗ , β̃n∗ ),
1
M0n (u∗ , v ∗ ) = Dn ∂θ∗ Mn (α̂n◦ , α0∗ , β̂n◦ , β0∗ )[w∗ ] + Dn ∂θ2∗ Mn (α̂n◦ , α0∗ , β̂n◦ , β0∗ )Dn [w∗ , w∗ ]
2


1 3
◦
∗
◦
∗
+ ∂θ∗ Mn (α̂n , α̃n , β̂n , β̃n ) (Dn w∗ )⊗3
6
1
(3.9)
= ∆n (θ0 )[w∗ ] + Γn (θ0 )[w∗ , w∗ ] + op (1),
2
where
Γn (θ0 ) := Dn ∂θ2∗ Mn (θ0 )Dn ,

∆n (θ0 ) := Dn ∂θ∗ Mn (θ0 ),

and where the small-order symbol op (1) in (3.9) is valid uniform in w∗ over each compact set
∗
∗
in Rp × Rq . Here in the second equality in (3.9), the sparse consistency of θ̂n◦ = (α̂n◦ , β̂n◦ ) was
used to extract ∆n (θ0 ) and Γn (θ0 ) through the mean-value theorem.
We need the joint weak convergence of these random sequences.
a∗

Assumption 3.14. There exist random variables ∆0 and Γ0 , and random functions ∆R0 (u∗ )
b∗
and ∆R0 (v ∗ ) such that




a∗
b∗
a∗
b∗
L
∆n (θ0 ), Γn (θ0 ), ∆Rn (·), ∆Rn (·) −
→ ∆0 , Γ0 , ∆R0 (·), ∆R0 (·)
∗

∗

in C(K0 × K1 ) for every compact K0 × K1 ⊂ Rp × Rq , and that the random function
1
a∗
b∗
M0 (u∗ , v ∗ ) := ∆0 [w∗ ] + Γ0 [w∗ , w∗ ] + ∆R0 (u∗ ) + ∆R0 (v ∗ )
2
has an a.s. unique minimum at (u∗ , v ∗ ) = (û∗0 , v̂0∗ ).
The argmax theorem concludes the following.
L

Theorem 3.15. We have (û∗n , v̂n∗ ) −
→ (û∗0 , v̂0∗ ) under Assumptions 3.1, 3.6, 3.11 and 3.14.
Note that under the foregoing assumptions, it follows from (3.9) that M0n has the LAQ
structure:
1
M0n (u∗ , v ∗ ) = ∆n (θ0 )[w∗ ] + Γ0 [w∗ , w∗ ] + rn (w∗ ),
2
p

∗ +q ∗

where the random function rn satisfies that supw∗ ∈K |rn (w∗ )| →
− 0 for each compact K ⊂ Rp

.

We now specialize the previous result by focusing on the case of asymptotically mixednormally distributed θ̂n∗ . To this end, we introduce some further notation and regularity conditions. For random elements µ = µ(ω) and Σ = Σ(ω) we use the symbol “M N (µ, Σ)” for a
“mixed-normal” distribution corresponding to the characteristic function



1
z 7→ E exp iµ[u] − Σ[u, u] ;
2
we may consider an extension of the original probability space whenever necessary. For random
Ls
variables Xn and X0 , we say that Xn (F-)stably converges in law to X0 , denoted by Xn −→
X0 ,
p

L

if (Xn , Fn ) −
→ (X0 , F0 ) for any F-measurable Fn such that Fn →
− F0 (see [15] for details).
11

Assumption 3.16.
L

s
1. ∆n (θ0 ) −→
∆0 ∼ M N (µ, Σ) for some (F-measurable) random variables µ ∈ Rp
∗
∗
∗
∗
p
+q
Σ∈R
⊗ Rp +q with the latter being a.s. positive definite;

∗ +q ∗

and

p

2. Γn (θ0 ) →
− Γ0 with P (Γ0 > 0) = 1.
3. The functions Ra and Rb are twice differentiable on R\{0}, and for each k 00 ∈ {1, . . . , p∗ }
a∗
b∗
and l00 ∈ {1, . . . , q ∗ } there correspond nonnegative random variables λ0,k00 and λ0,l00 such
p

a∗

p

b∗

b∗ →
that sn λa∗
− λ0,k00 and tn λn,l
00 − λ0,l00 .
n,k00 →

Corollary 3.17. Assume that we have Assumptions 3.1, 3.6, 3.11, and 3.16. Then, any minimizer ŵn∗ of Mn satisfies that
∗
ŵn∗ = −Γ−1
0 (∆n (θ0 ) + Λ ) + op (1),

(3.10)

where Λ∗ = (Λa∗ , Λb∗ )> with

>
a∗
a∗
∗
∗
Λa∗ := λ0,1 ∂Ra (α0,1
), . . . , λ0,p∗ ∂Ra (α0,p
,
∗)
>

b∗
b∗
∗
∗
), . . . , λ0,q∗ ∂Rb (β0,q
Λb∗ := λ0,1 ∂Rb (β0,1
.
∗)

(3.11)
(3.12)

In particular, we have the asymptotic mixed normality:

L
−1
−1
∗
.
(û∗n , v̂n∗ ) −
→ (û∗0 , v̂0∗ ) ∼ M N −Γ−1
0 (µ + Λ ), Γ0 ΣΓ0
Proof. Write Fn (θ∗ ) = Hn (α̂n◦ , α∗ , β̂n◦ , β ∗ ) − Hn (α̂n◦ , α0∗ , β̂n◦ , β0∗ ). Under the assumptions, on an
event whose probability tending to 1 it follows from the mean-value theorem that

−Dn ∂θ2∗ Fn (θ̌n )Dn [ŵn∗ ] = Dn ∂θ∗ Fn (θ0∗ )
(3.13)
p

for some random point θ̌n such that θ̌n →
− θ0∗ . Moreover, for suitable points (α̌n∗ , β̌n∗ ),
Dn ∂θ∗ Fn (θ0∗ ) = ∆n (θ0 ) + Λ∗ + op (1),


a∗
b∗
−Dn ∂θ2∗ Fn (θ̌n )Dn = −Γn (θ0 ) − diag s2n ∂α2 ∗ Rn (α̌n∗ ), t2n ∂β2∗ Rn (β̌n∗ ) + op (1)
= −Γ0 + op (1),
where, as we did in (3.9) before, the sparse consistency of θ̂n◦ was used to extract ∆n (θ0 ) and
Γn (θ0 ) in the right-hand sides. Hence (3.10) follows from (3.13). Then the latter claim is
trivial.
3.1.5

Uniform tail-probability estimate

We now proceed to the tail-probability estimate of the scaled estimator ŵn := (ûn , v̂n ), the
main concern of this paper; recall the definition (3.1). To this end, we will make use of the
PLDI result in [33], which is, though not specified, originally developed for the LAQ statistical
random fields. Throughout this section we assume that the matrix Γ0 is a.s. constant. Write
u = (u◦ , u∗ ) and v = (v ◦ , v ∗ ). We will proceed with the multi-step PLDI argument in [33] (see
also [19, Section 4]), which roughly goes as follows:

12

• At the first step we consider the mixed-rates random field
Mn (u, β) := Hn (α0 + sn u, β) − Hn (α0 , β)
a◦

a∗

a∗

= Mn (α0 + sn u, β) − Mn (α0 , β) + Rn (sn u◦ ) + Rn (α0∗ + sn u∗ ) − Rn (α0∗ )
◦
−2
=: s−2
n fn (u ) + tn gn (u, β),

where
a◦

◦
◦
s−2
n fn (u ) := Rn (sn u ),
a∗

a∗

∗
∗
∗
t−2
n gn (u, β) := Mn (α0 + sn u, β) − Mn (α0 , β) + Rn (α0 + sn u ) − Rn (α0 )

with regarding β as a nuisance parameter. We will derive the α-PLDI
sup sup rL P (|ûn | ≥ r) < ∞
r

n

through proving
"

#

L

sup sup r P

sup

r>0 n>0

{−Mn (u, β)} ≥ 0 < ∞

(3.14)

(u,β)∈{|u|≥r}×Θβ

for any L > 0, according to the facts that the probability in (3.14) is equal to or greater
than P (|ûn | ≥ r), that Mn (0, β) = 0, and that supβ∈Θβ {Hn (α0 , β) − Hn (α̂n , β)} ≥ 0.
• At the second step, by plugging-in α̂n we look at the mixed-rates random field
Mn (v) := Hn (α̂n , β0 + tn v) − Hn (α̂n , β0 )
b◦

b∗

b∗

= Mn (α̂n , β0 + tn v) − Mn (α̂n , β0 ) + Rn (tn v ◦ ) + Rn (β0∗ + tn v ∗ ) − Rn (β0∗ )
◦
−2
=: s−2
n fn (v ) + tn gn (v),

where
b◦

◦
◦
s−2
n fn (v ) := Rn (tn v ),
b∗

b∗

∗
∗
∗
t−2
n gn (v) := Mn (α̂n , β0 + tn v) − Mn (α̂n , β0 ) + Rn (β0 + tn v ) − Rn (β0 ).

Then, we will derive the β-PLDI
sup sup rL P (|v̂n | ≥ r) < ∞
r

n

from
"

#

sup sup rL P

sup {−Mn (v)} ≥ 0 < ∞.

r>0 n>0

|v|≥r

(3.15)

Combining (3.14) and (3.15) yields the joint-PLDI
sup sup rL P (|ŵn | ≥ r) < ∞,
r>0 n>0

so that the uniform tail-probability estimates of ŵn follows.

13

(3.16)

We see that the verification of α-PLDI reduces to that of the PLDI of the form
"
#
sup sup rL P
r>0 n>0

{−t−2
n gn (u, β)} ≥ 0 < ∞,

sup

(3.17)

(u,β)∈{|u|≥r}×Θβ

since we have
a◦

inf
u∈{|u|≥r}

Rn (sn u◦ ) ≥ 0

a.s.

and therefore
sup

{−Mn (u, β)} ≥ 0 ⇐⇒

(u,β)∈{|u|≥r}×Θβ

{−t−2
n gn (u, β)} ≥

sup
(u,β)∈{|u|≥r}×Θβ

=⇒

sup

inf
u∈{|u|≥r}

◦
s−2
n fn (u )

{−t−2
n gn (u, β)} ≥ 0.

(u,β)∈{|u|≥r}×Θβ

Typically −t−2
n gn (u, β) fails to be LAQ because of the presence of the regularization term
a∗ ∗
a∗
∗
Rn (α0 +sn u )−Rn (α0∗ ). However, we can apply the argument in [33] with regarding −t−2
n gn (u, β)
as “LAQ-like” structure (see (3.20)–(3.21) below). Similarly, the PLDI
"
#
sup sup rL P
r>0 n>0

sup

{−t2n gn (v)} ≥ 0 < ∞

(3.18)

v∈{|v|≥r}

verifies the β-PLDI through (3.15) since we have
inf
v∈{|v|≥r}

b◦

Rn (tn v ◦ ) ≥ 0

a.s.

First we set some regularity conditions on the part Mn .
Assumption 3.18.
fa : Θα → R and M
fb : Θβ → R, and positive constants
1. There exist nonrandom functions M
0
0
a
b
a
b
δ1 , δ1 , χ and χ such that for all K > 0,

n
o K

−2δ1a
2
a
f
sn Mn (α, β) − Mn (α0 , β) − M0 (α)
• supn>0 E supθ∈Θ sn
< ∞.

n
o K

−2δ b
fb (β)
< ∞.
• supn>0 E supβ∈Θβ tn 1 t2n Mn (α0 , β) − Mn (α0 , β0 ) − M
0
fa (α) ≥ χa |α − α0 |2 ,
• M
0

fb (β) ≥ χb |β − β0 |2 .
M
0

2. There exist nonrandom matrices C0 (β) and C0 > 0, and constants δ2a , δ2b ∈ (0, 1/2] such
that for all K > 0,
• inf β λmin (C0 (β)) > 0.


K 
−2δ2a 2 2
• supn>0 E supβ∈Θβ sn
sn ∂α Mn (α0 , β) − C0 (β)
< ∞.

K 
−2δ2b 2 2
• supn>0 E tn
tn ∂β Mn (α0 , β0 ) − C0
< ∞.


K
2(1−δ2b )
2
• supn>0 E supα∈Θα sn tn
∂α ∂β Mn (α, β0 )
< ∞.

14

3. For all K > 0,
h
i
h
i
• supn>0 E supβ∈Θβ |sn ∂α Mn (α0 , β)|K < ∞, supn>0 E |tn ∂β Mn (α0 , β0 )|K < ∞.


i
h
K
K
2
3
2
3
< ∞, supn>0 E supθ∈Θ tn ∂β Mn (θ)
< ∞.
• supn>0 E supθ∈Θ sn ∂α Mn (θ)


• supn>0 E supα∈Θα |sn tn ∂α ∂β Mn (α, β0 )|K < ∞.
Assumption 3.18 is borrowed from [33, Theorem 3(c)]. Although the description may seem
to be stringent at first glance, quite often we may verify them in a straightforward manner
under suitable stability conditions such as the (exponential) ergodicity and the boundedness of
moments; see [33, Section 6] and also Section 4 below. Further, we should note that the present
regularity conditions could be relaxed in compensation for more messy description; see Remark
3.21.
We also need moment conditions on the regularization terms.
Assumption 3.19. There exist constants ν a , ν b ∈ (0, 1/2) such that the following conditions
hold for any K > 0,
"
#



K

b
a
K
a
b
< ∞, sup E sup t1+2ν
1. sup E sup s1+2ν
< ∞;
Rn (α)
Rn (β)
n
n
n>0

n>0

α∈Θα

K
2. sup max
E[|sn λa∗
n,k00 | ] < ∞,
00
n>0 k

β∈Θβ

K
sup max
E[|tn λb∗
n,l00 | ] < ∞.
00
n>0

l

The following theorem gives us the joint-PLDI (3.16).
Theorem 3.20. For any L > 0, (3.16) holds under Assumptions 3.18 and 3.19. AdditionL

→ (û◦0 , û∗0 , v̂0◦ , v̂0∗ ) for some random vector
ally if we have the weak convergence (û◦n , û∗n , v̂n◦ , v̂n∗ ) −
◦
∗
◦
∗
(û0 , û0 , v̂0 , v̂0 ), then the moment convergence
E[f (ûn , v̂n )] → E[f (û◦0 , û∗0 , v̂0◦ , v̂0∗ )]

(3.19)

holds for all continuous f : Rp+q → R of at most polynomial growth.
Proof. At the first step, we will derive the PLDI (3.17). The Taylor expansion around u = 0
yields
1
−t−2
n gn (u, β) = ∆n (β)[u] − C0 (β)[u, u] + rn (u, β),
2
where
∆n (β) := −sn ∂α Mn (α0 , β),
Z 1

rn (u, β) := −
(1 − s) s2n ∂α2 Mn (α0 + sn su, β) − C0 (β) ds [u, u]

(3.20)

0
a∗

a∗

− {Rn (α0∗ + sn u∗ ) − Rn (α0∗ )}.

(3.21)

We will check the conditions [A1], [A4], [A6], [B1] and [B2] of [33, Theorem 1] (see Appendix A
for details).
From Assumptions 3.18 and 3.19, [B1] is trivial and for any θ ∈ Θ we have
Yan (α, β) := −s2n {Hn (α, β) − Hn (α0 , β)}
a

a∗

= −s2n {Mn (α, β) − Mn (α0 , β)} − s2n {Rn (α) − Rn (α0∗ )}
15

p
f0a (α) =: Ya0 (α) ≤ −χa |α − α0 |2 ,
→
− −M

hence [B2] also holds. We will derive [A1] and [A6], along with controlling the auxiliary parameters given in [A4]; for clarity, we will put the superscript “a” or “b” for them. In the sequel,
we will write xn . yn if supn (xn /yn ) < ∞.
First, we will verify [A1]. We have
Z

1


(1 − s) s2n ∂α2 Mn (α0 + sn su, β) − C0 (β) ds [u, u]
0


Z 1
Z 1
2 2
2 3
=
(1 − s) sn ∂α Mn (α0 , β) − C0 (β) +
sn ∂α Mn (α0 + sn stu, β) dt [sn su] ds [u, u] ,
0

0

hence
|rn (u, β)|
|u|2
s2 ∂ 2 Mn (α0 , β) − C0 (β)
.
1 + |u|2
1 + |u|2 n α
Z 1Z 1
|u|2
+
sn |u|
s2n ∂α3 Mn (α0 + sn stu, β) dtds
1 + |u|2
0
0
1
a∗
a∗
+
R (α∗ + sn u∗ ) − Rn (α0∗ ) .
1 + |u|2 n 0

(3.22)

Let

Una (r) := u ∈ Rp ; α0 + sn u ∈ Θα ,

r ≤ |u| ≤ sn−(1−ζ

a)

,

where ζ a ∈ (0, 1). From (3.22) and the Chebyshev’s inequality, the probability in [A1] can be
estimated as follows:
!
|rn (u, β)|
−ρa
P
sup
≥r 1
2
(u,β)∈Una (r)×Θβ 1 + |u|
( "
d #

2
|u|
2
2
dρa
sn ∂α Mn (α0 , β) − C0 (β)
.r 1 E
sup
2
(u,β)∈Una (r)×Θβ 1 + |u|
"
d #

Z 1Z 1
|u|2
+E
sup
s2n ∂α3 Mn (α0 + sn stu, β) dtds
s |u|
2 n
0
0
(u,β)∈Una (r)×Θβ 1 + |u|
"
d #)

1
a∗ ∗
a∗
Rn (α0 + sn u∗ ) − Rn (α0∗ )
(3.23)
+E
sup
2
u∈Una (r) 1 + |u|
for some d > 0. Letting ζ a ∈ (0, min{δ1a , 2δ2a , 1/2}) and ξ ∈ (0, ζ a /(1 − ζ a )). Note that −ζ a /2 +
(1 − ζ a )ξ/2 < 0, and therefore −δ2a + (1 − ζ a )ξ/2 < 0. Then, for the first term of the right-hand
side in (3.23), it follows from Assumption 3.18 that
"

d #
|u|2
sup E
sup
s2n ∂α2 Mn (α0 , β) − C0 (β)
2
1
+
|u|
a
n>0
(u,β)∈Un (r)×Θβ
#
( "

d )

d
|u|2 2δ2a ξ −ξ
−2δ2a 2 2
= sup E sup sn
sn ∂α Mn (α0 , β) − C0 (β)
sup
s |u| |u|
2 n
n>0
β∈Θβ
u∈Una (r) 1 + |u|
−2{−δ2a +(1−ζ a )ξ/2}d

. r−dξ sup sn
n>0

. r−dξ .

(3.24)

16

Further, for the second term of the right-hand side in (3.23), under Assumption 3.18 we obtain
the estimate:
"

d #
Z 1Z 1
|u|2
s |u|
s2n ∂α3 Mn (α0 + sn stu, β) dtds
sup E
sup
2 n
1
+
|u|
a
n>0
0
0
(u,β)∈Un (r)×Θβ

d
2
|u|
a
a
. sup sup
sn |u||u|ξ |u|−ξ
. r−dξ sup sn−2{−ζ /2+(1−ζ )ξ/2}d . r−dξ .
(3.25)
2
n>0 u∈Una (r) 1 + |u|
n>0
As for the third term of the right-hand side in (3.23), the local Lipschitz continuity (2.5) and
Assumption 3.19 yield
"
d #

1
a∗ ∗
a∗ ∗
∗
Rn (α0 + sn u ) − Rn (α0 )
sup E
sup
2
n>0
u∈Una (r) 1 + |u|


d 
∗
p
X
1

∗  
. sup E  sup 
λa∗

n,k00 sn |uk00 |
2
1
+
|u|
a
n>0
u∈Un (r)
00
k =1



∗

. sup E  sup

p
X

∗
λa∗
n,k00 sn |uk00 |

 . r−d .

1 + |u|2

u∈Una (r) k00 =1

n>0

!d 
(3.26)

Fix a ρa1 ∈ (0, ξ). Then from (3.23)–(3.26), we get for any L > 0
sup sup rL P
r>0 n>0

sup
(u,β)∈Una (r)×Θβ

|rn (u, β)|
a
≥ r−ρ1
2
1 + |u|

!
< ∞.

For condition [A4], we take ρa2 > 2ζ a and ν1a > 0 such that 1 − 2ν1a − ρa2 > 0. Then, the
condition [A6] is trivial since we have Assumptions 3.18 and 3.19, and
a
a∗
fa (α),
Yan (α, β) − Ya0 (α) = −s2n {Mn (α, β) − Mn (α0 , β)} − s2n {Rn (α) − Rn (α0∗ )} + M
0

i.e. we get

sup E 
n>0

"
sup E
n>0

!N1 
 < ∞,
sup |∆n (β)|

β∈Θβ

−2(1/2−ν1a ) a
sup sn
|Yn (θ)
θ∈Θ

N2 #

−

Ya0 (α)|

< ∞,

for N1 = L(1 − ρa1 )−1 > 0 and N2 = L(1 − 2ν1a − ρa2 )−1 > 0. Therefore [A4] holds with taking
the tuning parameters in [A4] as above. The proof of the α-PLDI (3.17) is complete.
At the second step of the multistep-PLDI argument for showing the β-PLDI (3.18), we can
take an analogous way to the derivation of the α-PLDI. We have
1
−t−2
n gn (v) = ∆n [u] − C0 [u, u] + rn (v),
2
where
∆n := −tn ∂β Mn (α̂n , β0 ),
Z 1

b∗
b∗
rn (v) := −
(1 − s) t2n ∂β2 Mn (α̂n , β0 + tn sv) − C0 ds [v, v] − {Rn (β0∗ + tn v ∗ ) − Rn (β0∗ )}.
0

17

Verification of [B1] and [B2] are the same as before: for any β ∈ Θβ , we have
Ybn (β) := −t2n {Hn (α̂n , β) − Hn (α̂n , β0 )}
b

b∗

= −t2n {Mn (α̂n , β) − Mn (α̂n , β0 )} − t2n {Rn (β) − Rn (β0∗ )}
p

f0b (β) =: Yb0 (β) ≤ −χb |β − β0 |2 .
→
− −M
Through expanding the ds-integrand of rn (v), we get the following as in (3.22):
Z 1Z 1
|v|2
|v|2
|rn (v)|
2 2
t ∂ Mn (α̂n , β0 ) − C0 +
t2n ∂β3 Mn (α̂n , β0 + tn stv) dtds
.
tn |v|
1 + |v|2
1 + |v|2 n β
1 + |v|2
0
0
1
b∗
b∗
Rn (β0 + tn v ∗ ) − Rn (β0∗ ) .
+
2
1 + |v|

−(1−ζ b )
Introducing Unb (r) = v ∈ Rq ; β0 + tn v ∈ Θβ , r ≤ |v| ≤ tn
for ζ b ∈ (0, 1), we can follow
the same line as in the derivation of (3.23)–(3.26) as before, to verify that for any L > 0
!
|rn (v)|
L
−ρb1
sup sup r P
sup
≥r
< ∞,
2
r>0 n>0
v∈Unb (r) 1 + |v|
for some number ρb1 ∈ (0, ξ) with ξ ∈ (0, ζ b /(1 − ζ b )) for ζ b ∈ (0, min{δ1b , 2δ2b , 1/2}). Note that
we made use of the moment bound
h
i
K
sup E s−1
(
α̂
−
α
)
< ∞ for all K > 0,
n
0
n
n>0

ensured by the inequality supr supn rL P (|ûn | ≥ r) < ∞ which has been previously obtained
from (3.17). Verification of [A6] along with [A4] can be done in the same ways as before. The
proof of the β-PLDI (3.18) is complete, concluding (3.16). The latter claim of the theorem is
trivial.
Remark 3.21. We note that, in view of [33, Theorem 1], the twice differentiability of the LAQ
part Mn and the constancy of Γ0 are not essential. All the assertions presented in this section can
also go for possibly non-differential Mn as long as statistical random fields associated with Mn
is of LAQ: Mn (θ0 + An u) − Mn (θ0 ) = ∆n [u] + 12 Γ0 [u, u] + rn (u) with (quasi-)score sequence ∆n ,
asymptotic (quasi-)information matrix Γ0 , and remainder term rn (u) = op (1) (locally uniformly
in u). The resulting set of conditions becomes somewhat less concise.

3.2

Standard case

In Section 3.1 we have focused on the sparse asymptotics, where the underlying statistical
random field is of mixed-rates type. Nevertheless, it is possible to treat the standard asymptotics
as well, where the localization via matrix (1.1):
Mn (u; θ0 ) = Hn (θ0 + An (θ0 )u) − Hn (θ0 ),
can completely determine the asymptotic distribution of θ̂n . In this case, we cannot take the
sparse consistency into account even if the true parameter θ0 has zero components. We note that
the random field Mn (·; θ0 ) still may not be LAQ according to choice of the regularization terms;
for example, the non-sparse asymptotics for the linear regression considered in [17, Theorem 2]
belongs to this standard-asymptotics category, while, for example, the ridge-type regularization
[13] under suitable stochastic-order conditions on the regularization coefficients λan,k and λbn,l
(recall (2.2)) leads to the LAQ structure (most often the locally asymptotically mixed-normal
one).
18

In the standard asymptotics, the underlying statistical random field is normally not of the
mixed-rates type. It is of course possible and easier to state results analogous to Theorems
3.4, 3.8, 3.15, Corollary 3.17, and Theorem 3.20, simply by removing the factor θ◦ , hence in
particular Assumption 3.11, in the foregoing arguments.
Let us be more specific. We consider (2.1) and (2.2) for the random function Hn consisting
a
b
of Mn (θ), Rn (α), and Rn (β) as before, except that we here do not impose the factorization
◦
∗
θ = (θ , θ ); the true value θ0 may be such that θ0,k 6= 0 for all k ∈ {1, . . . , p + q}, and we
no longer consider the decompositions (2.3) and (2.4). Then, by inspecting the corresponding
proofs in Section 3.1, it is straightforward to observe the following.
• The assertion of Theorem 3.4 holds as it is.
• Define Assumption 3.6? to be the same one as Assumption 3.6 except that Assumption
3.6.2 therein is replaced by “sn λan,k = Op (1) and tn λbn,l = Op (1) for each k and l”. Then,
the assertion of Theorem 3.8 holds under Assumptions 3.1 and 3.6? .
−1
• Turning to asymptotic distribution of (ûn , v̂n ) = (s−1
n (α̂n − α0 ), tn (β̂n − β0 )), we first note
that under Assumptions 3.1 and 3.6? we have

Mn (u, v) := Hn (α0 + sn u, β0 + tn v) − Hn (α0 , β0 )
1
a
b
= ∆n (θ0 )[w] + Γn (θ0 )[w, w] + ∆Rn (u) + ∆Rn (v) + op (1),
2
where w := (u, v) and we wrote, different from Section 3.1.4, ∆n (θ0 ) := Dn ∂θ Mn (θ0 )
a
a
and Γn (θ0 ) := Dn ∂θ2 Mn (θ0 )Dn for Dn := diag(sn Ip , tn Iq ), and also ∆Rn (u) := Rn (α0 +
a
b
b
b
sn u) − Rn (α0 ) and ∆Rn (v) := Rn (β0 + tn v) − Rn (β0 ). Let us define Assumption 3.14?
to be the same one as Assumption 3.14 with all the symbol “∗” therein removed from
the notation. Then, as a variant of Theorem 3.15, we have the following claim: we have
L
(ûn , v̂n ) −
→ (û0 , v̂0 ) under Assumptions 3.1, 3.6? , and 3.14? . Of course, the asymptotic
distribution may not be mixed-normal, as in the non-Gaussian limit specified in [17] for
the linear regression model. An analogue to Corollary 3.17 can be deduced similarly under
appropriate conditions.
• Finally as for Theorem 3.20, we partly modify Assumption 3.19 by removing the symbol
“∗” therein from the notation, and call it Assumption 3.19? . Then the claim is as follows:
under Assumptions 3.18 and 3.19? we have (3.16) for any L > 0, and additionally if we
L

have the weak convergence (ûn , v̂n ) −
→ (û0 , v̂0 ) for some random vector (û0 , v̂0 ), then the
moment convergence
E[f (ûn , v̂n )] → E[f (û0 , v̂0 )]
holds for all continuous f : Rp+q → R of at most polynomial growth.

3.3
3.3.1

Discussion
Specific choice of regularization term

P◦
Pp∗
a
a◦
a∗
a ◦
a∗
a ∗
Let us recall (2.3) and (2.4): Rn (α) = Rn (α◦ )+Rn (α∗ ) = pk0 =1 λa◦
n,k0 R (αk0 )+ k00 =1 λn,k00 R (αk00 )
P◦
Pq ∗
b
b◦
b∗
b ◦
b∗
b ∗
and Rn (β) = Rn (β ◦ ) + Rn (β ∗ ) = ql0 =1 λb◦
n,l0 R (βl0 ) +
l00 =1 λn,l00 R (βl00 ). As a simple explicit
component-wise regularization, we remark on a formal selection of the regularization terms
a
b
Rn (α) and Rn (β) in terms of maximizing the posterior probability in Bayesian setting. We will
think of the LAQ part Mn (θ) as the minus of a (quasi) log-likelihood quantity, with the prior

19

distribution of (α, β) of mutually independent components. Namely, we consider prior density,
say π a (α; λa ) and π b (β; λb ) of α and β, respectively, of the forms
◦

a

a

π (α; λ ) =

π b (β; λb ) =
a

∗

p
Y

C

a

(λa◦
k0 ) exp

−

a ◦
λa◦
k0 R (αk0 )



·

p
Y

k0 =1
q◦
Y

k00 =1
q∗
Y

l0 =1

l00 =1


b ◦
C b (λlb◦0 ) exp − λb◦
l0 R (βl0 ) ·


a∗ a ∗
C a (λa∗
k00 ) exp − λk00 R (αk00 ) ,


b∗ b ∗
C b (λb∗
l00 ) exp − λl00 R (βl00 ) .

b

We then set Rn (α) and Rn (β) to be − log π a (α; λan ) and − log π b (β; λbn ) up to additive constants
which do not depend on (α, β). At this stage, the functions Ra and Rb , and the regularization
a∗
b
b◦
b∗
intensity (possibly random) sequences λan = {(λa◦
n,k0 )k0 , (λn,k00 )k00 } and λn = {(λn,l0 )l0 , (λn,l00 )l00 }
are left unspecified. The functions Ra and Rb determine the class of regularization (such as the
ridge type, the lasso type, and so on). We suppose they are given and discuss about choice of
λan and λbn in computing θ̂n = θ̂n (λ); how to choose them is in general an unavoidable and very
often annoying problem in regularized estimation, and there are a lot of possibilities of reasoning
on this point from different points of view.2
As in [31], we take the root of the following equation with respect to λ = (λa , λb ):


∂λ Hn (θ) = −∂λa log π a (α; λa ), −∂λb log π b (β; λb ) = 0
a ◦
⇐⇒ ∂λa◦0 C a (λa◦
k0 ) = R (αk0 ),
k

b ◦
∂λb◦0 C b (λb◦
l0 ) = R (βl0 ),
l

a ∗
∂λa∗00 C a (λa∗
k00 ) = R (αk00 ),
k

b ∗
∂λb∗00 C b (λb∗
l00 ) = R (βl00 ).

(3.27)

l

That is to say, we formally regard λ = λ(θ) maximizing the posterior as an optimal choice.
We do not know how to set the value θ a priori, and want to use the non-regularized estimator
θ̃n = (θ̃n,k )k := θ̂n (0); then we plug-in the estimate θ̃n into (3.27) and solve the p+q simultaneous
equations with respect to λ, which produces a simple explicit data-driven choice of λn .
Let us take a closer look at the exponential power distribution, which is quite popular in
regularized estimation and corresponds to the bridge-type regularization:
p
1/γ
Y
γλk
π(θ; λ) =
exp(−λk |θk |γ )
2Γ(1/γ)
k=1

for some (given) γ > 0, where λ = (λk )k≤p , λk ≥ 0. We here only deal with γ ≤ 1. In this case,
the identity ∂λ∂ k log π(θ; λ) = 0 leads to the choice
λk =

1
.
γ|θk |γ

(3.28)

Note that this simple choice may fail to meet our technical conditions in the following two points;
let us only look at the case of αk , for the case of βl can be argued similarly.
2
Historically, development of statistical model-shrinkage methodologies goes back to the so-called L0 -type
regularization. Most popular among others are the classical model-selection criteria Akaike’s AIC (see Section
3.3.2) and Schwarz’s BIC, both of which provide theoretical bases of how to choose λ in significant-parameter
selection, or how to fix a significance level in the context of testing hypothesis in a nested model. These came
from different contexts, although the random functions to be optimized take rather similar forms: “the principal
part (typically log-likelihood)” plus “a regularization term”. See e.g. [6] for details.

20

• The first point is the stochastic-order conditions on the regularization terms. Suppose that
3
an s−1
n -consistent initial estimator (α̃n,k )k is available, then (3.28) results in

= Op (1) and not op (1) if α0,k 6= 0,



1
= Op (s−γ
if α0,k = 0 and s−1
λan,k =
(3.29)
n )
n α̃n,k 6= op (1),
γ
γ|α̃n,k | 

 p
→
− +∞
if α0,k = 0 and s−1
n α̃n,k = op (1),
for k = 1, . . . , p; typically, the first two cases are available. We should note that the rate
of convergence s−1
n of (α̃n,k )k is not essential: a different rate will just change divergence
rate of λan,k for α0,k = 0 in (3.29).
• The second one is that reciprocals of |α̃n,k | may not have finite high-order moments,
which
does matter when trying to verify Assumption 3.19: indeed, the typical case, where
√
n(α̃n,k − α0,k ) is asymptotically normally distributed, only moments of order less than
1 exist.
Below we discuss about these two points in more details.
Concerning the first point, let us first note that the first two cases in (3.29) are comparable
to the standard asymptotics (e.g. [17]), since then the random field (u, v) 7→ Hn (α0 + sn u, β0 +
tn v)−Hn (α0 , β0 ) can completely describe the asymptotic distribution of the regularized estimator
(α̂n , β̂n ). To consider the sparse asymptotics, we need to modify (3.28) more or less artificially,
such as
λan,k =

ln
γ|α̃n,k |γ

(3.30)

for some sequence ln → ∞ (such as log n, as seen below), as was already mentioned [31]. We do
not have any justification for this ad-hoc choice (3.30), however, it may enable us to apply, for
example, the bridge type regularization even when we do not know beforehand if the true value
α0 indeed has a zero component or not, since each regularization weight λan,k asymptotically
tells us if α0,k is zero or not. To show this, assume additionally that
sγn ln → 0.
Then, for (3.30) we have

p
γ

− 0
if α0,k 6= 0,
 = Op (sn ln ) →
γ a
sn λn,k
ln
p

→
− +∞ if α0,k = 0.
 =
−1
γ
γ|sn α̃n,k |
Note that the condition (3.7) in Remark 3.13 is fulfilled, and also the condition sn λan,k = op (1)
for α0,k 6= 0 as well. Thus, the modification (3.30) suffices if we are only concerned with the
weak convergence plus the sparse consistency of α̂n , without the tail-probability estimate.
From the above discussion, we can conclude the following claim concerning the conditions
on the regularization terms:
−1
Corollary 3.22. Let α̃n = (α̃n,k )k and β̃n = (β̃n,l )l be s−1
n -consistent and tn -consistent esti−1
mators of α and β, respectively, such that s−1
n α̃n,k 6= op (1) and that tn β̃n,l 6= op (1). If we set
(2.2) with
a

Ra (αk ) = |αk |γ ,
3

b

Rb (βl ) = |βl |γ ,

p

For a random variable Xn , we write Xn −
→ +∞ if P (Xn > K) → 1 for every K > 0.

21

(3.31)

λan,k =

lna
,
γ a |α̃n,k |γ a

λbn,l =

lnb
γ b |β̃n,l |γ b

for some lna , lnb → ∞ and γ a , γ b ∈ (0, 1] such that
a

b

sγn lna → 0,

tγn lnb → 0,

(3.32)

then Assumptions 3.1.3, 3.6.2 and 3.16.3 hold. If further we have (3.5) and (3.6), then Assumption 3.11 holds.
To further remedy the second point, where Assumption 3.19 comes into play in, we may for
example consider a variant of (3.30) of the form
λan,k =

ln
γ(|α̃n,k | ∨ an,k )γ

for a positive nonrandom sequence an,k such that
an,k  sn
for k = 1, . . . , p, in the sense that the sequences (an,k /sn ) and (sn /an,k ) are bounded. Then
−γ
|λan,k | . ln I(α0,k 6= 0) + s−γ
n ln I(α0,k = 0) . sn ln ,

so that Assumption 3.19 is satisfied if
γ ∈ (0, 1)

and

ln sn1−γ . 1.

a λa Ra (α ) + |s λa | . l s1+2νa −γ + l s1−γ . l s1−γ . 1.
Indeed, we then have supαk s1+2ν
n n
n n
n n
n n,k
k
n
n,k
a
A possible practical choice of n,k would be

an,k = Ca sn
for a fixed constant Ca > 0, which can be arbitrary but would more or less affect finite-sample
performance. The asymptotic theory does not tell us how to choose Ca , remaining as a nontrivial matter. Nevertheless, in practice we may for example set Ca so small that Ca sn equals
a threshold, say 10−5 , for which we basically have to make a subjective choice. Also, we may
regard an estimate of αk as exactly 0 if its estimate is less than an,k . Although this kind of
thresholding seems to be quite artificial, it would be necessary in numerical optimization, which
does not in principle return exact zero but just only very small number even when the target
value equals zero.
From the above discussion about the moment conditions on the regularization terms, we
conclude the following claim:
−1
Corollary 3.23. Let α̃n = (α̃n,k )k and β̃n = (β̃n,l )l be s−1
n -consistent and tn -consistent esti−1
mators of α and β, respectively, such that s−1
n α̃n,k 6= op (1) and that tn β̃n,l 6= op (1). If we set
(2.2) with (3.31) and

λan,k =

lna
,
γ a (|α̃n,k | ∨ an,k )γ a

λbn,l =

lnb
,
γ b (|β̃n,l | ∨ bn,l )γ b

where an,k and bn,l are positive nonrandom sequences such that
an,k  sn ,
22

bn,l  tn

for any k ∈ {1, . . . , p} and l ∈ {1, . . . , q} and where lna , lnb → ∞ and γ a , γ b ∈ (0, 1) satisfy (3.32)
and
a

b

s1−γ
lna → 0,
n

tn1−γ lnb → 0,

then Assumptions 3.1.3, 3.6.2, 3.16.3 and 3.19 hold. If further we have (3.5) and (3.6), Assumption 3.11 holds.
As has been well recognized, non-convex optimization such as the case of the bridge regularization with γ < 1 can be a hard task. Toward stabilization of numerical computation,
we could incorporate existing general devices for manipulating Ra , such as the local quadratic
approximation [9].
3.3.2

Prediction-related issues

Based on our moment-convergence results, it is straightforward to verify many kinds of predictionerror asymptotics, such as the asymptotic behavior of the mean-squared error E(|θ̂n − θ0 |2 ) =
V0 /n + o(1/n) in its typical form; among many others, we refer to [7] and [14] for some related
rigorous treatments. We may also apply our moment-convergence results to validate the celebrated AIC methodology ([3], and also [10], [26], and [27]) even under the sparse asymptotics.
Recent studies exactly in this direction contain [25] and [29], where the uniform integrability of
the sparse maximum-likelihood estimator with the bridge-like regularization played an important role for validating the asymptotic bias correction. Below we will briefly discuss how we
can extend the result of [29] to cover a broader range of statistical models with locally asymptotically normal structure. To keep things simple, we are only concerned here with correctly
specified models, keeping the multi-scaling setup with θ = (α, β); again we note that cases of
single scaling can be handled as well without any essential change, just by ignoring the β-part.
We keep adopting the basic setup and notation introduced in Section 2. In what follows, we
focus on a regularized minus log-likelihood function (2.1):
a

b

Hn (θ) = Hn (α, β) = Mn (α, β) + Rn (α) + Rn (β),
where {Mn (θ)}θ∈Θ is a correctly specified minus log-likelihood function, namely, Mn (θ0 ) corresponds to the minus likelihood function of true data-generating model. Recall the notation:
∆n (θ0 ) = Dn ∂θ∗ Mn (θ0 ) and Γn (θ) = Dn ∂θ2∗ Mn (θ)Dn . Building on the results in Section 3.1,
we here presuppose that Mn is smooth enough in θ ∈ Θ, and also that the following conditions
hold.
L

• ∆n (θ0 ) −
→ Np∗ +q∗ (0, Γ0 ) with the asymptotic covariance matrix Γ0 being nonrandom
p
and positive-definite, Γn (θ0 ) →
− Γ0 , and both {∆n (θ0 )}n and {supθ |Γn (θ)|}n are Lr (P )bounded for sufficiently large r > 0.
• We have the sparse consistency for the zero-parameters: P (θ̂n◦ = 0) → 1.
• Asymptotic normality for the non-zero parameters:


∗
∗
−1 ∗
∗
∗
ŵn∗ := s−1
(
α̂
−
α
),
t
(
β̂
−
β
)
= −Γ−1
n
n
0
n
n
0
0 (∆n (θ0 ) + Λ ) + op (1),
where Λ∗ is a constant vector.
• The uniform probability estimate
sup sup rL P (|ŵn | ≥ r) < ∞
r>0

n

−1
for some L > 2 is valid for ŵn := (s−1
n (α̂n − α0 ), tn (β̂n − β0 )).

23

(3.33)

−1
∗
The right hand side of (3.33) tends in distribution to the normal distribution Np∗ +q∗ (−Γ−1
0 Λ , Γ0 );
apart from the factor Λ∗ , this is the typical form of asymptotically efficient maximum-likelihood
estimation, where Γ0 is the Fisher information matrix of the non-zero parameter part.
Let us denote by X n data for constructing the estimator θ̂n = θ̂n (X n ), and X̃ n an independent copy of X n . For clarity, we will often write Mn (θ) = Mn (θ; X n ). The expected
Kullback-Leibler divergence from the true data-generating distribution to the prediction model
for X̃ n based on the regularized estimator θ̂n is given by
o
n
oi
h n
E Ẽ −Mn (θ0 ; X̃ n ) − Ẽ −Mn (θ̂n (X n ); X̃ n ) ,

where Ẽ denotes the expectation with respect to X̃ n . In principle, it is desirable to make this
quantity, equivalently E ⊗ Ẽ{Mn (θ̂n (X n ); X̃ n )}, as small as possible. A natural estimator for
the last quantity is Mn (θ̂n (X n ); X n ), but it is well-known that the statistics has a significant
bias due to the double usage of original data X n for both estimation and prediction. The bias
corrected version necessitates evaluation of the quantity
h
n
oi
Mn (θ̂n (X n ); X n ) − E Mn (θ̂n (X n ); X n ) − Ẽ Mn (θ̂n (X n ); X̃ n ) ,
but unfortunately, it is not a statistics in general. The crucial step in the classical AIC methodology for relative model assessment in terms of the best Kullback-Leibler divergence based
prediction is to derive and/or approximate the bias term
h
n
oi
Bn := −E Mn (θ̂n (X n ); X n ) − Ẽ Mn (θ̂n (X n ); X̃ n ) .
Arguing as in [29], we will show that
Bn = p∗ + q ∗ + o(1),

(3.34)

from which, with ignoring the o(1)-term, we arrive at the AIC type statistics (after multiplied
by 2 as usual)
An := 2Mn (θ̂n ) + 2(p∗ + q ∗ ).

(3.35)

Given more than one candidate models all belonging to our basic setup, we compute the quantity
An for each model, and then pick the one giving the minimum-An value: as was discussed in
[29] (also [21]), in some instances this criterion can be used to select the tuning parameter.
We are left to showing (3.34). By the usual mean-value theorem representation, we have:
for suitable θ̂n0 and θ̂n00 both tending in P -probability to θ0 ,
hn
o n
oi
Bn = −E ⊗ Ẽ Mn (θ̂n ; X n ) − Mn (θ0 ; X n ) − Mn (θ̂n ; X̃ n ) − Mn (θ0 ; X̃ n )


1 2
0
⊗2
= −E ⊗ Ẽ ∂θ Mn (θ0 )[θ̂n − θ0 ] + ∂θ Mn (θ̂n )[(θ̂n − θ0 ) ]
2


1
− ∂θ Mn (θ0 ; X̃ n )[θ̂n − θ0 ] + ∂θ2 Mn (θ̂n00 ; X̃ n )[(θ̂n − θ0 )⊗2 ] .
(3.36)
2
We observe the following two points.
• First we look at the weak convergence limit of the quantity inside the expectation sign
“−E ⊗ Ẽ”, say Gn = Gn (X n , X̃ n ). It follows from the sparse consistency together with
˜ n (θ0 ) = Dn ∂θ∗ Mn (θ0 ; X̃ n ),
the stochastic expansion (3.33) that, writing ∆

 

1
1
∗
∗
∗
∗
∗
∗
˜
Gn = ∆n (θ0 )[ŵn ] + Γ0 [ŵn , ŵn ] + op (1) − ∆n (θ0 )[ŵn ] + Γ0 [ŵn , ŵn ] + op (1)
2
2


˜ n (θ0 ) Γ−1 (∆n (θ0 ) + Λ∗ ) + op (1).
= − ∆n (θ0 ) − ∆
0
24

• Second, since (|ŵn |2 )n is assumed to be uniform integrable, we can readily deduce from
the expression (3.36) that (Gn )n is uniform integrable as well.
˜ n (θ0 ) into account we have the conTherefore, taking the independence between ∆n (θ0 ) and ∆
vergence of moments: for an appropriate probability with expectation operator E which can
˜ n (θ0 )), say (∆0 , ∆
˜ 0 ) ∼ Np∗ +q∗ (0, Γ0 ) ⊗ Np∗ +q∗ (0, Γ0 ), we
integrate the weak limit of (∆n (θ0 ), ∆
obtain
n

o
˜ n (θ0 ) Γ−1 (∆n (θ0 ) + Λ∗ ) + o(1)
Bn = E ⊗ Ẽ ∆n (θ0 ) − ∆
0
n

o
˜ 0 (θ0 ) Γ−1 (∆0 (θ0 ) + Λ∗ ) + o(1)
= E ∆0 (θ0 ) − ∆
0
= p∗ + q ∗ + o(1),
hence (3.34); we note that the constant factor Λ∗ has no asymptotic contribution to the nonnegligible leading term of Bn . As the results, we can conclude the following corollary which
gives a sufficient condition for (3.34), verifying the AIC type information criterion (3.35).
Corollary 3.24. Suppose that Mn (θ0 ) is the minus likelihood function of true data-generating
L

model, and that Assumptions 3.1, 3.6, 3.11, 3.16, 3.18 and 3.19 hold with ∆n (θ0 ) −
→ Np∗ +q∗ (0, Γ0 ),
p
where Γ0 is a constant positive-definite matrix such that Γn (θ0 ) →
− Γ0 given in Assumption 3.16.
Then (3.34) holds.

4

Regularized estimation of discretely observed ergodic diffusion process

In this section, we apply the results in Section 3.1 to the parametric ergodic-diffusion model,
keeping the basic notation of Section 2 in use. Let X = (Xt )t∈R+ be a d-dimensional solution
process to the stochastic differential equation
dXt = a(Xt , α)dwt + b(Xt , β)dt,

X0 = x0 ,

where α = (α1 , . . . , αp ) ∈ Θα ⊂ Rp , β = (β1 , . . . , βq ) ∈ Θβ ⊂ Rq , a : Rd × Θα → Rd × Rm ,
b : Rd × Θβ → Rd and wt , t ∈ [0, T ] is a standard Wiener process in Rm independent of the
initial variable x0 . We observe a discrete-time data Xn = (Xtj )0≤j≤n , where tj = ihn . We focus
on a sampling design such that
Tn := nhn → ∞ and

nh2n → 0

(n → ∞).

Let c(x, α) := a(x, α)⊗2 . We set some assumptions on X.
Assumption 4.1.
1. There exists a constant C such that for any x, y ∈ Rd
sup |a(x, α) − a(y, α)| + sup |b(x, β) − b(y, β)| ≤ C|x − y|.
β∈Θβ

α∈Θα

2. inf x,α |c(x, α)| > 0.
3. For all k ∈ {0, 1, 2} and l ∈ {0, 1, . . . , 4}, there exists a constant C = C(k, l) such that
sup

(1 + |x|)−C {|∂xk ∂αl a(x, α)| + |∂xk ∂βl b(x, β)|} < ∞.

(x,θ)∈Rd ×Θ

25

Assumption 4.2.
1. The process Xt is ergodic for θ = θ0 with invariant probability measure π0 :
Z
Z Tn
1
p
f (Xt )dt →
−
f (x)π0 (dx)
Tn 0
for every f ∈ L1 (π0 ).
2. If a(x, α) = a(x, α0 ) and b(x, β) = b(x, β0 ) for π0 -a.e. x, then α = α0 and β = β0 .
3. For all k > 0, supt E[|Xt |k ] < ∞.
Hereafter, we assume that Assumptions 4.1 and 4.2 hold. We note that
n

1X
p
f (Xtj−1 ) →
−
n

Z
f (x)π0 (dx)

j=1

for continuously differentiable f with the derivative ∂f of at most polynomial growth, since


Z Tn
n
n
q
X
1
1X
1


f (Xt )dt −
E
f (Xtj−1 ) .
sup
E[|Xs − Xtj−1 |2 ] → 0.
Tn 0
n
n
tj−1 ≤s≤tj
j=1

j=1

Now, let us define a regularized estimator θ̂n = (α̂n , β̂n ) as the minimizer of (2.1): Hn (θ) =
a
b
Mn (α, β) + Rn (α) + Rn (β), where

n 


1X
1 −1
⊗2
Mn (θ) =
log det(cj−1 (α)) +
c (α) (∆j X − hn bj−1 (β))
2
hn j−1
j=1

is a negative quasi-likelihood function up to an additive constant, ∆j X = Xtj −Xtj−1 , cj−1 (α) =
c(Xtj−1 , α) and bj−1 (β) = b(Xj−1 , β). This quasi-likelihood has been used by many researchers,
e.g., [11], [16], [32], and [33]. Following the route in Section 3, we will derive the asymptotic
√
√
behavior of θ̂n with setting sn = 1/ n and tn = 1/ Tn .
p

First, we derive the consistency θ̂n →
− θ0 by using Theorem 3.4. Note that in the present
setting we have the following convergence uniformly in θ under Assumptions 4.1 and 4.2 (see
[32, Section 6]):
n

1 X
1
{Mn (α, β0 ) − Mn (α0 , β0 )} =
n
2n
j=1


log

det(cj−1 (α))
det(cj−1 (α0 ))


1 −1
−1
⊗2
+
(c (α) − cj−1 (α0 ))[(∆j X − hn bj−1 (β0 )) ]
hn j−1

Z 

det(c(x, α))
p 1
−1
→
−
+ Tr c (x, α)c(x, α0 ) − Id π0 (dx),
log
2
det(c(x, α0 ))
n
1
1 X n hn −1
{Mn (α, β) − Mn (α, β0 )} =
c (α)[bi−1 (β)⊗2 − bj−1 (β0 )⊗ ]
Tn
Tn
2 i−1
j=1
o
− c−1
(α)[b
(β)
−
b
(β
),
∆
X]
j−1
j−1
0
j
j−1
Z
p 1
→
−
c−1 (x, α)[(b(x, β) − b(x, β0 ))⊗2 ]π0 (dx).
2
26

We set

Z 

1
det(c(x, α))
log
+ Tr c−1 (x, α)c(x, α0 ) − Id π0 (dx),
2
det(c(x, α0 ))
Z
1
b
M 0 (θ) =
c−1 (x, α)[(b(x, β) − b(x, β0 ))⊗2 ]π0 (dx).
2
a

M 0 (α) =

a

(4.1)
(4.2)

b

Then, we have M 0 (α) ≥ 0 and M 0 (θ) ≥ 0 with the inequalities holding if and only if θ = θ0 , so
that the following corollary is trivial from Theorem 3.4.
p

Corollary 4.3. We have θ̂n →
− θ0 under Assumptions 3.1.3, 4.1 and 4.2.
√
√
Next, in order to show the rates
√ ( n(α̂n − α0 ), Tn (β̂n − β0 )) = Op (1) we
√ of convergence
apply Theorem 3.8 with sn = 1/ n and tn = 1/ Tn . We assume Assumption 3.6.2, and will
check Assumption 3.6.1. Conditions (a) and (c) are derived from [33, Lemma 3, 6, 8 and 9].
Further, it is easy to show the conditions (b) and (d) since we have
sup
α

p
1
√ ∂α ∂β Mn (α, β0 ) = Op ( hn ),
n hn
n 
1 X
1 2
∂α Mn (θ0 ) =
∂α2 log det(cj−1 (α0 ))
n
2n
j=1




1 2 −1
⊗2
+
∂ c (α0 ) (∆j X − hn bj−1 (β0 ))
hn α j−1
Z
 2
p 1
→
−
∂α log det(c(x, α0 )) + ∂α2 c−1 (x, α0 )[c(x, α0 )] π0 (dx)
2
 Z
p
1
−1
−1
=: Γα0 (α0 ),
=
Tr{c (∂αi c)c (∂αj c)(x, α0 )}π0 (dx)
2
i,j=1

n 
X


1 2
1
1 −1
2
⊗2
∂ Mn (θ0 ) =
c (θ0 ) ∂β (∆j X − hn bj−1 (β0 ))
Tn β
2Tn
hn j−1
j=1
Z
p
→
−
c−1 (x, α0 )[(∂β bj−1 (β0 ))⊗2 ]π0 (dx) =: Γβ0 (θ0 ),
with λmin (Γα0 (α0 )) ∧ λmin (Γβ0 (θ0 )) > 0; both ∂α c(x, α) and ∂β b(x, β) are not identically zero
under the assumptions. As a result, we can conclude the following corollary:
√
√
Corollary 4.4. We have ( n(α̂n −α0 ), Tn (β̂n −β0 )) = Op (1) under Assumptions 3.1.3, 3.6.2,
4.1 and 4.2.
We proceed to the sparse consistency P (θ̂n◦ = 0) → 1; recall that we presupposed
√ that
both α and β do have zero components. Let Assumptions 3.1.3 and 3.6.2 hold; then ( n(α̂n −
√
α0 ), T√
n (β̂n − β0 )) = O
√p (1) from Corollary 4.4, from which we deduce (3.5) and (3.6) with
sn = 1/ n and tn = /1 Tn . We will check Assumption 3.11. We get
1
1 a◦ a ◦
◦
◦
∗
◦
∂α◦0 Mn (0, . . . , 0, α̂n,k
λ 0 R (α̂n,k0 )
0 , . . . , α̂n,p◦ , α̂n , β̂n )α̂n,k 0 +
n k
n n,k
◦
α̂n,k
0
1
1
◦
∗
◦
√ ◦
= √ ∂α◦0 Mn (0, . . . , 0, α̂n,k
0 , . . . , α̂n,p◦ , α̂n , β̂n )
◦
|α̂n,k0 | | nα̂n,k0 |
n k

1 ◦ −2 a◦ a ◦
◦
2
+ |α̂n,k
λn,k0 R (α̂n,k0 ) |α̂n,k
0|
0|
n
27




1
1
a◦
a ◦
◦
2
= Op (1) √ ◦
+ √ ◦ 2 λn,k0 R (α̂n,k0 ) |α̂n,k
0|
| nα̂n,k0 | | nα̂n,k
0|


√ ◦ −1 a◦ a ◦ 
1
◦
2
= √ ◦
Op (1) + | nα̂n,k0 | λn,k0 R (α̂n,k0 ) |α̂n,k
0| .
| nα̂n,k0 |
Therefore, if we assume

√ ◦ −1 a◦ a ◦
P | nα̂n,k
λn,k0 R (α̂n,k0 ) > K → 1
0|

(4.3)

p

◦ −1 b◦
◦
P | Tn β̂n,l
λn,l0 Rb (β̂n,l
→1
0|
0) > K

(4.4)

and similarly

for any K > 0, Assumption 3.11 holds; note that the conditions (4.3) and (4.4) put limitations
b◦
on specific form of the functions Ra and Rb , and also on the stochastic orders of λa◦
n,k0 and λn,l0 .
Theorem 3.12 concludes the following.
Corollary 4.5. We have P (θ̂n◦ = 0) → 1 under Assumptions 3.1.3, 3.6.2, 4.1, 4.2, (4.3) and
(4.4).
We now apply Corollary 3.17 to derive the asymptotic non-degenerate distribution of

√
p
n(α̂n∗ − α0∗ ), Tn (β̂n∗ − β0∗ ) .
(û∗n , v̂n∗ ) =
We use the same notation as in Section 3.1.4; recall that
M0n (u∗ , v ∗ ) := Mn (α̂n◦ , α0∗ + sn u∗ , β̂n◦ , β0∗ + tn v ∗ ) − Mn (α̂n◦ , α0∗ , β̂n◦ , β0∗ ).
By means of the Taylor expansion around (u∗ , v ∗ ) = (0, 0) together with the property P (θ̂n◦ =
0) → 1 and the martingale central limit theorem, we obtain
M0n (u∗ , v ∗ )

n 
1 X
= √
∂α∗ log det(cj−1 (α0 ))
2 n
j=1
h
i
1
⊗2
−1
+
∂α∗ cj−1 (α0 ) (∆j X − hn bj−1 (β0 ))
[u∗ ]
hn
n 
h
i
X
1
1 −1
⊗2
+ √
c
(α0 ) ∂β ∗ (∆j X − hn bj−1 (β0 ))
[v ∗ ]
2 nhn j=1 hn j−1
Z
o
1 n 2
+
∂α∗ log det(c(x, α0 ))[u∗ , u∗ ] + ∂α2 ∗ c−1 (x, α0 )[u∗ ⊗2 , c(x, α0 )] π0 (dx)
4
Z
1
c−1 (x, α0 )[∂β ∗ b(x, β0 )[v ∗ ], ∂β ∗ b(x, β0 )[v ∗ ]]π0 (dx) + op (1)
+
2
1
L
−
→ ∆0 (θ0 )[w∗ ] + Γ0 [w∗ , w∗ ],
2

where

22
∆0 (θ) ∼ N 0, diag(∆11
0 (θ0 ), ∆0 (θ0 )) ,
Z
1  2
11
∆0 (θ0 ) =
∂α∗ log det(c(x, α0 )) + ∂α2 ∗ c−1 (x, α0 )[c(x, α0 )] π0 (dx),
2
Z
22
∆0 (θ0 ) = c−1 (x, α0 )[∂β ∗ b(x, β), ∂β ∗ b(x, β0 )]π0 (dx),
28

 Z
1  2
∂α∗ log det(c(x, α0 )) + ∂α2 ∗ c−1 (x, α0 )[c(x, α0 )] π0 (dx),
Γ0 = diag
2

Z
−1
c (x, α0 )[∂β ∗ b(x, β0 ), ∂β ∗ b(x, β0 )]π0 (dx) .
Hence, Assumptions 3.16.1 and 3.16.2 hold. Obviously, Assumption 3.6.2 is implied by Assumption 3.16.3. The following corollary is then trivial from Corollary 3.17:
Corollary 4.6. Grant Assumptions 3.1.3, 3.16.3 with Λ∗ = (Λa∗ , Λb∗ ) given by (3.11) and
(3.12) being nonrandom, 4.1, 4.2, (4.3), and (4.4). Then, the asymptotic distribution of (û∗n , v̂n∗ )
is
L

−1
∗
(û∗n , v̂n∗ ) −
→ Np∗ +q∗ (−Γ−1
0 Λ , Γ0 ).

Finally, we look at the tail-probability estimate of (ûn , v̂n ) by means of Theorem 3.20. For
this purpose, we strengthen the ergodicity assumption and Assumption 4.2.2:
Assumption 4.7.
that

1. X is exponentially strong-mixing: there exists a constant C > 0 such
|P (A ∩ B) − P (A)P (B)| ≤ C −1 e−Ch ,

sup

sup

t∈R+

A∈σ(Xr :r≤t)
B∈σ(Xr :r≥t+h)

h > 0.

2. There exist positive constants χa and χb , such that
a

(a) M 0 (α) ≥ χa |α − α0 |2 for all α ∈ Θα ,
b

(b) M 0 (α0 , β) ≥ χb |β − β0 |2 for all β ∈ Θβ ,
a

b

where M 0 (α) and M 0 (θ) are given by (4.1) and (4.2).
Thanks to Assumption 4.7 and the results in [33, Section 6], we see that all the conditions
in Assumption 3.18 hold. Also noted is that Assumption 3.1.3 is implied by Assumption 3.19.
Hence we obtain the following result:
Corollary 4.8. For any L > 0, (3.16) holds under Assumptions 3.19, 4.1, 4.2 and 4.7. Additionally if we assume Assumption 3.16.3 with Λ∗ = (Λa∗ , Λb∗ ) being nonrandom, (4.3), and
−1
∗
(4.4), then (3.19) holds for û◦0 = 0, v̂0◦ = 0, (û∗0 , v̂0∗ ) ∼ Np∗ +q∗ (−Γ−1
0 Λ , Γ0 ), and all continuous
p+q
f :R
→ R of at most polynomial growth.
We thus conclude that, for example with a bridge type regularization terms for γ < 1 it is
possible to make the asymptotic variances of the zero parameters degenerate with leaving that
of non-zero part unbiased in the limit: see the discussions in Section 3.3.1 together with the
conditions (4.3) and (4.4).
Let us again note that non-sparse asymptotic is also in our scope (Section 3.2). For example,
as a formal variant of the drift estimation of a diffusion process we may consider the ridge-type
estimation of the drift coefficient with leaving the diffusion one completely unknown. That is,
for the model
dXt =

q
X

βl bl (Xt )dt + a(Xt )dwt ,

l=1

29

we could consider the regularized version of the least-squares type contrast function (e.g. [18]
and the references therein):
2 X
q
q
n 
X
1 X
∆j X − hn
βl bl (Xtj−1 ) +
Hn (β) :=
λn,l βl2 .
2hn
j=1

l=1

l=1

The associated regularized estimator is given by
X
−1 X
n
n
⊗2
β̂n =
bj−1 + 2diag(λn,1 , . . . , λn,q )
(∆j X)bj−1
j=1

j=1

with λn,k > 0 a.s., where bj−1 = (b1 (Xtj−1 ), . . . , bq (Xtj−1 )), would be more variance-stabilized if
P
−1 is unstable.
the computation of the random matrix ( nj=1 b⊗2
j−1 )

A

Polynomial type tail-supremum probability estimate

For convenience of reference, we here state a version of the PLDI of [33, Theorem 1]. We need
to introduce some notation. Given sets T and K ⊂ Θ × T , we let ξ0 := (θ0 , τ0 ) ∈ K be the
true value of parameter ξ. Let n = n (θ0 ) → 0 be a nonrandom positive sequence. For contrast
functions Hn : Ω × Θ × T → R, we define random functions

Mn (u, τ ; θ0 ) := Hn θ0 + n u, τ − Hn (θ0 , τ ),

Yn (θ, τ ; ξ0 ) := −2n Hn (θ, τ ) − Hn (θ0 , τ ) .
Also, let (θ, τ ) 7→ Y0 (θ, τ ; ξ0 ) be a random function. We consider the LAQ representation of
Mn :
1
Mn (u, τ ; θ0 ) = ∆n (τ ; ξ0 )[u] − Γ0 (τ ; ξ0 )[u, u] + rn (u, τ ; ξ0 )
2
for (u, τ ) ∈ {u ∈ Rp : θ0 + n u ∈ Θ} × T , where ∆n (τ ; ξ0 ) ∈ Rp , Γ0 (τ ; ξ0 ) ∈ Rp ⊗ Rp and
rn (u, τ ; ξ0 ) ∈ R are random variables. Finally, let ζ ∈ (0, 1) and Un (r, ξ0 ) := {u ∈ Rp : r ≤ |u| ≤
−(1−ζ)
n
}. We now introduce some conditions.
[A1] ∃ρ1 > 0, ∀L > 0,
L



sup sup r P
r>0 n>0

sup
(u,τ )∈Un (r,ξ0 )×T

|rn (u, τ ; ξ0 )|
≥ r−ρ1
1 + |u|2


< ∞.

[A2] ∀L > 0,
L



p

sup r P ∀(u, τ ) ∈ R × T , r
r>0

−ρ1


Γ0 (τ ; ξ0 )
|u| >
[u, u] < ∞.
4
2

[A3] ∀ξ0 ∈ K, ∃χ(ξ0 ) > 0 : random variable, ∃ρ = ρ(ξ0 ) > 0, ∀θ ∈ Θ, ∀τ ∈ T ,
Y0 (θ, τ ; ξ0 ) = Y0 (θ, τ ; ξ0 ) − Y0 (θ0 , τ ; ξ0 ) ≤ −χ(ξ0 )|θ − θ0 |ρ .
[A4] ζ ∈ (0, 1), ρ1 ∈ (0, 1), ζρ < ρ2 , ν ∈ [0, ∞), 1 − 2ν − ρ2 > 0.
[A5] ∀L > 0,


sup rL P χ(ξ0 ) ≤ r−(ρ2 −ζρ) < ∞.
r>0

30

[A6] ∀L > 0, N1 := L(1 − ρ1 )−1 , N2 := L(1 − 2ν − ρ2 )−1 ,

sup E

N1 
< ∞,
sup |∆n (τ ; ξ0 )|
τ ∈T

n>0


sup E
n>0

sup

−2(1/2−ν)
n

N2 
Yn (θ, τ ; ξ0 ) − Y0 (θ, τ ; ξ0 )

< ∞.

(θ,τ )∈Θ×T

The conditions [A2], [A3] and [A5] may be automatic when θ 7→ Y0 (θ, τ ; ξ0 ) is smooth
enough, and Γ0 (τ ; ξ0 ) > 0 and χ(ξ0 ) > 0 are nonrandom uniformly in the nuisance element τ .
This is the case for many kinds of ergodic models. We also introduce:
[B1] Γ0 (τ ; ξ0 ) is deterministic and positive-definite uniformly in τ ∈ T and ξ0 ∈ K.
[B2] ∃χ = χ(ξ0 ) > 0 : nonrandom, ∃ρ = ρ(ξ0 ) > 0, ∀θ ∈ Θ, ∀τ ∈ T ,
Y0 (θ, τ ; ξ0 ) ≤ −χ|θ − θ0 |ρ .
Then the following theorem holds:
Theorem A.1 (Yoshida [33], Theorems 1 and 3(a)). Assume either “[A1]–[A6]”, or “[A1],
[A4], [A6], [B1], and [B2]”. Then, for every L > 0


≥
r
< ∞.
sup sup rL P −1
(
θ̂
−
θ
)
n
0
n
r>0 n>0

The incorporation of the “nuisance” element τ plays an important role when the estimator
has components converging at different rates such as the case of ergodic diffusion treated in
Section 4. Then, the PLDI would be derived by repeated use of Theorem A.1; we refer to [33,
Section 5] and also [19, Section 4] for a detailed account.
Acknowledgements The authors are grateful to the referee for his helpful comments and
suggestion, all of which led to substantial improvement. This work was supported by JST,
CREST (H. Masuda) and JSPS (Y. Shimizu).

References
[1] G. Afendras and M. Markatou. Optimality of training/test size and resampling effectiveness
of cross-validation estimators of the generalization error. arXiv:1511.02980, 2015.
[2] G. Afendras and M. Markatou. Uniform integrability of the OLS estimators, and the
convergence of their moments. arXiv:1511.02962, 2015.
[3] H. Akaike. A new look at the statistical model identification. IEEE transactions on automatic control, 19(6):716–723, 1974.
[4] B. Antoine and E. Renault. Efficient minimum distance estimation with multiple rates of
convergence. J. Econometrics, 170(2):350–367, 2012.
[5] P. J. Bickel and B. Li. Regularization in statistics. Test, 15(2):271–344, 2006. With
comments and a rejoinder by the authors.
[6] K. P. Burnham and D. R. Anderson. Model selection and multimodel inference. SpringerVerlag, New York, second edition, 2002. A practical information-theoretic approach.

31

[7] N. H. Chan and C.-K. Ing. Uniform moment bounds of Fisher’s information with applications to time series. Ann. Statist., 39(3):1526–1550, 2011.
[8] A. De Gregorio and S. M. Iacus. Adaptive LASSO-type estimation for multivariate diffusion
processes. Econometric Theory, 28(4):838–860, 2012.
[9] J. Fan and R. Li. Variable selection via nonconcave penalized likelihood and its oracle
properties. J. Amer. Statist. Assoc., 96(456):1348–1360, 2001.
[10] D. F. Findley and C.-Z. Wei. AIC, overfitting principles, and the boundedness of moments
of inverse matrices for vector autoregressions and related models. J. Multivariate Anal.,
83(2):415–450, 2002.
[11] V. Genon-Catalot and J. Jacod. On the estimation of the diffusion coefficient for multidimensional diffusion processes. Ann. Inst. H. Poincaré Probab. Statist., 29(1):119–151,
1993.
[12] T. Hastie, R. Tibshirani, and J. Friedman. The elements of statistical learning. Springer
Series in Statistics. Springer, New York, second edition, 2009. Data mining, inference, and
prediction.
[13] A. E. Hoerl and R. W. Kennard. Ridge regression: Biased estimation for nonorthogonal
problems. Technometrics, 12(1):55–67, 1970.
[14] C.-K. Ing and C.-Y. Yang. Predictor selection for positive autoregressive processes. J.
Amer. Statist. Assoc., 109(505):243–253, 2014.
[15] J. Jacod and P. Protter. Discretization of processes, volume 67 of Stochastic Modelling and
Applied Probability. Springer, Heidelberg, 2012.
[16] M. Kessler. Estimation of an ergodic diffusion from discrete observations. Scand. J. Statist.,
24(2):211–229, 1997.
[17] K. Knight and W. Fu. Asymptotics for lasso-type estimators. Ann. Statist., 28(5):1356–
1378, 2000.
[18] H. Masuda. Simple estimators for parametric Markovian trend of ergodic processes based
on sampled data. J. Japan Statist. Soc., 35(2):147–170, 2005.
[19] H. Masuda. Parametric estimation of Lévy processes. In Lévy matters. IV, volume 2128 of
Lecture Notes in Math., pages 179–286. Springer, Cham, 2015.
[20] H. Masuda and Y. Shimizu.
arXiv:1406.6751v2, 2014.

Moment convergence in regularized estimations.

[21] Y. Ninomiya and S. Kawano. AIC for the Lasso in generalized linear models. Electron. J.
Stat., 10:2537–2560, 2016.
[22] B. M. Pötscher and H. Leeb. On the distribution of penalized maximum likelihood estimators: the LASSO, SCAD, and thresholding. J. Multivariate Anal., 100(9):2065–2082,
2009.
[23] P. Radchenko. Mixed-rates asymptotics. Ann. Statist., 36(1):287–309, 2008.
[24] Y. Sakamoto and N. Yoshida. Asymptotic expansion formulas for functionals of -Markov
processes with a mixing property. Ann. Inst. Statist. Math., 56(3):545–597, 2004.
[25] Y. Shimizu. Moment convergence of regularized least-squares estimator for linear regression
model. Ann. Inst. Statist. Math., 2016. Online First.
32

[26] M. Uchida and N. Yoshida. Information criteria in model selection for mixing processes.
Stat. Inference Stoch. Process., 4(1):73–98, 2001.
[27] M. Uchida and N. Yoshida. Asymptotic expansion and information criteria. SUT J. Math.,
42(1):31–58, 2006.
[28] M. Uchida and N. Yoshida. Adaptive estimation of an ergodic diffusion process based on
sampled data. Stochastic Process. Appl., 122(8):2885–2924, 2012.
[29] Y. Umezu, Y. Shimizu, H. Masuda, and Y. Ninomiya. AIC for non-concave penalized
likelihood method. arXiv:1509.01688v2, 2015.
[30] A. W. van der Vaart. Asymptotic statistics, volume 3 of Cambridge Series in Statistical
and Probabilistic Mathematics. Cambridge University Press, Cambridge, 1998.
[31] H. Wang, G. Li, and G. Jiang. Robust regression shrinkage and consistent variable selection
through the LAD-Lasso. J. Bus. Econom. Statist., 25(3):347–355, 2007.
[32] N. Yoshida. Estimation for diffusion processes from discrete observation. J. Multivariate
Anal., 41(2):220–242, 1992.
[33] N. Yoshida. Polynomial type large deviation inequalities and quasi-likelihood analysis for
stochastic differential equations. Ann. Inst. Statist. Math., 63(3):431–479, 2011.

33

