Asymptotic Theory of the Sparse Group LASSO
Benjamin Poignard∗

arXiv:1611.06034v4 [] 29 Nov 2016

December 1, 2016

Abstract
This paper proposes a general framework for penalized convex empirical criteria
and a new version of the Sparse-Group LASSO (SGL, Simon and al., 2013), called
the adaptive SGL, where both penalties of the SGL are weighted by preliminary
random coefficients. We explore extensively its asymptotic properties and prove
that this estimator satisfies the so-called oracle property (Fan and Li, 2001), that
is the sparsity based estimator recovers the true underlying sparse model and is
asymptotically normally distributed. Then we study its asymptotic properties in a
double-asymptotic framework, where the number of parameters diverges with the
sample size. We show by simulations that the adaptive SGL outperforms other
oracle-like methods in terms of estimation precision and variable selection.
Keywords: Asymptotic Normality, Consistency, Model Selection, Oracle Property.

∗

ENSAE - CREST & PSL - Paris Dauphine (CEREMADE). E-mail: benjamin.poignard@ensae.fr

1

1

Introduction

Model complexity is an obstacle when one models richly parameterized dynamics
such as multivariate nonlinear dynamic systems. For instance, dynamic variance
correlation processes of size N have an O(N 2 ) complexity as in the dynamic conditional correlation parametrization (DCC, Ding and Engle, 2001). Another issue
arises when the sample size, say T , is comparable to N , which may reduce the estimation performances. This is typically a high-dimensional statistical framework.
A significant literature developed on model penalization, which consists of reducing the number of parameters and performing variable selection. For instance,
the Akaike’s or Bayesian information criteria aim at selecting the size of a model.
However, these methods are unstable, computationally complex and their sampling
properties are difficult to study as Fan and Li (2001) pointed out mainly because
they are stepwise and subset selection procedures.
The LASSO procedure of Tibshirani (1996) overcomes these drawbacks as it simultaneously performs variable selection and model estimation. It then fosters sparsity and allows for continuity of the selected models. Other penalties were proposed
such as the smoothly clipped absolute deviation (SCAD) of Fan, which modifies
the LASSO to shrink large coefficients less severely. The elastic net regularization
procedure of Zou and Hastie (2005) was developed to overcome the collinearity between the variables, which hampers the LASSO to perform well. Their idea consists
of mixing a l1 penalty, which performs variable selection, with a l2 penalty, which
stabilizes the solution paths. The Group LASSO of Yuan and Lin (2006) fosters
sparsity and variable selection in a group of variables. Simon and al. (2013) designed the Sparse-Group LASSO (SGL) to foster sparsity both at a a group level
and within a group. Their penalization involves a l1 LASSO type penalty and a
mixed l1 /l2 penalty for group selection. All these procedures, together with the
algorithms designed for performing selection and estimation, were developed within
a linear framework. The penalized Ordinary Least Square (OLS) loss function is
typically used for linear models as it is convex, which makes the computation easier,
and allows for closed form solutions, such as the soft-thresholding operator for the
LASSO penalty. Furthermore, linear modeling allows for deriving non asymptotic
oracle inequalities straightforwardly: see Bühlmann and van de Geer (2011) on this
non-asymptotic framework.
Knight and Fu (2000) explored the asymptotic properties of the LASSO penalty
for OLS loss functions. Fan and Li (2001) proposed a penalization framework

2

for general convex functions and studied the asymptotic properties of the SCAD
penalty. They proved that the SCAD estimator satisfies the oracle property, that is
the sparsity based estimator recovers the true underlying sparse model and is asymptotically normally distributed. The LASSO as proposed by Tibshirani cannot enjoy
the oracle property. To fix this drawback, Zou (2006) proposed the adaptive LASSO
within an OLS framework, where adaptive weights are used to penalize different coefficients in the penalty. Nardi and Rinaldo (2008) applied the same methodology
for the Group LASSO estimator within an OLS framework and studied its oracle
property.
These theoretical studies were developed for fixed dimensional models with i.i.d.
data, a case where N does not depend on the sample size, and for least square type
loss functions, except Fan and Li (2001). Fan and Peng (2004) considered the general
penalized convex likelihood framework when the number of parameters grows with
the sample size and focused on the oracle property for general penalties. Zou and
Zhang (2009) also focused on the oracle property of the adaptive elastic-net within
the double-asymptotic framework. Their work highlights that adaptive weights
penalizing different coefficients are key quantities to enjoy the oracle property as one
can modify the convergence rate of the penalty terms. Nardi and Rinaldo (2008)
also proposed within the double-asymptotic setting selection consistency results,
which states that asymptotically the right set of relevant variables is selected.
In this paper, we develop the asymptotic theory of penalized M-estimators for
convex criteria and dependent variables and consider the asymptotic properties of
the Sparse-Group LASSO estimator. This penalty is relevant for problems where
one would like to foster sparsity for selecting active groups, that is a group for
which some of the corresponding coefficients are non zero, and active coefficients
within an active group, a situation where a coefficient is non zero within an active
group. Hence this is somehow a two step approach as first the active groups are
selected, and then the active variables within an active group are selected. We
prove that the SGL as proposed by Simon and al. (2013) does not enjoy the oracle
property. Then we propose a new version of the SGL, the adaptive SGL using the
same methodology of Zou (2006), which consists of penalizing different coefficients
and group of coefficients using random weights that are positive functions of a first
step estimator. This enables to alter the rate of convergence of the penalties such
that the adaptive SGL satisfies the oracle property. Our work is influenced by Fan
and Peng (2004) concerning the oracle property for general penalized convex loss
functions and by Zou and Zhang (2009) regarding the modeling of random weights

3

penalizing the coefficients differently. They both considered a double-asymptotic
framework. We also prove that the adaptive SGL enjoys the oracle property in a
double-asymptotic framework, a situation where the model complexity grows with
the sample size.
The rest of the paper is organized as follows. In section 2, we describe our
general framework for penalized convex empirical criteria and the SGL penalty. In
section 3, we derive the optimality conditions of the statistical criterion. In section
4, we derive the asymptotic properties of both the SGL and adaptive SGL when
the number of parameters is fixed. In section 5, we prove the oracle property of the
adaptive SGL in a double-asymptotic setting.

2

Framework and notations

We consider a dynamic system in which the criterion is written as an empirical
criterion, that is
θ 7→ GT l(θ) =

T
1X
l(ǫt ; θ),
T

(2.1)

t=1

such that l(.) is ”a general” known loss function on the sample space such that
for any process (ǫt ), θ 7→ l(ǫt ; θ) is convex. This framework encompasses for instance the maximum likelihood method, where the l(.) function corresponds to

l(ǫt ; θ) = − log f (ǫt ; θ), where f (ǫt ; θ) is the density of the observation (ǫt ) un(1)

der Pθ . Alternatively, a linear model would imply l(ǫt ; θ) = kǫt
(1)

(2)

(2)

− θ ′ ǫt kp , where

(ǫt , ǫt ) = ǫt . We denote the empirical score and Hessian of the empirical criterion
respectively as

ĠT l(θ) =

T
T
1X
1X
∇θ l(ǫt ; θ), G̈T l(θ) =
∇2 ′ l(ǫt ; θ).
T t=1
T t=1 θθ

The dependent nature of our framework requires the use of particular probabilistic
tools to study the asymptotic properties of M-estimators. We extensively use the
ergodic theorem and central limit theorem (Billingsley, 1961, 1995) to obtain convergence in probability of empirical quantities to their theoretical counterparts and
central limit theorems. To do so, we assume the stationarity and the ergodicity of
the underlying process (ǫt ): see assumption 1 in section 4.
In this setting, ǫt ∈ RN and θ ∈ Rd , a vector that can be split into m groups
m
P
ck = d. We suppose no overlap
Gk , k = 1, · · · , m, such that card(Gk ) = ck and
k=1

4

between these groups. We use the notation θ (l) as the subvector of θ, that is the
set {θk : k ∈ Gl }. Hence the vector θ = (θj , j = 1, · · · , d) can be written as
(k)

θ = (θi , k ∈ {1, · · · , m}, i = 1, · · · , ck ) 1 . We denote by θ0 the true parameter

vector of interest. Moreover, θ → E[l(ǫt ; θ)] is supposed to be a one-to-one mapping
and is minimized uniquely at θ = θ0 .

We denote by S := {k : θ (k) 6= 0} the set of indices for which the groups are

active. Let A := {j : θ0,j 6= 0} be the true subset model, which can be decomposed
(l)

into sub-groups of active sets as l ∈ S, Al = {(l, i) : θ0,i 6= 0}. Besides, there are
(l)

/ S} ⇔ {∀i =
inactive indices Gl \ Al = Acl = {(l, i) : θ0,i = 0}. We have {l ∈
(l)

1, · · · , cl , θ0,i = 0}. In this setting, A = ∪ Al such that for k 6= l, Ak ∩ Al = ∅.

Furthermore,

Ac

=

m

∪ Ac
l=1 l

l∈S

such that for k 6= l, Ack ∩ Acl = ∅.

Finally, we need the following notations: ĠT l(θ)(k) ∈ Rck is the ”score” vector

of the empirical criterion taken over group k of size ck , ĠT l(θ)(k),i ∈ R is the i-

th component of this score, and ĠT l(θ)A ∈ Rcard(A) is the score over the set of

active indices. G̈T l(θ)(k)(k) ∈ Mck ×ck (R) (resp. H(k)(k) ) is the empirical (resp.
theoretical) Hessian taken over the block representing group k, and G̈T l(θ)AA ∈

Mcard(A)×card(A) (R) is the Hessian over the set of active indices.

The statistical problem consists of minimizing over the parameter space Θ a
penalized criterion of the form
θ̂ = arg min {GT ϕ(θ)},

(2.2)

θ∈Θ

where
θ 7→ GT ϕ(θ) =

T
1P

{l(ǫt ; θ) + p1 (λT , θ) + p2 (γT , θ)}
T t=1
= GT l(θ) + p1 (λT , θ) + p2 (γT , θ).

and both penalties are specified as


 p1 : R+ × Rm
+ × Θ → R+ ,
1

−1

 (λT , α, θ) 7→ p1 (λT , θ) = λT T

m
P

αk kθ (k) k1 ,

k=1

p2 : R+ × Rm
+ × Θ → R+ ,

(γT , ξ, θ) 7→ p2 (γT , θ) = γT T −1

Formally, there is a one-to-one mapping between two ways for writing θ:

m
P

ξl kθ (l) k2 .

l=1

ψ : {1, · · · , d} → {(k, i), k = 1, . . . , m; i = 1, · · · , ck },
j 7→ ψ(j) = (kj , ij ).
In the rest of this paper, this mapping is implicit such that we allow such writings as j = (k, i) or j = ik
where k is clear.

5

Both αk and ξl are non negative scalar quantities for each group and the tuning
parameters λT and γT vary with T .
The estimator θ̂ obtained in (2.2) is not the minimum of the empirical unpenalized criterion GT l(.). Our main interest is to analyze the bias generated by the
penalties and how the oracle property can be achieved in the sense of Fan and Li
(2001). More precisely, the sparsity based estimator must satisfy
(i)Â = {i : θ̂i 6= 0} = A asymptotically, that is ”model selection consistency”.
√
d
(ii) T (θ̂A − θ0,A ) → N (0, V0 ) with V0 a covariance matrix related to the criterion of interest.
We highlight in Proposition 4.9, section 4 that actually the SGL as proposed
by Simon and al. (2013) cannot perform the oracle property. Hence in section 4,
we propose a new estimator based on the same idea as Zou (2006), the adaptive
Sparse Group LASSO, for which the oracle property is obtained when the weights
are randomized, as proved in Theorem 4.10.
This framework can be adapted to a broad range of problem. For instance, one
can penalize a subset of groups with a l1 penalty only, and the other groups with
a l1 /l2 penalty only. This framework encompasses the SGL, the LASSO and the
group LASSO for proper choices of α’s and ξ’s.
Let us motivate the interets of the SGL approach and illustrate our notations
through a simple linear example. In finance, finding the right set of explanatory
variables to predict future asset returns is a significant issue. For instance, one may
use Japanese companies indices, the Japanese GDP or the Japanese aggregated
dividend-price ratio to explain the Nikkei index return through a linear projection.
But one should also consider some foreign variables, such as the S&P 500 index or
the US yield curve. Consequently, some groups of variables naturally arise: group of
financial companies, tech companies, and the like; group of foreign components such
as American financial companies, and the like. Hence the set Gk may represent the
k-th (k ≤ m) group of Japanese financial companies, composed (as a shortcoming)

with Nomura (index 1), MUFG-Bank of Tokyo (index 2) and Sumitomo (index 3)
(k)

(k)

(k)

represented by the parameter vector θ (k) = (θ1 , θ2 , θ3 ); then k ∈ S if the whole
group has a statistically significant effect on the Nikkei index. Suppose the l1 /l2

penalty selects this group as active. Then Ak represents the set of active components
in Gk such that cAk = card(Ak ) ≤ card(Gk ) = ck . The l1 penalty fosters sparsity

within this selected group. If Nomura is the only variable that is expelled, then
1 ∈ Ack = Gk \ Ak , whereas {2, 3} ∈ Ak and cAk = 2.

6

3

Optimality conditions

The statistical problem consists of solving (2.2). Both GT l(.), p1 (λT , α, .) and
p2 (γT , ξ, .) are convex functions and there are no inequality constraints. Consequently, by the Karush-Kuhn-Tucker optimality conditions, which are necessary
and sufficient, the estimator θ̂ satisfies for a group k
ĠT l(θ̂)(k) + λT T −1 αk ŵ(k) + γT T −1 ξk ẑ (k) = 0,

(3.1)

for some vectors w(k) and z (k) satisfying

ŵ

(k)

=


sgn(θ̂ (k) )

(k)

if θ̂i

i

∈ {ŵ (k) : |ŵ(k) | ≤ 1}
i
i

(k)

if θ̂i

6= 0, i = 1, · · · , ck ,

ẑ

(k)

=

= 0, i = 1, · · · , ck .


θ̂ (k) /kθ̂ (k) k2

∈ {ẑ (k) : kẑ (k) k ≤ 1}
2

If θ̂ (k) = 0, we have kẑ (k) k2 ≤ 1. Then, from (3.1), we obtain for such a k ∈
/S

ck
ck
X
X
(k)
(k)
(γT T −1 ξk ẑ i )2 ≤ γT2 T −2 ξk2 kz (k) k22 .
(ĠT l(θ̂)(k),i + λT T −1 αk ŵi )2 =
i=1

i=1

Consequently, if the subgradient equations are satisfied for θ̂ (k) , then θ̂ (k) = 0 if
kĠT l(θ̂)(k) + λT T −1 αk ŵ(k) k2 ≤ γT T −1 ξk .
On the contrary, if this condition is not satisfied, then θ̂ (k) 6= 0. In this case, sparsity

is fostered by the l1 penalty as follows: using the optimality condition of (3.1), we
have for θ̂ (k) 6= 0
(k)

∀i = 1, · · · , ck , −ĠT l(θ̂)(k),i = λT T
(k)

If θ̂i

−1

(k)
αk ŵi

+ γT T

−1

ξk

θ̂i

kθ̂ (k) k2

.

(k)

= 0, then |ŵi | ≤ 1 and we obtain straightforwardly
|ĠT l(θ̂)(k),i | ≤ λT T −1 αk .

Bertsekas (1995) proposed the use of subdifferential calculus to characterize necessary and sufficient solutions for problems such as (2.2). The conditions we derived
are close to those of Simon and al. (2013) (obtained for a least square loss function).
They will be extensively used in the rest of the paper.

7

if θ̂ (k) 6= 0,
if θ̂ (k) = 0.

4

Asymptotic properties

To prove the asymptotic results, we make the following assumptions.
Assumption 1. (ǫt ) is a strictly stationary, ergodic and nonanticipative process.
Assumption 2. The parameter set Θ ⊂ Rd is convex and not necessarily compact.
Assumption 3. For any (ǫt ), the function θ 7→ l(ǫt ; θ) is strictly convex and
C ∞ (R, Θ).

Assumption 4. The function θ 7→ E[l(ǫt ; θ)] is uniquely minimized in θ0 . Moreover, (∇l(ǫt ; θ0 )) is a square integrable martingale difference.

Assumption 5. H := E[∇2θθ′ l(ǫt ; θ0 )] and M := E[∇θ l(ǫt ; θ0 )∇θ′ l(ǫt ; θ0 )] exist and
are positive definite.
Assumption 6. Let υt (C) =

{

sup

sup

k,l,m=1,··· ,d θ:kθ−θ0 k2 ≤νT C

|∂θ3k θl θm l(ǫt ; θ0 )|}, where C >

0 is a fixed constant and νT −→ 0, a quantity that will be made explicit. Then
T →∞

η(C) :=

T
1 X

T2

E[υt (C)υt′ (C)] < ∞.

t,t′ =1

Remark. Assumptions 1 and 4 allows for using the central limit theorem of Billingsley (1961). We remind this result stated as a corollary in Billingsley (1961).
Corollary 4.1. (Billingsley, 1982)
If (xt , Ft ) is a stationary and ergodic sequence of square integrable martingal incre-

ments such that σx2 = Var(xt ) 6= 0, then

T
1 X
d
√
xt → N (0, σx2 ).
T t=1

Note that the square martingale difference condition can be relaxed by α-mixing
and moment conditions. For instance, Rio (2013) provides a central limit theorem
for strongly mixing and stationary sequences.
Theorem 4.2. Under assumptions 1-3, if λT /T → λ0 ≥ 0 and γT /T → γ0 ≥ 0,
then for any compact set B ⊂ Θ such that θ0 ∈ B,
P

θ̂ −→ arg min{G∞ ϕ(x)},
x∈B

8

with

m
m
X
X
(k)
G∞ ϕ(x) = G∞ l(x) + λ0 αk kx k1 + γ0 ξl kx(l) k2 ,
k=1

l=1

where θ0∗ = arg min{G∞ ϕ(x)} is supposed to be a unique minimum, and G∞ l(.) is
x∈B
the limit in probability of GT l(.).
To prove this theorem, we remind of Theorem II.1 of Andersen and Gill (1982)
which proves that pointwise convergence in probability of random concave functions
implies uniform convergence on compact subspaces.
Lemma 4.3. (Andersen and Gill, 1982)
Let E be an open convex subset of Rp , and let F1 , F2 , . . . , be a sequence of random
P

concave functions on E such that Fn (x) −→ f (x) for every x ∈ E where f is some
n→∞

real function on E. Then f is also concave, and for all compact A ⊂ E,
P

sup|Fn (x) − f (x)| −→ 0.
n→∞

x∈A

The proof of this theorem is based on a diagonal argument and theorem 10.8 of
Rockafeller (1970), that is the pointwise convergence of concave random functions
on a dense and countable subset of an open set implies uniform convergence on any
compact subset of the open set. Then the following corollary is stated.
Corollary 4.4. (Andersen and Gill, 1982)
P

Assume Fn (x) −→ f (x),for every x ∈ E, an open convex subset of Rp . Suppose f
n→∞

P

has a unique maximum at x0 ∈ E. Let X̂n maximize Fn . Then X̂n −→ x0 .
n→∞

Newey and Powell (1987) use a similar theorem to prove the consistency of
asymmetric least squares estimators without any compacity assumption on Θ. We
apply these results in our framework, where the parameter set Θ is supposed to be
convex.
Proof. of Theorem 4.2.
By definition, θ̂ = arg min {GT ϕ(θ)}. In a first step, we prove the uniform converθ∈Θ

gence of GT ϕ(.) to the limit quantity G∞ ϕ(.) on any compact set B ⊂ Θ, idest
P

sup |GT ϕ(x) − G∞ ϕ(x)| −→ 0.
T →∞
x∈B

9

(4.1)

We define C ⊂ Θ an open convex set and pick x ∈ C. Then by Assumption 1, the
law of large number implies

P

GT l(x) −→ G∞ l(x).
T →∞

Consequently, if λT /T → λ0 ≥ 0 and γT /T → γ0 ≥ 0, we obtain the pointwise
convergence

P

|GT ϕ(x) − G∞ ϕ(x)| −→ 0.
T →∞

By Lemma 4.3 of Andersen and Gill, G∞ ϕ(.) is a convex function and we deduce
the desired uniform convergence over any compact subset of Θ, that is (4.1).
P

Now we would like that arg min {GT ϕ(.)} −→ arg min {G∞ ϕ(.)}. By assump-

tion 3, ϕ(.) is strictly convex, which implies

P

|GT ϕ(θ)| −→ ∞.
kθk→∞

Consequently, arg min{GT ϕ(x)} = O(1), such that θ̂ ∈ Bo (θ0 , C) with probabil-

ity approaching one for C large enough, with Bo (θ0 , C) an open ball centered at

θ0 and of radius C. Furthermore, as G∞ ϕ(.) is strictly convex, continuous, then
arg min {G∞ ϕ(x)} exists and is unique. Then by Corollary 4.4 of Andersen and
x∈B
Gill, we obtain
P

arg min{GT ϕ(x)} → arg min{G∞ ϕ(x)},
x∈B
x∈B
P

that is θ̂ −→ θ0∗ .
Theorem 4.5. Under Assumptions 1-3 and 6, the sequence of penalized estimators
θ̂ satisfies
kθ̂ − θ0 k = Op (T −1/2 + λT T −1 aT + γT T −1 bT ),
when λT = o(T ) and γT = o(T ), and aT := card(A).{max αk } and bT := card(A).{max ξl }
satisfy λT T −1 aT → 0 and γT T −1 bT → 0.

k

l

Proof. We denote νT = T −1/2 + λT T −1 aT + γT T −1 bT , with aT = card(A).{max αk }
k

and bT = card(A).{max ξl }. We would like to prove that for any ǫ > 0, there exists
l

Cǫ > 0 such that

P(

1
νT

kθ̂ − θ0 k > Cǫ ) < ǫ.

10

(4.2)

We have
P(

1
νT

kθ̂ − θ0 k > Cǫ ) ≤ P(∃u ∈ Rd , kuk2 ≥ Cǫ : GT ϕ(θ0 + νT u) ≤ GT ϕ(θ0 )).

Furthermore, kuk2 can potentially be large as it represents the discrepancy θ̂ − θ0
normalized by νT . Now based on the convexity of the objective function, we have

{∃u∗ , ku∗ k2 ≥ Cǫ , GT ϕ(θ0 +νT u∗ ) ≤ GT ϕ(θ0 )} ⊂ {∃ū, kūk2 = Cǫ , GT ϕ(θ0 +νT ū) ≤ GT ϕ(θ0 )},
(4.3)

a relationship that allows us to work with a fixed kuk2 . Let us define θ1 = θ0 + νT u∗

such that GT ϕ(θ1 ) ≤ GT ϕ(θ0 ). Let α ∈ (0, 1) and θ = αθ1 + (1 − α)θ0 . Then by
convexity of GT ϕ(.), we obtain

GT ϕ(θ) ≤ αGT ϕ(θ1 ) + (1 − α)GT ϕ(θ0 )
≤ GT ϕ(θ0 ).

We pick α such that kūk = Cǫ with ū := αθ1 + (1 − α)θ0 . Hence (4.3) holds, which

implies

P(kθ̂ − θ0 k > Cǫ νT ) ≤ P(∃u ∈ Rd , kuk2 ≥ Cǫ : GT ϕ(θ0 + νT u) ≤ GT ϕ(θ0 ))
≤ P(∃u, kuk2 = Cǫ : GT ϕ(θ0 + νT ū) ≤ GT ϕ(θ0 )).

Hence, we pick a u such that kuk2 = Cǫ . Using p1 (λT , α, 0) = 0 and p2 (γT , ξ, 0) = 0,
by a Taylor expansion to GT l(θ0 + νT u), under assumption 4, we obtain
νT2

u′ G̈T l(θ̄)u +

νT3

∇′ {u′ G̈T l(θ̄)u}u
2
6
+ p1 (λT , α, θT ) − p1 (λT , α, θ0 ) + p2 (γT , ξ, θT ) − p2 (γT , ξ, θ0 ),

GT ϕ(θ0 + νT u) − GT ϕ(θ0 ) = νT ĠT l(θ0 )u +

where θ̄ is defined as kθ̄ − θ0 k ≤ kθT − θ0 k. We want to prove
νT

E[u′ G̈T l(θ0 )u] +

νT

νT2

∇′ {u′ G̈T l(θ̄)u}u
2
2
6
+νT−1 {p1 (λT , α, θT ) − p1 (λT , α, θ0 ) + p2 (γT , ξ, θT ) − p2 (γT , ξ, θ0 )} ≤ 0) < ǫ,
(4.4)
d
P
uk ul {∂θ2k θl GT l(θ0 ) − E[∂θ2k θl GT l(θ0 )]}. By assumption 1, (ǫt )
where RT (θ0 ) =
P(∃u, kuk2 = Cǫ : ĠT l(θ0 )u +

RT (θ0 ) +

k,l=1

is a non anticipative stationary solution and is ergodic. As a square integrable
martingale difference,

√

d

T ĠT l(θ0 )u −→ N (0, u′ Mu),

11

by the central limit theorem of Billingsley (1961), which implies ĠT l(θ0 )u = Op (T −1/2 u′ Mu).
By the ergodic theorem of Billingsley (1995), we have
P

G̈T l(θ0 ) −→ H.
T →∞

This implies RT (θ0 ) = op (1).
Furthermore, we have by the Markov inequality and for b > 0 that

P(∃u, kuk2 = Cǫ :

sup
θ̄:kθ−θ0 k2 ≤νT Cǫ

|

νT2
6

∇′ {u′ G̈T l(θ̄)u}u| > b) ≤

νT4 Cǫ6
36b2

η(Cǫ ),

where η(Cǫ ) is defined in assumption 6. We now focus on the penalty terms. As
p1 (λT , α, 0) = 0, for the l1 norm penalty, we have
p1 (λT , α, θT ) − p1 (λT , α, θ0 ) = λT T −1

X

k∈S

(k)

(k)

αk {kθ0 + νT u(k) k1 − kθ0 k1 },

and |p1 (λT , α, θT ) − p1 (λT , α, θ0 )| ≤ card(S){max αk }λT T −1 νT kuk1 .
k∈S

(4.5)

As for the l1 /l2 norm, we obtain
p2 (γT , ξ, θT ) − p2 (γT , ξ, θ0 ) = γT T −1

X
(l)
(l)
ξl {kθT k2 − kθ0 k2 },
l∈S

X
and |p2 (γT , ξ, θT ) − p2 (γT , ξ, θ0 )| ≤ γT T −1 ξl νT ku(l) k2
l∈S

≤ card(S){max ξl }γT T −1 νT kuk2 . (4.6)
l∈S

Then denoting by δT = λmin (H)Cǫ2 νT , and using
deduce that (4.4) can be bounded as
νT

2

E[u′ G̈T l(θ0 )u] ≥ δT , we

νT2

∇′ {u′ G̈T l(θ̄)u}u
2
6
+νT−1 {p1 (λT , α, θT ) − p1 (λT , α, θ0 ) + p2 (γT , ξ, θT ) − p2 (γT , ξ, θ0 )} ≤ 0)

P(∃u, kuk2 = Cǫ : ĠT l(θ0 )u +

u′ G̈T l(θ0 )u +

νT

≤ P(∃u, kuk2 = Cǫ : |ĠT l(θ0 )u| > δT /8) + P(∃u, kuk2 = Cǫ :
νT2

∇′ {u′ G̈T l(θ̄)u}u| > δT /8)
6
+P(∃u, kuk2 = Cǫ : |p1 (λT , α, θT ) − p1 (λT , α, θ0 )| > νT δT /8)

+P(∃u, kuk2 = Cǫ : |

+P(∃u, kuk2 = Cǫ : |p2 (γT , ξ, θT ) − p2 (γT , ξ, θ0 )| > νT δT /8).

12

νT
2

|RT (θ0 )| > δT /8)

We also have for Cǫ and T large enough, and using norm equivalences that
P(∃u, kuk2 = Cǫ : |p1 (λT , α, θT ) − p1 (λT , α, θ0 )| > νT δT /8)

≤ P(∃u, kuk2 = Cǫ : card(S){max αk }λT T −1 νT kuk1 > νT δT /8) < ǫ/5,
k∈S

P(∃u, kuk2 = Cǫ : |p2 (γT , ξ, θT ) − p2 (γT , ξ, θ0 )| > νT δT /8)

≤ P(∃u, kuk2 = Cǫ : card(S){max ξl }γT T −1 νT kuk2 > νT δT /8) < ǫ/5.
l∈S

Moreover, if νT = T −1/2 + λT T −1 aT + γT T −1 bT , then for Cǫ large enough
P(∃u, kuk2 = Cǫ : |ĠT l(θ0 )u| > δT /8) ≤
≤

Cǫ2 Cst
T δT2
Cst
< ǫ/5.
Cǫ4

Moreover
P(∃u, kuk2 = Cǫ :

sup
θ̄:kθ̄−θ0 k2 <νT Cǫ

|

νT2
6

∇′ {u′ G̈T l(θ̄)u}u| > δT /8) ≤

Cst νT4 η(Cǫ )
δT2

≤ Cst νT2 Cǫ2 η(Cǫ )

where Cst > 0 is a generic constant. Consequently, we obtain, for T and Cǫ large
enough, we obtain

P(∃u, kuk2 = Cǫ : |ĠT l(θ0 )u| > δT /8) + P(∃u, kuk2 = Cǫ :
νT2

νT
2

|RT (θ0 )| > δT /8)

∇′ {u′ G̈T l(θ̄)u}u| > δT /8)
6
+P(∃u, kuk2 = Cǫ : |p1 (λT , α, θ0 ) − p1 (λT , α, θT )| > νT δT /8)

+P(∃u, kuk2 = Cǫ : |

+P(∃u, kuk2 = Cǫ : |p2 (γT , ξ, θ0 ) − p2 (γT , ξ, θT )| > νT δT /8) + 0
≤

Cst

Cǫ4
≤ ǫ,

+ νT2 Cǫ2 η(Cǫ )Cst + 3ǫ/5

for Cǫ sufficiently large, and T large enough. We then deduce
kθ̂ − θ0 k = Op (νT ) = Op (λT T −1 aT + γT T −1 bT + T −1/2 ).

Remark. We would like to highlight the use of the convexity property of GT ϕ(.).
It allowed us to obtain the upper bound (4.4). Otherwise, the inequality would have

13

been uniform over kuk2 ≥ Cǫ . A consequence is that kuk2 can take significantly

large values, which would have made the control of the random part in the Taylor

expansion hard. This issue is overcome thanks to the convexity that allows for
working with fixed kuk2 , as Fan and Li (2001), Fan and Peng (2004) or Nardi and
Rinaldo (2008) do.

We now focus on the distribution of the SGL estimator. Deriving the asymptotic distribution for M-estimators is standard in the case the objective function
is differentiable. It consists of characterizing the estimator by the orthogonality
conditions and derive a linear representation by Taylor expansions of the estimator.
But these techniques do not apply when the objective function is not differentiable.
In our case, ϕ(.) is not differentiable at 0 due to the penalty terms. In some specific
context, it may be possible to treat the non-differentiability of GT ϕ(.) by applying
the expectation operator E[.] to ϕ(.), which then becomes differentiable in θ0 . Then
Taylor expansions are feasible and one obtains the distribution, provided some regularity conditions of the empirical criterion, such as stochastic equi-continuity: see
Andrews (1994, a,b). This approach works for specific loss functions, such as the
LAD. But in our setting, the expectation operator fails at regularizing ϕ(.) due to
the penalty functionals.
Another approach to obtain the asymptotic distribution relies on the convexity
property of ϕ(.), and hence of GT ϕ(.), without assuming strong regularity conditions on ϕ(.). The intuition behind this rather strong statement is as follows.
Let FT (u) and F∞ (u), u ∈ Rd , be random convex functions such that their minimum are respectively uT and u∞ . Then if FT (.) converges in finite distribution to
F∞ (.), and u∞ is the unique minimum of F∞ with probability one, then uT converges weakly to u∞ . This method to prove the convergence of arg min processes
is called the convexity argument. It was developed by Davis and al. (1992), Hjort,
Pollard (1993), Geyer (1996a, 1996b) or Kato (2009). Chernozhukov and Huong
(2004), Chernozhukhov (2005) use this convexity argument to obtain the asymptotic distribution of quantile regression type estimators. The convexity argument
only requires the lower-semicontinuity and convexity of the empirical criterion. The
convexity Lemma, as in Chernozhukov (2005), can be stated as follows.
Lemma 4.6. ( Chernozhukov, 2005)
Suppose
(i) a sequence of convex lower-semicontinuous FT : Rd → R̄ marginally converges
to F∞ : Rd → R̄ over a dense subset of Rd ;

14

(ii) F∞ is finite over a nonempty open set E ⊂ Rd ;

(iii) F∞ is uniquely minimized at a random vector u∞ .
Then
d

d

arg min FT (z) −→ arg min F∞ (z), that is uT −→ u∞ .
z ∈Rd
z ∈Rd
Theorem 4.7. Under Assumptions 1-6, if λT T −1/2 → λ0 and γT T −1/2 → γ0 , then
√

d

T (θ̂ − θ0 ) −→ arg min {F∞ (u)},
u∈Rd

provided F∞ is the random function in Rd , where
ck
m
1
P
P
(k)
(k)
(k)
u′ Hu + u′ Z + λ0 αk {|ui |1θ(k) =0 + ui sgn(θ0,i )1θ(k) 6=0 }
2
0,i
0,i
i=1
k=1
′ (l)
(l)
m
u θ0
P
1θ(l) 6=0 },
+ γ0 ξl {ku(l) k2 1θ(l) =0 +
(l)
0
l=1
kθ0 k2 0

F∞ (u) =

with H = H(θ0 ) := E[∇2θθ′ l(ǫt ; θ0 )] and some random vector Z ∼ N (0, M), M =
M(θ0 ) := E[∇θ l(ǫt ; θ0 )∇θ′ l(ǫt ; θ0 )].

Proof. Let u ∈ Rd such that θ = θ0 + u/T 1/2 and we define the empirical criterion

FT (u) = T GT (ϕ(θ0 + u/T 1/2 ) − ϕ(θ0 )). First, we are going to prove the finite
distributional convergence of FT to F∞ . Then we use the convexity of FT (.) to

obtain the convergence in distribution of the arg min empirical criterion to the
√
arg min process limit. To do so, let u = T (θ − θ0 ). We have
FT (u) = T {GT (l(θ) − l(θ0 )) + p1 (λT , α, θ) − p1 (λT , α, θ0 ) + p2 (γT , ξ, θ) − p2 (γT , ξ, θ0 )}
m
√
P
(k)
(k)
αk [kθ0 + u(k) / T k1 − kθ0 k1 ]
= T GT (l(θ0 + u/T 1/2 ) − l(θ0 )) + λT
+

m
P

(l)
γT ξl [kθ0
l=1

+

√

u(l) /

T k2 −

k=1
(l)
kθ0 k2 ],

where FT (.) is convex and C 0 (Rd ). We now prove the finite dimensional distribution
of FT to F∞ to apply Lemma 4.6. For the l1 penalty, for any group k, we have for
T sufficiently large
(k)
kθ0

+u

(k)

ck
X
√
(k)
(k)
(k)
(k)
−1/2
{|ui |1θ(k) =0 + ui sgn(θ0,i )1θ(k) 6=0 },
/ T k1 − kθ0 k1 = T
i=1

15

0,i

0,i

which implies that
λT

ck
m
m
X
X
X
√
(k)
(k)
(k)
(k)
(k)
{|ui |1θ(k) =0 +ui sgn(θ0,i )1θ(k) 6=0 },
αk [kθ0 +u(k) / T k1 −kθ0 k1 ] −→ λ0 αk
T →∞

k=1

k=1

0,i

0,i

i=1

√
under the condition that λT / T → λ0 .
As for the l1 /l2 quantity, for any group l, we have
′

(l)
kθ0

(l)

√
u(l) θ0
(l)
+ u / T k2 − kθ0 k2 = T −1/2 {ku(l) k2 1θ(l) =0 + (l) 1θ(l) 6=0 } + o(T −1 ).
0
kθ0 k2 0
(l)

Consequently, if γT T −1/2 → γ0 ≥ 0, we obtain
(l)
m
m
X
X
√
u(l) θ0
(l)
(l)
(l)
(l)
γT
ξl [kθ0 +u / T k2 −kθ0 k2 ] = γ0 ξl {ku k2 1θ(l) =0 + (l) 1θ(l) 6=0 }+o(T −1 )γT .
0,k
kθ0 k2 0
l=1
l=1
′

Now for the unpenalized criterion GT l(.), by a Taylor expansion, we have
1
1
T GT (l(θ0 +u/T 1/2 )−l(θ0 )) = u′ T 1/2 ĠT l(θ0 )+ u′ G̈T l(θ0 )u+ 1/3 ∇′ {u′ G̈T l(θ̄)u}u,
2
6T
√
where θ̄ is defined as kθ̄ − θ0 k ≤ kuk/ T . Then by Assumption 4, we have the
central limit theorem of Billingsley (1961)
√

d

T ĠT l(θ0 ) −→ N (0, M),

and by the ergodic theorem
P

G̈T l(θ0 ) −→ H.
T →∞

Furthermore, we have by assumption 6
|∇′ {u′ G̈T l(θ̄)u}u|2 ≤
≤

T
d
d
1 P
P
P
l(ǫt ; θ̄).∂θ3k θl θm l(ǫt′ ; θ̄)|
uk ul um uk ul um |∂ 3
2
2 2
T 2 t,t′ =1k1 ,l1 ,m1 k2 ,l2 ,m2 1 1 1 2 2 2 θk1 θl1 θm1
T
d
d
1 P
P
P
uk ul um uk ul um υt (C)υt′ (C),
2
T t,t′ =1k1 ,l1 ,m1 k2 ,l2 ,m2 1 1 1 2 2 2

for C large enough, such that the υt (C) =

sup

{

k,l,m=1,··· ,d θ:kθ−θ0 k2 ≤νT C

with νT = T −1/2 + λT T −1 aT + γT T −1 bT . We deduce

∇′ {u′ G̈T l(θ̄)u}u = Op (kuk32 η(C)).

16

sup

|∂θ3k θl θm l(ǫt ; θ)|}

Consequently, we obtain
1
P
∇′ {u′ G̈T l(θ̄)u}u −→ 0.
1/3
T →∞
6T
d

Then we proved that FT (u) −→ F∞ (u), for a fixed u. Let us observe that
u∗T = arg min {FT (u)},
u
and FT (.) admits as a minimizer u∗T =

√

T (θ̂ − θ0 ). As FT is convex and F∞ is

continuous, convex and has a unique minimum, then by the convexity Lemma 4.6,
we obtain

√
d
T (θ̂ − θ0 ) = arg min{FT } −→ arg min{F∞ }.
u
u

Theorem 4.8. Under assumptions 1-6, if γT T −1 → 0 and γT T −1/2 → ∞ such that

λT γT−1 → µ0 , with µ0 ≥ 0, then
T

d

(θ̂ − θ0 ) −→ arg min {K∞ (u)},
γT
u
provided K∞ is a uniquely defined random function in Rd , where
m
1
P
′
(k)
u′ Hu + µ0 αk {ku(k) k1 1θ(k) =0 + u(k) sgn(θ0 )1θ(k) 6=0 }
2
0
0
k=1
′ (l)
(l)
m
u θ0
P
ξl {ku(l) k2 1θ(l) =0 +
+
1θ(l) 6=0 }.
(l)
0
l=1
kθ0 k2 0

K∞ (u) =

The limit quantity K∞ (.) is non-random, which implies that the convergence in
T
P
distribution implies the convergence in probability
(θ̂ − θ0 ) −→ arg min {K∞ (u)}
γT
u
by Shiryaev (ex 7, p 259, 1995).
Proof. To prove this convergence result, we proceed as in Theorem 4.7. To do so,
we define θ = θ0 + uγT /T and we prove that F̃T (u) = GT (ϕ(θ0 + uT /γT ) − ϕ(θ0 ))

converges in finite distribution to K∞ (.). We have

F̃T (u) = T {GT (l(θ) − l(θ0 )) + p1 (λT , α, θ) − p1 (λT , α, θ0 ) + p2 (γT , ξ, θ) − p2 (γT , ξ, θ0 )}
m
P
(k)
(k)
αk [kθ0 + u(k) γT /T k1 − kθ0 k1 ]
= T GT (l(θ0 + uγT /T ) − l(θ0 )) + λT
+

m
P

(l)
γT ξl [kθ0
l=1

+

u(l) γT /T k2

−

k=1
(l)
kθ0 k2 ].

17

For the unpenalized empirical criterion, we have the expansion

T GT (l(θ0 + uγT /T ) − l(θ0 )) = γT ĠT l(θ0 )u +

γT2
2T

′

u G̈T l(θ0 )u +

γT3
6T 2

where θ̄ lies between θ0 and θ0 + uγT /T . This implies F̃T (u) =
KT (u) =
+

∇{u′ G̈T l(θ̄)u}u,

γT2
T

KT (u), where

√
γT ′
( T ĠT l(θ0 )u) + 21 u′ G̈T l(θ̄)u + 6T
∇ {u′ G̈T l(θ̄)u}u
m
m
P
P
(l)
(l)
(k)
T
(k) γ /T k − kθ (k) k ] + T
ξl [kθ0 + u(l) γT /T k2 − kθ0 k2 ].
α
[kθ
+
u
λ
T
1
1
T
k
2
0
0
γT
γ

√

T
γT
T

l=1

k=1

We first focus on the penalty terms. For the l1 part, for any group k, we have
(k)
kθ0

(k)

+u

γT /T k1 −

(k)
kθ0 k1

= γT T

−1

ck
X
(k)
(k)
(k)
{|ui |1θ(k) =0 + ui sgn(θ0,i )1θ(k) 6=0 }.
0,i

i=1

0,i

We deduce that
ck
X
T
(k)
(k)
(k)
(k)
(k)
(k)
λT αk [kθ0 +u γT /T k1 −kθ0 k1 ] → µ0 αk {|ui |1θ(k) =0 +ui sgn(θ0,i )1θ(k) 6=0 },
2
0,i
0,i
γT
i=1
under the condition λT γT−1 → µ0 .
As for the l1 /l2 quantity, for any group l, we have
′

(l)
kθ0

(l)

+ u γT /T k2 −

(l)
kθ0 k2

= γT T

−1

(l)

{ku k2 1θ(l) =0 +
0

(l)

u(l) θ0

(l)
kθ0 k2

1θ(l) 6=0 } + o(T −1 ).
0

Consequently, we obtain
′

(l)

u(l) θ0
T
(l)
(l)
ξl [kθ0 + u(l) γT /T k2 − kθ0 k2 ] → ξl {ku(l) k2 1θ(l) =0 + (l) 1θ(l) 6=0 }.
γT
0
kθ0 k2 0
Now for the unpenalized part, by the central limit theorem of Billingsley (1961),
√
T ĠT l(θ0 ) is asymptotically normal, then γT T −1/2 → ∞ implies by the Slutsky
theorem

√
T √
P
( T ĠT l(θ0 )u) −→ 0.
T →∞
γT

Furthermore, by the ergodic theorem of Billingsley (1961), we have
P

G̈T l(θ0 ) −→ H.
T →∞

As for the third order term, by assumption 6 and using the same reasoning as the

18

proof of Theorem 4.7, we have
γT ′ ′
P
∇ {u G̈T l(θ̄)u}u −→ 0,
T →∞
6T
d

using γT = o(T ). Then we proved that KT (u) −→ K∞ (u), for a fixed u ∈ Rd . We
have

u∗T = arg min {KT (u)},
u
and KT (.) admits as a minimizer u∗T =

T
γT

(θ̂ − θ0 ). KT (.) is convex and K∞ (.) is

continuous, then by the convexity Lemma, we deduce
T
γT

d

(θ̂ − θ0 ) = arg min {KT } −→ arg min {K∞ }.

Remark. The convergence rate of θ̂ is slower than

√
T . Furthermore, the limiting

function is not random.
We now turn to the oracle property of the SGL. Model selection consistency
consists of evaluating the probability that {Â = A}, for T large enough. That

means we check that the regularization asymptotically allows for identifying the
right model.
Proposition 4.9. Under assumption 1-6, if λT T −1/2 → λ0 and γT T −1/2 → γ0 ,
then

lim sup P(Â = A) ≤ c < 1,
T →∞

where c is a constant depending on the true model.
Proof. In Theorem 4.7, we proved
√

d

T (θ̂ − θ0 ) := arg min{FT } −→ arg min{F∞ },
u∈Rd
u∈Rd

√
√
under the assumption λT / T → λ0 and γT / T → γ0 . The limit random function
is

ck
m
1
P
P
(k)
(k)
(k)
u′ Hu + u′ Z + λ0 αk {|ui |1θ(k) =0 + ui sgn(θ0,i )1θ(k) 6=0 }
2
0,i
0,i
i=1
k=1
′ (l)
(l)
m
u θ0
P
+ γ0 ξl {ku(l) k2 1θ(l) =0 +
1θ(l) 6=0 }.
(l)
0
l=1
kθ0 k2 0

F∞ (u) =

19

First, let us observe that
(k)

{Â = A} = {∀k = 1, · · · , m, i ∈ Ack , θ̂i

(k)

= 0} ∩ {∀k = 1, · · · , m, i ∈ Âck , θ0,i = 0}.

Both sets describing {Â = A} are symmetric, and thus we can focus on
(k)

{Â = A} ⇒ {∀k = 1, · · · , m, i ∈ Ack , T 1/2 θ̂i

= 0}.

Hence
(k)

P(Â = A) ≤ P(∀k = 1, · · · , m, ∀i ∈ Ack , T 1/2 θ̂i

= 0).

√
d
Denoting by u∗ := arg min{F∞ (u)}, Theorem 4.7 corresponds to T (θ̂A −θ0,A ) −→
u∈Rd
u∗A . By the Portmanteau Theorem (see Wellner and van der Vaart, 1996), we have
(k)

lim sup P(∀k = 1, · · · , m, ∀i ∈ Ack , T 1/2 θ̂i
T →∞

(k)∗

= 0) ≤ P(∀k = 1, · · · , m, ∀i ∈ Ack , ui

= 0),

as θ0,Ac = 0. Consequently, we need to prove that the probability of the right hand
side is strictly inferior to 1, which is upper-bounded by
(k)∗

P(∀k = 1, · · · , m, ∀i ∈ Ack , ui

= 0) ≤

(k)∗

min(P(k ∈
/ S, u(k)∗ = 0), P(k ∈ S, ∀i ∈ Ack , ui

= 0)).

(4.7)

If λ0 = γ0 = 0, then u∗ = −H−1 Z, such that Pu∗ = N (0, H−1 MH−1 ). Hence ,
c = 0.
If λ0 6= 0 or γ0 6= 0, the necessary and sufficient optimality conditions for a

group k tell us that u∗ satisfies


(k)


 (Hu∗ + Z) + λ α p(k) + γ ξ θ0
= 0,
0 k
0 k
(k)
(k)
kθ
k
2
0


 (Hu∗ + Z) + λ α w(k) + γ ξ z (k) = 0,
0 k
0 k
(k)

k ∈ S,

(4.8)

otherwise,

where w (k) and z (k) are the subgradients of ku(k) k1 and ku(k) k2 given by
(k)

wi


sgn(u(k) ) if u(k) =
i
i 6 0,
=
∈ {w (k) : |w(k) | ≤ 1} if u(k) = 0,
i
i
i
(k)

and pi

(k)

(k)

z (k)

(k)


(k)


 u
if u(k) 6= 0,
= ku(k) k2


∈ {z (k) : kz (k) k ≤ 1} if u(k) = 0,
2

= ∂ui {|ui |1θ(k) =0 + ui sgn(θ0,i )1θ(k) 6=0 }.
0,i

0,i

20

If u(m)∗ = 0, ∀m ∈
/ S, then the optimality conditions (4.8) become
(

HSS u∗S + Z S + λ0 τS + γ0 ζS = 0,

(4.9)

k − H(l)S u∗S − Z (l) − λ0 αl w(l) k2 ≤ γ0 ξl , as kz (l) k2 ≤ 1, l ∈ S c ,
(k)

with τS = vec(k ∈ S, αk

p(k) )

and ζS = vec(k ∈ S, ξk

Rcard(S) .
(k)

For k ∈ S, that is the vector θ0





θ0

(k)

kθ0 k2

), which are vectors of

is at least non-zero, then
(k)

θ0,i

(k)
λ0 αk sgn(θ0,i )

(Hu∗

+ Z)i +
+ γ0 ξk (k) = 0, if k ∈ S, i ∈ Ak ,
kθ0 k2


 (Hu∗ + Z) + λ α w(k) = 0, i ∈ Ac .
i
0 k i
k
(k)∗

Consequently, if ui






(4.10)

= 0, ∀i ∈ Ack , with k ∈ S, then the conditions (4.10) become

HAk Ak u∗Ak + Z Ak + λ0 αk sgn(θ0,Ak ) + γ0 ξk
| − (HAck Ak u∗Ak + Z Ack )i | ≤ λ0 αk .

θ0,Ak
kθ0,Ak k2

= 0,

Combining relationships in (4.9), we obtain
(l)
c
kH(l)S H−1
SS (Z S + λ0 τS + γ0 ζS ) − Z (l) − λ0 αl w k2 ≤ γ0 ξl , l ∈ S .

The same reasoning applies for active groups with inactive components, such that
combining relationships in (4.10), we obtain
|(HAck Ak H−1
Ak Ak (Z Ak + λ0 αk sgn(θ0,Ak ) + γ0 ξk

θ0,Ak
kθ0,Ak k2

) − Z Ack )i | ≤ λ0 αk .

Hence we deduce
(k)∗

P(∀k = 1, · · · , m, ∀ ∈ Ack , ui

min(P(k ∈
/

S, u(k)∗

= 0) ≤

(k)∗

= 0), P(k ∈ S, ∀i ∈ Ack , ui

= 0)) := min(a1 , a2 ).

Under the assumption that λ0 < ∞ and γ0 < ∞, we obtain
(l)
a1 = P(l ∈ S c , kH(l)S H−1
SS (Z S + λ0 τS + γ0 ζS ) − Z (l) − λ0 αl w k2 ≤ γ0 ξl ) < 1,

a2 = P(k ∈ S, i ∈ Ack , |(HAck Ak H−1
Ak Ak (Z Ak + λ0 αk sgn(θ0,Ak ) + γ0 ξk

21

θ0,Ak

kθ0,Ak k2

) − Z Ack )i | ≤ λ0 αk ) < 1.

Thus c < 1, which proves (4.7), that is proposition 4.9.
Remark. The result in Proposition 4.9 highlights that the SGL as proposed by
Simon and al. (2013) cannot achieve the oracle property since the penalties cannot
recover the unknown set of active indices A, which is called model selection consistency. To fix this drawback in an ordinary least square framework, Zou (2006)

proposes the adaptive LASSO, where random weights are used to penalize different
coefficients and proves that the adaptive LASSO estimator satisfies the oracle property in the sense of Fan and Li (2001), that is asymptotic normality and selection
(k)

consistency for a proper choice of λT and αi . That is also the case for the adaptive group LASSO model proposed by Nardi and Rinaldo (2008), where adaptive
weights are used to penalize grouped coefficients differently. We propose the same
approach than Zou (2006) and use adaptive weights in the penalties such that the
adaptive SGL enjoys the oracle property in the sense of Fan and Li (2001) as proved
in Theorem 4.10.
The adaptive specification of the proposed estimator now becomes
θ̂ = arg min {GT ψ(θ)},

(4.11)

θ∈Θ

where
θ 7→ GT ψ(θ) =

T
1P

l(ǫt ; θ) + p1 (λT , θ̃, θ) + p2 (γT , θ̃, θ)
T t=1
= GT l(θ) + p1 (λT , θ̃, θ) + p2 (γT , θ̃, θ),

such that both penalties are specified as
p1 (λT , θ̃, θ) = λT T −1

ck
m
m X
X
X
(k)
(k)
α(θ̃i )|θi |, p2 (γT , θ̃, θ) = γT T −1 ξ(θ̃ (l) )kθ (l) k2 .
k=1 i=1

l=1

These penalties are now randomized through the θ̃ argument in the weights α’s and
ξ’s. This first step estimator θ̃ is supposed to be a T 1/2 -consistent estimator of
θ0 . For instance, it can be defined as an M-estimator of the unpenalized empirical
criterion GT l(.), that is
θ̃ = arg min GT l(θ).
θ∈Θ

Adaptive weights are also used by Zou and Zhang (2009), who plug the elastic-net
estimator in the adaptive weight and then estimate a new elastic net model using
these weights, that is the adaptive elastic net.

22

k
The weights we use are now random and for any group k or l, α(θ̃ (k) ) ∈ Rc
+ ,

ξ(θ̃ (l ) ∈ R+ are specified as

(k)

(k)

αT := α(θ̃ (k) ) = (|θ̃i |−η , i = 1, · · · , ck ), ξT,l := ξ(θ̃ (l ) = kθ̃ (l) k−µ
2 ,
for some constants η > 0 and µ > 0 (to be specified).
Remark. Theorem 4.5 can be adapted to the adaptive specification of the SGL in
(4.11), where the bound in probability of the error would be
kθ̂ − θ0 k = Op (T −1/2 + λT T −1 aT + γT T −1 bT ),
P

(k)

with aT = card(A).{max (max αT,i )}, bT = card(A).{max ξT,l }, such that λT T −1 aT →
k∈S

l∈S

i∈Ak

P

0 and γT T −1 bT → 0. The proof follows exactly the same steps as for Theorem (4.5),
except aT and bT are random quantities.

Theorem 4.10. Under assumptions 1-6, if λT T −1/2 → 0, γT T −1/2 → 0, T (η−1)/2 λT →

∞ and T (µ−1)/2 γT → ∞, then θ̂ obtained in (4.11) satisfies

lim P(Â = A) = 1, and
T →∞
√
d
−1
T (θ̂A − θ0,A ) −→ N (0, H−1
AA MAA HAA ).
Proof. We start with the asymptotic distribution and proceed as in the proof of
Theorem 4.7, where we used Lemma 4.6. To do so, we prove the finite dimensional
convergence in distribution of the empirical criterion FT (u) to F∞ (u) with u ∈ Rd ,

where these quantities are respectively defined as

√
FT (u) = T GT (ψ(θ0 + u/ T ) − ψ(θ0 ))
ck (k) (k)
m P
√
P
(k) √
(k)
αT,i [|θ0,i + ui / T | − |θ0,i |]
= T GT (l(θ0 + u/ T ) − l(θ0 )) + λT
+

m
P

(l)
γT ξT,l [kθ0
l=1

+

√

u(l) /

T k2 −

k=1i=1
(l)
kθ0 k2 ],

and


1 ′
u H u + u′A Z A
F∞ (u) = 2 A AA A


∞

if ui = 0, when i ∈
/ A, and

(4.12)

otherwise,

with Z A ∼ N (0, MAA ). By Lemma 4.6, the finite dimensional convergence in
d

distribution implies arg min{FT (u)} −→ arg min{F∞ (u)}. We first consider the
u∈Rd
u∈Rd

23

unpenalized empirical criterion of FT (.), which can be expanded as
√
1
1
T GT (ψ(θ0 +u/ T )−ψ(θ0 )) = T 1/2 ĠT l(θ0 )u+ u′ G̈T l(θ̄)u+ 1/3 ∇′ {u′ G̈T l(θ̄)}u,
2
6T
√
where θ̄ lies between θ0 and θ0 + u/ T . First, using the same reasoning on the
third order term, we obtain
1

P

6T 1/3

∇′ {u′ G̈T l(θ̄)}u −→ 0.
T →∞

P

By the ergodic theorem, we deduce G̈T l(θ0 ) −→ H and by assumption 4,
T →∞

N (0, M).

√

d

T ĠT l(θ0 ) −→
(k)

(k)

We now focus on the penalty terms of (4.11), we remind that αT,i = |θ̃i |−η ,
(k) P

such that for i ∈ Ak , k ∈ S, θ̃i
√

(k)

→ θ0,i 6= 0. Note that

√
P
(k)
(k)
(k)
(k)
T (|θ0 + u(k) / T | − |θ0 |] −→ ui sgn(θ0,i )1θ(k) 6=0 .
T →∞

0,i

This implies that, for i ∈ Ak , k ∈ S, we have
λT T

−1/2

ck
X
√
√
P
(k)
(k)
(k)
(k)
αT,i T (|θ0,i + ui / T | − |θ0,i |) −→ 0,
T →∞

i=1

(k)

(k)

under the condition λT T −1/2 → 0. For i ∈ Ack , θ0,i = 0, then T η/2 (|θ̃i |)η = Op (1).

Hence under the assumption λT T (η−1)/2 → ∞, we obtain

(k) √
(k)
(k) √
(k)
(k)
λT T −1/2 αT,i T (|θ0,i + ui / T | − |θ0,i |) = λT T −1/2 |ui |

T η/2

P

−→ ∞.

(k)
(T 1/2 |θ̃i |)η T →∞

(4.13)

As for the l1 /l2 quantity, we remind that ξT,l = kθ̃ (l) k−µ
2 , such that for l ∈ S,
P

(l)

θ̃ (l) → θ0 , and in this case
√

′

(l)
T {kθ0

(l)

√
u(l) θ0
(l)
+ o(T −1/2 ).
+ u / T k2 − kθ0 k2 } =
(l)
kθ0 k2
(l)

Consequently, using γT T −1/2 → 0, and for l ∈ S, we obtain
√
√
P
(l)
(l)
γT T −1/2 T ξT,l (kθ0 + u(l) / T k2 − kθ0 k2 ) −→ 0.
T →∞

(k)

Combining the fact k ∈ S and θ0

is partially zero, that is i ∈ Ack , we obtain the

24

(l)

divergence given in (4.13). Furthermore, if l ∈
/ S, that is θ0 = 0, then
√

√
(l)
(l)
T {kθ0 + u(l) / T k2 − kθ0 k2 } = ku(l) k2 ,

and T µ/2 (kθ̃ (l) k2 )µ = Op (1), then under the assumption γT T (µ−1)/2 → ∞, we obtain
√
√
(l)
(l)
γT T −1/2 ξT,l T [kθ0 +u(l) / T k2 −kθ0 k2 ] = γT T −1/2 ku(l) k2

T µ/2

P

−→ ∞.

(T 1/2 kθ̃ (l) k2 )µ T →∞

d

We deduce the pointwise convergence FT (u) −→ F∞ (u), where F∞ (.) is given

in (4.12). As FT (.) is convex and F∞ (.) is convex and has a unique minimum
(H−1
AA Z A , 0Ac ), by Lemma 4.6, we obtain
√

d

T (θ̂ − θ0 ) = arg min{FT (u)} −→ arg min{F∞ (u)},
u∈Rd
u∈Rd

that is to say
√

d

T (θ̂A − θ0,A ) −→ H−1
AA Z A , and

√

d

T (θ̂Ac − θ0,Ac ) −→ 0Ac .

We now prove the model selection consistency. Let i ∈ Ak , then by the asymp(k)

totic normality result, θ̂i
consists of proving

P

(k)

−→ θ0 , which implies P(i ∈ Âk ) → 1. Thus the proof

T →∞

∀k = 1, · · · , m, ∀i ∈ Ack , P(i ∈ Âk ) → 0.
This problem can be split into two parts as
∀k ∈
/ S, P(k ∈ Ŝ) → 0, and ∀k ∈ S, ∀i ∈ Ack , P(i ∈ Âk ) → 0.

(4.14)

Let us start with the case k ∈
/ S. If k ∈ Ŝ, by the optimality conditions given by

the Karush-Kuhn-Tucker theorem applied on GT ψ(θ̂), we have

ĠT l(θ̂)(k) +

λT
T

(k)

αT ⊙ ŵ (k) +

γT
T

ξT,k

θ̂ (k)
kθ̂ (k) k2

⊙ is the Hadamard product and
(k)

ŵi


sgn(θ̂ (k) ) if θ̂ (k) 6= 0,
i
i
=
(k)
(k)
∈ {ŵ : |ŵ | ≤ 1} if θ̂ (k) = 0.
i
i
i

25

= 0,

Multiplying the unpenalized part by T 1/2 , we have the expansion
T 1/2 ĠT l(θ̂)(k) = T 1/2 ĠT l(θ0 )(k) + T 1/2 G̈T l(θ0 )(k)(k) (θ̂ − θ0 )(k)
+ T 1/2 ∇′ {(θ̂ − θ0 )′(k) G̈T l(θ̄)(k)(k) (θ̂ − θ0 )(k) },

which is asymptotically normal by consistency, assumption 6 regarding the bound
on the third order term, the Slutsky theorem and the central limit theorem of
Billingsley (1961). Furthermore, we have

γT T

−1/2

ξT,k

θ̂ (k)
kθ̂ (k) k2

= γT T

(µ−1)/2

(T

1/2

kθ̃

(k)

−µ

k2 )

θ̂ (k)

P

−→ ∞.

kθ̂ (k) k2 T →∞

(k)

We obtain the same when adding λT T −1/2 αT ⊙ ŵ(k) . Therefore, we have
∀k ∈
/ S, P(k ∈ Ŝ) ≤ P(−ĠT l(θ̂)(k) =

λT
T

(k)
αT

⊙

(k)
ŵi

+

γT
T

ξT,k

θ̂ (k)
kθ̂ (k) k2

) → 0.

We now pick k ∈ S and consider the event {i ∈ Âk }. Then the Karush-Kuhn-

Tucker conditions for GT ψ(θ̂) are given by

(ĠT l(θ̂))(k),i +

λT
T

(k)
(k)
αT,i sgn(θ̂T,i )

γT

+

T

(k)

ξT,k

θ̂i

kθ̂ (k) k2

= 0.

Using the same reasoning as previously, T 1/2 (ĠT l(θ̂))(k),i is also asymptotically
P

(k)

normal, and θ̃ (k) −→ θ0
T →∞

for k ∈ S, and besides
(k)

(k)

λT T −1/2 αT,i sgn(θ̂i ) = λT

T (η−1)/2

P
−→
(k)
(T 1/2 |θ̃i |)η T →∞

∞,

(k)

such that we obtain the same when adding γT

T −1/2 ξ

∀k ∈ S, ∀i ∈
/ Ak , P(i ∈ Âk ) ≤ P(−(ĠT l(θ̂))(k),i =
We have proved (4.14).

26

λT
T

θ̂i
T,k

kθ̂ (k) k2

. Therefore, we have

γT
(k)
(k)
αT,i sgn(θ̂i )+ ξT,k
T

(k)

θ̂i

kθ̂ (k) k2

) → 0.

5

Double asymptotic

In the previous sections, we worked with a fixed dimension d, where d =

m
P

ck .

k=1

From now on, let us consider the case where d = dT , such that dT → ∞ as T → ∞.

Note that card(S) = O(card(A)) = O(dT ). The speed of growth of the dimension is

supposed to be dT = O(T c ) for some q2 < c < q1 . In this section, we prove that the
adaptive SGL enjoys the oracle property, that is model selection consistency and
optimal rate of convergence for proper choices of 0 ≤ q1 < q2 < 1. We highlight

that our general framework unfortunatly hampers a high degree of flexibility on the
behavior of dT , that is c cannot be set in (0, 1). This issue was encountered by Fan
and Peng (2004) in an i.i.d. and non-adaptive framework. This lack of flexibility
is a necessary cost to cope with the random remainder of the Taylor expansions
as we should take the third order term into account. This problem is moved aside
when considering the simple linear model, where the third order derivative is zero.
For instance, Zou and Zhang (2009) proved the oracle property of the adaptive
elastic-net in a double-asymptotic framework for linear models where 0 ≤ c < 1.
For the asymptotic normality, we use the method of Fan and Peng (2004) and Zou

and Zhang (2009), where we derive the asymptotic distribution of the discrepancy
√
T (θ̂ − θ0 )A times a matrix sequence (QT ) of size r × card(A), r being arbitrary

but finite. This allows for switching from infinite dimensional distribution to finite
dimensional distribution, where we can apply usual tools of asymptotic analysis.
In this section, we provide the conditions to achieve the oracle property as in Fan
and Peng (2004) or Zou and Zhang (2009). In this double asymptotic framework,
the quantities depend on dT , hence on T . They should be indexed by T , which

expresses that the dimension depend on the sample size. In the rest of the paper,
we denote HT := E[∇2θθ l(ǫt ; θ0 )] and MT := E[∇θ l(ǫt ; θ0 )∇θ′ l(ǫt ; θ0 )]. To make the
reading easier, we do not index other quantities by T , which will be implicit. We
remind that the criterion is
θ̂ = arg min{GT l(θ) + p1 (λT , θ̃, θ) + p2 (γT , θ̃, θ)}
θ∈Θ

= arg min{
θ∈Θ

(k)

T
1P

T t=1

l(ǫt ; θ) +

ck (k) (k)
m P
m
γT P
λT P
αT,i |θi | +
ξT,l kθ (l) k2 },
T k=1i=1
T l=1

(k)

(5.1)

with αT,i = |θ̃i |−η and ξT,l = kθ̃ (l) k−µ
2 , where η > 0, µ > 0, and θ̃ is a first step

estimator satisfying

θ̃ = arg min{GT l(θ)}.
θ∈Θ

27

The double asymptotic framework implies that the empirical criterion can be
viewed as a sequence of dependent random variables for which we need refined
asymptotic theorems for dependent sequence of arrays. Shiryaev (1991) proposes
a version of the central limit theorem for dependent sequence of arrays, provided
this sequence is a square integrable martingale difference satisfying the so-called
Lindeberg condition. A similar theorem can be found in Billingsley (1995, theorem
35.12, p.476). We provide here the theorem of Shiryaev (see Theorem 4, p.543 of
Shiryaev, 1991) that we will use to derive the asymptotic distribution of the adaptive
SGL estimator.
Theorem 5.1. (Shiryaev, 1991)
Let a sequence of square-integrable martingale differences ξ T = (ξT,t , FtT ), T ≥ 0,
with FtT = σ(ξT,s , s ≤ t), satisfy the Lindeberg condition, for ǫ > 0, given by
T
X
P
2
T
E[ξT,t
1|ξT,t |>ǫ |Ft−1
] −→ 0,
T →∞

t=0

then if

T
P

t=0

P

2 |F T ] −→ σ 2 , or
E[ξT,t
t
t−1
T →∞

T
P

t=0

P

2
−→ σt2 , then
ξT,t
T →∞

T
P

t=0

d

ξT,t −→ N (0, σt2 ).

Remark. Note that central limit theorems relaxing the stationarity and martingale
difference assumptions for sequences of arrays exist. Neumann (2013) proposes such
a central limit theorem for weakly dependent sequences of arrays. Such sequences
should also satisfy a Lindeberg condition and conditions on covariances. In the rest
of the paper, we use Shiryaev’s result.
We consider problem (5.1), which is the adaptive SGL estimator. In the first
step, we study the convergence rate of the first step unpenalized estimator, which is
plugged in the adaptive specification. The convergence rate of a classic M-estimator
is T 1/2 , for d fixed. For d diverging, we need some additional assumptions.
The two next assumptions are similar to condition (F) of Fan and Peng (2004)
and allows for controlling the minimum and maximum eigenvalues of the limits of
the empirical Hessian and the score cross-product. We denote by λmin (M ) and
λmax (M ) the minimum and maximum eigenvalues of any positive definite square
matrix M .
Assumption 7. HT and MT exist. HT is nonsingular, and there exist b1 , b2 with
0 < b1 < b2 < ∞ and c1 , c2 with 0 < c1 < c2 < ∞ such that, for all T ,
b1 < λmin (MT ) < λmax (MT ) < b2 , c1 < λmin (HT ) < λmax (HT ) < c2 .

28

−1
Let VT = H−1
T MT HT , we deduce there exist a1 , a2 with 0 < a1 < a2 < ∞ such that,

for all T ,

a1 < λmin (VT ) < λmax (VT ) < a2 .
Assumption 8. E[{∇θ l(ǫt ; θ0 )∇θ′ l(ǫt ; θ0 )}2 ] < ∞, for every dT (and then of T ).
Assumption 9. There exist some functions Ψ(.) such that, for all T ,
sup
k=1,··· ,dT

E[∂θk l(ǫt ; θ)∂θk l(ǫt′ ; θ)] ≤ Ψ(|t − t′ |),

and
T
1X

sup

T

T

Ψ(|t − t′ |) < ∞.

t,t′ =1

Assumption 10. Let ζkl,t := ∂θ2k θl l(ǫt ; θ0 ) − E[∂θ2k θl l(ǫt ; θ0 )]. There exist some func-

tions χ(.) such that

|E[ζkl,tζk′ l′ ,t′ ]| ≤ χ(|t − t′ |),
and
sup
T

T
1X

T

Assumption 11. Let υt (C) :=

χ(|t − t′ |) < ∞.

t,t′ =1

sup

{

sup

k,l,m=1,··· ,dT θ:kθ−θ0 k2 ≤νT C

|∂θ3k θl θm l(ǫt ; θ)|}, where

C > 0 is a fixed constant and νT = (dT /T )1/2 . Then

η(C) :=

T
1 X

T2

E[υt (C)υt′ (C)] < ∞.

t,t′ =1

Theorem 5.2. Under Assumptions 1-3, 7-11 and if d4T = o(T ), the sequence of
unpenalized M-estimators solving θ̃ = arg min {GT l(θ)} satisfies
θ∈Θ

kθ̃ − θ0 k2 = Op ((

dT
T

1

) 2 ).

Both vectors θ̃ and θ0 depend on T such that θ̃ = θ̃T and θ0 = θ0,T := θ0,∞ .eT .
Remark. Note that this consistency result requires at most d4T = o(T ), as Theorem
1 of Fan and Peng (2004).
Proof. We proceed as in the proof of Theorem 4.5. We denote νT = (dT /T )1/2 and
we would like to prove that, for any ǫ > 0, there exists Cǫ > 0 such that
P(kθ̃ − θ0 k2 /ν̃T > Cǫ ) < ǫ.

29

(5.2)

To prove (5.2), it is sufficient to show that for any ǫ > 0, there exists Cǫ > 0 such
that
P(kθ̃ − θ0 k2 > Cǫ νT ) ≤ P(∃u ∈ RdT , kuk2 ≥ Cǫ : GT l(θ0 + νT u) ≤ GT l(θ0 ))

= P(∃u ∈ RdT , kuk2 = Cǫ : GT l(θ0 + νT u) ≤ GT l(θ0 )),

by convexity. By a Taylor expansion of GT l(θ0 + νT u), we obtain

GT l(θ0 + νT u) = GT l(θ0 ) + νT ĠT l(θ0 )u +

νT2
2

′

u G̈T l(θ0 )u +

νT3
6

∇′ {u′ G̈T l(θ̄)u}u,

where θ̄ ∈ Θ such that kθ̄ − θ0 k2 ≤ Cǫ νT . We would like to prove
dT

P(∃u ∈ R , kuk2 = Cǫ : νT ĠT l(θ0 )u+

νT2
2

′

u G̈T l(θ0 )u+

νT3
6

∇′ {u′ G̈T l(θ̄)u}u ≤ 0) < ǫ.
(5.3)

To do so, we focus on each quantity of the Taylor expansion to extract the dominant
term. First, for a > 0 and the Markov inequality, we have for the score term
P( sup |ĠT l(θ0 )u| > a) ≤ P( sup kĠT l(θ0 )k2 kuk2 > a)
u:kuk2 =Cǫ
u:kuk2 =Cǫ
≤ P(kĠT l(θ0 )k2 > Caǫ )
≤ ( Caǫ )2 E[kĠT l(θ0 )k22 ]
dT
P
≤ ( Caǫ )2 E[(∂θk GT l(θ0 ))2 ]
k=1

= ( Caǫ )2 T12

≤
By assumption 9,

sup
k=1,··· ,dT

dT
T P
P

E[∂θk l(ǫt ; θ0 )∂θk l(ǫt′ ; θ0 )]

t,t′ =1k=1
T
P
( Caǫ )2 { T12
Ψ(|t
t,t′ =1

− t′ |)}.dT .

E[∂θk l(ǫt ; θ0 )∂θk l(ǫt′ ; θ0 )] ≤ Ψ(|t − t′ |) and

t′ |) < ∞. This implies
Cǫ2 dT
P( sup |ĠT l(θ0 )u| > a) ≤
K1 ,
T a2
u:kuk2 =Cǫ
for some constant K1 > 0.
We now focus on the hessian quantity that can be rewritten as
u′ G̈T l(θ0 )u = u′ E[G̈T l(θ0 )]u + RT (θ0 ),

30

1
T

T
P

Ψ(|t −

t,t′ =1

where RT (θ0 ) =

dT
P

uk ul {∂θ2k θl GT l(θ0 ) − E[∂θ2k θl GT l(θ0 )]}. We have

k,l=1

dT
X

T
1 X
E[RT (θ0 )] = 0, Var(RT (θ0 )) = 2
T ′
t,t

uk uk′ ul ul′ E[ζkl,t.ζk′ l′ ,t′ ],

=1k,k ′ ,l,l′ =1

where ζkl,t = ∂θ2k θl l(ǫt ; θ0 ) − E[∂θ2k θl l(ǫt ; θ0 )]. Let b > 0, we deduce by the Markov
inequality and assumption 10,

P(|RT (θ0 )| > b) ≤

1
b

E[R2T (θ0 )]
2

≤

K2 kuk42 d2T
b2

T

≤

K2 Cǫ4 d2T
b2 T

,

where K2 > 0. Furthermore, by assumption 7,
u′ E[G̈T l(θ0 )]u ≥ λmin (HT )u′ u.
As for the third order term, we have
T
1 P
P P
|uk uk uk ul ul ul ||∂ 3
l(ǫt ; θ̄).∂θ3l θl θl l(ǫt′ ; θ̄)|
2
1 2 3
T t,t′ =1k1 ,k2 ,k3 l1 ,l2 ,l3 1 2 3 1 2 3 θk1 θk2 θk3

|∇{u′ G̈T l(θ̄)u}u|2 ≤

≤ kuk62 d3T

T
1 P
υt (Cǫ )υt′ (Cǫ ),
T 2 t,t′ =1

where
υt (C0 ) = sup {

sup

k1 k2 k3 θ:kθ−θ0 k2 ≤νT C0

|∂θ3k

θ θ
1 k2 k3

l(ǫt ; θ)|}.

Note that υt (C0 ) depends on dT and C0 . By assumption 11, we have

η(C0 ) :=

T
1 X

T2

t,t′ =1

E[υt (C0 )υt′ (C0 )] < ∞.

By the Markov inequality, for c > 0, we conclude that

P(∃u, kuk2 = Cǫ :

νT2
6

sup
kθ̄−θ0 k2 ≤νT Cǫ

|∇{u′ G̈T l(θ̄)u}u| > c) ≤

νT4 d3T Cǫ6
36c2

η(Cǫ ).

We can now bound (5.3) thanks to proper choices of a, b, c and Cǫ . We denote

31

by δT = λmin (HT )Cǫ2 νT , and using

νT
2

E[u′ G̈T l(θ0 )u] ≥ δT , we have
νT

νT2

∇{u′ G̈T l(θ̄)u}u ≤ 0)
T l(θ0 )u +
2
6
≤ P(∃u ∈ RdT , kuk2 = Cǫ : |ĠT l(θ0 )u| > δT /4) + P(∃u ∈ RdT , kuk2 = Cǫ : ν2T |RT (θ0 )| > δT /4)

P(∃u ∈

RdT , kuk2

= Cǫ : ĠT l(θ0 )u +

+P(∃u ∈ RdT , kuk2 = Cǫ :
≤
≤

νT2
6

u′ G̈

|∇{u′ G̈T l(θ̄)u}u| > δT /4)

sup

θ̄:kθ̄−θ0 k2 <νT Cǫ
2
2
4
4νT dT Cǫ
16νT4 d3T Cǫ6
+
+
η(Cǫ )
T δT2
36δT2
dT
d2T
+ C3 νT2 d3T Cǫ2 η(Cǫ ),
C1
+
C
2
T
T Cǫ2 νT2

16Cǫ2 dT K1
T δT2

where C1 , C2 , C3 are strictly positive constants. We chose νT = (

dT

deduce
P(∃u ∈ RdT , kuk2 = Cǫ : νT ĠT l(θ0 )u +
≤

C1

d2T

Cǫ

T

+ C2
2

+

d4T Cǫ2
T

νT2
2

u′ G̈T l(θ0 )u +

νT3
6

T

1

) 2 , we then

∇{u′ G̈T l(θ̄)u}u ≤ 0)

η(Cǫ ).

Now we fix Cǫ sufficiently large enough, such that C1 /Cǫ2 < ǫ/3. Once this constant is fixed, there exists a T0 such that for T > T0 we have C2
d4 C 2
C3 TT ǫ η(Cǫ )

d2T
T

< ǫ/3 and

< ǫ/3 under the assumption that d4T = o(T ). Consequently, we obtain

P(∃u ∈ RdT , kuk2 = Cǫ : GT l(θ0 )+νT ĠT l(θ0 )u+
This proves (5.2), that is kθ̃ − θ0 k2 = Op ((

dT
T

νT2
2

u′ G̈T l(θ0 )u+

νT3
6

∇{u′ G̈T l(θ̄)u}u ≤ 0) < ǫ.

1

) 2 ).

The first step estimator used for the adaptive weights is (T /dT )1/2 -consistent.
However, the estimated quantities on Ac converge to zero by consistency. We then
˜
propose a slight modification of the first step estimator, denoted θ̃, which disappears
asymptotically as follows
θ̃˜ = θ̃ + eT ,
such that eT → 0 is a strictly positive quantity. We choose eT = T −κ with κ > 0.

This means we add in the adaptive weights a power of T to the first step estimator,
that is
(k)
˜(k)
˜
αT,i = |θ̃i | = |θ̃ + T −κ |−η , ξT,l = kθ̃ (l) k2 = kθ̃ (l) + T −κ k−µ
2 .

32

γT c
Theorem 5.3. Under assumptions 1-3, 7-11, if d4T = o(T ), and if √ T 2 +κµ −→
T →∞
T
λT
0, √ T κη −→ 0, then the sequence of penalized estimators θ̂ solving 5.1 satisfies
T →∞
T
kθ̂ − θ0 k2 = Op ((

dT

1

) 2 ).

T

Remark. Note that d4T = o(T ) is as in Fan and Peng (2004), Theorem 1. The
asymptotic behaviors of the regularization terms provide a condition such that the
penalty terms are negligible with respect to (dT /T )1/2 .
Proof. We proceed as we did for proving Theorem 5.2. Let νT = (dT /T )1/2 . We
would like to prove that for any ǫ > 0, there exists Cǫ > 0 such that
P(kθ̂ − θ0 k2 /νT > Cǫ ) < ǫ.

(5.4)

To prove (5.4), we show
νT
νT2
P(∃u ∈ RdT , kuk2 = Cǫ : ĠT l(θ0 )u + u′ G̈T l(θ0 )u + ∇′ {u′ G̈T l(θ̄)u}u
2
6
˜ θ + ν u) − p (λ , θ̃,
˜ θ ) + p (γ , θ̃,
˜ θ + ν u) − p (γ , θ̃,
˜ θ }) ≤ 0) < ǫ.
+νT−1 {p1 (λT , θ̃,
0
T
T
0
T
0
T
0
1
2
2 T
(5.5)
a relationship obtained by convexity and a Taylor expansion.
The score quantity can be upper bounded as

|ĠT l(θ0 )u| ≤ kĠT l(θ0 )k2 kuk2 = Op ((

dT
T

1

) 2 )kuk2 = Op (νT )kuk2 ,

where we used assumption 9 to obtain the bound in probability of the score.
As for the third order term, we have by the Cauchy-Schwartz inequality
|∇′ {u′ G̈T l(θ̄)u}u|2 ≤ kuk62 d3T

dT
dT
T2
1 P
P
P
l(ǫt ; θ̄)∂θ3k θl θm l(ǫt′ ; θ̄)}
∂3
{
2
2 2
T 2 t,t′ =1 k1 ,l1 ,m1 =1k2 ,l2 ,m2 =1 θk1 θl1 θm1

= kuk62 d3T η(Cǫ ).
This implies

3/2

∇′ {u′ G̈T l(θ̄)u}u = Op (dT kuk32 ).

33

Hence by the Markov inequality
dT

P(∃u ∈ R , kuk2 = Cǫ :

|νT2 ∇′ {u′ G̈T l(θ̄)u}u|

νT4 Cǫ6 d3T

> a) ≤

a2

η(Cǫ ).

where we used assumption 11.
Finally, the hessian quantity can be treated as in the proof of Theorem 5.2. We
dT
P
denote by RT (θ0 ) =
uk ul {∂θ2k θl GT l(θ0 ) − E[∂θ2k θl GT l(θ0 )]}. We have
k,l=1

u′ G̈T l(θ0 )u = u′ E[G̈T l(θ0 )]u + RT (θ0 ).
By assumption 10 and the Markov inequality, for any κ > 0, we obtain

P(|RT (θ0 )| > κ) ≤

K2 Cǫ4 d2T
K2 kuk42 d2T
2
≤
,
E[R
(θ
)]
≤
0
T
κ2
κ2
T
κ2 T
1

with K2 > 0. This relationship holds for any κ > 0. Then for T large enough, we
deduce that |RT (θ0 )| = op (1). Consequently
νT2
2

′

u G̈T l(θ0 )u ≥

νT2
2

λmin (HT )kuk22 + op (1)νT2 kuk22 .

We focus on the penalty terms. We have
˜
˜
p1 (λT , θ̃, θ0 + νT u) − p1 (λT , θ̃, θ0 )

= λT T −1

P P

k∈S i∈Ak

(k)

(k)

(k)

(k)

αT,i {|θ0,i + νT ui | − |θ0,i |},

P P (k)
(k)
˜
˜
αT,i νT |ui |.
and |p1 (λT , θ̃, θ0 + νT u) − p1 (λT , θ̃, θ0 )| ≤ λT T −1
k∈S i∈Ak

As for the l1 /l2 norm, we obtain
˜
˜
p2 (γT , θ̃, θ0 + νT u) − p2 (γT , θ̃, θ0 )

= γT T −1

P

(l)

l∈S1

P
˜
˜
and |p2 (γT , θ̃, θ0 + νT u) − p2 (γT , θ̃, θ0 )| ≤ γT T −1 ξT,l νT ku(l) k2 .
l∈S

34

(l)

ξT,l {kθ0 + νT uk2 − kθ0 k2 }

For the l1 norm penalty, using { min

k∈S,i∈Ak

λT T −1

P P

k∈S i∈Ak

˜(k)
|θ̃i |}−η ≤ T κη , then

(k)

(k)

αT,i νT |ui | ≤ λT T −1 νT {

P P ˜(k) −2η 1/2
|θ̃i | } kuk2

k∈S i∈Ak

≤ λT

T −1 ν

√
dT

T

˜(k)
{ min |θ̃i |}η
k∈S,i∈Ak
√
≤ λT T −1 νT dT T κη kuk2 ,

kuk2

c

by the Cauchy-Schwartz inequality. Then if λT T 2 −1+κη is bounded, we obtain
˜
˜
p1 (λT , θ̃, θ0 + νT u) − p1 (λT , θ̃, θ0 ) = O(νT2 )kuk2 .
As for the l1 /l2 term, using {min kθ̃˜(l) k2 }−µ ≤ T κµ , we obtain
l∈S

γT T −1

m
P

P ˜(l) −2µ 1/2
kθ̃ k2 } kuk2
l∈S
√
dT
≤ γT T −1 νT
kuk2
˜
{min kθ̃ (l) k2 }µ
l∈S
√
−1
≤ γT T νT dT T κµ kuk2 ,

ξT,l νT ku(l) k2 ≤ γT T −1 νT {

l=1

c

by the Cauchy-Schwartz inequality. Then if γT T 2 −1+κµ is bounded, we obtain
˜
˜
p2 (γT , θ̃, θ0 + νT u) − p2 (γT , θ̃, θ0 ) = O(νT2 )kuk2 .
We now can prove (5.5). Let δT = λmin (HT )Cǫ2 νT and using

νT
2

E[u′ G̈T l(θ0 )u] ≥ δT ,

we have
P(∃u ∈ RdT , kuk2 = Cǫ : ĠT l(θ0 )u + νT u′ G̈T l(θ0 )u/2 + νT2 ∇{u′ G̈T l(θ̄)u}u/6
˜ θ + ν u) − p (λ , θ̃,
˜ θ ) + p (γ , θ̃,
˜ θ + ν u) − p (γ , θ̃,
˜ θ )} ≤ 0)
+νT−1 {p1 (λT , θ̃,
0
T
0
0
T
0
1 T
2 T
2 T

≤ P(∃u ∈ RdT , kuk2 = Cǫ : |νT u′ G̈T l(θ0 )u/2| ≤ |ĠT l(θ0 )u| + |νT2 ∇{u′ G̈T l(θ̄)u}u/6|
˜ θ ) − p (λ , θ̃,
˜ θ + ν u)| + |p (γ , θ̃,
˜ θ ) − p (γ , θ̃,
˜ θ + ν u)|})
+νT−1 {|p1 (λT , θ̃,
0
0
T
0
0
T
1 T
2 T
2 T
≤ P(∃u ∈ RdT , kuk2 = Cǫ : |ĠT l(θ0 )u| > δT /8) + P(∃u ∈ RdT , kuk2 = Cǫ :
+P(∃u ∈

RdT , kuk2

= Cǫ : |

νT2

∇{u′ G̈

> δT /8)
˜
˜
+P(∃u ∈ RdT , kuk2 = Cǫ : |p1 (λT , θ̃, θ0 ) − p1 (λT , θ̃, θ0 + νT u)| > νT δT /8)
˜
˜
+P((∃u ∈ RdT , kuk2 = Cǫ : |p2 (γT , θ̃, θ0 ) − p2 (γT , θ̃, θ0 + νT u)| > νT δT /8)
≤

Cst

Cǫ2
< ǫ,

6

+ Cst {νT2 Cǫ6 d3T η(Cǫ )} +

T l(θ̄)u}u|

Cst νT2 d2T Cǫ4
T δT2

35

+ ǫ/5 + ǫ/5

νT
2

|RT (θ0 )| > δT /8)

with Cst > 0 a generic constant. We used d4T = o(T ) and for Cǫ large enough
˜ θ ) − p (λ , θ̃,
˜ θ + ν u)| > ν δ /8) ≤ ǫ/5,
P(∃u ∈ RdT , kuk2 = Cǫ : |p1 (λT , θ̃,
0
0
T
T T
1 T
˜
˜
d
P(∃u ∈ R T , kuk2 = Cǫ : |p2 (γT , θ̃, θ0 ) − p2 (γT , θ̃, θ0 + νT u)| > νT δT /8) ≤ ǫ/5.
c

Thus we obtain for Cǫ and T large enough, with the conditions γT T 2 −1+κµ → 0
c

and λT T 2 −1+κη → 0 that

kθ̂ − θ0 k2 = Op (νT ) = Op ((

dT
T

1

) 2 ).

To achieve the oracle property, we need some additional assumptions regarding
the adaptive penalty components.
Assumption 12. For any T , there exists β such that 0 < β < min θ0,i,Ak , k ∈ S.
i∈Ak

Moreover,

1/2

β −1 T −1 {λT dT E[ max

k∈S,i∈Ak

αT,Ak ,i ] + γT E[max ξT,k ]} −→ 0.
k∈S

T →∞

Assumption 13. The model complexity is assumed to behave as d5T = o(T ), which
implies that 0 < c <

1
5.

The regularization parameters are chosen such that they

satisfy
γT c
√ T 2 +κµ −→ 0,
T →∞
T
λT
√ T κη −→ 0,
T →∞
T
γT (1+µ)(1− c −κη)−1
2
−→ ∞.
T
T →∞
λ1+µ
T

γT 1
√ T 2 [(1+µ)(1−c)−1] −→ ∞,
T →∞
T
λT 1
√ T 2 [(1+η)(1−c)−1] −→ ∞,
T →∞
T

Remark. The main condition is d5T = o(T ), which is the same as Fan and Peng
(2004). This condition comes from the control for the third order derivative of the
empirical criterion. Note that simple cases allow for a framework where 0 ≤ c < 1.

Moreover, these asymptotic behaviors are closely related to condition (A5) of Zou
and Zhang (2009). In section 6, we provide further details about the calibration of
the adaptive weights and κ.
Assumption 14. Let FtT = σ(XT,s , s ≤ t) with XT,t =

√

−1/2

−1
T QT VT,AA HT,AA
ĠT lt (θ0 )A ,
P

(QT ) is a sequence of r × card(A) matrices such that QT × Q′T → C, for some

36

−1
r × r nonnegative symmetric matrix C, VT,AA = (H−1
T MT HT )AA and ĠT lt (θ0 )A =
1
T ∇A l(ǫt ; θ0 ).

E[

Then XT,t is a martingale difference and we have

sup

i,j=1,··· ,dT

T
E[{∂θi l(ǫt ; θ0 )∂θj l(ǫt ; θ0 )}2 |Ft−1
]λmax,t−1 (HTt−1 )] ≤ B̄ < ∞,

with
T
HTt−1 := E[∇l(ǫt ; θ0 )∇′ l(ǫt ; θ0 )|Ft−1
] ≤ λmax (HTt−1 ) < ∞.

Theorem 5.4. Under assumptions 1-3, and assumptions 7-14, the sequence of
adaptive estimator θ̂ solving (5.1) satisfies
lim P(Â = A) = 1, and
√
d
−1/2
T QT VT,AA (θ̂A − θ0,A ) → N (0, C),

T →∞

P

where (QT ) is a sequence of r × card(A) matrices such that QT × Q′T → C, for some

−1
r × r nonnegative symmetric matrix C and VT,AA = (H−1
T MT HT )AA .

Proof. Model selection consistency consists of proving that the probability of the
event {Â = A} tends to one asymptotically. This event is
(k)

(k)

{Â = A} = {∀k ∈ S, ∀i ∈ Ak , |θ̂i | > 0} ∩ {∀k = 1, · · · , m, ∀i ∈ Ack , θ̂i

= 0}.

Hence we prove
(k)

(k)

P({∀k ∈ S, ∀i ∈ Ak , |θ̂i | > 0} ∩ {∀k = 1, · · · , m, ∀i ∈ Ack , θ̂i

= 0}) −→ 1. (5.6)
T →∞

Model selection consistency can be decomposed into two parts: recovering the active indices by estimating nonzero coefficients; discarding the inactive indices by
shrinking to zero the related coefficients. Now (5.6) can be proved by first showing
that for any T , there exists β such that 0 < β < min θ0,i,Ak , with k ∈ S and
i∈Ak

P(kθ̂A − θ0,A k2 < β) −→ 1.
T →∞

(5.7)

The second part regarding nonactive indices can be proved as


 P( ∩ c {kẑ (k) k2 < 1}) −→ 1,
k∈S


∩ ∩c
 P(k∈S
i∈A

k

(k)
{|ŵi |

T →∞

< 1}) −→ 1,

(5.8)

T →∞

where ẑ (k) (resp. ŵ(k) ) is the subgradient of kθ̂ (k) k2 (resp. kθ̂ (k) k1 ) given in (3.1).

37

Hence (5.7) and (5.8) prove (5.6).
We first focus on (5.7), which is equivalent to
P(kθ̂Ak − θ0,Ak k2 > β) −→ 0.
T →∞

By the Karush-Kuhn-Tucker optimality conditions, we have
ĠT l(θ̂)A + λT T −1 αT,A ⊙ sgn(θ̂A ) + γT T −1 ςT = 0,
θ̂Ak

, k ∈ S). We denote by αT,Ak = (αT,i , i ∈ Ak ), a vector
kθ̂Ak k2
of size RcAk . By a Taylor expansion of the gradient component around θ0,A , we

where ςT = vec(ξT,k

have
1
ĠT l(θ0 )A + HT,AA (θ̂A − θ0,A ) + PT (θ0 )(θ̂A − θ0,A ) + ∇′A {(θ̂A − θ0,A )′ G̈T l(θ̄)(θ̂A − θ0,A )}
2
+λT T −1 αT,A ⊙ sgn(θ̂A ) + γT T −1 ςT = 0
−1 α
−1 ς
⇔ θ̂A = θ0,A − H−1
T,A ⊙ sgn(θ̂A ) + γT T
T
T,AA (ĠT l(θ0 )A + λT T

1
−1
′
′
−H−1
T,AA ∇A {(θ̂A − θ0,A ) G̈T l(θ̄)AA (θ̂A − θ0,A )} − HT,AA PT (θ0 )(θ̂A − θ0,A ),
2
where kθ̄−θ0 k2 ≤ kθ̂−θ0 k2 , PT (θ0 ) = G̈T l(θ0 )AA −HT,AA and HT,AA = E[∇2θθ′ l(ǫt ; θ0 )]AA .
Then using kθ̂A − θ0,A k2 = Op ((

dT
T

1

) 2 ), we obtain

−1
−1 α
P(kθ̂A − θ0,A k2 > β) ≤ P(kH−1
T,A ⊙ sgn(θ̂A )k2
T,AA ĠT l(θ0 )A k2 + kHT,AA k2 kλT T
−1 ς k
+ kH−1
T 2
T,AA k2 kγT T

′
′
+ kH−1
T,AA k2 k∇A {(θ̂A − θ0,A ) G̈T l(θ̄)AA (θ̂A − θ0,A )}/2k2

+ kH−1
T,AA k2 kPT (θ0 )(θ̂A − θ0,A )k2 > β)

−1
−1 kα
≤ P(λ−1
T,A k2
min (HT )kĠT l(θ0 )A k2 + λmin (HT )λT T

−1 kς k + λ−1 (H )C 2 (d /2T )k∇′ {G̈ l(θ̄)
+ λ−1
T
AA }k2
T
T 2
0 T
A
min
min (HT )γT T

1/2 kP (θ )k > β) + P(kθ̂ − θ
1/2 C ),
+ λ−1
T 0 2
A
0,A k2 > (dT /T )
0
min (HT )C0 (dT /T )

−1
for C0 > 0 large enough, and we used kH−1
T xk2 ≤ λmin (HT )kxk2 for any vector

x ∈ RdT .

Let us proceed element-by-element. We have by the Markov inequality
P(λ−1
min (HT )C0

r

2
36λ−2
dT
β
min (HT )C0 dT
kPT (θ0 )k2 > ) ≤
E[kPT (θ0 )k22 ].
T
6
T β2

38

We have
E[kPT k22 ]

=

T
X X
1 X

T2

E[ζkl,tζk′ l′ ,t′ ],

t,t′ =1k,k ′ ∈Al,l′ ∈A

where ζkl,t = ∂θ2k θl l(ǫt ; θ0 ) − E[∂θ2k θl l(ǫt ; θ0 )]. By assumption 10, we obtain
P(λ−1
min (HT )C0

r

2 3
36λ−2
dT
β
min (HT )C0 dT
kPT (θ0 )k2 > ) ≤
.
T
6
β2
T2

As for the third order term, by the Markov inequality
4 2
9λ−2
β
1
min (HT )C0 dT
2 dT
′
(H
)C
k∇
{
G̈
l(
θ̄)
}k
>
)
≤
E[k∇′A {G̈T l(θ̄)AA }k22 ].
P( λ−1
T
T
AA
2
0
A
2 min
T
6
T2

We obtain
E[k∇′A {G̈T l(θ̄)AA }k22 ] ≤
≤

T
1 P
P
P
E[|∂θ3k θk θk l(ǫt ; θ0 ).∂θ3l θl θl l(ǫt ; θ0 )l(ǫt′ ; θ0 )|]
2
1 2 3
1 2 3
T t,t′ =1k1 ,k2 ,k3 ∈Al1 ,l2 ,l3 ∈A

1

T

d3
2 T

T
P

E[υt (C0 )υt′ (C0 )] = η(C0 )d3T ,

t,t′ =1

by assumption 11, where υt (C0 ) = sup

k1 k2 k3

duce that

supq

θ:kθ−θ0 k2 ≤

dT
T

C0

|∂θ3k

θ θ
1 k2 k3

l(ǫt ; θ0 )|. We de-

4 5
9λ−2
β
1 −1
min (HT )C0 dT
2 dT
′
η(C0 ).
P( λmin (HT )C0 k∇A {G̈T l(θ̄)AA }k2 > ) ≤
2
T
6
4
T2

We now turn to the score quantity. By the Markov inequality and assumption
9, we have
P(λ−1
min (HT )kĠT l(θ0 )A k2 > β/6) ≤
≤
≤
≤

λ−2
min (HT )36

E[kĠT l(θ0 )A k2 ]
β2
T
λ−2
min (HT )36 1 P P
E[∂θk l(ǫt ; θ0 )∂θk l(ǫt′ ; θ0 )]
2
2
β
T t,t′ =1k∈A
T
λ−2
min (HT )36 1 1 P
{
Ψ(|t − t′ |)}dT
β2
T T t,t′ =1

λ−2
min (HT )36KdT
T β2

39

,

with K > 0. Hence we deduce
−1
−1 kς k > β/2)
−1 kα
P(kθ̂A − θ0,A k2 > β) ≤ P(λ−1
T 2
T,A k2 + λmin (HT )γT T
min (HT )λT T

1/2 kP (θ )k > β/6)
+ P(kθ̂A − θ0,A k2 > (dT /T )1/2 C0 ) + P(λ−1
T 0 2
min (HT )C0 (dT /T )
′
2
+ P(λ−1
min (HT )C0 (dT /2T )k∇A {G̈T l(θ̄)AA }k2 > β/6)

+ P(λ−1
min (HT )kĠT l(θ0 )A k2 > β/6)
≤
+

2λ−1
min (HT )
1/2
{λT T −1 dT E[ max αT,Ak ,i ] + γT T −1 E[max ξT,k ]}
k∈S
k∈S,i∈Ak
β
−2
−2
−2
4
5
36λmin (HT )KdT 9λmin (HT )C0 dT
36λmin (HT )C02 d3T
+
η(C
)
+
+ ǫ.
0
T β2
4
T2
β2
T2

For T and C0 large enough, if d5T = o(T ), by assumption 12, that is if
1/2

β −1 T −1 {λT dT E[ max

k∈S,i∈Ak

αT,Ak ,i ] + γT E[max ξT,k ]} −→ 0,
T →∞

k∈S

then
P(kθ̂A − θ0,A k2 > β) −→ 0.
T →∞

We now turn to the second step of model selection consistency. First we prove
P( ∩ c {kẑ (k) k2 < 1}) −→ 1 ⇔ P( ∪ c {kẑ (k) k2 ≥ 1}) −→ 0.
T →∞

k∈S

(5.9)

T →∞

k∈S

This is equivalent to proving
(k)

P( ∪ c {kĠT l(θ̂)(k) + λT T −1 αT ⊙ ŵ(k) k2 ≥ γT T −1 ξT,k }) −→ 0.
T →∞

k∈S

We have for k ∈ S c that kŵ (k) k∞ ≤ 1, which implies by the optimality conditions

of Karush-Kuhn-Tucker that

(k)

P( ∪ c {kĠT l(θ̂)(k) + λT T −1 αT ⊙ ŵ(k) k2 ≥ γT T −1 ξT,k })
k∈S

(k)

≤ P( ∪ c {kĠT l(θ̂)(k) k2 ≥ γT T −1 ξT,k − λT T −1 kαT k2 }).
k∈S

By a Taylor expansion around θ0 , let θ̄ such that kθ̄ − θ0 k ≤ kθ̂ − θ0 k, we have
(k)

P( ∪ c {kẑ (k) k2 ≥ 1}) ≤ P( ∪ c {kĠT l(θ0 )(k) k2 ≥ γT T −1 ξT,k − λT T −1 kαT k2
k∈S

k∈S

− kG̈T l(θ0 )(k)(k) k2 kθ̂ − θ0 k2 − k∇′ {G̈T l(θ̄)(k)(k) }(k) k2 kθ̂ − θ0 k22 })
˜(k) −η
˜
−1 d1/2 max (|θ̃
≤ P( ∪ c {kĠT l(θ0 )(k) k2 ≥ γT T −1 kθ̃ (k) k−µ
2 − λT T
i | )
T
c
k∈S

− kG̈T l(θ0 )(k)(k) k2 kθ̂ − θ0 k2 −

40

k∇′ {G̈

k∈S ,i∈Gk
2
T l(θ̄)(k)(k) }(k) k2 kθ̂ − θ0 k2 }),

where we used kG̈T l(θ0 )(k)(k) (θ̂−θ0 )k2 ≤ kG̈T l(θ̄)(k)(k) k2 kθ̂−θ0 k2 and kG̈T l(θ0 )(k)(k) k2 =
kG̈T l(θ0 )(k)(k) ks . Let ǫ > 0, and Kǫ strictly positive constants, we proved for T large
enough that

P(kθ̂ − θ0 k2 > Kǫ (dT /T )1/2 ) < ǫ/6.
We deduce that
˜
−1 d1/2 max
P( ∪ c {kẑ (k) k2 ≥ 1}) ≤ P( ∪ c {kĠT l(θ0 )(k) k2 ≥ γT T −1 kθ̃ (k) k−µ
2 − λT T
T
c
k∈S

k∈S ,i∈Gk

k∈S

− kG̈T l(θ0 )(k)(k) k2 (dT /T )1/2 Kǫ − k∇′ {G̈T l(θ̄)(k)(k) }(k) k2 (
Let M1,T = (

γT
T

dT
T

˜(k)
(|θ̃i |−η )

)2 Kǫ2 }) + ǫ/6.

1

) 1+µ , then we obtain

P( ∪ c {kẑ (k) k2 ≥ 1}) ≤
k∈S

P

k∈S c

−1 d1/2 max
{P(kĠT l(θ0 )(k) k2 ≥ γT T −1 kθ̃˜(k) k−µ
2 − λT T
T
c

k∈S ,i∈Gk

(|θ̃˜i |−η )
(k)

dT
1
˜
− kG̈T l(θ0 )(k)(k) k2 (dT /T ) 2 Kǫ − k∇′ {G̈T l(θ̄)(k)(k) }(k) k2 ( )2 Kǫ2 , kθ̃ (k) k2 ≤ M1,T )
T
+ P(kθ̃˜(k) k > M )} + ǫ/6.
2

1,T

Consequently, we have the relationship
P( ∪ c {kẑ (k) k2 ≥ 1}) ≤
k∈S

P

k∈S c

−µ
{P(kĠT l(θ0 )(k) k2 ≥ γT T −1 M1,T
/4)
1/2

+ P(λT T −1 dT

max
c

k∈S ,i∈Gk

˜(k)
−µ
(|θ̃i |−η ) > γT T −1 M1,T
/4)

−µ
+ P(kG̈T l(θ0 )(k)(k) k2 (dT /T )1/2 Kǫ > γT T −1 M1,T
/4)

dT

−µ
)2 Kǫ2 > γT T −1 M1,T
/4)
5
P
˜
+ P(kθ̃ (k) k2 > M1,T )} + ǫ/6 :=
Ti + ǫ/6.

+ P(k∇′ {G̈T l(θ̄)(k)(k) }(k) k2 (

T

i=1

We then focus on each Ti . We have by the Markov inequality

T1 :=

P

k∈S c

−µ
P(kĠT l(θ0 )(k) k2 > γT T −1 M1,T
/4) ≤

≤
≤

P 16E[kĠT l(θ0 )(k) k22 ]

k∈S c

−µ 2
{γT T −1 M1,T
}

16E[kĠT l(θ0 )k22 ]
−µ 2
{γT T −1 M1,T
}

16dT

−µ 2
T {γT T −1 M1,T
}

γT 1
− 2
= O(( √ T 2 [(1+µ)(1−c)−1] ) 1+µ ).
T

41

˜(k)
Furthermore, using |θ̃i |−η ≤ T κη , we have for T2 that
1/2

P(λT T −1 dT

max

k∈S c ,i∈Gk

1/2
˜(k)
−µ
−µ
/4)
(|θ̃i |−η ) > γT T −1 M1,T
/4) ≤ P(λT T −1 dT T κη > γT T −1 M1,T
1/2

µ
−µ
T κη } < 0).
≤ P(γT T −1 M1,T
/4{1 − 4λT γT−1 dT M1,T
(5.10)
−1/2

The quantity of interest is γT λ−1
T dT

−µ −κη
M1,T
T
that has to converge to ∞ such

that (5.10) converge to zero for T sufficiently large enough. We have
−1/2

γT λ−1
T dT

−µ −κη
M1,T
T
→∞⇔

γT

− 1+µ
2

dT
λ1+µ
T

T −κη(1+µ)+µ → ∞.

As for T3 , we have by the Markov inequality
P
−µ
P((kHT,(k)(k) k2 + kRT,(k) (θ0 )k2 )(dT /T )1/2 Kǫ > γT T −1 M1,T
/4)
T3 :=
c
k∈S
P
−µ
P((kRT,(k) (θ0 )k2 (dT /T )1/2 Kǫ > γT T −1 M1,T
/4 − kHT,(k),(k) k2 (dT /T )1/2 Kǫ )
≤
c
k∈S
P
−µ
−µ
{P((kRT,(k) (θ0 )k2 (dT /T )1/2 Kǫ > γT T −1 M1,T
/8) + P(kHT,(k),(k) k2 (dT /T )1/2 Kǫ > γT T −1 M1,T
/8)}
≤
k∈S c

P 64Kǫ2 dT E[kRT,(k) (θ0 )k22 ] 64Kǫ2 dT kHT,(k)(k) k22
{
≤
+
}
−2µ
−2µ
T γT2 T −2 M1,T
T γT2 T −2 M1,T
k∈S c

≤
≤
≤

64Kǫ2 dT kHT k22
−2µ
γT2 T −1 M1,T

+

64Kǫ2 E[kRT (θ0 )k22 ]

64Kǫ2 dT λ2max (HT )
−2µ
γT2 T −1 M1,T

−2µ
γT2 M1,T

+

64Kǫ2 d3T
−2µ
γT2 M1,T

64Kǫ2 λ2max (HT )
−1/2

{γT T −1/2 dT

−µ 2
M1,T
}

+

64Kǫ2
−3/2

{γT dT

−µ 2
M1,T
}

γT 1
γT 1
− 2
− 2
= O(( √ T 2 [(1+µ)(1−c)−1] ) 1+µ ) + O(( √ T 2 [(1+µ)(2−3c)−1] ) 1+µ ).
T
T
We obtain for T4 by the Markov inequality

T4 :=
≤
≤
≤

P

k∈S c

P(k∇′ {G̈T l(θ̄)(k)(k) }(k) k2 (

dT
T

−µ
)2 Kǫ2 > γT T −1 M1,T
/4)

P 16Kǫ4 d2T E[k∇′ {G̈T l(θ̄)(k)(k) }(k) k22 ]
−2µ
T 2 γT T −2 M1,T

k∈S c

16Kǫ4 d5T E[k∇′ {G̈T l(θ̄)}k22 ]
−2µ
γT2 M1,T

16Kǫ4 d5T η(Kǫ )
−2µ
γT2 M1,T

=

16Kǫ4 η(Kǫ )

γT 1 [(1+µ)(2−5c)−1] − 2
√
) 1+µ ).
=
O((
T2
−5/2
−µ 2
T
}
{γT dT M1,T

42

Finally, we have for T5 that

T5 :=

P

k∈S c

˜
P E[kθ̃ (k) k22 ]
2
M1,T
k∈S c
E[kθ̃˜ − θ k2 ]

P(kθ̃˜(k) k2 > M1,T ) ≤

0 2

≤

2
M1,T

γT 1
− 2
= O(( √ T 2 [(1+µ)(1−c)−1] ) 1+µ ).
T
Hence we obtain from these relationships and using assumption 13
γT

T
λ1+µ
T

µ−( 2c +κη)(1+µ)

−→ ∞,

T →∞

γT 1
√ T 2 [(1+µ)(1−c)−1] −→ ∞,
T →∞
T
such that the latter implies
γT 1
√ T 2 [(1+µ)(2−3c)−1] −→ ∞,
T →∞
T
γT 1
√ T 2 [(1+µ)(2−5c)−1] −→ ∞.
T →∞
T
Consequently each Ti converges to zero for T large enough. Hence
P( ∪ c {kẑ
k∈S

(k)

5
X
Ti + ǫ/6 −→ ǫ.
k2 ≥ 1}) ≤
T →∞

i=1

For ǫ → 0, we prove P( ∪ c {kẑ (k) k2 ≥ 1}) → 0 for T large enough.
k∈S

As for the second part of the model selection procedure, we prove that
(k)

∩ {|ŵ i | < 1}) −→ 1 ⇔ P( ∪

P( ∩

k∈S i∈Ack

(k)

∪ {|ŵ i | ≥ 1}) −→ 0.

k∈S i∈Ack

T →∞

T →∞

By the optimality conditions, we have
P( ∪

(k)

∪ c {|ŵ i | ≥ 1}) = P( ∪

k∈S i∈Ak

(k)

∪ c {|ĠT l(θ̂)(k),i | ≥ λT T −1 αT,i }).

k∈S i∈Ak

43

(5.11)

Then by a Taylor expansion around θ0 , with θ̄ between θ̂ and θ0 , we have
P( ∪

(k)

P 2
GT l(θ0 )(θ̂j − θ0,j )]i
∪ c {|ĠT l(θ0 )(k),i + [ ∂ij

∪ c {|ŵ i | ≥ 1}) = P( ∪

k∈S i∈Ak

+

k∈S i∈Ak
T
P
P
3 l(ǫ ; θ̄)(θ̂
[ T −1 ∂ijk
t
j
t=1
j,k

j

(k)

− θ0,j )2 /2]i | ≥ λT T −1 αT,i })

P 2
(k)
∪ c {|ĠT l(θ0 )(k),i | ≥ λT T −1 αT,i − [ ∂ij
GT l(θ0 )(θ̂j − θ0,j )]i

≤ P( ∪

k∈S i∈Ak

j

T
P
P
3 l(ǫ ; θ̄)(θ̂ − θ )2 /2] |}).
− [ T −1 ∂ijk
t
j
0,j
i
t=1

j,k

Let M2,T = (

λT

1

) 1+η . Then using kθ̂ − θ0 k2 = Op ((

dT

T
inequality, we obtain
P( ∪

(k)

∪ {|ŵ i | ≥ 1}) ≤

k∈S i∈Ack

T

1

) 2 ) and the Cauchy-Schwartz

P P

P 2
(k)
GT l(θ0 )(θ̂j − θ0,j )]i
{P(|ĠT l(θ0 )(k),i | ≥ λT T −1 αT,i − [ ∂ij

j,k

t=1

P P

P 2
−η
{P(|ĠT l(θ0 )(k),i | ≥ λT T −1 M2,T
− { (∂ij
GT l(θ0 ))2 }1/2 Kǫ (dT /T )1/2

k∈S i∈Ack

j

T
P
P
˜(k)
˜(k)
3 l(ǫ ; θ̄)(θ̂ − θ )2 /2] |, |θ̃
− [ T −1 ∂ijk
t
j
0,j
i
i | ≤ M2,T ) + P(|θ̃i | > M2,T )}

≤

k∈S i∈Ack

− {
+
≤

P

j

T −2

j,k,l,m
(k)
P(|θ̃˜i |

P P

k∈S i∈Ack

T
P

3 l(ǫ ; θ̄)∂ 3 l(ǫ ′ ; θ̄)}1/2 K 2 (d /T ))
∂ijk
t
t
ǫ T
ilm

t,t′ =1

> M2,T )} + ǫ/5
−η
{P(|ĠT l(θ0 )(k),i | ≥ λT T −1 M2,T
/3)

P 2
−η
+ P({ (∂ij
GT l(θ0 ))2 }1/2 Kǫ (dT /T )1/2 > λT T −1 M2,T
/3)
j

+ P({

P

T −2

j,k,l,m

T
P

t,t′ =1

3 l(ǫ ; θ̄)∂ 3 l(ǫ ′ ; θ̄)}1/2 K 2 (d /T ) > λ T −1 M −η /3)
∂ijk
T
t
t
ǫ T
ilm
2,T

4
P
(k)
+ P(|θ̃˜i | > M2,T )} + ǫ/5 :=
Ti + ǫ/5.
i=1

We proceed as for inactive groups. For T1 , we have by the Markov inequality

T1 :=

P P

k∈S i∈Ack

P(|ĠT l(θ0 )(k),i | ≥

−η
λT T −1 M2,T
/3)

≤
≤

P P 9E[|ĠT l(θ0 )(k),i |2 ]

k∈S i∈Ack

−η 2
{λT T −1 M2,T
}

9E[kĠT l(θ0 )k22 ]
−η 2
{λT T −1 M2,T
}

λT 1
2
= O(( √ T 2 [(1+η)(1−c)−1] )− 1+η ).
T

44

As for T2 , we have
T2 :=
=

P P

k∈S i∈Ack

P P

k∈S i∈Ack

≤
≤

P 2
−η
P({ (∂ij
GT l(θ0 ))2 }1/2 Kǫ (dT /T )1/2 > λT T −1 M2,T
/3)
j

P
P
−η
P(({ PT,(k),j (θ0 )}1/2 + { H2T,(k),j }1/2 )Kǫ (dT /T )1/2 > λT T −1 M2,T
/3)
j

j

2
36dT kHT k22
P P P 36dT E[PT,(k),j (θ0 )]
}
+
{
−η 2
−η 2
T {λT T −1 M2,T
}
T {λT T −1 M2,T
}
k∈S i∈Ack j

36dT λ2max (HT )

−η 2
T {λT T −1 M2,T
}

36dT E[kPT (θ0 )k22 ]

+

−η 2
T {λT T −1 M2,T
}

λT 1
λT 1
− 2
− 2
= O(( √ T 2 [(1+η)(1−c)−1] ) 1+η ) + O(( √ T 2 [(1+η)(2−3c)−1] ) 1+η ).
T
T
Furthermore, for the third order term in T3 , we have
T3 :=
≤

P P

k∈S i∈Ack

P({

P

T −2

j,k,l,m

9d2T E[k∇′ {G̈T l(θ̄}k22 ]
−η 2
T 2 {λT T −1 M2,T
}

T
P

3 l(ǫ ; θ̄)∂ 3 l(ǫ ′ ; θ̄)}1/2 K 2 (d /T ) > λ T −1 M −η /3)
∂ijk
T
t
t
ǫ T
ilm
2,T

t,t′ =1

λT 1
2
= O(( √ T 2 [(1+η)(2−5c)−1] )− 1+η ).
T

Finally, we have for T4 that

T4 :=

P

i∈Ack

˜
P P E[|θ̃i |2 ]
2
k∈S i∈Ack M2,T
λT 1 [(1+η)(1−c)−1] − 2
E[kθ̃˜ − θ0 k22 ]
√
T2
=
O((
) 1+η ).
≤
2
M2,T
T
(k)

˜(k)
P(|θ̃i | > M2,T ) ≤

λT 1
We have from these relationships and by assumption 13, √ T 2 [(1+η)(1−c)−1] −→ ∞
T →∞
T
implies
λT 1
√ T 2 [(1+η)(2−3c)−1] −→ ∞,
T →∞
T
λT 1
√ T 2 [(1+η)(2−5c)−1] −→ ∞.
T →∞
T
We deduce
P( ∪

(k)

∪ {|ŵ i | ≥ 1}) −→ ǫ,

k∈S i∈Ack

T →∞

for T sufficiently large enough. We have then concluded the model selection consistency.

45

We now focus on the asymptotic normality. Model selection implies that
(k)

P({k ∈ S, i ∈ Ak , : θ̂i

6= 0} = A) −→ 1.
T →∞

As a consequence, the next relationship holds
P(∀k ∈ S, ĠT l(θ̂)Ak + λT T −1 αT,Ak ⊙ sgn(θ̂Ak ) + γT T −1 ξT,k

θ̂Ak
kθ̂Ak k2

= 0) −→ 1.
T →∞

By a Taylor expansion of the gradient term around θ0,A , we obtain
1
∇′ {(θ̂A − θ0,A )′ G̈T l(θ̄)AA (θ̂A − θ0,A )}
2
+ λT T −1 αT,A ⊙ sgn(θ̂A ) + γT T −1 ηT = 0) −→ 1,

P(ĠT l(θ0 )A + G̈T l(θ0 )AA (θ̂A − θ0,A ) +

T →∞

where ηT = vec(ξT,k
we have

θ̂Ak
kθ̂Ak k2

, k ∈ S) and kθ̄ − θ0 k2 ≤ kθ̂ − θ0 k2 . As a consequence,

1
P(θ0 )(θ̂A − θ0,A ) + HT,AA (θ̂A − θ0,A ) = −ĠT l(θ0 )A − ∇′ {(θ̂A − θ0,A )G̈T l(θ̄)AA (θ̂A − θ0,A )}
2
− λT T −1 αT,A ⊙ sgn(θ̂A ) − γT T −1 ηT + op (1),
where P(θ0 ) = G̈T l(θ0 )AA − HT,AA and HT,AA = E[∇2θθ′ l(ǫt ; θ0 )]AA . Then multi√
−1/2
plying by T QT VT,AA , we obtain
√

√
−1/2
−1/2
−1 α
−1 η )
T QT VT,AA (θ̂A − θ0,A ) = − T QT VT,AA H−1
T,A ⊙ sgn(θ̂A ) + γT T
T
T,AA (λT T
√
−1/2 −1
−
T QT VT,AA HT,AA ĠT l(θ0 )A
√
−1/2
′
′
T /2QT VT,AA H−1
−
T,AA ∇ {(θ̂A − θ0,A ) G̈T l(θ̄)AA (θ̂A − θ0,A )}
√
−1/2
−
T QT VT,AA H−1
T,AA P(θ0 )(θ̂A − θ0,A ) + op (1).

We focus on the l1 penalty term, which can be upper bounded as
λT
√
−1/2
−1/2
−1/2 max α
N1,T := | T QT VT,AA H−1
αT,A ⊙ sgn(θ̂A ))| ≤ |QT VT,AA ||H−1
T,i,A
T,AA |λT T
T,AA (
k∈S,i∈Ak
T
−1/2
˜(k) −η
−1/2 { min |θ̃
≤ |QT VT,AA |λ−1
i |}
min (HT,AA )λT T
k∈S,i∈Ak

−1/2

1

κη− 2
.
≤ |QT VT,AA |λ−1
min (HT,AA )λT T

If λT T κη → 0, then N1,T = op (1).

46

As for the l1 /l2 penalty, it can be upper bounded as
γT
√
−1/2
−1/2
−1
−1/2 kη k
N2,T := | T QT VT,AA H−1
T 2
T,AA T ηT | ≤ |QT VT,AA ||HT,AA |γT T
rP
−1/2
˜
−1/2
kθ̃ (k) k−2µ
≤ |QT VT,AA ||H−1
2
T,AA |γT T
≤

k∈S
1/2
−1/2
−1
−1/2
dT {minkθ̃˜(k) k2 }−µ
|QT VT,AA |λmin (HT,AA )γT T
k∈S
1/2

−1/2

κµ .
−1/2 d
≤ |QT VT,AA |λ−1
min (HT,AA )γT T
T T

Using dT = O(T c ), if γT T

c−1
+κµ
2

N1,T + N2,T = op (1).

→ 0, then N2,T = op (1). Consequently, we have

We now turn to the hessian quantity of the Taylor expansion and prove the
discrepancy P(θ0 ) converges uniformly to zero in probability. For any ǫ > 0, by the

Markov’s inequality, we have

P(kG̈T l(θ0 )AA − HT,AA k22 > (ǫ/dT )2 ) ≤
≤

d2T
ǫ2 T 2
d4T
ǫ2 T 2

E[

P

{∂θ2k θl l(ǫt ; θ0 ) − E[∇2θk θl l(ǫt ; θ0 )]}2 ]

(k,l)∈A

λ2max (HT,AA ).

As for the third order term, by the Cauchy-Schwartz inequality
k∇′ {(θ̂A − θ0,A )′ G̈T l(θ̄)AA (θ̂A − θ0,A )}k22 ≤
≤

T
1 P
P
l2 (ǫt ; θ̄)}kθ̂A − θ0,A k42
∂3
{
2
T t=1 (k,l,m)∈A θk θl θm T
T
1 P
P
ψ 2 (ǫt )}kθ̂A − θ0,A k42
{
T 2 t=1 (k,l,m)∈A T

= Op (

We now prove XT,t =

√

d5T

1
)
=
o
(
).
p
T2
T

−1/2

T QT VT,AA H−1
T,AA ĠT lt (θ0 )A , t = 1, · · · , T, is asymptot-

ically normal by checking the Lindeberg-Feller’s condition for applying Shiryaev’s
Theorem 5.1. We remind that ĠT lT,t (θ0 ) is the t-th point of the score of the empirical criterion. Let β > 0, and to use Shiryaev’s Theorem, we need to prove that for
any ǫ > 0, we have
T
X
T
] > ǫ) −→ 0.
P( E[kXT,t k22 1kXT,t k2 >β |Ft−1
T →∞

t=0

47

By the Markov inequality, we obtain
P(

T
P

t=0

T ] > ǫ) ≤
E[kXT,t k22 1kXT,t k2 >β |Ft−1

≤
≤

T
1P
T ]
E[kXT,t k22 1kXT,t k2 >β |Ft−1
ǫ
t=0
T
1P
T ]1/2 P(kX k > β|F T )1/2 ]
E[E[kXT,t k42 |Ft−1
T,t 2
t−1
ǫ
t=0
T
1P
T ]}1/2
E[{ CTst2 E[k∇l(ǫt ; θ0 )∇′ l(ǫt ; θ0 )k22 |Ft−1
ǫ
t=0
√
−1/2
2
T 1/2 ],
. β1 E[k T QT VT,AA H−1
T,AA ĠT lt (θ0 )A k2 |Ft−1 ]

−1/2

with Cst > 0. First, let KT = QT VT,AA H−1
T,AA , we have
√
T ] =
E[k T KT ĠT lt (θ0 )A k22 |Ft−1
=
=
≤

1
T
1
T
1
T
1
T

T ]
E[∇′ l(ǫt ; θ0 )K′T KT ∇l(ǫt ; θ0 )|Ft−1
T ]
E[Trace(∇′ l(ǫt ; θ0 )K′T KT ∇l(ǫt ; θ0 ))|Ft−1
T ]K′ K )
Trace(E[∇l(ǫt ; θ0 )∇′ l(ǫt ; θ0 )|Ft−1
T T

λmax (HTt−1 )C̃st ,

where C̃st > 0. Furthermore, we have
T ] = E[
E[k∇l(ǫt ; θ0 )∇′ l(ǫt ; θ0 )k22 |Ft−1

dT
P

T ]
{∂θi l(ǫt ; θ0 )∂θj l(ǫt ; θ0 )}2 |Ft−1

i,j=0

≤ d2T

sup

i,j=1,··· ,dT

T ].
E[{∂θi l(ǫt ; θ0 )∂θj l(ǫt ; θ0 )}2 |Ft−1

By assumption 14, we have
P(

T
P

t=0

T ] > ǫ)
E[kXT,t k22 1kXT,t k2 >β |Ft−1

1

≤

1

T
Cst2 C̃st2 dT P
3

T2

t=0

E[

sup

i,j=1,··· ,dT

1

T ]λ
T
E[{∂θi l(ǫt ; θ0 )∂θj l(ǫt ; θ0 )}2 |Ft−1
max (Ht−1 )] ≤

1

Cst2 C̃st2 B̄T dT
3

T2

Consequently, we obtain
T
X
t=0

T
E[kXT,t k22 1kXT,t k2 >β |Ft−1
] = op (1).

We deduce that XT,t satisfies the Lindeberg-Feller condition, and by Theorem 5.1,
√
−1/2
T QT VT,AA H−1
T,AA ĠT l(θ0 )A is asymptotically normally distributed. The asymptotic distribution of Theorem 5.4 follows.

48

.

6

Simulation Experiments

In this section, we carry out a simulation study to explore the finite sample performance of the adaptive Sparse Group Lasso. We first focus on the calibration of
the adaptive weights entering the penalties. The regularization parameters must
satisfy conditions to achieve the oracle property in the double asymptotic case. To
do so, we suppose λT = T β and γT = T α , where β and α are both strictly positive
constant. Regarding assumption 13, we obtain the conditions


α+






 α−









c
2
1
2

+ κµ −

1
2

< 0,

+ 12 [(1 + µ)(1 − c) − 1] > 0,
β + κη − 21 < 0,
β − 21 + 12 [(1 + η)(1 − c) − 1] > 0,
(1 + µ)[1 − 2c − κη − β] + α − 1 >

0.

This system allows for flexibility when choosing µ and η once κ, c, α and β are fixed.
For instance, for c = 1/6, κ = 0.05, α = 1/10 and β = 1/10, then µ ∈ [0.4, 6.3] and

η ∈ [0.6, 7.9]. If α = β = 1/5 and for c = 1/6 and κ = 0.05, then µ ∈ [0.4, 4.3] and
η ∈ [0.3, 5.9].

We consider 6 methods in the experiment: the Lasso (L), the Adaptive Lasso
(AL), the Group Lasso (GL), the Adaptive Group Lasso (AGL), the Sparse Group
Lasso (SGL) and the Adaptive Sparse Group Lasso (ASGL).
There are several methods to numerically solve the non-differentiable statistical
problem (5.1). Fan and Li (2001) proposed a local quadratic approximation (LQA)
of the first order derivative of the penalty function and a Newton-Raphson type
algorithm. To circumvent numerical instability, they suggest to shrunk to zero
coefficients that are close to zero, that is a coefficient |θj | < ǫ, with ǫ > 0 to be
calibrated. The drawback is that once it is set to zero, it will be excluded at any step
of the LQA algorithm. Hunter and Li (2005) proposed a more sophisticated version
of the LQA algorithm to avoid the drawback of the stepwise selection and numerical
instability. They also studied the convergence properties of the LQA method. Zou
and Li (2008) proposed a local linear approximation (LLA) of the penalty function
such that the estimated coefficients have naturally a sparse representation, under
the condition that the penalty function enjoys the continuity condition. Zou (2006)
or Zou and Zhang (2009) use the LQA algorithm for their empirical study. Other
approaches are also possible such as gradient descent methods.
When one consider the OLS loss function, closed form algorithm can be applied

49

to our problem. Bühlmann and van de Geer (2011) compiled these methodologies
for solving the Lasso and the Group Lasso using gradient descent methods for general penalized convex empirical function. We used these algorithms in our study
for solving the group LASSO. As for the LASSO, we applied the shooting algorithm developed by Fu (1998), which is a particular case of the gradient descent
method. Simon and al. (2013) proposed an algorithm for solving the SGL that can
accommodate likelihood criteria. This is a ”two-step” method, where we first check
whether the group is active, and then, if active, check if the coefficient within this
group is active. In this simulation study, we used the alternative direction method
of multipliers provided by Li and al. (2014).
We used a cross-validation procedure to select both parameters λT and γT such
that both terms are defined by λT = T β and γT = T α , and β = α = 1/8. The
adaptive weights are computed as follows: we first compute an OLS estimator θ̃
˜
such that the adaptive weights entering the penalties correspond to θ̃ = θ̃ + T −κ ,
with κ = 0.2. As for the adaptive weights, they are chosen such that the above
system is satisfied: we set η = 3.5 and µ = 2.5.
We report the variable selection performance through the number of zero coefficients correctly estimated, denoted as C and, the number of nonzero coefficients
incorrectly estimated, denoted IC. Besides, the mean squared error is reported as
an estimation accuracy measure.
Simulated experiment. We consider a data generating process
y=

X (l)
β0 X(l) + ση,
l

where η is a strong white noise, normally distributed, centered with unit variance
and σ = 0.3. The matrices X(l) follow cl - dimensional multivariate normal distributions, centered and with variance covariance Σ(l) such that the entries are defined as
(l)

Σij = ρ|i−j| , 1 ≤ j, i ≤ cl . The correlation parameter ρ is randomly chosen among
{0.5, 0.8, 0.9}. Moreover, the dimension dT = [x × T 1/6 ] with T = 500, 2000, 4000

and x = 10, 30, 50 respectively for the values of T . As dT = O(T c ) with c = 1/6, we
can multiply by x to consider more realistic settings. The number of groups is defined as Ng = 4 (resp. Ng = 8, resp. Ng = 18) for n = 500 (resp. for n = 2000, resp.
for n = 4000) and the size of each of them is randomly chosen among {5, · · · , 30}.

The number of active groups is defined as |S| = 2aT with aT = [Ng /3]. Moreover,

zero coefficients are randomly chosen among the whole vector β for active groups,
such that the total number of zeros -both the zero subvectors for inactive groups

50

and zero components for active groups - matches the total number of inactive indices. The total number of active indices is defined as |A| = 3bT with bT = [dT /9].

Finally, we generate the active indices among a uniform law U ([0.1, 0.99]). Zou and

Zhang (2009) experiment influenced our framework.

Table 1: Model selection and precision accuracy based on 100 replications
T
500

dT
28

2000 106

4000 199

Ng
4

8

18

|S|
2

4

12

|A|
9

33

66

Model
Truth
Lasso
aLasso
GLasso
AGLasso
SGL
ASGL
Truth
Lasso
aLasso
GLasso
AGLasso
SGL
ASGL
Truth
Lasso
aLasso
GLasso
AGLasso
SGL
ASGL

MSE
0.0178
0.0118
0.0146
0.0129
0.0183
0.0101

C
19
13.13
17.98
12.77
13.57
12.97
18.83

IC
0
0
0
0
0
0
0

0.0118
0.0103
0.0150
0.0160
0.0125
0.0095

73
49.65
70.95
57.48
60.78
58.88
72.70

0
0
0
0
0
0
0

0.0105
0.0093
0.0140
0.0150
0.0102
0.0094

133
87.17
131.33
113.42
113.17
98.92
133

0
0
0
0
0
0
0

We can highlight some interesting remarks from this simulation study. First,
the adaptive versions of the Lasso, the Group Lasso or the SGL outperfom their
non adaptive versions. The difference is significant for the adaptive Lasso and
the adaptive SGL. This is in line with the asymptotic theory. The adaptive SGL
performs well as it can discard inactive groups and inactive indices among active
groups and outperform other adaptive penalization methods.

51

Acknowledgements. I would like to thank Jean-Michel Zakoı̈an and Christian
Francq for all the theoretical references they provided. And I thank warmly JeanDavid Fermanian for his significant help and helpful comments. I finally gratefully
acknowledge the Ecodec Laboratory.

52

References
[1] Anderson, P.K. and Gill, R.D. (1994,a): Cox’s Regression Model for Counting Processes: A Large Sample Study. The Annals of Statistics, Vol. 10, No. 4,
1100-1120.
[2] Andrews, D.W.K. (1994,a): Asymptotics for Semiparametric Econometric
Models Via Stochastic Equicontinuity. Econometrica, Vol. 62, No. 1, 43-72.
[3] Andrews, D.W.K. (1994,b): Empirical Process Methods in Econometrics.
Handbook of econometrics, Elsevier.
[4] Bertsekas, D. (1995): Nonlinear Programming. Athena Scientific, Belmont,
MA.
[5] Billingsley, P. (1961): The Lindeberg-Levy theorem for martingales. Proceedings of the American Mathematical Society 12, 788792.
[6] Billingsley, P. (1995): Probability and Measure. New York: John Wiley and
Sons, Inc.
[7] Bühlmann, P. and van de Geer, S. (2011): Statistics for High-Dimensional
Data: Methods, Theory and Applications. Springer Series in Statistics.
[8] Chernozhukov, V. and Hong, H. (2004): Likelihood Estimation and Inference in a Class of Nonregular Econometric Models. Econometrica, Vol. 72, No.
5, 1445-1480.
[9] Chernozhukov, V. (2005): Extremal Quantile Regression. The Annals of
Statistics, Vol. 33, No. 2, 806-839.
[10] Davis, R.A., Knight, K., Liu, J. (1992): M-Estimation for Autoregressions
with Infinite Variance. Stochastic Processes and their Applications, 40, 145-180.
[11] Fan, J. and Li, R. (2001): Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties. Journal of the American Statistical Association,
96:456, 1348-1360.
[12] Fan, J. and Peng, H. (2004): Nonconcave Penalized Likelihood with a Diverging Number of Parameters. The Annals of Statistics, Vol. 32, No. 3, 928-961.
[13] Geyer, C.J. (1994): On the Asymptotics of Constrained M-Estimation. The
Annals of Statistics, Vol. 22, No. 4, 1993-2010.
[14] Geyer, C.J. (1996): On the Asymptotics of Convex Stochastic Optimization.
Unpublished manuscript.

53

[15] Hjort, N.L. and Pollard, D. (1993): Asymptotics for Minimisers of Convex Processes. Unpublished manuscript.
[16] Hunter, D.R and Li, R. (2005): Variable Selection Using MM Algorithms.
The Annals of Statistics, Vol. 33, No. 4, 1617-1642.
[17] Kato, K. (2009): Asymptotics for Argmin Processes: Convexity Arguments.
Journal of Multivariate Analysis, 1816-1829.
[18] Knight, K. and Fu, W. (2000): Asymptotics for LASSO-type Estimators.
The Annals of Statistics, Vol. 28, No. 5, 1356-1378.
[19] Li, X., Mo, L., Yuan, X., Zhang, J. (2014): Linearized Alternating Direction Method of Multipliers for Sparse Group and Fused LASSO models. Computational Statistics and Data Analysis, 79, 203-221.
[20] Nardi, Y. and Rinaldo, A. (2008): On the Asymptotic Properties of the
Group LASSO Estimator for Linear Models. Electronic Journal of Statistics,
Vol. 2, 605-633.
[21] Neumann, M.H. (2013): A Central Limit Theorem for Triangular Arrays of
Weakly Dependent Random Variables, with Applications in Statistics. Probability and Statistics, Vol. 17, 120-134.
[22] Newey, W.K. and Powell, J.L. (2008): Asymmetric Least Squares Estimation and Testing. Econometrica, Vol. 55, No. 4, 819-847.
[23] Rio, E. (2013): Inequalities and limit theorems for weakly dependent sequences.
3 ème cycle, pp.170.
[24] Rockafeller, R.T. (1970): Convex analysis. Princeton University Press,
Princeton.
[25] Shiryaev, A.N. (1991): Probability. Second Edition, Springer.
[26] Simon, N., Friedman, J., Hastie, T., Tibshirani, R. (2013): A sparse
group lasso. Journal of Computational and Graphical Statistics, 22:2, 231-245.
[27] Tibshirani, R. (1996): Regression Shrinkage and Selection via the Lasso.
Journal of the Royal Statistical Society. Series B, Vol. 58, No. 1, pp. 267-288.
[28] Yuan, M. and Lin, Y. (2006): Model Selection and Estimation in Regression
with Grouped Variables. Journal of the Royal Statistical Society. Series B, Vol.
68, No. 1, pp. 49-67
[29] Zou, H. (2006): The Adaptive LASSO and its Oracle Properties. Journal of
the American Statistical Association, 101:476, 1418-1429.

54

[30] Zou, H. and Li, R. (2008): One-Step Sparse Estimates in Nonconcave Penalized Likelihood Models. The Annals of Statistics, Vol. 36, No. 4, 1509-1533.
[31] Zou, H. and Zhang, H.H. (2009): On the Adaptive Elastic-Net with a Diverging Number of Parameters. The Annals of Statistics, Vol. 37, No. 4, 1733-1751.

55

