On the Distribution, Model Selection Properties and
Uniqueness of the Lasso Estimator in Low and High
Dimensions
Ulrike Schneider and Karl Ewald

arXiv:1708.09608v2 [] 29 Jan 2018

Vienna University of Technology
We derive expressions for the finite-sample distribution of the Lasso estimator in the context of
a linear regression model with normally distributed errors in low as well as in high dimensions by
exploiting the structure of the optimization problem defining the estimator. In low dimensions we
assume full rank of the regressor matrix and present expressions for the cumulative distribution
function as well as the densities of the absolutely continuous parts of the estimator. Additionally,
we establish an explicit formula for the correspondence between the Lasso and the least-squares
estimator. We derive analogous results for the distribution in less explicit form in high dimensions
where we make no assumptions on the regressor matrix at all. In this setting, we also investigate
the model selection properties of the Lasso and show that possibly only a subset of models might
be selected by the estimator, completely independently of the observed response vector. Finally,
we present a condition for uniqueness of the estimator that is necessary as well as sufficient.

1

Introduction

The distribution of the Lasso estimator (Tibshirani, 1996) has been an object of study in the
statistics literature for a number of years. The often cited paper by Knight & Fu (2000) gives the
asymptotic distribution of the Lasso in the framework of conservative model selection in a lowdimensional (fixed-p) framework by listing the limit of the corresponding stochastic optimization.
Pötscher & Leeb (2009) derive explicit expressions of the distribution in finite samples as well
as asymptotically for all large-sample regimes of the tuning parameter (“conservative” as well as
“consistent model selection”) in the framework of orthogonal regressors. More recently, Zhou (2014)
gives high-level information on the finite-sample distribution for arbitrary designs in low and high
dimensions, geared towards setting up a Monte-Carlo approach to infer about the distribution.
In Ewald & Schneider (2015), the large-sample distribution of the Lasso is derived in a lowdimensional framework for the large-sample regime of the tuning parameter not considered in
Knight & Fu (2000). Moreover, Jagannath & Upadhye (2016) consider the characteristic function
of the Lasso to obtain approximate expressions for the marginal distribution of one-dimensional
components of the Lasso when these components are “large”, therefore not having to consider the
atomic part of the estimator.
In this paper, we exactly and completely characterize the distribution of the Lasso estimator
in finite samples in the context of a linear regression model with normal errors. In low dimensions,
we give formulae for the cumulative distribution function (cdf), as well as the density functions
conditional on which components of the estimator are non-zero. We do so assuming full column
rank of the regressor matrix. We also exactly quantify the correspondence between the Lasso and
least-squares (LS) estimator. In a high-dimensional setting, we make absolutely no assumptions
on the regressor matrix. We give formulae for the probability of the Lasso estimator falling into a
given set and exactly quantify the relationship between the Lasso estimator and the data object
X 0 y. Through this relationship, we also learn that the Lasso may never select certain models,
this property depending only on the regressor matrix and the penalization weights and being
independent of the observed response vector. In fact, we can characterize a so-called structural set

1

that contains all covariates that are part of a Lasso model for some response vector. This structural
set can be identified by how the row space of the regressor matrix intersects the cube at the origin
whose side lengths are determined by the penalization weights. The set may not contain all indices
in which case the Lasso estimator will rule out certain covariates for all possible observations of the
dependent variable. Finally, we present a condition for uniqueness of the Lasso estimator that is
both necessary and sufficient, again related to how the row space of the regressor matrix intersects
the above mentioned cube. All our results are based on properties of the optimization problem
defining the estimator and do not hinge on the assumption of Gaussian errors.
The paper is organized as follows. We introduce setting and notation in Section 2. The lowdimension case is treated in Section 3 whereas we consider the high-dimensional case in Section 4.
We conclude in Section 5.

2

Setting and Notation

Consider the linear model
y = Xβ + ε,

(1)

where y is the observed n × 1 data vector, X the n × p regressor matrix which is assumed to be
non-stochastic, β ∈ Rp is the true parameter vector and ε the unobserved error term with independent and identically distributed components that follow a N (0, σ 2 )-distribution. We consider
the weighted Lasso estimator β̂L , defined as the unique solution to the minimization problem
minp L(β) = minp ky − Xβk2 + 2

β∈R

β∈R

p
X

λn,j |βj |,

(2)

j=1

where λn,j , are non-negative user-specified tuning parameters that will typically depend on n. To
ease notation, however, we shall suppress this dependence for the most part and write λn,j = λj
for each j. Note that if λj = 0 for all j, the weighted Lasso estimator is equal to the ordinary
LS estimator β̂LS and that λ1 = · · · = λp > 0 corresponds to the classical Lasso estimator as
proposed by Tibshirani (1996), to which case we also refer to by uniform tuning. For later use, let
λ = (λ1 , . . . , λp )0 and define M0 = {j : λj = 0}, the index set of all unpenalized coefficients. If
M0 6= ∅, we speak of partial tuning. We stress dependence on the unknown parameter β when it
occurs, but do not specify dependence on X, y or λ as these quantities are available to the user.
The following notation will be used throughout the paper. Let ej denote the j-th unit vector
in Rp and let φ(µ,Σ) denote the Lebesgue-density of a normally distributed random variable with
mean µ and covariance matrix Σ and Φ be the cdf of a univariate standard normal distribution.
For a vector m ∈ Rp and an index set I ⊆ {1, . . . , p}, the vector mI ∈ R|I| contains only the
components of m corresponding to the elements of I and we write |I| for the cardinality of I and
I c for {1, . . . , p} \ I, the complement of I. The 1-norm of m is denoted by kmk1 whereas the 2norm is simply denoted by kmk. For x ∈ R, let sgn(x) = 1{x>0} − 1{x<0} where 1 is the indicator
function. For a set A ⊆ Rp , the set m + A = A + m is defined as {m + z :Q
z ∈ A} with a analogous
definitions for A − m and m − A. We denote the Cartesian product by
and the column space
and rank of a matrix C by col(C) and rk(C), respectively. The columns of C are denoted by Cj
whereas CI , for some index set I, is the matrix containing the |I| columns of C corresponding to
the indices in I only. We use R>0 for the positive and R≥0 for the non-negative real numbers.
Let {D− , D+ , D0 } be a partition of {1, . . . , p} into three sets, some of which may be empty. It
will be convenient to also describe this partition by a vector d ∈ {−1, 0, 1}p with dj = 1{j∈D+ } −
1{j∈D− } . For such d, we denote by Od = {z ∈ Rp : sgn(zj ) = dj for j = 1, . . . , p} = {z ∈ Rp : zj <
0 for j ∈ D− , zj > 0 for j ∈ D+ , zj = 0 for j ∈ D0 }. Note that m + β ∈ Od is short-hand notation
+
for mj < −βj for j ∈ D− , mj > −βj for j ∈ D+ and mj = −βj for j ∈ D0 . We write D−
for
D− ∪ D+ .
Finally, the directional derivative of a function g : Rp → R at m in direction r ∈ Rp with
krk = 1 is defined as
g(m + hr) − g(m)
∂g(m)
= lim
.
h&0
∂r
h
2

3

The Low-dimensional Case

Throughout this section, we assume that X has full column rank p, implying that we are considering
the low-dimensional setting where p ≤ n. For our arguments, we use a reparametrized version of
the objective function. Define
Vβ (u) = L(u + β) − L(β) = u0 X 0 Xu − 2u0 W + 2

p
X

λj [|uj + βj | − |βj |] ,

(3)

j=1

where W = X 0 ε ∼ N (0, σ 2 X 0 X) and note that Vβ is minimized at û = β̂L − β. We are interested
in the distribution of the estimation error û. To state the main theorem, we need the following
lemma that characterizes a solution to the minimization problem.
Lemma 1. Let m ∈ Rp . The following two statements are equivalent.
(a)

∂Vβ (m)
∂r

≥ 0 ∀r with krk = 1

(b)

∂Vβ (m)
∂ej

≥ 0 and

∂Vβ (m)
∂(−ej )

≥ 0 for j = 1, . . . , p.

Proof. Only (b) ⇒ (a) needs to be proved. Let d ∈ {−1, 0, 1}p such that m + β ∈ Od and let
{D− , D+ , D0 } be the corresponding partition of {1, . . . , p}. A straight-forward calculation shows
that
p
X

∂Vβ (m)
= 2r0 X 0 Xm − 2r0 W + 2
λj −1{j∈D− } rj + 1{j∈D+ } rj + 1{j∈D0 } |rj |
∂r
j=1

=

p
X



1{rj ≥0} (2X 0 Xm − 2W )j + 2λj (−1{j∈D− } + 1{j∈D+ ∪D0 } ) rj

j=1



+ 1{rj <0} −(2X 0 Xm − 2W )j + 2λj (1{j∈D− ∪D0 } − 1{j∈D+ } ) (−rj )
=

p
X

p

1{rj ≥0}

j=1

X
∂Vβ (m)
∂Vβ (m)
(−rj ) ≥ 0.
rj +
1{rj <0}
∂ej
∂(−ej )
j=1

This complete characterization of minima of Vβ now allows to state the following theorem on
the distribution of the estimation error û = β̂L − β.
Theorem 2. Let z ∈ Rp . Let d = sgn(z + β) ∈ {−1, 0, 1}p and let {D− , D+ , D0 } be the corresponding partition of {1, . . . , p}. Then
P (ûj ≤ zj for j ∈ D− , ûj ≥ zj for j ∈ D+ , ûj = zj for j ∈ D0 )
Z
=

Z

Z

Z Z

Z

···

···

···

sj ∈[−λj ,λj ]
j∈D0

mj ≥zj
j∈D+

mj ≤zj
j∈D−

φ(0,σ2 X 0 X) (X 0 Xmβ + sλ ) dmD− dmD+ dsD0 ,

where mβ and sλ ∈ Rp are given by (mβ )D+ = mD+ , (mβ )D0 = −βD0 and (sλ )D− = −λD− , sD+ =
−
−
λD+ , (sλ )D0 = sD0 , respectively.
∂V (m)

Proof. Since the function Vβ is convex, m ∈ Rp is a minimizer of Vβ if and only if β∂r ≥ 0 for
all r ∈ R with krk = 1. We wish to find all minimizers m satisfying mj ≤ zj for j ∈ D− , mj ≥ zj
for j ∈ D+ and mj = zj for j ∈ D0 . Note that this implies that m + β ∈ Od since z + β ∈ O by
∂Vβ (m)
∂Vβ (m)
assumption. By Lemma 1 together with the fact that the condition ∂e
≥ 0 and ∂(−e
≥0
j
j)

3

∂V (m)

β
= 0 if Vβ is differentiable at m with respect to the j-th component, we get the
reduces to ∂u
j
following necessary and sufficient conditions for such m to be a minimizer of Vβ .

0

for j ∈ D−
 Wj = (X Xm)j − λj
0
(4)
Wj = (X Xm)j + λj
for j ∈ D+


0
0
(X Xm)j − λj ≤ Wj ≤ (X Xm)j + λj for j ∈ D0

Therefore, m satisfying m + β ∈ Od is a minimizer of Vβ if and only if W lies in the set
{s ∈ Rp : sj = (X 0 Xm)j − λj for j ∈ D− , sj = (X 0 Xm)j + λj for j ∈ D+ ,
(X 0 Xm)j − λj ≤ sj ≤ (X 0 Xm)j + λj for j ∈ D0 },
which equals
X 0 Xm + {sλ : (sλ )D− = −λD− , (sλ )D+ = λD+ , |sλ,j | ≤ λj for j ∈ D0 }.
Since we are interested in all minimizers m of Vβ that satisfy mj ≤ zj for j ∈ D− , mj ≥ zj for
j ∈ D+ and mj = zj for j ∈ D0 (that is, m − z ∈ Od ), we let
Ad = {X 0 Xm : m − z ∈ Od } + {s ∈ Rp : sD− = −λD− , sD+ = λD+ , |sj | ≤ λj for j ∈ D0 }.
As W follows a N (0, σ 2 X 0 X)-distribution, the sought-after probability is clearly given by
Z
φ(0,σ2 X 0 X) (u) du,
Ad

which is what was claimed.
Results on the distribution of β̂L itself rather than the estimation error are of course a direct
consequence of Theorem 2 and summarized in the following corollaries, the latter one giving the
probability of the extreme event of the Lasso setting all components to zero.
Corollary 3. Let z ∈ Rp and let d = sgn(z) with {D− , D+ , D0 } being the corresponding partition
of {1, . . . , p}.
P (β̂L,j ≤ zj for j ∈ D− , β̂L,j ≥ zj for j ∈ D+ , β̂L,j = 0 for j ∈ D0 )
Z
=

Z
···

Z

Z
···

Z

Z
···

φ(0,σ2 X 0 X) (X 0 Xmβ + sλ ) dmD− dmD+ dsD0 ,

sj ∈[−λj ,λj ] mj ≥zj −βj mj ≤zj −βj
j∈D+
j∈D−
j∈D0

where mβ and sλ ∈ Rp are given by (mβ )D+ = mD+ , (mβ )D0 = −β and (sλ )D− = −λD− , sD+ =
−
−
λD+ , (sλ )D0 = sD0 , respectively.
Corollary 4.
Z

λp

Z

λ1

···

P (β̂L = 0) =
−λp

φ(X 0 Xβ,σ2 X 0 X) (s) ds
−λ1

Remark 1. To illustrate the structure behind the proof of Theorem 2, we rewrite Corollary 3 as
P (β̂ ∈ Bz ) = P (W ∈ Aβ (Bz ))
with Bz = {b ∈ Rp : bj ≤ Szj for j ∈ D− , bj ≥Qzj for j ∈ D+ , bj = 0 for j ∈ D0 }, W ∼
p
N (0, σ 2 X 0 X), and Aβ (Bz ) = b∈Bz X 0 X(b − β) + j=1 Bj (bj ) where
(
{sgn(bj )λj } bj 6= 0
Bj (bj ) =
[−λj , λj ]
bj = 0.
4

The events {β̂ ∈ Bz } and {W ∈ Aβ (Bz )} are shown to be equivalent through Lemma 1. This
equivalence holds due to the structure of the optimization problem defining β̂L and does not depend
on the distribution of W = X 0 ε. In this sense, the distributional results do not hinge on the
normality assumption of the errors and can easily be generalized to other error distributions. The
relationship and shape of the sets Bz and Aβ (Bz ) are illustrated in Figure 1. Note that Aβ depends
on λ whereas Bz does not.
We exploit the structure of the optimization problem by characterizing the minimum through
directional derivatives rather than using the Kuhn-Karush Tucker conditions as in the high-level
approach of Zhou (2014). This has the advantage that our distributional results give the joint
distribution of the estimator only and are not “augmented” by the subdifferential vector.
Theorem 2 now puts us into a position to fully specify the distribution of the Lasso estimator. In
case λj > 0 for all j = 1, . . . , p, one easily sees from the preceding corollary that this distribution is
not absolutely continuous with respect to the p-dimensional Lebesgue-measure and thus no density
exists. One can, however, represent the distribution through Lebesgue-densities after conditioning
on which components of the estimator are negative, positive, and equal to zero – which we shall
do in the sequel.
Proposition 5. The distribution of û = β̂L − β, conditional on the event {β̂L ∈ Od }, can be
represented by a kdk1 -dimensional Lebesgue-density given by
f d (zD+ ) =

1{zβ + β ∈ Od }
P (β̂L ∈ Od )

−

Z

Z
···

φ(0,σ2 X 0 X) (X 0 Xzβ + sλ ) dsD0 ,

sj ∈[−λj ,λj ]
j∈D0

where zβ is defined by (zβ )D+ = zD+ and (zβ )D0 = −βD0 , and sλ is defined by (sλ )D− = −λD− ,
−

−

(sλ )D+ = λD+ , and (sλ )D0 = sD0 . Note that the constants P (β̂L ∈ Od ) can be calculated using
Corollary 3.
Proof. Observe that


d

f (zD+ ) =
−

∂
∂zj


+
j∈D−



+
P ûj ≤ zj for j ∈ D−
|β̂L ∈ Od ,

and note that by Theorem 2 for any z ∈ Rp we have with z + β ∈ Od we have


P ûj ≤ zj for j ∈ D− , ûj ≥ zj for j ∈ D+ |β̂L ∈ Od


1
P ûj ≤ zj for j ∈ D− , ûj ≥ zj for j ∈ D+ , β̂L,j = 0 for j ∈ D0
=
P (β̂L ∈ Od )
Z
Z Z
Z Z
Z
1
···
···
· · · φ(0,σ2 X 0 X) (X 0 Xmβ + sλ ) dmD− dmD+ dsD0 ,
=
P (β̂L ∈ Od )
sj ∈[−λj ,λj ]
j∈D0

mj ≤zj
j∈D+

mj ≥zj
j∈D−

where mβ ∈ Rp is defined by (mβ )D+ = mD− ∪D+ , and (mβ )D0 = −βD0 and sλ ∈ Rp is defined by
−

+
(sλ )D− = −λD− , (sλ )D+ = λD+ , and (sλ )D0 = sD0 . Differentiating with respect to zj : j ∈ D−
and taking the absolute value gives the density, thus completing the proof.

Besides the conditional densities, we can also specify the full cdf of û = β̂L − β which is done
in the following theorem.
Theorem 6. The cdf of û = β̂L − β is given by
X

F (z) = P (û1 ≤ z1 , . . . , ûp ≤ zp ) =

d∈{−1,0,1}p

5

Z

Z
···

mj ≤zj
+
j∈D
−

hd (mD+ ) dνkdk1 ,
−

(a) z1 , z2 > 0

(b) z1 , z2 < 0

(c) z1 = 0 and z2 > 0

Figure 1: The sets Bz are displayed on the left-hand side, the corresponding sets Aβ (Bz ) are
displayed on the right-hand side. Illustrated for p = 2 and various values of z, see Remark 1 for
details.

6

where νk denotes k-dimensional Lebesgue-measure and where
Z
Z
hd (mD+ ) = 1{mβ + β ∈ Od }
···
φ(0,σ2 X 0 X) (X 0 Xmβ + sλ ) dsD0 ,
−

sj ∈[−λj ,λj ]
j∈D0

with mβ ∈ Rp given by (mβ )D+ = mD− ∪D+ , and (mβ )D0 = −βD0 and sλ ∈ Rp given by (sλ )D− =
−
−λD− , (sλ )D+ = λD+ , and (sλ )D0 = sD0 .
Proof. It is easily seen that
P (û1 ≤ z1 , . . . , ûp ≤ zp ) =

X

P (β̂L ∈ Od )

d∈{−1,0,1}p

Z

Z
···

f d (mD+ ) dνkdk1 .
−

mj ≤zj
+
j∈D
−

Plugging in the formula for f d completes the proof.
For illustration of Proposition 5 and Theorem 6, consider Figures 2 and 3 which display an
example of the distribution of û = β̂L − β. One can see that the Lasso estimation error follows a
shifted normal distribution conditional on the event ûj 6= −βj (β̂L,j 6= 0) for each j = 1, . . . , p with
the shift depending on the signs of β̂L as is to be seen in Figure 2. Figure 3 displays the mass which
lies on the set {z ∈ R2 : z1 = −β1 , z2 6= 0}, that is, the density functions h(0,1) and h(0,−1) on
their corresponding domains. The mass on the set {z ∈ R2 : z1 6= 0, z2 = −β2 } looks qualitatively
similar to Figure 3. Note that we also have point-mass at −β, as is pointed out by Corollary 4.

Figure 2: The contour lines of the absolutely continuous part with respect to 2-dimensional
1 0.5 ), λ = (0.75, 0.75)0 and β =
Lebesgue-measure of the distribution of β̂L − β for X 0 X = ( 0.5
1
0
(0, −0.25) . Note that the blue lines as well as the blue point also carry probability mass.

3.1

Shrinkage Areas

Using the conditions for minimality from the proof of Theorem 2, we can establish a direct relationship between the LS and the Lasso estimator in the following sense. For any b ∈ Rp , there
exists a set S(b) ⊆ Rp , such that the Lasso estimator assumes the value b if and only if the LS
estimator lies in S(b). We refer to the set S(b) as shrinkage area since the Lasso estimator can be
viewed as a procedure that shrinks the LS estimates from the set S(b) to the point b. Note that
by shrinkage we mean that kbk1 ≤ kzk1 for each z ∈ S(b), but |bj | > |zj | could hold for certain
components. The explicit form of S(b) is formalized in the following theorem.
7

0

0

1 0.5 ), λ = (0.75, 0.75)0
Figure 3: The functions h(0,−1) (in blue) and h(0,1) (in red) for X 0 X = ( 0.5
1
0
and β = (0, −0.25) , corresponding to the absolutely continuous part.

Theorem 7. For each b ∈ Rp there exists a set S(b) ⊆ Rp , such that
β̂L = b ⇐⇒ β̂LS ∈ S(b).
Moreover, for b ∈ Od , the set S(b) is given by
+
S(b) = {z ∈ Rp : (X 0 Xz)j = (X 0 Xb)j + sgn(bj )λj for j ∈ D−
, |(X 0 X(z − b))j | ≤ λj for j ∈ D0 }

Clearly, the sets S(b) are disjoint for different b’s.
Proof. Note that we have β̂LS − β = (X 0 X)−1 X 0 ε = (X 0 X)−1 W . With the minimality conditions
in (4) from the proof of Theorem 2 together with the fact that W = X 0 X(β̂LS − β) and some
rearranging, we get that m = β̂L − β minimizes Vβ if and only if β̂LS satisfies

0
0

 (X X β̂LS )j = (X X β̂L )j − λj for j ∈ D−
(X 0 X β̂LS )j = (X 0 X β̂L )j + λj for j ∈ D+


|(X 0 X(β̂LS − β̂L ))j | ≤ λj
for j ∈ D0 ,
or, β̂LS ∈ S(b) for β̂L = b, as required.
Remark 2. Clearly, if b ∈ Rp satisfies bj 6= 0 for all j = 1, . . . , p, then S(b) is the singleton
S(b) = {b − (X 0 X)−1 λ̃},
where λ̃j = − sgn(bj )λj for j = 1, . . . , p. This implies that in case β̂L,j 6= 0 for all j, the Lasso
estimator is given by
β̂L = β̂LS − (X 0 X)−1 λ̃.
Note that aside from b, S(b) depends on X and λ only.
Given Theorem 7, we can identify areas in which components of the LS estimator are shrunk to
zero by the Lasso. For p = 2, this leads to the image displayed in Figure 4. Clearly, the shrinkage
areas are related to the polyhedral selection areas of Lee et al. (2016) but yield a different level of
information. Our results can identify the regions that lead to a given value b of β̂L rather than to
a general model (with given signs of the non-zero components of the estimator).

8

Figure 4: The shrinkage areas from Theorem 7 for p = 2. The blue parallelogram equals the set
S(0). The dark gray area should consists
of lines parallel to the adjacent edge of the parallelogram

where each line equals a set S b02 for b2 6= 0. Analogously, the light gray area consists of lines
parallel to the adjacent edge of the parallelogram and each of those lines equals a set S b01 for

b1 6= 0. The yellow areas contain all singletons S bb12 with b1 , b2 6= 0 as described in Remark 2.
1 0.5 ) and λ = (0.75, 0.75)0 .
In this example, X 0 X = ( 0.5
1

4

High-Dimensional Case

We now turn to the main case of this this article, the high-dimensional setting where p > n. We
make no assumptions on the regressor matrix X in this section. Using similar arguments as in
the case p ≤ n, we can again start by characterizing the distribution of the Lasso, albeit in a
somewhat less explicit form. Note that we have rk(X) < p and that the true parameter is not
identified without further assumptions. We denote by B0 the set of all β ∈ Rp that yield the model
given in (1), that is, B0 = {β ∈ Rp : Xβ = E(y) = µ}. Furthermore, it is important to note that
the Lasso solution need not be unique anymore. We give necessary and sufficient conditions for
uniqueness later on in Section 4.3.
Note that for any fixed β ∈ B0 , the function Vβ defined in (3) is minimized at β̂L − β, where
β̂L may be any solution of (2). All findings in this section also hold when p ≤ n, but more explicit
results for this case are found in Section 3. We start with a high-level result on the distribution.
Theorem 8. For any set M ⊆ Rp and any β ∈ B0 , we have
P (arg min Vβ (u) ∩ M 6= ∅) = P (W ∈ Āβ (M )),
u∈Rp

S
Qp
where W ∼ N (0, σ 2 X 0 X) and Āβ (M ) = m∈M Āβ (m) with Āβ (m) = X 0 Xm + j=1 Bβ,j (mj )
and
(
{sgn(mj + βj )λj } mj + βj 6= 0
Bβ,j (mj ) =
[−λj , λj ]
mj + βj = 0.
Proof. Using the same necessary and sufficient conditions for m ∈ Rp to be a minimizer of Vβ as
in (4), we see that
m ∈ arg min Vβ (u) ⇐⇒ W ∈ Aβ (m).
u∈Rp

While the distribution of β̂L −β depends on the choice of β ∈ B0 , the distribution of β̂L does not,
as it is determined by y ∼ N (µ, σ 2 In ). This is formalized in the following corollary. As mentioned
9

before, β̂L need not be unique. Also remember that β̂L itself minimizes the function L(β) defined
in (2).
Corollary 9. For any set B ⊆ Rp and any β ∈ B0 , we have
P (arg min L(β) ∩ B 6= φ) = P (W ∈ A0 (B)),
β∈Rp

where W ∼ N (0, σ 2 X 0 X) and A0 (B) =

S

b∈B

(
Bj (bj ) =

A0 (b) with A0 (b) = X 0 X(b − β) +

{sgn(bj )λj }
[−λj , λj ]

Qp

j=1

Bj (bj ) and

bj =
6 0
bj = 0.

In particular, the distribution of the estimator β̂L does not depend on the choice of β ∈ B0 .
Proof. First note that arg minu Vβ (u) = arg minβ L(β) − β. We thus have
b ∈ arg min L(β) ⇐⇒ b − β ∈ arg min Vβ (u).
u

β

Using Theorem 8, we get for any β ∈ B0
b − β ∈ arg min Vβ (u) ⇐⇒ W ∈ X 0 X(b − β) +
u

p
Y

Bj (bj ) = A0 (b).

(5)

j=1

Finally, note that A0 depends on β only through X 0 Xβ which assumes the same value for all
β ∈ B0 .
As the random variable W = X 0 ε has singular covariance matrix, some care needs to be
taken when computing the probability from Corollary 9 through the appropriate integral of the
corresponding density function. This is specified in Corollary 10.
Corollary 10. Let the columns of U form a basis of col(X 0 ). The probability that a Lasso solution
lies in the set B ∈ Rp can be written as
Z
P (arg min L(β) ∩ B 6= φ) = 1{col(X 0 ) ∩ A0 (B) 6= ∅}
φ(0,σ2 U 0 X 0 XU ) (s)ds
β∈Rp

U 0 A0 (B)

Proof. Note that U 0 W ∼ N (0, σ 2 U 0 X 0 XU ) and that U 0 X 0 XU is invertible. Let N be a matrix
whose columns form a basis of col(X 0 )⊥ , then N 0 W has covariance matrix σ 2 N 0 X 0 XN = 0 and
N 0 W = 0 almost surely. We therefore have
W ∈ A0 (B) ⇐⇒ (U, N )0 W ∈ (U, N )0 A0 (B) ⇐⇒ U 0 W ∈ U 0 A0 (B) and 0 ∈ N 0 A0 (B)
⇐⇒ U 0 W ∈ U 0 A0 (B) and col(X 0 ) ∩ A0 (B) 6= ∅,
which proves the claim.

4.1

Shrinkage Areas and Model Selection Properties

For the low-dimensional case, Theorem 7 gives shrinkage areas of the Lasso with respect to β̂LS .
In the high-dimensional case, similar results are derived in Theorem 11 whose proof is based on
Corollary 9. These shrinkage areas are now given with respect to X 0 y as this quantity is always
uniquely defined in the high-dimensional case.
Theorem 11. For each b ∈ Rp there exists a set S̄(b) ⊆ Rp such that
b ∈ arg min L(β) ⇐⇒ X 0 y ∈ S̄(b),
β∈Rp

Moreover, S̄(b) is given by
S̄(b) = X 0 Xb +

p
Y

Bj (bj ).

j=1

with

(
{sgn(bj )λj }
Bj (bj ) =
[−λj , λj ]
10

bj 6= 0
bj = 0.

Proof. As X 0 y = X 0 Xβ + X 0 ε = X 0 Xβ + W , this follows immediately from (5).
Remark 3. Inspecting the sets S̄(b) from Theorem 11 more closely, we see that they are in general
not disjoint for different values of b ∈ Rp . This illustrates the fact that the Lasso solution need not
be unique in high dimensions anymore. (This is of course in contrast to the low-dimensional case
as can be seen in Theorem 7.) Indeed, we can have S̄(b) ∩ S̄(b0 ) 6= ∅ as long as b − b0 ∈ ker(X)
and {sgn(bj ), sgn(b0j )} =
6 {−1, 1} for all j = 1, . . . , p. This also makes apparent that b and b0 may
be Lasso solutions not corresponding to the same model which has been noted by Tibshirani (2013)
for the case of λ1 = · · · = λp > 0. We get deeper into the issue of (non-)uniqueness in Section 4.3.
Moreover, just as for the low-dimensional case, note that aside from b, S̄(b) depends on X and
λ only.
Theorem 11 also sheds some light on which models M ⊆ {1, . . . , p} may in fact be chosen by
the Lasso estimator, where the model chosen by β̂L is given by {j : β̂L,j 6= 0}. We find that some
models will in fact never be selected by the Lasso.
0
Let BM = {b ∈ Rp : bj 6= 0 if and only if j ∈ M}. Since X 0 y lies
S in col(X ), any model
0
M ⊆ {1, . . . , p} that satisfies that col(X ) ∩ S̄(BM ) = ∅ with S̄(BM ) = b∈BM S̄(b) can be never
be selected by the Lasso estimator. Looking at the definition of S̄(b) in Theorem 11 and noting
that X 0 Xb ∈ col(X 0 ), we get
col(X 0 ) ∩ S̄(BM ) = ∅ ⇐⇒ col(X 0 ) ∩ BM = ∅,
where
BM =

p
[ Y
b∈BM

(
sgn(bj )λj
[−λj , λj ]
j=1

(
p
Y
j∈M
{−λj , λj }
=
j∈
/M
[−λj , λj ]
j=1

j∈M
j∈
/ M.

We summarize this in the following corollary.
Corollary 12. Let X ∈ Rn×p and λ ∈ Rp≥0 be given. There exist y ∈ Rn such that the corresponding Lasso solution selects model M ⊆ {1, . . . , p} if and only if
col(X 0 ) ∩ BM 6= ∅,
where

(
BM =

{−λj , λj }
[−λj , λj ]

j∈M
j∈
/ M.

which satisfies BM̃ ⊆ BM for M ⊆ M̃.
What do the sets BM look like? If M0 = ∅, B∅ is the p-dimensional λ-box, B{j} is the union
of two opposite facets of the λ-box, for 1 < |M| < p, BM is the union of p − |M| dimensional
faces of the λ-box, and B{1,...,p} simply contains the corners of the λ-box.
For partial tuning with M0 6= ∅, B∅ is p − |M0 |-dimensional and we have BM ⊆ BM0 for all
M ⊆ {1, . . . , p} as well as
{0} ⊆ col(X 0 ) ∩ BM0 6= ∅,
so that, not surprisingly, there always exist y such that the non-penalized components will be part
of the model chosen by the Lasso solution.
We illustrate the results above by the following simplistic yet instructive example.
Example 1. Suppose X = (1, 2), so that n = 1, p = 2 and let λ1 = λ2 = λ̄ (uniform tuning).
Note that
col(X 0 ) ∩ B{1} = ∅
for all λ̄ > 0, so that by Corollary 12, β̂L,1 = 0 for any value of y, in fact independent of B0 and
σ 2 ! This yields
P (β̂L,1 = 0) = 1.

11

To say something about the remaining distribution of β̂L using Corollary 9, note that W = X 0 ε =
( 12 ) ε. We therefore get P (β̂L = 0) = P (W ∈ A0 (0)) with




λ̄,λ̄]
1 ) + [−λ̄,λ̄] ,
A0 (0) = −X 0 Xβ + [−
=
−(β
+
2β
)
(
1
2
2
[−λ̄,λ̄]
[−λ̄,λ̄]
so that the event {W ∈ A0 (0)} is equivalent to the event that W lies in the set
{(s − µ) ( 12 ) : s ∈ [−λ̄/2, λ̄/2]}
with µ = Xβ = β1 + 2β2 , whose probability is given by


P (β̂L = 0) = Φ (λ̄/2 − µ)/σ − Φ (−λ̄/2 − µ)/σ .
Next, for z < 0, we have
P (β̂L,1 = 0, β̂L,2 ≤ z) = P (W ∈

[



[

λ̄,λ̄]
A0 ( b02 ) ) = P (W ∈
b2 ( 24 ) + [−
)
{−λ̄}

b2 <z

= P (W ∈

[

b2 <z

(2b2 − λ̄/2)( 12 )) = P (ε ≤ 2z − λ̄/2) = Φ(2z − λ̄/2).

b2 <z

Similarly, we get that for z > 0
P (β̂L,1 = 0, β̂L,2 > z) = 1 − Φ(2z + λ1 /2).
The distribution of β̂L is thus given by
β̂L,1 = 0
and β̂L,2 following the distribution given by
dF (z) = (Φ(λ1 /2) − Φ(−λ1 /2)) dδ0 (z) + 1{z<0} 2φ(2z − λ1 /2) dz + 1{z>0} 2φ(2z + λ1 /2) dz.
It is interesting to note that the distribution of β̂L,2 is the same as the one of the Lasso estimator
with the same penalization parameter in the smaller model yi = 2β2 + ε where the first regressor
is taken out. Indeed, using the Lasso in the smaller model is equivalent to using the Lasso in
the larger model in our example, as the procedure only considers models that do not contain the
first regressor. This fact is, of course, only valid for the specific form of X and λ. However, the
models which are considered by the Lasso estimator in the first place do not depend on β and ε
in the sense that certain values of X and λ may immediately rule out certain models completely
independently of y and is thus not to be considered a purely data-driven model selection procedure.
The choice between the location model (when β̂L,1 = 0 and β̂L,2 6= 0) and the pure noise model
(when β̂L,1 = β̂L,2 = 0) in our example does of course very much depend on β and ε. These
considerations are also illustrated in Figure 5 which depicts in which areas the quantity X 0 y has
to fall into in order for the Lasso estimator to choose a certain model. For clarification, note that
the Lasso is always unique in this example.
The considerations sparked by Example 1 suggest that in the high-dimensional setting, model
selection by the Lasso estimator may, indeed, not be a purely data-driven procedure in the sense
that there is a structural model or structural set M ⊆ {1, . . . , p} determined by X and λ only,
satisfying β̂L,j = 0 for any j ∈
/ M for all observations y ∈ Rn . In particular, the true parameter
β ∈ B0 as well as the distribution of ε do not have any influence on this set. In other words,
some models are considered by the model selection procedure, completely independently of the
data vector y. Put yet differently again, this means that for a given regressor matrix X, one can
restrict or choose the class of models considered by choice of λ.
Given all the considerations above, one might ask whether such a structural model M always
satisfies |M | ≤ n under certain conditions. Clearly, uniqueness would be a meaningful requirement
in this context, as then all Lasso solutions will choose models of cardinality of at most n as has been
shown in Tibshirani (2013)1 . If this was indeed the case, the Lasso estimator would be equivalent
to a low-dimensional Lasso procedure restricted to this structural model M this could be used to
employ results from low-dimensional models also for inference in high-dimensional settings.
1 Note that this fact alone does not imply that the structural set has cardinality of at most n since the active sets
may certainly vary over y.

12

Figure 5: The shrinkage areas with respect to X 0 y from Theorem 11 for Example 1. Displayed in
0
0
red is span(X 0 ), the area on which the probability mass of X
 y is concentrated. The set S̄ ( 0 ) is
displayed in blue, while the yellow lines correspond to S̄ bb12 with b1 , b2 6= 0. The light gray area


is S̄ b02 with b2 6= 0, whereas the dark gray area equals S̄ b01 with b1 6= 0.
Example 1 (continued). Here, M = {2} and the distribution of the Lasso estimator is equal to
the distribution of the Lasso in the model
y = 2β2 + ε.
Using this property, one can construct a confidence set for β2 based on the Lasso using results from
Pötscher & Schneider (2010).
In Example 1 the Lasso solution is always unique (for any value of y) since the column span
of X 0 does not intersect overlapping shrinkage areas. It is not difficult, however, to construct an
example where the Lasso solution is not unique anymore.
Example 2. Again, take the model from Example 1 with X = (1, 2). This time, choose λ = (1, 2)0 .
It can easily be seen using Theorem 11 that for each y < −1,



0
y+1
β̂L = y+1
,
β̂
=
, and β̂L = ( y+1−2c
) for (y + 1)/2 < c < 0.
L
c
0
2
all are Lasso solutions for the same value of y. Similarly, for y > 1,



0
y−1
β̂L = y−1
,
β̂
=
, and β̂L = ( y+1−2c
) for 0 < c < (y + 1)/2.
L
c
0
2
all are Lasso solutions for the same value of y. (Note that β̂L = 0 for all y with |y| ≤ 1.) The
corresponding shrinkage areas are illustrated in Figure 6.
Example 2 shows an already known property of the Lasso from another perspective: The
solution to the Lasso problem is, in general, not unique. Moreover, if the solution is not unique,
then, by convexity of the problem, there exists an uncountable set of solutions2 . This example
shows, moreover, that the set of y which yield non-unique Lasso solutions is not a null set, in fact,
in this example it occurs with probability 2Φ(−1).
Of course, this problem could be overcome by slightly altering the choice of the tuning parameters, even though, this would imply to make a choice of the class of models under consideration,
as pointed out previously in this section.
2 This

fact has been pointed out by Tibshirani (2013) in Lemma 1 for the case of uniform tuning.

13

Figure 6: The shrinkage areas with respect to X 0 y from Theorem 11 for Example 2. Displayed in
0
0
red is span(X 0 ), the area on which the probability mass of X
 y is concentrated. The set S ( 0 ) is
displayed in blue, while the yellow lines correspond to S bb12 with b1 , b2 6= 0. The light gray area


is S b02 with b2 6= 0, whereas the dark gray area equals S b01 with b1 6= 0. The red line passes
through S ( 00 ) where the solution is unique but also through the line where the light gray, the dark
gray and the yellow areas intersect.

4.2

Structural Sets

Clearly, Example 2 shows that the structural set may also be equal to the entire set of explanatory
variables. In fact, It is easy to see that for n = 1 and p = 2, the Lasso estimator will always
have a structural set with cardinality n = 1 whenever we have uniqueness. The question is, of
course, whether the same can be said in general. Before answering this question, we show how the
structural set can fully be determined given X and λ by counting how many faces of the λ-box
B∅ are intersected by col(X 0 ).
Theorem 13. Let X ∈ Rn×p and λ ∈ Rp≥0 be given. Let M be the structural set of X and λ that
contains all j ∈ {1, . . . , p} such that there exist y ∈ Rn so that the corresponding Lasso solution
β̂L satisfies β̂L,j 6= 0, that is, the set of all regressors that are part of a Lasso solution for some
observation y. This set is given by

M = M (X, λ) = j ∈ {1, . . . , p} : B{j} ∩ col(X 0 ) 6= ∅ .
Proof. By Corollary 12, there exist y ∈ Rn such that the corresponding Lasso solution chooses
model M if and only if BM ∩ col(X 0 ) 6= ∅. For any {j} ⊆ M, we have, by Corollary 12 also,
BM ⊆ B{j} , so that B{j} ∩ col(X 0 ) 6= ∅ also.
Remark 4. As indicated in Theorem 13 and as discussed above, the structural set M depends
on X and λ only. Moreover, it can easily be seen that it depends on the tuning parameters λ
only through the penalization weighting in the sense that whenever λ = λ̄ω for some λ̄ > 0 and
ω ∈ Rp≥0 , M (X, λ) = M (X, ω) follows. This implies that, in particular, in the common case of
uniform tuning with λ̄ = λ1 = · · · = λp , the structural set only depends on X!
Coming back to the conjecture whether the structural set always satisfies |M | ≤ min{n, p} in
case the solutions are unique, using Theorem 13, we can list the following simple example with
n = 2 and p = 3 to show that this cannot be the case in general. However, note that Theorem 13
allows to compute the structural set and that whenever |M | ≤ n, the resulting Lasso estimator is,
in fact, just equivalent to a low-dimensional procedure!

14

Figure 7: The intersection of col(X 0 ) (in gray and yellow) with the λ-cube (in orange). The
upper left edge is contained in B{1,2} whereas the upper back edge is contained in B{2,3} . (Each
B{i,j} contains four parallel edges.) To view this figure in terms of shrinkage areas, note that the
areas corresponding to single-regressor models are displayed in gray while the shrinkage areas that
correspond to two-regressor models are displayed in yellow. The intersection of the λ-cube with
col(X 0 ) which which corresponds to the zero estimator is displayed in blue.
Example 3. Let

X=

1
0

1
1


0
1

and λ = (1, 1, 1)0 . Then the structural set is clearly given by
M = {1, 2, 3},
as (1, 1, 0)0 ∈ col(X 0 ) ∩ B{1,2} and (0, 1, 1)0 ∈ col(X 0 ) ∩ B{2,3} and BM ⊆ B{j} for any j ∈ M
by Corollary 12, see Figure 7 for illustration. Yet the Lasso solutions for this X and λ are always
unique which can be checked on the basis of Theorem 14 in the subsequent section.

4.3

A Necessary and Sufficient Condition for Uniqueness

We now turn to some results revolving around uniqueness of the Lasso estimator which can be
obtained with a similar geometric approach, that is, studying the intersection of the λ-cube with
col(X 0 ). Note that by uniqueness we mean that for a given X ∈ Rn×p and a given λ ∈ Rp , the
Lasso solutions are unique for all observations y ∈ Rn .
Tibshirani (2013) showed that for a given regressor matrix X, Lasso solutions are unique
for all observations y if the columns of X are in general position 3 which occurs when no kdimensional affine4 subspace for any k < min(n, p) contains more than k + 1 elements of the
3 Note that general position does not mean that any selection of n columns of X is linearly independent, as has
sometimes been suggested in the literature, these two concepts are in fact unrelated.
4 In Tibshirani (2013), the word “affine” is missing which has caused some confusion in the literature.

15

the set {±X1 , . . . , ±Xp }, excluding antipodal pairs (see p. 1463 in Tibshirani, 2013). In fact, the
solutions are then unique for all choices of the tuning parameter, provided that all components
are tuned uniformly. As the condition is sufficient, one may ask whether it is also necessary. The
answer to this question is, in fact, no, as can easily be seen from the example below.
When can exist non-unique solutions? For a given X ∈ Rn×p and λ ∈ Rp this occurs if and
only if there exist b, b̃ ∈ Rp with b 6= b̃ and
S(b) ∩ S(b̃) ∩ col(X 0 ) 6= ∅.
More concretely, by Theorem 11, and since the Lasso fit Xb is always unique5 , this means that
X 0 Xb + v = X 0 X b̃ + v,
where v ∈ col(X 0 )∩BM for some M ⊆ {1, . . . , p} and b̃Mc = bMc = 0. Moreover, for j ∈ M\M0 ,
we have sgn(bj ) = sgn(vj ) whenever bj 6= 0 as well as sgn(b̃j ) = sgn(vj ) whenever b̃j 6= 0. Note
that we therefore have Xb = XM bM = XM b̃M = X b̃, implying that the columns of XM must be
linearly dependent. So non-uniqueness occurs if and only if col(X 0 ) ∩ BM 6= ∅ for M ⊆ {1, . . . , p}
with linearly dependent columns in XM . The following example now immediately shows that the
columns of X being in general position is not necessary for uniqueness.
Example. Let

X=

1
0

1
0

2
1


0
.
3

Clearly, the columns are not general position, however, all Lasso solutions are unique for any
choice of the tuning parameter when the components are tuned uniformly. This is the case since
col(X 0 ) ∩ BM = ∅ whenever {1, 2} ⊆ M or |M | > 2 which can easily be checked using the fact
that v ∈ col(X 0 ) if and only if v 0 w1 = v 0 w2 = 0 for ker(X) = span{w1 , w2 }.
We find that the following criterion is in fact sufficient as well as necessary for uniqueness of
all Lasso solutions for a given X and λ.
Theorem 14 (Uniqueness). Let X ∈ Rn×p and λ ∈ Rp≥0 . The Lasso solution is unique for all
y ∈ Rn if and only if
col(X 0 ) ∩ BM = ∅ for all M ⊆ {1, . . . , p} with |M| > rk(X).
Proof. ( =⇒ ) Assume the condition is not satisfied. Then there exists v ∈ BM with |M| > rk(X)
and v = X 0 z for some z ∈ Rn . We show that there is a y ∈ Rn such that the corresponding Lasso
problem is not uniquely solvable.
If Xj = 0 for some j ∈ M0 , we are done as the corresponding coefficient may be arbitrary.
Note that Xj = 0 for j ∈ M \ M0 is not possible since that would imply vj = 0 as v ∈ col(X 0 ),
but that contradicts v ∈ BM . We therefore assume that Xj 6= 0 for all j ∈ M.
Since |M| > rk(X), there is a column of XM , say Xj (Xj 6= 0), that can be written as a linear
combination of the other columns, in particular,
X
dXj =
cl Xl
l∈M\{j}

where d = sgn(vj ) if λj 6= 0 and d = 1 if λj = 0.
b ∈ Rp by

d

 2c
bl = sgn(vl )


0

Moreover, let c = maxl∈M\{j} |cl | > 0. Define
l=j
l ∈ M \ {j}
l∈
/ M.

Then b is a Lasso solution for y = z + Xb since
X 0 y = X 0 Xb + X 0 z = X 0 Xb + v ∈ S(b).
5 This has been shown in Lemma 1 in Tibshirani (2013) for uniform tuning and can easily be extended to nonuniform and partial tuning.

16

We now construct b̃ ∈ Rp with b̃ 6= b that is also a Lasso solution for the same y by
(
cl
sgn(vl ) + 2c
l ∈ M \ {j}
b̃l =
0
l = j or l ∈
/ M.
Clearly, b 6= b̃, sgn(bl ) = sgn(b̃l ) = sgn(vj ) for l ∈ M \ {j} and
Xb =

X
l∈M

bl Xl =

X
l∈M\{j}

bl Xl +

d
Xj =
2c

X

(bl +

l∈M\{j}

cl
)Xl = X b̃.
2c

We therefore get
X 0 y = X 0 Xb + v = X 0 X b̃ + v ∈ S(b̃)
also, implying that both b and b̃ are Lasso solutions for the the given y.
( ⇐= ) We now prove the other direction. Assume that there exists y ∈ Rn such that non-unique
Lasso solutions b 6= b̃ exist. As discussed above, this implies the existence of v ∈ BM ∩ col(X 0 ) for
some M ⊆ {1, . . . , p} with XM bM = XM b̃M and bMc = b̃Mc = 0, entailing that the columns of
XM are linearly dependent.
If |M| > rk(X), we are done. If |M| ≤ rk(X), we do the following. Since we have rk(XM ) <
|M| ≤ rk(X), we can pick z ∈ Rn such that z ∈ col(XM )⊥ \ col(XMc )⊥ . This is possible since
col(XM )⊥ \ col(XMc )⊥ = ∅ ⇐⇒ col(XM )⊥ ⊆ col(XMc )⊥ ⇐⇒ col(XMc ) ⊆ col(XM )
⇐⇒ col(XM ) = col(XM , XMc ) = col(X) ⇐⇒ rk(XM ) = rk(X),
which is not the case. This z satisfies (X 0 z)M = (XM )0 z = 0 and (X 0 z)Mc = (XMc )0 z 6= 0, so
that we can find c ∈ R such that
ṽ = v + c X 0 z ∈ BM̃ ∩ col(X 0 )
with M ⊆ M̃ and |M| < |M̃|. As long as |M̃| ≤ rk(X), repeat the steps above with v = ṽ and
M = M̃.
Remark. Note that just as for Theorem 13, the result from the above theorem depends on λ only
through the penalization weights, meaning that for any M ⊆ {1, . . . , p}, whenever λ = λ̄ω for
some λ̄ > 0 and ω ∈ Rp≥0 , we have col(X 0 ) ∩ BM (λ) = ∅ if and only if col(X 0 ) ∩ BM (ω) (when
indicating the dependence of BM on the tuning parameters).
As mentioned in the preamble of Section 4, Theorem 14 does not require p > n, so that it
also covers the low-dimensional case. Clearly, the condition for uniqueness is trivially satisfied if
rk(X) = p.

5

Conclusion

We give explicit formulae regarding the distribution of the Lasso estimator in finite-samples assuming Gaussian errors. In the low-dimensional case, we consider the cdf as well as the density
functions conditional on “active sets” of the estimator. Our results exploit the structure of the
underlying optimization problem of the Lasso estimator and do not hinge on the normality assumption of the error term. We also explicitly characterize the correspondence between the Lasso
and the LS estimator: It is shown that the Lasso estimator essentially creates shrinkage areas
around the axes inside which the probability mass of the LS estimator is “compressed” into lowerdimensional densities that can be specified conditional on the active set of the estimator. As a
result, the distribution looks like a pieced-together combination of Gaussian-like densities with
each active set having its own distributional piece whose dimension depends on the number of
nonzero components in that it is given by the number of nonzero components, resulting also in
point mass at the origin and mass being distributed along the axes.
The form of the distribution is even more intricate in the high-dimensional case in which the
estimator may not be unique anymore. We quantify the relationship between a Lasso solution
17

and the quantity X 0 y (rather than the LS estimator as in the low-dimensional case). We gain
valuable insights into the behavior of the estimator by illustrating that some models may never be
selected by the estimator. In fact, a structural set based on the regressor matrix and penalization
weights only can be computed that contains all covariates that are part of a Lasso solution for
some response vector y. In case this structural set has cardinality less than or equal to n, the Lasso
is equivalent to a low-dimensional procedure and results from the p ≤ n-framework can be used
for inference. It remains to be seen whether on can quantify that this is in some sense a generic
case.
Finally, the previous insights allow us to close a gap in the literature by providing a condition
for uniqueness of the Lasso estimator that is both necessary and sufficient.

Acknowledgements
The authors gratefully acknowledge support from the Deutsche Forschungsgemeinschaft (DFG)
through grant FOR 916 and thank Thomas Hack and Nikos Dafnis for very insightful discussions
on the geometric aspects of this paper.

References
Ewald, K. & Schneider, U. (2015). Confidence sets based on the Lasso. Tech. Rep. 1507.05315,
arXiv.
Jagannath, R. & Upadhye, N. S. (2016). The Lasso estimator: Distributional properties. Tech.
Rep. 1605:03280, arXiv.
Knight, K. & Fu, W. (2000). Asymptotics of Lasso-type estimators. Annals of Statistics 28,
1356–1378.
Lee, J. D., Sun, D. L., Sun, Y. & Taylor, J. E. (2016). Exact post-selection inference with
an application to the Lasso. Annals of Statistics 44, 907–927.
Pötscher, B. M. & Leeb, H. (2009). On the distribution of penalized maximum likelihood
estimators: The LASSO, SCAD, and thresholding. Journal of Multivariate Analysis 100, 2065–
2082.
Pötscher, B. M. & Schneider, U. (2010). Confidence sets based on penalized maximum
likelihood estimators in Gaussian regression. Electronic Journal of Statistics 4, 334–360.
Tibshirani, R. (1996). Regression shrinkage and selection via the Lasso. Journal of the Royal
Statistical Society Series B 58, 267–288.
Tibshirani, R. J. (2013). The Lasso problem and uniqueness. Electronic Journal of Statistics 7,
1935–1490.
Zhou, Q. (2014). Monte carlo simulation for lasso-type problems by estimator augmentation.
Journal of the American Statistical Association 109, 1495–1516.

18

