Penalized Sieve GEL for Weighted Average Derivatives of Nonparametric Quantile IV Regressions
Xiaohong Chen Demian Pouzo James L. Powell









Penalized Sieve GEL for Weighted Average Derivatives of Nonparametric
Quantile IV Regressions
We appreciate discussions with Roger Koenker about quantiles and other
interesting topics. Roger's creativity, curiosity and kindness have inspired
us all these years. We thank guest editors and anonymous referees for their patience and helpful comments. Any errors are the responsibility of the authors. First version: August 2017.
    Xiaohong Chen
Tel.: +1 203 432 5852; Email: xiaohong.chen@yale.edu. , Demian
Pouzo
Corresponding author. Tel.: +1 510 642 6709. Email: dpouzo@econ.berkeley.edu.,
 and James
L. Powell 
Tel.: +1 510 643 0709. Email: powell@econ.berkeley.edu.
    December 30, 2023
==========================================================================================================================================================================================================================================================================================================================================================================================================================================




This paper considers estimation and inference for a
weighted average derivative (WAD) of a nonparametric quantile instrumental
variables regression (NPQIV). NPQIV is a non-separable and
nonlinear ill-posed inverse problem, which might be why there is no
published work on the asymptotic properties of any estimator of its WAD.
We first characterize the semiparametric
efficiency bound for a WAD of a NPQIV, which, unfortunately, depends on an unknown
conditional derivative operator and hence an unknown degree of ill-posedness, making
it difficult to know if the information bound is singular or not. In either case,
we propose a penalized sieve generalized empirical likelihood (GEL) estimation
and inference procedure, which is based on the unconditional WAD moment
restriction and an increasing number of unconditional moments that are implied by the conditional NPQIV restriction, where the unknown quantile function is approximated by a penalized sieve. Under some regularity conditions, we show that the self-normalized penalized sieve GEL estimator of the WAD of a NPQIV is asymptotically standard normal. We also show that the quasi likelihood ratio statistic based on the
penalized sieve GEL criterion is asymptotically chi-square distributed
regardless of whether or not the information bound is singular.





JEL Classification: C14; C22



Keywords: Nonparametric quantile instrumental variables;
Weighted average derivatives; Penalized sieve generalized empirical
likelihood; Semiparametric efficiency; Chi-square inference.






§ INTRODUCTION


Since the seminal paper by <cit.>, quantile
regressions and functionals of quantile regressions have been the subjects
of ever-expanding theoretical research and applications in economics,
statistics, biostatistics, finance, and many other science and social
science disciplines. See <cit.> and the forthcoming
Handbook of Quantile Regression (2017) for the latest theoretical advances and empirical applications.

The presence of endogenous regressors is common in many empirical
applications of structural models in economics and other social sciences.
The Nonparametric Quantile Instrumental Variable (NPQIV) regression, E[1{ Y ≤ h_0(W) } - τ|X]=0, was, to our knowledge, first proposed in
<cit.> and <cit.>. This model is a
leading important example of nonlinear and non-separable ill-posed inverse
problems in econometrics, which has been an active research topic following the Nonparametric (mean) Instrumental Variables (NPIV) regression, E[ Y  - h_0(W)|X]=0, studied by <cit.>, <cit.>, <cit.>, <cit.>, <cit.> and others. See, for example, <cit.>,
<cit.>, <cit.>, <cit.>, <cit.> and others for recent work on the NPQIV and its
various extensions.


In this paper, we consider

estimation and inference for a Weighted
Average Derivative (WAD) functional of a NPQIV. For models without nonparametric
endogeneity, WAD functionals of nonparametric (conditional) mean regression, E[ Y  - h_0(X)|X]=0, and of quantile regression, E[1{ Y ≤ h_0(X) } - τ|X]=0, have been extensively studied in both statistics and
econometrics. In particular, under some mild regularity conditions, plug-in estimators for WADs of any nonparametric mean and quantile regressions can be shown to be semiparametrically efficient
and root-n asymptotically normal (where n is the sample size). See, for example, <cit.>,
<cit.>, <cit.>, <cit.> and the references therein. Although unknown functions of endogenous
regressors occur frequently in empirical work, due to the ill-posed nature
of NPIV and NPQIV, there is not much research on WAD functionals of NPIV and
NPQIV yet. In fact, even for the simpler NPIV model that is a linear and
separable ill-posed inverse problem, it is still a difficult question whether a
linear functional of a NPIV could be estimated at the root-n rate; see, e.g.,
<cit.> and <cit.>. Although <cit.> provide low-level sufficient conditions for a root-n
consistent and asymptotically normal estimator of the WAD of the NPIV model, and <cit.> provide a semiparametric efficient estimator of WAD
for that model, to our knowledge, there is no published work on semiparametric efficient estimation
of the WAD for the NPQIV model yet.

We first characterize the semiparametric efficiency bound for the WAD functional of a
NPQIV model. Unfortunately, the bound depends on an unknown conditional derivative
operator and hence an unknown degree of ill-posedness. Therefore, it is difficult to
know if the semiparametric information bound is singular or not. Further, even if a researcher
assumes that the information bound is non-singular and the WAD is root-n
consistently estimable, the results in <cit.> and <cit.> show that a simple plug-in estimator of a WAD
might not be semiparametrically efficient. This is in contrast to the
results of <cit.> and <cit.>
who show that plug-in estimators of a WAD of a nonparametric mean and quantile
regression are semiparametrically efficient.

We then propose penalized sieve Generalized Empirical Likelihood
(GEL) estimation of the WAD for the NPQIV model, which is based on the unconditional
WAD moment restriction and an increasing number of unconditional moments implied by the conditional moment restriction of the NPQIV model, where the unknown quantile function is approximated by a flexible penalized sieve.

Under some regularity conditions, we show that the self-normalized penalized sieve GEL estimator of the WAD of a NPQIV is asymptotically standard normal. We also show that the Quasi Likelihood Ratio (QLR) statistic based on the penalized sieve GEL criterion is asymptotically chi-squared distributed
regardless of whether the information bound is singular or not; this can be
used to construct confidence sets for the WAD of NPQIV without the need to
estimate the variance nor the need to know the precise convergence rates of
the WAD estimator.

Our estimation procedure builds upon <cit.>, who approximate a
conditional moment restriction E[ρ (Y,θ_0 )|X]=0 by an increasing
sequence of unconditional moment restrictions, and then consider estimation of the Euclidean parameter θ_0 (of fixed and finite dimension) and specification
tests based on GEL (and related) procedures. For the same model E[ρ (Y,θ_0 )|X]=0, <cit.>
directly estimate the conditional moment restriction via kernel and then
apply a kernel-based conditional empirical likelihood (EL) to estimate θ_0. However, the model considered in these papers does not contain any
unknown functions  (say h()) and the residuals ρ (.,θ) are assumed
to be twice continuously differentiable with respect to θ at θ_0. For the semiparametric conditional moment restriction E[ρ (Y,θ_0,
h_0 (·) )|X]=0 when the
unknown function h(· ) could depend on an endogenous variable, <cit.> and <cit.> consider a
sieve conditional EL extension of <cit.>, and <cit.> provides a sieve unconditional GEL extension of <cit.>, where the unknown function h(.) is approximated by a finite dimensional linear sieve (series) as in <cit.>. However, like
<cit.>, all these papers assume twice continuously differentiable residuals ρ(.,θ, h(.)) with respect to (θ_0, h_0(.)), and hence rule out
the NPQIV model.

<cit.> study GEL properties for non-smooth residuals g(.,.) in the unconditional moment models E[g(Y,θ _0)]=0, but
require the dimensions of both g(.,.) and θ _0 to be fixed and finite.

Finally, <cit.>, <cit.>, <cit.>, and <cit.>  do include the NPQIV model, but none of these papers addresses the issues of 
estimation and inference for the WAD of the NPQIV.

The rest of the paper is organized as follows. Section <ref>
introduces notation and the model. Section <ref> characterizes the
semiparametric efficiency bound for the WAD of the NPQIV model. Section <ref> introduces a flexible penalized sieve GEL procedure. Section <ref> derives the consistency and the convergence rates of the
penalized sieve GEL estimator for the NPQIV model. Section <ref>
establishes the asymptotic distributions of the WAD estimator and of the QLR statistic based on penalized sieve GEL for the WAD of a NPQIV. Section <ref> concludes with a discussion
of extensions.




§ PRELIMINARIES AND NOTATION




Let Z≡ (Y,W,X) be the observable data vector, where Y is the outcome
variable, W is the endogenous variable and X is the instrumental variable (IV); we assume the observable data, Z, is distributed according to a probability distribution 𝐏. In order to simplify the exposition, we restrict attention to real-valued continuous random variables, i.e., we assume 𝐏 has a density 𝐩 with support given by ℤ≡𝕐×𝕎×𝕏⊆ℝ^3; extending our results to vector-valued endogenous and instrumental variables would be straightforward but cumbersome in terms of notation.


Notation. For any subset, ℤ, of an Euclidean space let 𝒫(ℤ) be the class of Borel probability measures over ℤ. For any P∈𝒫(ℤ), we use p to denote its probability density function (pdf) (with respect to Lebesgue (Leb) measure) and supp(P) to denote its support. We also use P_X (p_X) to denote the marginal probability (pdf) of a random variable X; and P_Y|X (p_Y|X) to denote the conditional probability (pdf) of Y given X. For expectation, we write E_Q[.] to be explicit about the fact that Q is the measure of integration; throughout we sometimes use E[.]≡ E_𝐏[.] when 𝐏 is the true probability of the data. The term “wpa1" stands for “with probability approaching one (under 𝐏)"; for any two real-valued sequences (x_n,y_n)_n x_n≾ y_n denotes x_n≤ C y_n for some C finite and universal; ≿ is defined analogously. For any q ≥ 1, we use L^q(Q)≡ L^q(ℤ,Q) to denote the class of measurable functions f : ℤ↦ℝ such that ||f||_L^q(Q)=( ∫_z∈ℤ
|f(z)|^qQ(dz)) ^1/q < ∞; as usual L^∞(Leb) denotes the class of essentially bounded real-valued functions. We use ||.||_e to denote the Euclidean norm, ℝ_+=[0,∞ ) and ℝ_++=(0,∞ ).

For any subset S of a vector space (𝕊,||.||_S), lin {S} denotes the smallest
linear space containing S; for any subspace A ⊆𝕊, A^⊥ denotes its orthogonal complement in (𝕊,||.||_S). For any linear operator, M : (𝕊_1,||.||_1 ) → (𝕊_2,||.||_2 ), let Kernel(M) ≡{ x∈𝕊_1  M[x] = 0  } and Range(M) ≡{ y ∈𝕊_2 ∃ x ∈𝕊_1, M[x] = y  }; it is bounded if and only if sup_x∈𝕊_1 : ||x||_1=1 ||M[x]||_2 <∞. For any linear bounded operator M, M^+ denotes its generalized inverse; see, e.g., <cit.>.




 §.§ The WAD of the NPQIV model


Let 𝔸≡ℝ×ℍ, where
ℍ = { h ∈ L^2(Leb)  h^' exists and ||h^'||_L^2(Leb) < ∞}, i.e., ℍ is a Sobolev space of order 1, here h^' should be viewed as a weak derivative of h (see <cit.>). We note that ℍ is a Hilbert space under the norm ||h||_ℍ≡ ||h||_L^2(Leb)+||h'||_L^2(Leb), and 𝔸 is a Hilbert space under the norm ||(θ, h)||_𝔸≡ ||θ||_e + ||h||_ℍ. In this paper we measure convergence in 𝔸 using another norm ||(θ, h)||≡ ||θ||_e + ||h|| for ||h||≤ ||h||_ℍ (such as ||h||=||h||_L^2(Leb)). The parameter set is given by 𝒜≡Θ×ℋ⊆𝔸, where Θ is bounded and convex  and ℋ is a set that contains additional restrictions on h ∈ℍ which will be specified below. We assume that 𝐏 is such that there exists a parameter α_0≡ (θ_0,h_0) ∈𝒜 that satisfies

    0 =     E_𝐏[1{ Y ≤ h_0(W) } - τ|X] 
    θ_0 =     E_𝐏[μ(W)h^'_0(W)]

for τ∈ (0,1), where μ is a nonnegative, continuously differentiable scalar function in L^∞(Leb) ∩ℍ and should be viewed as the weighting function of the average derivative, θ_0, of h_0.


The following assumption ensures that the conditions above uniquely identify α_0; it will be maintained throughout the paper and will not be explicitly referenced in the results below.


 There is a unique α _0∈ int(𝒜) that
satisfies model (<ref>)-(<ref>).


The interior assumption is needed only for the asymptotic distribution results in Section <ref>. In cases where ℋ has an empty interior, one can use
the concept of relative interior of ℋ.
This assumption is clearly high level. The goal of this paper is to
characterize the asymptotic behavior of a modified GEL estimator of α
, taking as given the identification part; for a discussion of primitive
conditions for Assumption <ref>, we refer the reader to <cit.> and
references therein.


The following assumption
imposes additional restrictions over the primitives: μ, 𝐏 and α _0.


 (i) 𝐏 has a continuously differentiable pdf, 𝐩, such that: the marginal density 𝐩_W of W is uniformly
bounded, zero at the boundary of the support and 𝐩^'_W∈ L^2(Leb); the marginal density 𝐩_X of X is uniformly bounded away from 0 on its support; sup_y,w,x ∈ℤ𝐩_Y|WX(y
| w, x) < ∞, sup_y,w,x ∈ℤd𝐩_Y|WX(y | w, x)/dy < ∞
;
(ii) ℋ is convex and such that for all h ∈ℋ, sup_w ∈𝕎 |μ(w) h(w)| < ∞;
(iii) Var_𝐏(μ(W)h^'(W)) > 0 for all h ∈ℋ in a ||· ||-neighborhood of h_0.


Part (i) of this condition imposes differentiability and boundedness
restrictions on different elements of 𝐩; part (ii) ensures that lim_w →±∞𝐩_W(w) μ (w) h(w) = 0 which allows for an alternative representation for θ_0 using integration by parts (see expression <ref>  below); part (iii) is a high level
assumption and essentially implies Var_𝐏(μ(W)h^'_0(W)) > 0 as
well as continuity of h ↦ Var_𝐏(μ(W)h^'(W)).





§ EFFICIENCY BOUND FOR Θ_0




By definition of ℍ, Assumption <ref> and integration
by parts, it follows that

    θ_0 = E[μ(W)h_0^'(W)] = - ∫ℓ(w) h_0 (w) dw

where 
    w ↦ℓ(w) ≡μ^'(w) 𝐩_W(w) + μ(w)
    𝐩_W^'(w).


For the derivations of the
efficiency bound, it is important to recall that ℓ depends on p_W,
so we sometimes use ℓ_𝐏 to denote ℓ. Finally, observe that under our assumptions over μ and 𝐩_W, ℓ∈ L^2(Leb).

The formal definition of the efficiency bound for the unknown parameter θ _0 is given at the beginning of Appendix <ref>. Loosely speaking, the efficiency bound is a lower bound for the asymptotic variance of all locally regular and asymptotically linear estimators of θ_0; see <cit.> for details and formal definitions. If it is infinite, then the parameter θ_0 cannot be estimated at root-n rate by these estimators. We now derive this bound. For this, we introduce some useful
notation. For any (y,w,α )∈𝕐×𝕎×𝔸, let 
    ρ (y,w,α )≡(ρ_1(y,w,α),ρ_2(y,w,h))^T≡(θ -μ (w)h^'(w), 1{y ≤ h(w) } - τ)^T.

Let 𝐓:ℍ→ L^2(𝐏_X) be given by

    𝐓[g](x)=∫𝐩_Y|WX(h_0(w)|w,x)g(w)𝐩_W|X(w|x)dw

for all x∈𝕏 and g∈ℍ. The fact that 𝐓 maps into L^2(𝐏_X) follows from
Jensen inequality and the fact that sup_w,x𝐩_YW| X(h_0(w),w|
x)<∞ (see Assumption <ref>). Its adjoint operator is denoted
as 𝐓^∗:L^2(𝐏_X)→ L^2(𝐏_W). Finally, let

    x↦Γ (x)≡ E[ρ _1(Y,W,α _0)ρ _2(Y,W,h_0)|X=x]/(τ (1-τ ))

and z↦ϵ (z)≡ρ
_1(y,w,α _0)-Γ (x)ρ _2(y,w,h _0). Then E[ϵ (Z)ρ _2(Y,W,h_0)|X]=0 and E[ϵ (Z)]=0.



 Suppose Assumptions <ref> and <ref>
hold and ℓ∈ Kernel(𝐓)^⊥. Then



  * The efficiency bound of θ_0 is finite iff ℓ∈ Range
(𝐓^∗).


  * If it is finite, its efficient variance V_0 is given by

    V_0= ||ϵ(·) ||^2_L^2(𝐏) +
    ‖𝐓(𝐓^∗𝐓)^+ [ ℓ - 𝐓^∗[Γ] ] ‖^2_L^2(𝐏).





	 	See Appendix <ref>.
	 

The first result in Theorem <ref> is obtained following the
approach of <cit.>.  The condition ℓ∈ Kernel(𝐓)^⊥ ensures that only the “identified part" of h_0 — that is, the part of h_0 that is orthogonal to the kernel of 𝐓 —  matters for computing the weighted average derivative; we refer the reader to Appendix <ref> and the paper by  <cit.> for further discussion.



<cit.> provides an analogous result to Theorem <ref>(1) for linear functionals
in a nonparametric linear IV regression model. Our condition ℓ∈ Range(𝐓^∗), is analogous to theirs, but with a subtle yet
important difference. In <cit.>, the object that plays the role of ℓ does not depend on 𝐏, whereas in our case it does. This
observation changes the nature of our condition vis-a-vis theirs, because,
in our setup, ℓ∈ Range(𝐓^∗) implies a restriction on 𝐏
since both quantities, ℓ and 𝐓 depend on it.[
It is worth pointing out that this restriction was not imposed as one of the
conditions that defined the model used to construct the tangent space; see
Appendix <ref> for a definition.] It is also important to note that, if 𝐓 is compact, then the range of 𝐓^∗ is a strict subset of L^2(𝐏_W) so that ℓ∈
Range(𝐓^∗) may not hold. Hence, in this case the weighted average
derivative may not be root-n estimable, and, moreover, the condition that
determines the finiteness of the efficiency bound depends on unknown
quantities. This observation highlights a difference with the no-endogeneity
case, where the efficiency bound is always finite, provided that ℓ∈
L^2(𝐏_W) (see <cit.>).

Another discrepancy between the no-endogeneity case and ours is that in the
former case the “plug in" is always efficient (see <cit.>, <cit.>) due to the fact that the
tangent space is the whole of { f ∈ L^2(𝐏)  E[f] = 0 }. On
the other hand, for NPQIV <cit.> show that the
closure of the tangent space is the whole space iff the Range(𝐓) is dense
in L^2(𝐏_X), which in turn is equivalent to Kernel(𝐓^∗) = { 0}
. This last condition is comparable to a completeness condition on the
conditional distribution of the exogenous variable given the endogenous
ones, which may or may not hold for a particular 𝐏.[
In the NPIV setting, Kernel(𝐓^∗) = { 0} is equivalent to the pdf of
X given W satisfying a completeness condition.]

The second result in Theorem <ref> follows from projecting the
influence function onto the closure of the tangent space (see <cit.> and <cit.> and references therein). So as
to shed some light on the expression for the efficiency bound, we point out
that it corresponds to the efficiency bound of the semiparametric
sequential conditional moment model via the “orthogonalized moments" approach in <cit.>.




In their notation, let ε_2(z,α) ≡ρ_2(y,w,h) and
ε_1(z,α) ≡ρ_1(y,w,α) - Γ(x)ρ_2(y,w,h).
Note that E[ε_1(Z,α_0)ε_2(Z,α_0) | X] = 0 (and ε_1(z,α_0 )=ϵ (z)). The model (<ref>)-(<ref>) becomes equivalent to their orthogonalized moment model:

    E[ε_2(Z,α_0)|X] = 0 ,   E[ε_1(Z,α_0)]=0.

The expression in our Theorem <ref>(2) coincides with their theorem 2.3 semiparametric efficient variance bound for θ_0 of the model (<ref>). Also see proposition 3.3 in <cit.> for the semiparametric efficient variance bound for the WAD of a NPIV model.




§ THE PENALIZED-SIEVE-GEL ESTIMATOR




In this section we introduce our estimator for α_0∈𝒜≡Θ×ℋ⊆𝔸≡Θ×ℍ. In order to do
this, it will be useful to define some quantities. Given the i.i.d. sample (Z_i)_i=1^n, let P_n be the corresponding empirical probability. Let (q_k)_k∈ℕ be a complete basis in L^2(𝕏,Leb). For any J∈ℕ, let q^J(x)=(q_1(x),...,q_J(x))^T be J× 1 vector-valued function of x, and for any (z,α )∈ℤ×𝒜, let

    g_J(z,α )≡ (ρ _1(y,w,α ),ρ _2(y,w,α)q^J(x)^T)^T
    = (θ -μ (w)h^'(w), [ 1{y ≤ h(w) } - τ ]q^J(x)^T)^T .

Let 𝒮⊆ℝ be an open interval that contains 0. For any P ∈𝒫(ℤ),  any α∈𝒜 and any J
∈ℕ, denote Λ_J(α,P)≡∩_z ∈ supp (P){λ∈ℝ^J+1λ^Tg_J(z,α) ∈𝒮}, and Λ̂_J(α)≡Λ_J(α,P_n).

Let s:𝒮→ℝ be strictly
concave, twice-continuously differentiable with Lipschitz continuous second
derivative; and s^'(0)=s^''(0)=-1; see, e.g., <cit.> and <cit.> for examples of such s(.) functions. For any λ∈Λ_J(α,P), let

    S_J(α,λ,P) ≡ E_P[s(λ^Tg_J(Z,α))] - s(0) ,   Ŝ_J(α,λ)≡ S_J(α,λ,P_n).

If 𝒜 were a finite-dimensional compact set with (𝒜) ≤ J+1, then α_0 could be estimated by the GEL procedure: min_α∈𝒜sup_λ∈Λ̂_J(α )Ŝ_J(α ,λ ) (see, e.g., <cit.>).

Due to the presence of the infinite-dimensional nuisance parameter h_0 ∈ℋ in the NPQIV model (<ref>), the parameter space 𝒜≡Θ×ℋ is  an infinite-dimensional function space that is typically non-compact subset in (𝔸, ||.||) and hence the identifiable uniqueness condition needed for consistency in ||.||-norm might fail; see, e.g., <cit.> and <cit.>. The above GEL procedure needs to be regularized to regain consistency and/or to speed up rate of convergence in ||.||-norm. 
To this end, we introduce a regularizing structure, which, jointly with (q_k)_k ∈ℕ, consists of a sequence of sieve spaces (𝒜_k ≡Θ×ℋ_k)_k∈ℕ in (𝔸, ||.||), and a sequence of penalties (γ_k× Pen (·))_k ∈ℕ with tuning parameters γ_k↓ 0 and a penalty function Pen : 𝔸→ℝ_+.

The Penalized-Sieve-GEL (PSGEL) estimator
is defined as

    α̂_L,n∈min_α∈𝒜_K[sup_λ∈Λ̂_J(α )Ŝ_J(α ,λ )+γ
    _KPen(α )],

for any (L=(J,K),n)∈ℕ^3. If the “arg min" in the previous expression is empty, one can replace it by an approximate minimizer.

The following assumption imposes restrictions over the regularizing structure {(q_k, ℋ_k,γ_kPen)_k ∈ℕ}. Let (φ _k)_k∈ℕ be a basis functions in ℍ, and ∇φ
^K=(φ _1^',...,φ _K^')^T.


 (i) (q_k)_k∈ℕ is a basis in L^2(𝐏_X), and E[q^J(X)q^J(X)^T]=I for each finite J;
(ii) For all K, ℋ_K⊆ lin{φ _1,...,φ _K} is
closed and convex, and ∪ _kℋ_k⊇ℋ, i.e., for any α∈𝒜≡Θ×ℋ there is an Π_Kα∈𝒜_K ≡Θ×ℋ_K such that ||Π_Kα - α ||=o(1); and for some finite C≥ 1,
C^-1I≤ E[ ( φ ^K(W)) (
φ ^K(W)) ^T+( ∇φ ^K(W)) (
∇φ ^K(W)) ^T] ≤ CI;
(iii) (a) Pen : 𝔸→ℝ_+
is lower semi-compact (in ||.||), |Pen (Π_Kα_0 ) - Pen (α_0)|=O(1), Pen (α_0 )<∞, and γ_k↓ 0, and (b) there exists an M<∞ such that for any m≥ M, any K and any α∈𝒜_K,
if Pen(α )≤ m then sup_w ∈𝕎 |μ(w) h^'(w)| ≤ m.


Condition (i) is mild (see <cit.> (DIN) and the discussion therein).
Condition (ii) essentially defines the sieve space. Part (a) of Condition (iii) is standard in ill-posed problems (see <cit.>); Part (b) is not. If ℋ_K is ||· ||_L^∞(𝕎,μ) bounded, then the condition is vacuous.
If this is not the case, then the condition requires Pen to be “stronger"
than the ||· ||_L^∞(𝕎,μ) norm. The need to bound || h^' ||_L^∞(𝕎,μ) arises from the fact that, in many
instances, in the proofs we need to control ρ(y,w,α) uniformly on (y,w) (e.g., see Lemma <ref> in the Supplemental Material <ref>). Additionally, in our setup, is useful to link Pen to ||· ||_L^∞(𝕎,μ) because the structure of the problem implies a natural bound for Pen(.) — and thus, through Assumption <ref>(iii), a bound for ||· ||_L^∞(𝕎,μ) —, as shown in the following lemma.


 For any L=(J,K) ∈ℕ^2 and any α∈𝒜_K,

    γ_K Pen(α̂_L,n) ≤sup_λ∈Λ̂
    _J(α)Ŝ_J(α,λ) + γ_K Pen(α)   wpa1.




	See Appendix <ref>.


The bound, however, may depend on (J,K,n) and thus may affect the convergence
rate. Below, we will set α in the right-hand-side (RHS) to a particular value in 𝒜_K and use the resulting bound to construct what we call an “effective sieve space".





§ CONSISTENCY AND CONVERGENCE RATES OF THE PSGEL ESTIMATOR



This section establishes the consistency and the rates of convergence of the PSGEL estimator α̂_L,n to the true parameter α_0 under a given norm ||.|| over 𝔸. In this and the next section, we note that the implicit constants inside the O_𝐏 do not depend on (J,K,n).



 §.§ Effective sieve space


Throughout the paper we use the following notation. Let θ≡sup_θ∈Θ |θ| <∞; and  b_ρ,J≡ (E[||q^J(X)||^ρ_e])^1/ρ for any ρ > 0. For any L=(J,K) ∈ℕ^2, let

    Γ_L,n≡{g̅^2_L,0/n +
    ||E[g_J(Z,Π_K α_0)]||^2_e + γ_K Pen(Π_K α_0)
    },   g̅^2_L,0≡θ + ||μ (Π_K h_0)^'||^2_L^2(𝐏) + b_2,J^2.

Let (l_n)_n be a slowly diverging positive sequence, e.g., l_n =loglog n, which is introduced solely to avoid keeping track of constants. Finally we let

    𝒜̅_L,n≡{α∈𝒜_K
    Pen(α) ≤℧_L,n} ,   where  ℧_L,n≡ l_nγ^-1_KΓ_L,n .






The sequence of sets, (𝒜̅_L,n)_L,n, can be viewed as
the sequence of “effective" sieve spaces, because, as the following lemma
shows, wpa1 the estimator (and, trivially, the sieve approximator Π_K α_0∈𝒜_K) both
belong to it.


 (i) b^4_4,J/n = o(1); (ii) δ_n = o(1), δ_n×℧_L,n = o(1), b_ϱ,J^ϱ n δ^ϱ_n= o(1) for some ϱ > 0; (iii) √(g̅^2_L,0/n + ||E[g_J(Z,Π_K α_0)]||^2_e)
= o(δ_n).



	 Let Assumptions <ref>, <ref>, <ref> and <ref> hold. Then, for any L ∈ℕ^2, α̂
	_L,n∈𝒜̅_L,n wpa1.



	See Appendix <ref>.


The proof of this Lemma follows from Lemma <ref> with α
= Π_K α_0 and Lemma <ref> with  α = Π_K α_0 and P=P_n in
Appendix <ref>. The latter lemma provides a bound for sup_λ∈Λ̂_J(Π_K α_0)Ŝ_J(Π_K α_0,λ) in terms
of ||n^-1∑_i=1^n g_J(Z_i,Π_K α_0) ||^2_e and γ_K Pen(Π_K α_0). With this in mind, the components of ℧_L,n are intuitive: g̅^2_L,0/n is related to the
“variance" of n^-1∑_i=1^n g_J(Z_i,Π_K α_0), where g̅^2_L,0 is a bound for ||g_J(.,Π_K α_0)||^2_e. The term ||E[g_J(Z,Π_K α_0)]||_e is related to the “bias" and reflects the
fact that Π_K α_0∈𝒜_K is a sieve approximate to α_0.



	As explained above, Lemma <ref> and Assumption <ref>(iii) are used to ensure that || h^'||_L^∞(
		𝕎,μ) is bounded. If the construction of ℋ_K directly
	implies || h^'||_L^∞(𝕎,μ)≤℧ for some
	fixed constant ℧ < ∞, then ℧ should replace ℧_L,n in
	the definition of 𝒜̅_L,n. This is applicable every time ℧_L,n appears below. 





 §.§ Relation to Penalized Sieve GMM



As expected, the asymptotic properties of the PSGEL estimator are closely related to an approximate minimizer of a GMM criterion associated to the following expression: for any J ∈ℕ and any P ∈𝒫(ℤ), let

    α↦ Q_J(α,P) ≡
    E_P[g_J(Z,α)]^TH_J(α_0,𝐏)^-1E_P[g_J(Z,α)]

where (α,P) ↦ H_J(α,P) ≡ E_P[g_J(Z,α)
g_J(Z,α)^T]. That is, Q_J(.,𝐏) is the optimally weighted (population) GMM criterion
function associated with the vector of moments E[g_J(Z,·)].

For what follows, it will be useful to define the following intermediate
quantity which can be viewed as a (sequence) of pseudo-true parameters. For
each L≡ (J,K)∈ℕ^2, let

    α _L,0≡min_α∈𝒜̅_L,nQ_J(α
    ,𝐏).

We note that α _0∈min_α∈𝒜Q_J(α ,𝐏) for any J; but as we
restrict to the effective sieve space 𝒜̅_L,n, it could be
that α _L,0≠α _0 for any L∈ℕ^2. The following lemma guarantees that α _L,0 is in fact non-empty.


 Let Assumptions <ref> and <ref> hold. Then, for each L=(J,K) ∈ℕ^2, α_L,0 is non-empty.



		See Appendix <ref>.
	

While this lemma shows that α _L,0 is non-empty, it may not be a singleton. Nevertheless, for model (<ref>)-(<ref>), it is easy to choose some finite-dimensional linear sieve ℋ_K and some strict convex penalty Pen such that α _L,0 is in fact a singleton. Therefore the next assumption is effectively a way to suggest choices of a regularizing structure:


	For any L ∈ℕ^2, α _L,0 is single-valued.



Let m_2(X,α) ≡ E_𝐏[ρ_2(Y,W,α) | X]. For each J ∈ℕ, the L^2(𝐏) projection of m_2(·, α) onto the linear span of q^J(X) is denoted as Proj_J[m_2(·,α)](X), where

    Proj_J[m_2(·,α)](X) =     E_𝐏[ m_2(X,α) q^J(X)^T] (E_𝐏[q^J(X)q^J(X)^T])^-1 q^J(X) 
    
    =      E_𝐏[ ρ_2(Y,W,α) q^J(X)^T] q^J(X)

where (E_𝐏[q^J(X)q^J(X)^T])^-1 = I by Assumption <ref>.



















The next lemma provides sufficient conditions that ensure convergence of α_L,0 to the true parameter α_0.


 Let Assumptions <ref>, <ref>, <ref> and <ref> hold. Suppose lim_n →∞sup_α∈𝒜̅_L_n,n||Proj_J_n[m_2(·,α)] - m_2(·,α)||_L^2(𝐏) = 0. Then: ||α_L_n,0-α_0||=o(1).



		See Appendix <ref>.
	



 §.§ Convergence rates


A crucial part of establishing the convergence rate of α̂_L,n is to bound the rate of ||α̂_L,n - α_L,0||. For this it is important to quantify how well the population sieve GMM criterion function Q_J separates points in (𝒜̅_L,n,||.||) around α_L,0. To do this, we define, for each, (L,n)∈ℕ^3, ϖ_L,n : ℝ
_+→ℝ_+ as

    t ↦ϖ_L,n(t) ≡inf_{α∈𝒜̅_L,n ||α - α_L,0 || ≥ t } Q_J(α,𝐏) -
    Q_J(α_L,0,𝐏).


The function ϖ_L,n is analogous to the one used in the standard identifiable
uniqueness condition (see <cit.>, <cit.>). Within the
ill-posed inverse literature this function is akin to the notion of sieve
measure of ill-posedness used in <cit.> and <cit.>. The following lemma establishes some useful properties.


	 Let Assumptions <ref>, <ref> and <ref> hold. Then: for each (L,n) ∈ℕ^3, ϖ_L,n(t) = 0
	iff t = 0 and ϖ_L,n is continuous and non-decreasing in t.



	See Appendix <ref>.


It is worth noting that even though ϖ_L,n(t) > 0 for all t>0,
it could happen that ϖ_L,n(t) → 0 as L diverges. This
behavior reflects the ill-posed nature of the problem.

We now present some high-level assumptions used to establish the convergence rate of the PSGEL estimator. The first of these assumptions introduces, and imposes restrictions on, a positive real-valued sequence (δ _n)_n∈ℕ that is common in the GEL literature (see the Appendix in <cit.>). It ensures that the ball {λ∈ℝ^J+1 ||λ||_e≤δ_n} belongs to Λ̂_J(α) for any α∈𝒜̅_L,n (see Lemma <ref> in the Supplemental Material <ref>).  The assumption also restricts the rates of (b_ρ,J)_ρ∈ℝ,J∈ℕ, and the rate at which L=(J,K) ∈ℕ^2 diverges relative to n:


 (i) Assumption <ref> holds; (ii)  δ_n l_n = o(1), b^3_3,Jδ^ϱ_n= o(1) for some ϱ > 0, and (℧_L,n)^4 b^4_4,J/n = o(1).



Recall that the sequence (l_n)_n diverges
arbitrary slowly like loglog n, and the bound ℧_L,n is allowed to grow
(slowly) at the rate of l_n. Assumption <ref> slightly strengthens Assumption <ref>.


The following assumption is a high-level condition that
controls the supremum of the process f ↦𝔾_n[f] ≡ n^-1/2∑_i=1^n{f(Z_i) -E[f(Z_i)]} over  the classes 𝒜̅_L,n and 𝒢_L≡{ (y,w) ↦ρ_2(y,w,α) α∈𝒜̅_L,n}.


 There exists a positive real-valued sequence, (Δ_L,n)_L,n ∈ℕ^3, such that, for any L ∈ℕ
^2, sup_(θ,h) ∈𝒜̅_L,n |𝔾_n[μ·
h^']| =O_𝐏(Δ_L,n) and for all 1 ≤ j ≤ J, sup_g ∈𝒢_L |𝔾_n[g · q_j]|
=O_𝐏(Δ_L,n).


For instance, if {μ· h^' h ∈ℋ} and { (y,w) ↦ 1{ y ≤ h(w) } h ∈ℋ} are P-Donsker, then (Δ_L,n)_L,n ∈ℕ^3
is uniformly bounded.[Restrictions on the “complexity" of these classes are implicit restrictions on the “complexity" of ℋ; see <cit.> and <cit.>.]  But if this is not the case, then (Δ_L,n)_L,n
∈ℕ^3 may diverge as L (or n) grows.

The next theorem establishes the convergence rate of the PSGEL estimator; in particular it establishes the rate for the estimator of the infinite dimensional component h_0 ∈ℋ.


	 Suppose Assumptions <ref>, <ref>, 
	<ref>, <ref> and <ref> hold. For any (δ_n,l_n)_n
	satisfying Assumption <ref>, there exists a finite constant M>0 such that
	
    ||α̂_L,n - α_0|| = O_𝐏( ϖ_L,n^-1( M (δ_1,L,n + δ_2,L,n) ) ) + ||α_L,0 -
    	α_0 ||,

	where 
    δ _1,L,n≡√(J/n)×Δ _L,n×( θ+℧ _L,n+b_2,J) ,  δ _2,L,n≡℧ _L,n^2{δ _n+δ
    	_n^-1Γ _L,n}




	See Appendix <ref>.


The rate of convergence of the PSGEL estimator is composed of two standard
terms reflecting the “approximation error" ||α_L,0 - α_0 ||
and the “sampling error" ϖ_L,n^-1(M(δ_1,L,n +
δ_2,L,n)). The component ϖ^-1_L,n(.), reflects the ill-posed nature of the estimation
problem. As noted previously, even though, for a fixed L, ϖ_L,n(t) >
0 for t>0, this relationship can deteriorate as L diverges, which
implies that ϖ_L,n^-1(t) may diverge as L diverges.

Below, we present an heuristic description of the proof that sheds light on the role of the sequences (δ_1,L,n,δ_2,L,n)_L,n and of ϖ_L,n.



 §.§ Heuristics


By the triangle inequality it suffices to bound the rate of ||α̂_L,n - α_L,0||. We do this by linking the PSGEL estimator to the population sieve GMM problem defined by Q_J(·,𝐏). The first step to do this is to show that the PSGEL is an approximate minimizer of the sample sieve GMM criterion Q_J(.,P_n) with the rate given by δ _2,L,n.

	 Let Assumptions <ref>
	and <ref> hold. For any (δ _n,l_n)_n∈ℕ satisfying Assumption <ref>, we have:
	
    Q_J(α̂_L,n,P_n)=O_𝐏(δ _2,L,n) ,   with  δ _2,L,n = ℧ _L,n^2{δ _n+δ_n^-1Γ _L,n} .




	See Appendix <ref>.


The Lemma illustrates not only the role of δ_2,L,n but its nature. The two terms inside the curly brackets are completely analogous to those appearing in <cit.>. The
scaling by ℧^2_L,n is not present in <cit.> and its
appearance here is due to the fact that the bound of ρ(.,α_L,0)
may depend, in principle, on n and L=(J,K). In <cit.>, on the
other hand, the upper bound ℧_L,n can be taken to be a fixed constant
due to their Assumption 6.

Lemma <ref> implies that, for some finite M, the event Q_J(α̂_L,n,P_n) - Q_J(α_L,0,P_n) ≤ M δ_2,L,n occurs wpa1. The next step is to link the empirical GMM criterion function, Q_J(·,P_n), to its population analog, Q_J(·,𝐏) for which we can quantify its behavior (around α_L,0) using ϖ_L,n. The next lemma provides such a link by showing that Q_J(·,P_n) converges to its population analog.

	 Let Assumptions <ref>, <ref> and <ref> hold. Then: for any L≡ (J,K)∈ℕ^2,
	
    sup_α∈𝒜̅_L,n|Q_J(α ,P_n)-Q_J(α,𝐏)|=O_𝐏(δ _1,L,n) ,   with  δ _1,L,n=√(J/n)×Δ _L,n×( θ +℧ _L,n+b_2,J).

	


	See Appendix <ref>.


The rate (δ_1,L,n)_L,n has several components. The component √(J/n) reflects the pointwise convergence rate of ||n^-1∑_i=1^n g_J(Z_i,α) - E[g_J(Z,α)] ||_e, while the
factor of Δ_L,n reflects the fact that we need uniform
convergence of that term. Finally, the term ( θ +
℧_L,n + b_2,J) is essentially the (uniform) bound for α↦ ||n^-1∑_i=1^n g_J(Z_i,α)||_e and α↦ ||E[g_J(Z,α)] ||_e over 𝒜̅_L,n.

With this result at hand and simple algebra, one can show that for some finite M the set A ≡{Q_J(α̂_L,n,𝐏) - Q_J(α_L,0,𝐏) ≤ M (δ_1,L,n + δ_2,L,n)} occurs wpa1. Therefore, by standard laws of probabilities, it follows that the probability of the set ||α̂_L,n - α_0|| ≥ M^'ϖ_L,n^-1( M (δ_1,L,n + δ_2,L,n) ) (for any M^') is — up to a vanishing term — less or equal than the probability of the intersection of the same set with A. Therefore, it only remains to show that the latter probability is naught for sufficiently large M^'. This follows because this latter probability is in turn bounded above by the probability of ϖ_L,n( M^'ϖ_L,n^-1( M (δ_1,L,n + δ_2,L,n) ) ) ≤ M (δ_1,L,n + δ_2,L,n). By the fact that ϖ_L,n is non-decreasing (see Lemma <ref>), this probability is naught by sufficiently large M^', proving the result of Theorem <ref>.



 §.§ Discussion of the elements in the Convergence Rate



We now present some observations regarding the main components of the convergence rate in Theorem <ref>, namely, the rates (δ_1,L,n,δ_2,L,n)_L,n and ϖ_L,n defined in expression <ref>. Regarding the latter, we first need to specify the norm ||.||. We start by taking (φ_k)_k ∈ℕ to be an orthogonal basis with respect to the Lebesgue measure over ℍ. Thus, for any α = (θ,h) ∈𝔸, there exists a real-valued sequence, (π_l)_l=0^∞, such that   α = (θ,h) = (π_0φ̅_0, ∑_l=1^∞π_lφ̅_l) where φ̅_0 = 1 and π_0 = θ, and, for any k ≥ 1, φ̅_k = φ_k and π_k is the “Fourier" coefficient of h with respect the basis (φ_k)_k ∈ℕ. This representation gives rise to the following norm over 𝔸, α↦√(∑_l=0^∞π^2_l) = √(θ^2 + ||h||^2_L^2(Leb)).  The aforementioned norm presents itself as a “natural" norm under which we can establish convergence rate and thus we set ||.|| as this norm; our result can be extended to norms other than this by specifying how the desired norm relates to α↦√(|θ|^2 + ||h||^2_L^2(Leb)).


We now shed light on the behavior of ϖ_L,n under our choice of ||.||. In particular, we will illustrate how this function is linked to the curvature of the criterion function α↦Q̅_J(α,𝐏). To do this, it is convenient to use local approximations, so we take, for each L=(K,L) ∈ℕ^2, 𝒜_K to be convex, α_L,0 to be such that ARC_K(α_L,0) ≡{α_L,0 + t ζζ∈𝒜_K∖{α_L,0} and t∈ [0,1]  }⊆𝒜_K,  and require that Pen to be convex and twice continuously differentiable. By the mean value theorem and the fact that α_L,0 is a minimizer — and thus satisfies that dQ̅_J(α_L,0,𝐏)/dα[·]=0 —, it follows that for any α∈𝒜̅_L,n,

    Q̅_J(α,𝐏) - Q̅_J(α_L,0,𝐏) ≥1/2inf_η∈ ARC_K(α_L,0)d^2Q̅_J(η ,𝐏)/dα^2[α - α_L,0,α - α_L,0].

By the sieve representation discussed above, the RHS in this expression can be cast as

    Q̅_J(α,𝐏) - Q̅_J(α_L,0,𝐏) ≥     (π^K+1 - π^K+1_L,0) ℐ_L (π^K+1 - π^K+1_L,0) ^T,

where π^K+1 denotes the first K+1 coefficients of the representation of α; π^K+1_L,0 is the same but for α_L,0, and ℐ_L is a (K+1) × (K+1) matrix where the (i,j)-th component is given by

    ℐ_L[i,j] ≡1/2inf_η∈ ARC_K(α_L,0)d^2Q̅_J(η ,𝐏)/dα^2[φ̅_i,φ̅_j].

This result implies that ϖ_L,n(t) ≥ t^2 e_min(ℐ_L ) (e_min(A) is the minimal eigenvalue of the matrix A). If e_min(ℐ_L)>0, then

    ||α̂_L,n - α_0 || = O_𝐏( (e_min(ℐ_L ))^-1/2√(δ_1,L,n + δ_2,L,n)  +  ||α_L,0 - α_0|| ).

The scaling factor (e_min(ℐ_L ))^-1/2 summarizes the ill-posed nature of the problem, because, even though we require e_min(ℐ_L )>0 for each L, we do not impose this restriction uniformly on L, i.e., we allow that e_min(ℐ_L ) → 0 as L →∞.[The condition e_min(ℐ_L )>0 for each L essentially ensures that the “identifiable uniqueness" condition <ref> holds for each L; this requirement is common in the ill-posed inverse literature (e.g., <cit.>).] The speed at which this occurs depends on the local curvature of Q̅_J(·, 𝐏) (at α_L,0) and the growth of 𝒜_K; see <cit.> and <cit.> for a more thorough discussion.

We next discuss the rate components (δ_1,L,n,δ_2,L,n)_L,n. As mentioned in Remark <ref> above, if sup_h ∈ℋ ||h^'||_L^∞(𝕎,μ) is finite, then ℧_L,n can be replaced by a fixed constant ℧; this fact and some algebra implies that δ_2,L,n = O(δ_n + δ^-1_n (b^2_2,J/n + ||E[g_J(Z,Π_Kα_0)]||^2_e + γ_K Pen(Π_Kα_0) ) ), where Π_Kα_0 is the projection of α_0 onto 𝒜_K (see Lemma <ref> in the Supplemental Material). By taking δ_n to balance both terms, it follows δ_2,L,n = O(√(b^2_2,J/n + {||E[g_J(Z,Π_Kα_0)]||^2_e + γ_K Pen(Π_Kα_0)})). Ignoring the term inside the curly brackets, this is the same rate than the one obtained by DIN in Lemma A.14 (note that in their setup b^2_2,J≤ J); the additional term inside the curly brackets stems from the fact that our estimation problem needs to be regularized and consequently α_L,0 is not the true parameter that nullifies the moments.

 The sequence (δ_1,L,n)_L,n is somewhat more standard within the semi-/non-parametric literature, e.g. <cit.>, and its components essentially impose restrictions on the “complexity" of ℋ. For instance, if 𝒜 = Θ×ℋ is such that the classes {ρ_2(·,·,α) α∈𝒜} and {μ h ^' h ∈ℋ} are P-Donsker, then Δ_L,n = O(1) and δ_1,L,n = O (  √(J/n)× b_2,J).

To further simplify the expression, suppose b^ρ_ρ,J≤ J^ρ/2; cf. Assumption 2 in <cit.> (see that paper for details and further references). Thus, under these conditions, the result in Theorem <ref> simplifies to

    ||α̂_L,n - α_0 || = O_𝐏( (e_min(ℐ_L ))^-1/2(  J/√(n)  + ||E[g_J(Z,Π_Kα_0)]||_e + √(γ_K Pen(Π_Kα_0)))^1/2 +  ||α_L,0 - α_0|| );
 
a rate governed by the degree of ill-posedness, the number J of moment functions, the number K of series terms and the bias arising from α_L,0.





§ ASYMPTOTIC DISTRIBUTION THEORY


 We now define the LR-type test statistic for the null
hypothesis θ_0 = ν. For any ν∈Θ and any (L,n) ≡ (J,K,n)
∈ℕ^3, let

    ℒ̂_L,n(ν) ≡ 2 {inf_{α∈𝒜
    _Kθ = ν}[ sup_λ∈Λ̂_J(α)Ŝ_J(α,λ) + γ_K Pen(α)] - inf_α∈𝒜_K[sup_λ∈Λ̂_J(α)Ŝ
    _J(α,λ) + γ_K Pen(α)] }.


The goal of this section is to show that this statistic is asymptotically
chi-square distributed with one degree of freedom. The proof of this result
relies on a local quadratic approximation of the criterion function Ŝ
_J and a representation for the parameter of interest. To derive these
results, we define the following quantities: For any α∈𝒜
_K and any (θ, ζ ) ∈𝔸, let

    G(α)[(θ,ζ)] = dE[g_J(Z,α)]/dθθ + dE[g_J(Z,α)]/dh[ζ] = [
    [ θ; 0 ]] + [
    [                     E[ℓ(W) ζ(W)]; E[𝐩_Y|WX(h(W)|W,X) ζ(W) q^J(X) ] ]]

where 0 is a J × 1 vector of zeros. By assumption <ref> these quantities are well-defined.


For any L∈ℕ^2 and for any (θ, ζ ) ∈𝔸, we define another norm over 𝔸 as,

    ||(θ, ζ )||_w^2≡ (G(α _L,0)[(θ, ζ )])^TH_L^-1(G(α
    _L,0)[(θ, ζ ) ]),

where H_L≡ H_J(α _L,0,𝐏). This norm acts as the
so-called “weak norm" in <cit.>.




 §.§ Alternative Representation for the Weighted Average Derivative



Lemma <ref> in Appendix 
<ref> shows that, over lin{𝒜_K} for any L=(J,K)∈ℕ^2, ||α ||_w=0 iff α =0. This fact implies that
linear functionals are always bounded in the space (lin{𝒜
_K},||.||_w). Since θ can be interpreted as a linear
functional of α, the following representation for θ holds:
For all L=(J,K)∈ℕ^2, there exists a v_L,n^∗∈𝒜_K such that for any α =(θ ,h)∈𝒜_K,

    θ =⟨ v_L,n^∗,α⟩ _w, and ||v_L,n^∗||_w=sup_a=(θ ,h)∈ lin{𝒜_K}, a≠ 0|θ|/||a||_w.


We note that, even though for each fixed L ∈ℕ^2, ||v^∗_L,n||_w < ∞, this quantity may diverge as L diverges
if θ is not root-n estimable. Hence, we scale v^∗_L,n by its
norm, and define u^∗_L,n≡
v^∗_L,n/||v^∗_L,n||_w. Then

    θ̂_L,n - θ_L,0/||v^∗_L,n||_w=⟨ u^∗_L,n , α̂_L,n - α_L,0⟩_w .




	The weak norm of the Riesz representer, ||v_L,n^∗||_w, is the efficiency bound of θ_0 in a model with J+1 unconditional moments functions, E_P[g_J(Z,· )], and K+1 parameters (which define α∈𝒜_K).[<cit.> established this claim for a richer model with conditional moments and infinite dimensional parameters.] For a suitably chosen sequence L≡ L(n) that increases as n does — since (q_j)_j is dense in L^2(𝕏,Leb) — one expects the sequence of unconditional moment functions to approximate the moments (<ref>)-(<ref>) defining the model. Thus, by the results in <cit.> (see also lemma 3.3, lemma 4.1 and appendix A.1 in <cit.>)  one expects (||v_L(n),n^∗||_w)_n to converge to the efficiency bound presented in Theorem <ref> provided it is finite. If the efficiency bound is infinite, the sequence (||v_L(n),n^∗||_w)_n will diverge; this fact reflects the non root-n estimability of the weighted average derivative within the original model (<ref>)-(<ref>).  




 §.§ The Asymptotic distributions of θ̂_L,n and LR statistic



For any positive real-valued sequences (η_L,n,η_w,L,n)_L,n ∈ℕ^3 (they will be restricted below) and any (L,n) ∈ℕ
^3, let

    𝒩_L,n≡{α∈𝒜̅_L,n
    ||α - α_L,0 || ≤η_L,n and ||α - α_L,0
    ||_w≤η_w,L,n}.



In what follows, for any (L,n) ∈ℕ^3, let α̂
^ν_L,n be the argument that minimizes the restricted criterion
function, i.e., α̂^ν_L,n∈min_{α∈𝒜_Kθ = ν}sup_λ∈Λ̂
_J(α)Ŝ_J(α,λ). We impose the following
assumption that restricts the convergence rate of the unrestricted and
restricted PSGEL estimators.



For any L ∈ℕ^2 and α∈{α̂_L,n , α̂^ν_L,n}, if ν=θ_0: (i) α∈ int(𝒩
_L,n); (ii) γ_Ksup_t  |t| ≤ l_n n^-1/2 |Pen(α)
- Pen(α + t u^∗_L,n)| = o_𝐏(n^-1); (iii) There exists a C< ∞ such that for any h ∈ℍ, ||h||_L^2(Leb)≤ C ||(0,h)||.


Part (i) of this assumption ensure that both estimators — the restricted
and unrestricted ones — converge to α_L,0 faster than η_L,n
and η_w,L,n in the respective norms. One can use the results in
Section <ref> to verify this assumption.[
The results in Section <ref> apply to the restricted
estimator, under the null, with minimal changes.] Part (ii) ensures that the
penalty term is negligible (see also <cit.>). Finally part (iii) states a relationship between the norm h ↦ ||(0,h)|| — used in Section <ref> — and the L^2(Leb) norm over ℍ.

In the following assumption we let 𝒢̅_L,n≡{f(.,α)=ρ _2(.,.,α
)-ρ _2(.,.,α _L,0)α∈𝒩_L,n}.


 There exists positive sequence, (Δ
_2,L,n)_L,n∈ℕ^3, such that, for any L=(J,K)∈ℕ
^2, sup_α=(θ,h) ∈𝒩_L,n𝔾_n[μ· (h^'-h_L,0^')]=O_𝐏(Δ _2,L,n) and for all 1≤ j≤ J
, sup_f∈𝒢̅_L,n𝔾_n[f·
q_j]=O_𝐏(Δ _2,L,n).


This is a high-level assumption that controls one of the terms in the
remainder of the quadratic approximation in Lemma <ref> below. As 𝒩_L,n is shrinking, one would expect Δ _2,L,n=o(1); the
exact rate, however, depends on the complexity of 𝒜̅_L,n.


 There exists a positive real-valued sequence, (Ξ_L,n)_L,n ∈ℕ^3, such that, for any L=(J,K) ∈ℕ^2, sup_α∈𝒩_L,n‖
H_J(α,P_n) - H_J(α_L,0,P_n) - { H_J(α,𝐏) -
H_J(α_L,0,𝐏) }‖_e = O_𝐏(Ξ_L,n).


This high-level assumption implies stochastic equi-continuity of the process
H_J(· ,P_n), and it is used to control one of the terms in the
remainder of the quadratic approximation in Lemma <ref> below.

The final two assumptions impose additional restrictions on (η
_L,n,η _w,L,n)_L,n∈ℕ^3, (b_ρ ,J)_ρ∈ℝ,J∈ℕ, (δ _n)_n∈ℕ and the rate
at which L=(J,K) diverges relative to n.


 (i) √(n)/||v^∗_L,n||_w
||E[g_J(Z,α_L,0)]||_e = o(1); (ii) √(n)/
||v^∗_L,n||_w |θ_L,0-θ_0|= o(1).


This assumption implies that the “bias" terms arising from working with α_L,0, as opposed to α_0, are small relative to the rate we
are using to scale the leading term of the asymptotic expansions below √(n)/||v^∗_L,n||_w. Similar assumptions have been
imposed in the literature, e.g. <cit.> and reference therein.


 (i) n δ^3_n ( ℧_L,n + b_3,J )^3 =
o(1), nδ^2_n( {θ_L,0^2 + || h^'_L,0||^2_L^∞(𝕎,μ)}√(b_4,J/n) + ℧_L,nη_L,n + Ξ_L,n) = o(1) and nδ_n( √(J/n)Δ_2,L,n + η_L,n^2 b_2,J) = o(1); (ii) √(g̅^2_L,0/n + ||E[g_J(Z,α_L,0)]||^2_e) + η_w,L,n
=o(δ_n) and ( √(J/n)Δ_2,L,n +
η_L,n^2 b_2,J) = o(δ_n); (iii) There exists a ϱ>0 such that ||h^'_L,0||^2+ϱ_L^∞(𝕎,μ)/n^2+ϱ = o(1) and b^2+ϱ_2+ϱ,J/n^2+
ϱ = o(1); (iv) A_L,0≡ E[𝐩_Y|WX(h_L,0(W)| W,X)
q^J(X)φ^K(W)^T ] has full rank K and n^-1/2
e_min(A_L,0^TA_L,0)^-1 = o(η_L,n); (v) ||h^'_L,0||^2_L^∞(𝕎, μ) b^2_4,J/√(n) = o(1).


Part (i) ensures that the remainder term for the asymptotic quadratic
representation of Ŝ_J is negligible (see Lemma <ref>). The sequence (δ _n)_n in part (ii) was discussed after Assumption <ref>. Part (iii) is used to show asymptotic normality of the leading
term in Lemma <ref> by means of a Lyapounov condition. Finally,
part (iv) ensures that the weak norm is proportional to the strong norm over
𝒜_K (even though the constant of proportionality may vanish as
L diverges) and that deviations of the form α
+l_nn^-1/2u_L,n^∗ stay in 𝒩_L,n (see Lemma <ref> in Appendix <ref>). These deviations play a crucial
role in the proof of Lemma <ref>.


	While parts (iii)-(v) are fairly easy to check and interpret, parts (i)-(ii) are not as easy. The goal of this remark is to illustrate the restrictions imposed by these parts on the different rates (δ_n,η_L_n,n,η_w,L_n,n,Δ_2,L_n,n,Ξ_L_n,n)_n where (L_n)_n is a diverging sequence in ℕ^2. To do this, we take as the point of departure the setting described in Section <ref>, which allows us to simplify some expressions. Under this setup, part (i) imposes δ_n = o( n^-1/3 J^-1/2_n). Given this, the restrictions in parts (i)-(ii) imply that
	 η_L_n,n = O(min{ n^-1/2δ^-1/2_nJ^-1/2_n, n^-1/6 J^-3/4_n})
 and η_w,L_n,n = o(n^-1/3 J^-1/2_n); we note that by imposing  a polynomial rate of decay, this condition rules out the so-called severely ill-posed case wherein the rate of for (η_L_n,n)_n decays slower than polynomial order (see <cit.> and references therein). Parts (i)-(ii) also imply that Δ_2,L,n = o( (√(nJ_n)δ^2_n)^-1  ) and Ξ_L,n = o(n^-1δ_n^-2); for the “worst case" where δ_n = ( n^-1/3 J^-1/2_n)/l_n, it follows that Δ_2,L,n = O( J_n n^-1/6  ) and Ξ_L,n = O(n^-1/3 J_n ), but the restriction can be relaxed if (δ_n)_n decays faster. Finally, parts (i)-(ii) impose restrictions on the growth of (L_n)_n: J_n = O(n^-1/6) and √(J_n||E[g_J_n(Z,α_L_n,0)]||^2_e) = o(n^-1/3). 



The following result characterizes the asymptotic distribution of the LR test statistic under the null. This characterization holds regardless of whether the parameter θ_0 is root-n estimable or not.


 Let Assumptions <ref>-<ref>
and <ref>-<ref> hold. Then, under the null θ_0 = ν,

    ℒ̂_L,n(θ_0) ⇒χ^2_1.




	See Appendix <ref>.


This result extends those in <cit.> to a non-parametric setup where the GEL is constructed
using an increasing number of moment conditions, and wherein the parameter
of interest may not be root-n estimable. Using a related estimator —an
EL-based on conditional moments a la <cit.> — <cit.> derived an analogous result but her assumptions rule out non-smooth residuals, relevant for the quantile IV model considered here.

As a by-product of the derivations used to prove Theorem <ref>, an asymptotic linear representation for the estimator of the WAD is obtained.


	 Let Assumptions <ref>-<ref>, <ref> (for α̂_L,n), <ref>, <ref> and <ref> hold. Then
	
    θ̂_L,n - θ_L,0/||v^∗_L,n||_w=
    	n^-1∑_i=1^n (G(α_L,0)[u^∗_L,n])^TH_L^-1
    	g_J(Z_i,α_L,0) + o_𝐏(n^-1/2).

	Further, under Assumption <ref>, we have
	
    √(n)(θ̂_L,n - θ_0)/||v^∗_L,n||_w⇒ N(0,1) .







The proof is the same as that Lemma <ref> in Appendix <ref> so it is omitted. This result illustrates the role of ||v^∗_L,n||_w as the appropriate scaling of our estimator. If the sequence (||v^∗_L,n||_w)_n is uniformly bounded, then this theorem implies that θ̂_L,n is √(n) asymptotically Gaussian. On the other hand, if the sequence diverges, Gaussianity is still preserve but the rate is slower and given by √(n)/||v^∗_L,n||_w.





 §.§ Heuristics


The idea is to show that, asymptotically, ℒ̂_L,n is a quadratic form of Gaussian random variables. The first step is to provide a
quadratic approximation for the criterion function Ŝ_J(α,·) as a function of λ, as shown in the following lemma.

 Let Assumptions <ref>-<ref>, <ref>, <ref> and <ref>(v) hold. Then uniformly over (α,λ) ∈𝒩_L,n×{λ∈ℝ^J+1 ||λ||_e≤δ_n},
for any L=(J,K) ∈ℕ^2

    Ŝ_J(α,λ) =    - λ^T(α) - 1/2λ^T H_Lλ
        + O_𝐏( δ^3_n ( θ + l_nγ_K^-1Γ_L,n + b_3,J )^3) 
        + O_𝐏( δ^2_n( ( θ + ||h^'_L,0||_L^∞(𝕎,μ) )^2√(b_4,J/n) + ℧_L,nη_L,n + Ξ_L,n) ) 
        + O_𝐏( δ_n( √(J/n)Δ_2,L,n +
    η_L,n^2 b_2,J) ).

where (α) ≡ n^-1∑_i=1^n g_J(Z_i,α_L,0) +
G(α_L,0)[α - α_L,0].



	See Appendix <ref>.


The “remainder" terms in the RHS (the O_𝐏(.) terms) are fairly intuitive: the order δ _n^3
-term requires boundedness of the third derivative of Ŝ_J(α
,· ); the δ _n^2-term arises because the expansion yields a
quadratic term with H_J(α ,P_n) as opposed to H_L; and the δ _n-term is the error of approximating n^-1∑_i=1^ng_J(Z_i,α ) with (α ). This last
part handles the non-smooth nature of the residuals ρ _2 by using E[g_J(Z,· )], which is a smooth function. Assumption <ref>(i) ensures that these `remainder" terms are in fact o_𝐏(n^-1). This fact, and the fact that Λ̂
_J(α) contains a δ_n-ball (see Lemma <ref>
in the Supplemental Material <ref>), imply that the expression in the Lemma provides an asymptotic characterization for
sup_λ∈Λ̂_J(α)Ŝ_J(α,λ)
in terms of ((α))^TH^-1_L((α)), which is
a quadratic form in α.

With this result at hand and Assumption <ref>, one can obtain lower and upper bounds for ℒ̂_L,n(θ_0) of the form,

    ℒ̂_L,n(θ_0)  ≥ ((α̂_L,n))^TH^-1_L((α̂_L,n)) - ((α̂_L,n) + t u^∗_L,n)^TH^-1_L((α̂_L,n + t u^∗_L,n)) + o_𝐏(1),

for appropriately chosen t ∈ℝ, and

    ℒ̂_L,n(θ_0)  ≤ ((α̂^θ_0_L,n  + t u^∗_L,n ))^TH^-1_L((α̂^θ_0_L,n  + t u^∗_L,n)) - ((α̂^θ_0_L,n)^TH^-1_L((α̂^θ_0_L,n)) + o_𝐏(1),

for appropriately chosen t ∈ℝ. Since α↦(α) is an affine function, the RHS in the previous expression is fairly easy to characterize. The following lemma formalizes these steps (its proof presents the explicitly choice for t in the previous two displays).

 Let Assumptions <ref>-<ref>, <ref>-<ref> hold. Then, under the
null ν = θ_0,

    ℒ̂_L,n(θ_0) - ( n^-1/2∑_i=1^n
    (G(α_L,0)[u^∗_L,n])^TH_L^-1 g_J(Z_i,α_L,0)
    )^2
       ≥ 2√(n)(θ_0 - θ_L,0)/||v^∗_L,n||_w( n^-1/2∑_i=1^n
    (G(α_L,0)[u^∗_L,n])^TH_L^-1 g_J(Z_i,α_L,0)
    )+ o_𝐏(1).

and

    ℒ̂_L,n(θ_0) - ( n^-1/2∑_i=1^n
    (G(α_L,0)[u^∗_L,n])^TH_L^-1 g_J(Z_i,α_L,0)
    )^2
       ≤ 2√(n)(θ_0 - θ_L,0)/||v^∗_L,n||_w( n^-1/2∑_i=1^n
    (G(α_L,0)[u^∗_L,n])^TH_L^-1 g_J(Z_i,α_L,0)
    ) 
        + ( √(n)(θ_0 - θ_L,0)/||v^∗_L,n||_w)^2 + o_𝐏(1).




	See Appendix <ref>.


This lemma shows the reason for Assumption <ref> in our analysis,
as this assumption ensures that

    ℒ̂_L,n(θ_0) = ( n^-1/2∑_i=1^n
    (G(α_L,0)[u^∗_L,n])^TH_L^-1 g_J(Z_i,α_L,0)
    )^2 + o_𝐏(1).


Under mild assumptions and Assumption <ref>, the object
inside the parenthesis is asymptotically Normal with mean 0 and variance 1.
Here we see the importance of the “optimal weight", H_L^-1. If H_L differed from E[g_J(Z,α
_L,0)g_J(Z,α _L,0)^T], then the variance of the term inside
the parenthesis will not be equal to 1, and the test statistic will only be
proportional to a χ _1^2 in the limit; see <cit.> for a
more thorough discussion and results for this case.




§ CONCLUSION




Since the seminal work by Koenker and Bassett about 40 years ago (<cit.>), quantile regression models have become ubiquitous in econometrics and statistics; see <cit.> for a recent survey. The original linear quantile regression model has been extended in several directions; in particular to the general non-parametric IV framework that allows for “flexible functional forms" and endogeneity of the regressors. This type of model, while very general, presents technical challenges arising from the non-smooth nature of the criterion function as well as its ill-posedness. One goal of this paper is to shed some light on how the nonlinear ill-posedness of the non-parametric quantile IV (NPQIV) model affects not only the speed of convergence to the conditional quantile function but also the accuracy for estimating even simple linear functionals. For this, we derive the semiparametric efficiency bound for a particular linear functional of the NPQIV — the weighted average derivative (WAD).

To estimate the parameters of interest — the NPQIV function and its WAD —  we propose a general penalized sieve GEL procedure based on the unconditional WAD moment restriction and an increasing number of
unconditional moments that are asymptotically equivalent to the conditional moment defining the NPQIV model (<ref>). We show that the QLR statistic based on the penalized sieve GEL is asymptotically chi-square distributed regardless of whether or not the information bound of the WAD is singular. This result can be used to construct confidence sets for the WAD without the need to estimate the variance of the estimator of the WAD. We hope these results extend even further the scope of quantile regression models.

The penalized sieve GEL procedure is more generally applicable to any
semi/nonparametric conditional moment restrictions and unconditional moment
restrictions, say of the following form:

    E[ρ _2(Y,W;θ _02,h_01(· ),...,h_0q(· ))|X]    =0,   a.s.-X,
    
    E[ρ _1(Y,W;θ _01,θ _02,h_01(· ),...,h_0q(·
    ))]    =0.

Here Y denotes dependent (or endogenous) variables, X denotes
conditioning (or instrumental) variables and W could be either endogenous or subset of X, θ =(θ _1^',θ _2^')^' denotes a vector of finite dimensional
parameters, and h(· )=( h_1(· ),...,h_q(· )) a
q× 1 vector of real-valued measurable functions of Y, W, X and other
unknown parameters. The residual functions ρ _j(y,w;θ ,h(· ))
, j=1,2, could be nonlinear, pointwise non-smooth with respect to (θ
,h). And some of the θ could have singular information bound. This
is a valuable alternative to classical semiparametric two-step GMM
when the second step finite dimensional parameter θ might not be root-
n estimable.

In an old unpublished  draft, <cit.> study the asymptotic properties of
another estimation procedure, optimally weighted penalized Sieve Minimum
Distance (SMD) based on orthogonalized residuals for model (<ref>)-(
<ref>). Under a set of regularity conditions, including the assumption that the WAD of a NPQIV has a positive information bound, <cit.> establish that their optimally weighted penalized SMD
estimator of the WAD is root-n asymptotically normal and semiparametrically efficient. It would
be interesting to compare this paper's estimator against theirs, and
we leave this to future work.

plainnat






	Appendix





§ PROOF OF THEOREM <REF>





To show Theorem <ref> we need some more detailed notation and
definitions. Let ℳ be the set of Borel probability measures over ℤ such
that for each P ∈ℳ: (1) there exists a (θ(P),h(P)) ∈
𝒜 for which equations (<ref>)-(<ref>) hold for P; (2) the conditions of the Theorem are satisfied for P.

Given a Q ∈ℳ, we use (θ(Q),h(Q)) to denote the
parameters that satisfy equation <ref> and θ(Q) = -
E_Q[ℓ_Q(W)h(Q)(W)]. For the true 𝐏, we simply use (θ_0,h_0) = (θ(𝐏),h(𝐏)).

Henceforth, let L^2_0(P) ≡{ g ∈ L^2(P)  E_P[g(Z)] =
0}. A curve in ℳ at P is a mapping [0,1] ∋ t ↦
P[t] ∈ℳ such that there exists a g ∈ L^2_0(P) such
that

    lim_t → 0∫( √(P[t](dz)) - √(P(dz))/t
    - 0.5 g(z) √(P(dz)))^2 = 0.

We call g the tangent of the curve; we typically use t ↦ P[t](g)
to denote a curve with tangent g. The set of tangents for all curves in ℳ at P is called the tangent set; the linear span of the set is
called the tangent space of ℳ at P∈ℳ, and we
denote it as 𝒯.



The efficiency bound of θ_0 is defined as (e.g., see <cit.>)

    ℰ(𝐏) ≡sup_g ∈𝒯|θ̇(𝐏)[g]|/||g||_L^2(𝐏)

where θ̇(𝐏) is the G-derivative of P ↦θ(P) at 𝐏, i.e.,

    g ↦θ̇(𝐏)[g] = lim_t → 0θ(𝐏[t](g)) -θ(𝐏)/t.




Henceforth, we use 𝐓_P to denote the operator 𝐓 under the probability measure P; the notation 𝐓 is reserved for 𝐓_𝐏.

(1) From the expression for ℰ(𝐏), it follows that finiteness of the efficiency bound is equivalent to boundedness of the linear functional θ̇(𝐏). In order to show this, we note that, since ℓ_P∈ Kernel(𝐓_P)^⊥, it follows that, for any P ∈ℳ,

    θ(P) = - ∫ℓ_P(w) h_id(P)(w) dw

where h_id(P) is the “identified part" of h(P) under 𝐓_P, i.e., h_id(P) is such that h(P) = h_id(P) + ν where h_id(P) ∈ Kernel(𝐓_P)^⊥ and ν∈ Kernel(𝐓_P). Thus, it is enough to characterize the G-derivative of the RHS, and we do it in the following lemma; for this, let A_𝐏: 𝒯→ L^2(𝐏_X) be defined as

    g ↦ A_𝐏[g](·) ≡∫ρ_2(y,w,h(𝐏))	g(y,w,x) 𝐏_YW | X(dy,dw|· ).



	 For any g ∈𝒯,
	
    θ̇(𝐏)[g] =    θ(g ·𝐏) - E_𝐏[ ℓ_𝐏(W) ḣ_id(𝐏)[g](W) ] 
    
    	=    ⟨μ h^'(𝐏), g ⟩_L^2(𝐏) - ⟨ℓ_𝐏 ,
    	ḣ_id(𝐏)[g] ⟩_L^2(𝐏)

	and ḣ_id(𝐏)[g] = (𝐓^∗𝐓)^+𝐓^∗ A_𝐏[g], where  (𝐓^∗𝐓)^+ be the generalized inverse of 𝐓^∗𝐓; for a definition see <cit.> Ch 2.



	See Section <ref>.



	This lemma illustrates the role that the condition ℓ_P∈ Kernel(𝐓_P)^⊥ plays in our proof. The previous lemma uses the conditional moment <ref> to characterize the G-derivative of P ↦ h(P), and this only allow us to characterize the G-derivative of h_id, since the part of h in the Kernel of 𝐓 vanishes. Under condition, ℓ_P∈ Kernel(𝐓_P)^⊥, however, this is enough for characterizing the G-derivative of P ↦θ(P). 


Therefore,

    ℰ(𝐏) = sup_g ∈𝒯| ⟨μ h^'(𝐏) , g   ⟩_L^2(𝐏) - ⟨ℓ_𝐏 ,  ḣ(𝐏)[g]  ⟩_L^2(𝐏)  |/||g||_L^2(𝐏),

and

    sup_g ∈𝒯| ⟨ℓ_𝐏 ,  ḣ(𝐏)[g]  ⟩_L^2(𝐏)  |/||g||_L^2(𝐏) = sup_g ∈𝒯| ⟨ℓ_𝐏 ,  (𝐓^∗𝐓)^+𝐓^∗ A_𝐏 [g]  ⟩_L^2(𝐏)  |/||g||_L^2(𝐏).
	

We now show that, if ℓ_𝐏∈ Range(𝐓), then ℰ(𝐏) <∞. By the triangle inequality, it suffices to show that  | ⟨μ h^'(𝐏) , g   ⟩_L^2(𝐏)   |/||g||_L^2(𝐏) < ∞ and sup_g ∈𝒯| ⟨ℓ_𝐏 ,  (𝐓^∗𝐓)^+𝐓^∗ A_𝐏 [g]  ⟩_L^2(𝐏)  |/||g||_L^2(𝐏) <∞. The former  follows because μ is uniformly bounded and h^'(𝐏) ∈ L^2(𝐏). We now show that the latter holds. As ℓ_𝐏∈ Range(𝐓^∗), then (𝐓^∗𝐓)^+[ℓ_𝐏] is well-defined. And thus

    ⟨ℓ_𝐏, (𝐓^∗𝐓)^+𝐓^∗ A_𝐏 [g]  ⟩_L^2(𝐏)  =    ∫𝐓 (𝐓^∗𝐓)^+[ℓ_𝐏](x) ρ_2(y,w,α_0) g(y,w,x) 𝐏(dy,dw,dx) 
    
    =    ⟨𝐓 (𝐓^∗𝐓)^+[ℓ_𝐏] ·ρ_2  ,g  ⟩_L^2(𝐏).


Also, || 𝐓 (𝐓^∗𝐓)^+[ℓ_𝐏] ·ρ_2||_L2(𝐏)≤ 2 || 𝐓 (𝐓^∗𝐓)^+[ℓ_𝐏] ||_L^2(𝐏) < ∞ because 𝐓(𝐓^∗𝐓)^+ is bounded and ||ℓ_𝐏||_L^2(𝐏)≾ ||ℓ_𝐏||_L^2(Leb)  < ∞ under Assumption <ref>. Therefore sup_g ∈𝒯| ⟨ℓ_𝐏 ,  (𝐓^∗𝐓)^+𝐓^∗ A_𝐏 [g]  ⟩_L^2(𝐏)  |/||g||_L^2(𝐏) < ∞ when  ℓ_𝐏∈ Range(𝐓^∗), as desired.


We now show that if ℓ_𝐏∉ Range(𝐓^∗) then ℰ(𝐏) = ∞.  To show this, observe that by the triangle inequality ℰ(𝐏) ≥| ⟨μ h^'(𝐏) , g   ⟩_L^2(𝐏) - ⟨ℓ_𝐏 ,  ḣ_id(𝐏)[g]  ⟩_L^2(𝐏)  |/||g||_L^2(𝐏)≥|⟨ℓ_𝐏 ,  ḣ_id(𝐏)[g]  ⟩_L^2(𝐏)  |/||g||_L^2(𝐏) - C for some constant C < ∞ and any g ∈𝒯. Since ℓ_𝐏∉ Range(𝐓^∗), g ↦⟨ℓ_𝐏 ,  (𝐓^∗𝐓)^+𝐓^∗ A_𝐏 [g]  ⟩_L^2(𝐏) is not bounded and thus

    sup_g ∈𝒯| ⟨ℓ_𝐏 ,  (𝐓^∗𝐓)^+𝐓^∗ A_𝐏 [g]  ⟩_L^2(𝐏)  |/||g||_L^2(𝐏) = ∞

so the result follows by choosing the g that achieves this supremum (or a subsequence which yields a value arbitrarly close to it).



(2)  To prove part (2), we assume that ℓ_𝐏∈ Range(𝐓^∗). Let ḣ^∗_id(𝐏) : L^2(𝐏) →𝒯^∗ be the adjoint of ḣ_id(𝐏) and is given by

    g ↦ḣ^∗_id(𝐏)[g](y,w,x) = 𝐓 (𝐓^∗𝐓)^+[g](x) ρ_2(y,w,α(𝐏))

for any (y,w,x) ∈ℤ.

It is well known (<cit.> p. 363) that the efficiency bound (when it exists) is the variance of the projection of the influence function onto the tangent space, i.e.,

    ℰ(𝐏) =      ||Proj_𝒯[ μ h^'(𝐏) - ḣ^∗_id(𝐏)[ℓ_𝐏]  ]||_L^2(𝐏)
    
    =     ||Proj_𝒯[ μ h^'(𝐏) - 𝐓 (𝐓^∗𝐓)^+[ℓ_𝐏] ·ρ_2]||_L^2(𝐏)

where Proj_𝒯 : L^2(𝐏)  →𝒯̅ is the projection operator onto the closure of the tangent space. This operator is characterized in the following lemma:


	 For any f ∈ L^2(𝐏),
	
    (y,w,x) ↦ Proj_𝒯[f](y,w,x) =     f - Proj_L^2(𝐏_X)[f](x) 
        - ρ_2(y,w,α(𝐏))/γ(1-γ)· (I -
    	𝐓(𝐓^∗𝐓)^+𝐓^∗)Proj_L^2(𝐏_X)[ρ_2· f](x).




	See Section <ref>.


Let 𝐌≡ (I - 𝐓(𝐓^∗𝐓)^+𝐓^∗)/(τ(1-τ)) and Γ(x) = E[ρ_1(Y,W,α(𝐏))ρ_2(Y,W,α(𝐏))| X=x]/(τ(1-τ)), then we can write

    Proj_𝒯 [μ h^'(𝐏)](y,w,x) =    (θ(𝐏) - μ(w) h^'(𝐏)(w)) - E[θ(𝐏) - μ(W)h^'(𝐏)(W) | X=x] 
        - ρ_2(y,w,α(𝐏))𝐌[E[μ(W) h^'(𝐏)(W) ρ_2(Y,W,α(𝐏)) |·]](x) 
    
    =    ρ_1(y,w,α(𝐏)) - E[ρ_1(Y,W,α(𝐏))| X=x] 
        -  ρ_2(y,w,α(𝐏))𝐌[E[ρ_2(Y,W,α(𝐏))ρ_1(Y,W,α(𝐏)) |· ] ](x) 
    
    =    ρ_1(y,w,α(𝐏)) - E[ρ_1(Y,W,α(𝐏))| X=x] -  ρ_2(y,w,α(𝐏)) Γ(x) 
        - ρ_2(y,w,α(𝐏)) 𝐓(𝐓^∗𝐓)^+𝐓^∗ [Γ](x)

where the second line follows from the fact that E[ρ_2(Y,W,α(𝐏))ρ_1(Y,W,α(𝐏)) | X] equals
 E[μ(W)h^'(𝐏)(W)ρ_1(Y,W,α(𝐏)) | X].

In addition,

    Proj_𝒯 [𝐓(𝐓^∗𝐓)^+[ℓ_𝐏] ·ρ_2](y,w,x) 
    
    =   𝐓(𝐓^∗𝐓)^+[ℓ_𝐏](x) ρ_2(y,w,α(𝐏)) 
        - ρ_2(y,w,α(𝐏)) 𝐌[ E[𝐓(𝐓^∗𝐓)^+[ℓ_𝐏](·)  ρ_2^2|· ]  ](x)
    
    =   𝐓(𝐓^∗𝐓)^+[ℓ_𝐏](x) ρ_2(y,w,α(𝐏)) - ρ_2(y,w,α(𝐏)) 𝐌𝐓(𝐓^∗𝐓)^+[ℓ_𝐏](x) τ (1-τ) 
    	
    =    𝐓(𝐓^∗𝐓)^+[ℓ_𝐏](x) ρ_2(y,w,α(𝐏)) - ρ_2(y,w,α(𝐏)) 𝐓(𝐓^∗𝐓)^+[ℓ_𝐏](x)  
        + ρ_2(y,w,α(𝐏)) 𝐓(𝐓^∗𝐓)^+𝐓^∗𝐓(𝐓^∗𝐓)^+[ℓ_𝐏](x) 
    
    =    𝐓(𝐓^∗𝐓)^+[ℓ_𝐏](x) ρ_2(y,w,α(𝐏))
	
where the last line follows from properties of the generalized inverse, namely, (𝐓^∗𝐓)^+𝐓^∗𝐓(𝐓^∗𝐓)^+  =(𝐓^∗𝐓)^+; see <cit.>  Proposition 2.3.

Therefore, for all (y,w,x) ∈ℤ,

    Proj_𝒯[ μ h^'(𝐏) - ḣ_id^∗(𝐏)[ℓ_P]  ](y,w,x) =    ϵ(y,w,x) - E[ϵ(Y,W,x) | X=x] 
        + ρ_2(y,w,α(𝐏))𝐓(𝐓^∗𝐓)^+[ ℓ_𝐏 - 𝐓^∗Γ](x).
	
where (y,w,x) ↦ϵ(y,w,x) ≡ρ_1(y,w,α(𝐏)) - ρ_2(y,w,h(𝐏)) Γ(x). Note that E[ϵ(Y,W,x) | X=x] = E[ρ_1(Y,W,α(𝐏)) | X = x].

The proof concludes by showing that ϵ(Y,W,X) - E[ϵ(Y,W,X) | X] is orthogonal to

    ρ_2(Y,W,α(𝐏))𝐓(𝐓^∗𝐓)^+[ ℓ_𝐏 - 𝐓^∗Γ](X).

This follows because, conditional on X, ϵ(Y,W,X) - E[ϵ(Y,W,X) | X] is orthogonal to ρ_2(Y,W,α(𝐏)) by construction. □






§ APPENDIX FOR SECTION <REF>



This Appendix contains the proofs of all the Lemmas presented in Section <ref>.


	Note that for any α∈𝒜, Λ̂_J(α) ∋ 0 wpa1, hence sup_λ∈Λ̂_J(α)Ŝ_J(α,λ) ≥ 0 wpa1; in particular this applies to α = α̂_L,n. Therefore, for any α∈𝒜_K,
	
    γ_K Pen(α̂_L,n) ≤sup_λ∈Λ̂
    		_J(α̂_L,n)Ŝ_J(α̂_L,n,λ)  + γ_K Pen(α̂_L,n)≤sup_λ∈Λ̂
    		_J(α)Ŝ_J(α,λ) + γ_K Pen(α),

	wpa1, where the second inequality is due to the definition of the minimizer α̂_L,n∈𝒜_K.










§ PROOF FOR THEOREM <REF>



Consider an L that satisfies the Assumptions of the theorem. By the triangle inequality, it suffices to show that for any ϵ>0, there exists constants M_1,M>0 and N ∈ℕ such that

    𝐏( ||α̂_L,n - α_L,0 || ≥ M ϖ_L,n^-1(M_1δ̅_L,n)   ) ≤ϵ

for all n ≥ N, where δ̅_L,n≡δ_1,L,n + δ_2,L,n. Henceforth, let A_n(M_1,M) ≡{  ||α̂_L,n - α_L,0 || ≥ M ϖ_L,n^-1(M_1δ̅_L,n)    }.

From the proof of Lemma <ref>, γ_KPen(α̂_L,n) = O_𝐏 ( Γ_L,n ) = o_𝐏( l_n(δ_1,L,n + δ_2,L,n)). This fact,  Lemma <ref> and the fact that Q_J(α_L,0,P_n) ≥ 0, imply that there exists an M_0 and an N_0 such that

    𝐏( Q_J(α̂_L,n,P_n) - Q_J(α_L,0,P_n) ≥ M_0δ_2,L,n) ≤ϵ

for all n ≥ N_0. This result and Lemma <ref> in turn imply that

    𝐏( Q_J(α̂_L,n,𝐏) - Q_J(α_L,0,𝐏) ≥ 2 M_0{δ_1,L,n + δ_2,L,n}) ≤ϵ

for all n ≥ N_0. Let B_n≡{  Q_J(α̂_L,n,𝐏) - Q_J(α_L,0,𝐏) ≤ 2 M_0{δ_1,L,n + δ_2,L,n}}.

The previous display implies that for any n ≥ N_0, 𝐏(A_n(M_1,M)) ≤𝐏(A_n(M_1,M) ∩ B_n) + ϵ. By definition of ϖ_L,n, for any history of data (Z_i)_i in A_n(M_1,M) ∩ B_n it follows that

    ϖ_L,n( M ϖ_L,n^-1(M_1δ̅_L,n) ) ≤ 2 M_0{δ_1,L,n + δ_2,L,n}  M ϖ_L,n^-1(M_1δ̅_L,n) ≤ϖ_L,n^-1( 2 M_0{δ_1,L,n + δ_2,L,n})

where the equivalence follows from the fact that t ↦ϖ_L,n(t) is non-decreasing (see Lemma <ref>).
By setting M_1 = 2M_0 and M>1 this display implies that 𝐏(A_n(2M_0,M) ∩ B_n) = 0 thus proving the desired result. □




§ APPENDIX FOR SECTION <REF>



This Appendix contains the proofs of all the Lemmas presented in Section <ref>.

The following lemmas are used to prove the Lemmas in Section <ref>; the proofs are relegated to the Supplementary Material <ref>.


 Let Assumption <ref> hold. Suppose (α,P) ∈𝔸×𝒫(ℤ) and δ>0 are such that: There exists finite C>0 such that (1) sup_λ∈ B(δ)sup_z ∈ supp(P) s^''(λ^T g_J(z,α)) ≤ - √(C), (2) e_min(H_J(α,P)) ≥√(C), (3) 2C^-1
||E_P[g_J(Z,α)]||_e < δ, and (4) the hypothesis of Lemma 
<ref> are satisfied for ϵ>0.
Then, with probability higher than 1-ϵ,



  * max_λ∈Λ_J(α,P) S_J(α,λ,P)
= max_λ∈ B(δ) S_J(α,λ,P) = {λ_J(α,P) }; d
S_J(α,λ_J(α,P),P)/d λ = 0.


  * sup_λ∈Λ(α,P)S(α,λ,P) ≤ 2C^-1
||E_P[g_J(Z,α)]||^2_e.


  * ||λ_J(α,P)||_e≤ 2 C^-1
||E_P[g_J(Z,α)]||_e.




	 Let Assumption <ref> hold. Then, for any
	(non-random) α∈𝒜 and any J ∈ℕ,
	
    ||E_P_n[g_J(Z,α)]||_e = O_𝐏( √(θ + ||μ h^'||^2_L^2(𝐏) + b^2_2,J/n +
    		||E_𝐏[g_J(Z,α)]||^2_e)).

	(the constant implicit in the O_𝐏 does not depend on J).



	 Suppose Assumptions <ref> and <ref>
	hold. Then for any α in a ||· ||- neighborhood of α_0,
	
	
		
  * ||H_J(α,P_n) - H_J(α,𝐏)||_e = O_𝐏({θ^2 + ||h^'||^2_L^∞(𝕎,μ)}√(
			b^4_4,J/n)).[
			For matrices, ||.||_e is the operator norm induced by the Euclidean norm.
		]
		
		
  * If {θ^2 + ||h^'||^2_L^∞(𝕎,μ)}√(b^4_4,J/n) = o(1), there exists a C < ∞ such that wpa1, 
    1/C ≤ e_min(H_J(α,P)) ≤ e_max(H_J(α,P)) ≤ C

		for P ∈{P_n, 𝐏}.
	




 Suppose Assumption <ref> and <ref>
hold. For any (L=(J,K),n), and any positive real-valued sequence (δ_n)_n satisfying Assumption <ref>(i)(iii)(iv)(v) and δ_n = o(1), it follows that

    ||n^-1∑_i=1^n g_J(Z_i,α̂_L,n) ||_e≾
    C_L,n{δ_n + δ^-1_n{
    ||E_P_n[g_J(Z,α_L,0)]||^2_e + γ_K Pen(α_L,0)
    }}.




	Suppose Assumption <ref>(iii). Then for P ∈{ P_n , 𝐏},
	
    sup_α∈𝒜̅_L,n ||E_P[g_J(Z,α)]||_e≾θ + l_nγ^-1_KΓ_L,n + E_P[||q^J(X)||_e]

	wpa1.




	By Lemma <ref> and the fact that Π_K α_0∈𝒜_K, we have
	
    Pen(α̂_L,n) ≤γ^-1_Ksup_λ∈Λ_J(Π_K α_0,P_n) S(Π_K α_0,λ,P_n) + Pen(Π_K α_0).

	
	By Lemma <ref> applied to (Π_K α_0,P_n) (by Lemma <ref> in the Supplementary Material <ref>, the conditions of the Lemma <ref> hold wpa1) it follows that
	
    Pen(α̂_L,n) ≾γ^-1_K ||E_P_n[g_J(Z,Π_K α_0)]||^2_e + Pen(Π_K α_0)   wpa1.

	By Lemma <ref>, wpa1, ||E_P_n[g_J(Z,Π_K α_0)]||^2_e≤ l_n( g̅_L,0/n + ||E_P[g_J(Z,Π_K α_0)]||^2_e), and thus the result follows.




	Throughout, fix L=(J,K). We show that the “argmin" is non-empty by invoking the Weierstrass Theorem (see <cit.>). For this, note that α↦ Q_J(α,𝐏) is a continuous transformation of

    α↦ E[g_J(Z,α)]^T = (θ + E[ℓ(W)h(W)],E[E[(F_Y|WX(h(W)| W,X)-τ)|X]q^J(X)^T] )

(here F_Y|WX is the conditional cdf of Y given W,X associated to 𝐏 ). Since ℓ∈ L^2(Leb) and 𝐩_W is bounded (see Assumption <ref>), then h ↦ E[ℓ(W)h(W)] is continuous with respect to ||.||_L^2(Leb). Also under Assumption <ref>,
	
    |E[F_Y|WX(h_1(W)| W,x) - F_Y|WX(h_2(W)| W,x)|x]| ≾   ∫ |h_1(w) - h_2(w)|  𝐩_W|X(w|x)dw 
    ≤   inf_x 1/𝐩_X(x) ∫ |h_1(w) - h_2(w)|    𝐩_W(w) dw  
    ≾    ||h_1 - h_2||_L^2(Leb),

	so h ↦ E[E[(F_Y|WX(h(W)| W,X)-τ)|X]q^J(X)] is also continuous with respect to  ||.||_L^2(Leb). Under assumption <ref>, α↦ Q_J(α,𝐏) + γ_K Pen(α) is lower semi-compact, and 𝒜_K=Θ×ℋ_K is a finite dimensional and closed set. So all the assumptions of the Weierstrass Theorem hold.



	In the proof we simply use L instead of L_n. We first present some intermediate results.
	

	
	From the definition of Proj_J it follows that
	
    E_𝐏[g_J(Z,α)]  = E_𝐏[M_J(X) ρ(Y,W,α)] = (E_𝐏[ρ_1(Y,W,α)],E_𝐏[ρ_2(Y,W,α)q^J(X)^T])^T.

	This in turn implies that
	
    ( E_𝐏[g_J(Z,α)] )^T M_J(X) =     (E_𝐏[ρ_1(Y,W,α)],E_𝐏[ρ_2(Y,W,α)q^J(X)^T])M_J(X) 
    
    	=     (E_𝐏[ρ_1(Y,W,α)],Proj_J[m_2(·,α)](X)).

	Since E_𝐏[M_J(X) M_J(X)^T] = I_J+1 under Assumption <ref> the previous result implies that
	
    (E_𝐏[ρ_1(Y,W,α)])^2 + ||Proj_J[m_2(·,α)]||^2_L^2(𝐏) =     E_𝐏[ (E_𝐏[g_J(Z,α)])^T M_J(X) M_J(X)^T  (E_𝐏[g_J(Z,α)])  ] 
    
    	=     ||E_𝐏[g_J(Z,α)]||^2_e.

	
	This observation and the proof of Lemma <ref> (applied to α_0) implies that
	
    Q_J(α,𝐏) ≥    c^-1  E_𝐏[g_J(Z,α)^T] (E_𝐏[M_J(X)M_J(X)^T])^-1  E_𝐏[g_J(Z,α)] 
    
    	=     c^-1 ||E_𝐏[g_J(Z,α)]||^2_e
    
    	=     c^-1{ (E_𝐏[ρ_1(Y,W,α)])^2 + ||Proj_J[m_2(·,α)]||^2_L^2(𝐏)}

	and
	
    Q_J(α,𝐏) ≤    c { (E_𝐏[ρ_1(Y,W,α)])^2 + ||Proj_J[m_2(·,α)]||^2_L^2(𝐏)}
	
	for some c>1.
	
	By the conditions in the lemma, uniformly over α∈𝒜̅_L,n,
	
    lim inf_J →∞   Q_J(α,𝐏) ≥ c^-1{ (E_𝐏[ρ_1(Y,W,α)])^2 + ||m_2(· ,α)||^2_L^2(𝐏)}

	and
	
    lim sup_J →∞   Q_J(α,𝐏) ≤ c { (E_𝐏[ρ_1(Y,W,α)])^2 + ||m_2(· ,α)||^2_L^2(𝐏)}.
	
	
	We now turn to the proof of the claim in the Lemma. We show this by contradiction. That is, suppose that there exists a c>0 such that ||α_L,0 - Π_Kα_0 || ≥ c i.o. Therefore, for any L for which the expression holds, it follows that
	
    inf_α∈𝒜̅_L,n || α - Π_Kα_0 || ≥ c Q_J(α,𝐏) ≤ Q_J(Π_Kα_0,𝐏).

	
	Under Assumption <ref>, {α∈𝒜̅_L,n || α - Π_Kα_0 || ≥ c } is compact under ||.|| and by the proof of Lemma <ref>, α↦ Q_J(α,𝐏) is continuous under ||.||. Thus, there exists a α_L∈𝒜̅_L,n such that (i) || α_L - Π_Kα_0 || ≥ c and (ii) Q_J(α_L,𝐏) ≤ Q_J(Π_Kα_0,𝐏).
	
	
	By the first part of the proof and our conditions, Q_J(Π_Kα_0,𝐏) = o(1) so this implies that Q_J(α_L,𝐏) = o(1). By the first part of the proof, this result implies that
	
    (E_𝐏[ρ_1(Y,W,α_L)])^2 + ||m_2(·,α_L)||^2_L^2(𝐏) = o(1).

	
	Since, under our conditions, ℧_L,n≤ K < ∞, it follows that the sequence (α_L)_L belongs to {α Pen(α) ≤ K  } which is compact under ||.|| by Assumption <ref>. Thus, there exists a convergent subsequence with limit α^∗. On the one hand, || α_L - Π_Kα_0 || ≥ c, it follows that α^∗α_0. But, on the other hand, continuity of α↦ 	(E_𝐏[ρ_1(Y,W,α)])^2 + ||m_2(·,α)||^2_L^2(𝐏) and the previous display imply that
	
    (E_𝐏[ρ_1(Y,W,α^∗)])^2 + ||m_2(·,α^∗)||^2_L^2(𝐏) =0

	but this contradicts the identification condition in Assumption <ref>.




	The proof of Lemma <ref> implies that α↦ Q_J(α,𝐏) is continuous and that t ↦{α∈𝒜̅_L,n ||α - α_L,0 || ≥ t } is a compact-valued correspondence which is also continuous. Thus by the Theorem of the Maximum, ϖ_L,n is continuous. This also implies that the “inf" is achieved; this and the definition of α_L,0 and assumption <ref> imply that  ϖ_L,n(t) = 0 iff t = 0. The fact that it is non-decreasing is trivial to show.





	Observe that
	
    Q_J(α̂_L,n,P_n) ≤   1/e_min(H_J(α_0,𝐏)) ||E_P_n[g_J(Z,α̂_L,n)]||^2_e .

	By Lemma <ref>(2), it follows that
	
    Q_J(α̂_L,n,P_n) ≤    C^-1 ||E_P_n[g_J(Z,α̂_L,n)]||^2_e.
	
By Lemma <ref>
	
    Q_J(α̂_L,n,P_n) ≾  C_L,n{δ_n + δ^-1_n{
    ||E_P_n[g_J(Z,α_L,0)]||^2_e + γ_K Pen(α_L,0)
    }}.

By Lemma <ref> and definition of δ_2,L,n,
	
    Q_J(α̂_L,n,P_n) =  O_𝐏(δ_2,L,n).





	Note that
	
    |Q_J(α,P_n) - Q_J(α,P)| ≤    ||H_J(α_0,𝐏)^-1/2 (E_P_n[g_J(Z,α)] - E_𝐏[g_J(Z,α)])||_e
       × ||H_J(α_0,𝐏)^-1/2(E_P_n[g_J(Z,α)] + E_𝐏[g_J(Z,α)])||_e

	This bound and Lemma <ref>(2) (applied to α_0) imply that it suffices to show that
	
    sup_α∈𝒜̅_L,n Term_2,J(α,P_n) = O_𝐏( θ + l_nγ_K^-1Γ_L,n + b_2,J),
    
    			and    sup_α∈𝒜̅_L,n Term_1,J(α,P_n) = O_𝐏( √(J/n)Δ_L,n)

	where
	
    Term_1,J(α,P_n) ≡ ||E_P_n[g_J(Z,α)] - E_𝐏[g_J(Z,α)]||_e
    
    		Term_2,J(α,P_n) ≡ ||E_P_n[g_J(Z,α)] + E_𝐏[g_J(Z,α)]||_e.

	
	By Lemma <ref>,
	
    sup_α∈𝒜̅_L,n ||E_P_n[g_J(Z,α)]||_e≾θ + l_nγ_K^-1Γ_L,n + E_P_n[||q^J(X)||_e]

	wpa1; and similarly for sup_α∈𝒜̅_L,n ||E_𝐏[g_J(Z,α)]||_e. Hence,
	
    sup_α∈𝒜̅_L,n Term_2,J(α,P_n) = O_𝐏( θ + l_nγ_K^-1Γ_L,n + b_2,J).

	

    Regarding Term_1,J(α,P_n), by definition of g_J and simple algebra,
    
    sup_α∈𝒜̅_L,n Term_1,J(α,P_n) ≤    n^-1/2sup_(θ,h) ∈𝒜̅_L,n |𝔾_n[μ· h']|
        + sup_α∈𝒜̅_L,n || n^-1∑_i=1^nρ_2(Y_i,W_i,α)q^J(X_i) - E_𝐏[ρ_2(Y,W,α)q^J(X)] ||_e.

 By Assumption <ref>, the first term in the RHS is O_𝐏(n^-1/2Δ_L,n); the second term in the RHS can be bounded above by √(J/n)max_1 ≤ j ≤ Jsup_g ∈𝒢_K|𝔾_n[g · q_j ]|, which by Assumption <ref> is O_𝐏( √(J/n)Δ_L,n).
	
	




§ PROOF OF THEOREM <REF>



By Assumption <ref>(i) and Lemma <ref> in Appendix <ref>,  n^-1/2∑_i=1^n (G(α_L,0)[u^∗_L,n])^TH_L^-1 g_J(Z_i,α_L,0)  ⇒ N(0,1). This fact, Lemma <ref> and Assumption <ref>(ii), imply the desired result. □




§ APPENDIX FOR SECTION <REF>




For any L=(J,K) ∈ℕ^2, let M_L∈ℝ^(1+J)× (1+K) be defined as

    M_L =    [
    [                                        1                          E[ℓ(W)φ^K(W)^T];                                        0 E[𝐩_Y|WX(h_L,0(W) | W,X) q^J(X)φ^K(W)^T] ]]

where 0 is a J × 1 vector of zeros. The following eight lemmas are proved in the Section <ref> in the Supplemental Material.


 Let Assumptions <ref>-<ref> and <ref>(iv)(v) hold. Then, there exists a c>0 such that for any L = (J,K) ∈ℕ^2 and any α =
(θ,π^Tφ^K) some π∈ℝ^K (i.e., α∈ lin{𝒜_K}), it follows that

    || α||_w≥ c ×√(e_min(M_L^T M_L) )
    ||(θ,π)||_e

where e_min(M_L^T M_L)>0.



 Let Assumptions <ref>-<ref>, <ref> and <ref>(v) hold. Then

    sup_α∈𝒩_L,n ||H_J(α,P_n) -
    H_J(α_L,0,𝐏)||_e = O_𝐏(Ξ_1,L,n)

with

    Ξ_1,L,n = O_𝐏( ( θ + ||h^'_L,0||_L^∞(𝕎,μ) )^2√(b_4,J/n) + ℧_L,nη_L,n + Ξ_L,n)




 For any L=(J,K) ∈ℕ^2,

    sup_α∈𝒩_L,n‖ n^-1∑_i=1^n
    g_J(Z_i,α) - g_J(Z_i,α_L,0) - E[g_J(Z,α) -
    g_J(Z,α_L,0)] ‖_e
       ≾√(J/n)( sup_(θ,h) ∈𝒩_L,n𝔾_n[μ· h^'] + max_1 ≤ j ≤ Jsup_h ∈𝒢̅_L,n𝔾_n[g · q_j] ).




 Let Assumptions <ref>-<ref>, <ref>(ii)(iii) and <ref>(iv)(v) hold. If α∈ int(𝒩_L,n), then α + t
u^∗_L,n∈𝒩_L,n for all |t| ≤ l_nn^-1/2.



 Let Assumptions <ref>-<ref>,
 <ref>, <ref> and <ref> hold. Then
all the conditions of Lemma <ref> hold for (α̂
_L,n,P_n) and (α̂^ν_L,n,P_n) wpa1.



 Let Assumptions <ref>-<ref> and <ref>(v) hold. For
any L=(J,K) ∈ℕ^2, any γ>0 and any α∈𝒜̅_L,n such that ||α - α_L,0||_w≤γ,
it follows that

    ||H^-1_L(α)||_e = O_𝐏( √(g̅
    _L,0^2/n + ||E[g_J(Z,α_L,0)]||^2_e) + γ).




 Let Assumptions <ref>-<ref>,
and <ref> hold. Then

    ⟨ u^∗_L,n , α̂_L,n - α_L,0⟩_w =
    n^-1∑_i=1^n (G(α_L,0)[u^∗_L,n])^TH_L^-1
    g_J(Z_i,α_L,0) + o_𝐏(n^-1/2).




 Let Assumptions <ref>, <ref> and <ref> hold. Then,

    n^-1/2∑_i=1^n (G(α_L,0)[u^∗_L,n])^TH_L^-1{g_J(Z_i,α_L,0) - E[g_J(Z,α_L,0)] }⇒ N(0,1).




  		By the calculations in the proof of Lemma <ref>, for all (α,λ) ∈𝒩_L,n× B(δ_n),
  		
    Ŝ_J(α,λ) ≥ - λ^T E_P_n[g_J(Z,α)] - 1/2λ^T H_J(α,P_n) λ + O( δ^3 E_P_n[||g_J(Z,α)||^3_e] );

  		the reverse inequality can also be shown in similar fashion.
  		
  		By Lemma <ref>, sup_α∈𝒩_L,n ||H_J(α,P_n) - H_J(α_L,0,𝐏)||_e = O_𝐏(Ξ_1,L,n). Hence
   		
    Ŝ_J(α,λ) = - λ^T E_P_n[g_J(Z,α)] - 1/2λ^T H_J(α_L,0,𝐏) λ + O( δ^3 E_P_n[||g_J(Z,α)||^3_e] + δ^2Ξ_1,L,n ).

   		
   	By Lemma <ref> and Assumption <ref>, it follows that
    sup_α∈𝒩_L,n‖ n^-1∑_i=1^n g_J(Z_i,α) - g_J(Z_i,α_L,0)  - E[g_J(Z,α) - g_J(Z,α_L,0)] ‖_e = O_𝐏( √(J/n)Δ_2,L,n).

   	Hence
   	
    Ŝ_J(α,λ) =     - λ^T{ E_P_n[ g_J(Z,α_L,0)]  + E[g_J(Z,α) - g_J(Z,α_L,0)]   } - 1/2λ^T H_J(α_L,0,𝐏) λ
        + O ( δ^3 E_P_n[||g_J(Z,α)||^3_e] + δ^2Ξ_1,L,n + δ√(J/n)Δ_2,L,n).


  By the mean value theorem,
  
    E[g_J(Z,α) - g_J(Z,α_L,0)] =     G(α_L,0)[α - α_L,0] 
        +  ∫_0^1{ G(α_L,0 + t(α - α_L,0))[α - α_L,0] - G(α_L,0)[α - α_L,0] } dt.

  Note that, for any α and α_1,
  
    G(α)[α_1] - G(α_L,0)[α_1] = [
      	[                                                               0; E[{𝐩_Y|WX(h(W)|W,X) -  𝐩_Y|WX(h_L,0(W) | W,X) } h_1(W) q^J(X) ] ]].

  By Assumption <ref>, |d𝐩_Y|WX(y|w,x)/dy | ≤ C some finite C and thus, for each j,
  
    |E[{𝐩_Y|WX(h(W)|W,X) -  𝐩_Y|WX(h_L,0(W) | W,X) } h_1(W) q_j(X) ]|
    ≤    C E[|h(W) - h_L,0(W)| × |h_1(W)| |q_j(X)| ] 
    ≤    C sup_x E[|h(W) - h_L,0(W)| × |h_1(W)| |X=x] × E[|q_j(X)|].

  Therefore,
  
    ||	G(α)[α_1] - G(α_L,0)[α_1] ||_e≾sup_x E[|h(W) - h_L,0(W)| × |h_1(W)| |X=x] × E[||q^J(X)||_e].


  Applying these observations to the last term in the RHS of expression (<ref>), it follows that
  
    || E[g_J(Z,α) - g_J(Z,α_L,0)] - G(α_L,0)[α - α_L,0] ||_e =     O (sup_x E[|h(W) - h_L,0(W)|^2 |X=x] × E[||q^J(X)||_e]) 
    
      	= O     (||h - h_L,0||^2_L^2(Leb) E[||q^J(X)||_e])

  where the last line follows by Assumption <ref>. Therefore,
   	
    Ŝ_J(α,λ) =     - λ^T{ E_P_n[ g_J(Z,α_L,0)]  + G(α_L,0)[α - α_L,0]    } - 1/2λ^T H_J(α_L,0,𝐏) λ
        + O ( δ^3 E_P_n[||g_J(Z,α)||^3_e] + δ^2Ξ_1,L,n + δ{√(J/n)Δ_2,L,n + ||h - h_L,0||^2_L^2(Leb) b_2,J}).

	Since α̂_L,n∈𝒜̅_L,n wpa1 (Lemma <ref>), it follows by the proof of Lemma <ref> that ||g_J(Z,α̂_L,n)||^3_e≤ (θ + l_nγ_K^-1Γ_L,n + ||q^J(X)||_e )^3. So by the Markov inequality
   	
    E_P_n[||g_J(Z,α̂_L,n)||^3_e] = O_𝐏( (θ + l_nγ_K^-1Γ_L,n + b_3,J )^3).

   	
	


		
		Step 1. We show that, for α∈{α̂_L,n , α̂^ν_L,n},
		
    sup_λ∈Λ̂_J(α)Ŝ_J(α,λ) = 1/2(α)^TH_L^-1(α) + o_𝐏(n^-1).

		
		We first note that, for any α∈{α̂_L,n , α̂^ν_L,n}, by Lemma <ref> and Lemmas <ref> { B(δ_n) ⊆Λ̂_J(α)  } wpa1. Also, by Assumption <ref>, under the null, {α̂_L,n , α̂^ν_L,n}⊆𝒩_n. Hence, under Assumption <ref>, Lemma <ref> implies that
		
    sup_λ∈Λ̂_J(α)Ŝ_J(α,λ) ≤   sup_λ∈ B(δ_n) - λ^T(α) - 0.5 λ^T H_L^-1λ + o_𝐏(n^-1) 
    ≤   1/2(α)^TH_L^-1(α) + o_𝐏(n^-1)

		where the first inequality is valid, because sup_λ∈Λ̂_J(α)Ŝ_J(α,λ) = sup_λ∈ B(δ_n)Ŝ_J(α,λ) by Lemmas <ref> and Lemma <ref>; the last inequality follows because the RHS is obtained by maximizing over the whole ℝ^J+1 not only Λ̂_J(α).
		
		By Lemma <ref> and Lemma <ref>,
		
    sup_λ∈Λ̂_J(α)Ŝ_J(α,λ) ≥ - λ^T(α) - 0.5 λ^T H_L^-1λ + o_𝐏(n^-1),

		for all λ∈ B(δ_n). By Lemma <ref> and Assumption <ref>, the maximizer of the RHS, λ^∗, is such that ||λ^∗||_e = O_𝐏( √(g̅_L,0^2/n + ||E[g_J(Z,α_L,0)]||^2_e) + η_w,L,n). Hence, by Assumption <ref>,  λ^∗∈ B(δ_n). Therefore,
		
    sup_λ∈Λ̂_J(α)Ŝ_J(α,λ) ≥1/2(α)^TH_L^-1(α) + o_𝐏(n^-1).

		
		Step 2. We now show that
		
    ℒ̂_L,n(ν)  ≥     o_𝐏(n^-1) +  ( n^-1∑_i=1^n (G(α_L,0)[u^∗_L,n])^TH_L^-1 g_J(Z_i,α_L,0) )^2
        + 2(ν - θ_L,0)/||v^∗_L,n||_w( n^-1∑_i=1^n (G(α_L,0)[u^∗_L,n])^TH_L^-1 g_J(Z_i,α_L,0) ).
 		
		
		
		Using the results in step 1,
		
    ℒ̂_L,n(ν) ≥   {(α̂^ν_L,n)^TH_L^-1(α̂^ν_L,n) - (α)^TH_L^-1(α)  } +γ_K{Pen(α̂^ν_L,n) - Pen(α) }
         +  o_𝐏(n^-1).

		For any α∈𝒜_K. In particular α = α̂^ν_L,n - tu^∗_L,n with t = (G(α_L,0)[u^∗_L,n])^TH_L^-1 E_P_n[g_J(Z,α_L,0)]. By Lemma <ref> and Assumption <ref>, |t| = O_𝐏(n^-1/2), so by Lemma <ref>, under Assumption <ref>, this choice of α belongs to 𝒩_L,n. Moreover, by Assumption <ref>, γ_K{Pen(α̂^ν_L,n) - Pen(α) } = o_𝐏(n^-1). Hence, after some simple calculations,
		
    ℒ̂_L,n(ν)  ≥     o_𝐏(n^-1) -  { t^2 ||u^∗_L,n||^2_w - 2 t (G(α_L,0)[u^∗_L,n])^TH_L^-1(α̂^ν_L,n)  }.

		Note that (α̂^ν_L,n) = E_P_n[g_J(Z,α_L,0)] + G(α_L,0)[α̂^ν_L,n - α_L,0],	so 
    2  (G(α_L,0)[u^∗_L,n])^TH_L^-1(α̂^ν_L,n)  =      2  (G(α_L,0)[u^∗_L,n])^TH_L^-1 E_P_n[g_J(Z,α_L,0)] 
        +  2  (G(α_L,0)[u^∗_L,n])^TH_L^-1(G(α_L,0)[α̂^ν_L,n - α_L,0]) 
    
    		=      2  (G(α_L,0)[u^∗_L,n])^TH_L^-1 E_P_n[g_J(Z,α_L,0)] 
        +  2 (ν - θ_L,0)/||v^∗_L,n||_w,

		where the last line follows because (G(α_L,0)[u^∗_L,n])^TH_L^-1(G(α_L,0)[α̂^ν_L,n - α_L,0]) = ⟨ u^∗_L,n , α̂^ν_L,n - α_L,0] ⟩_w = (θ̂_L,0 - θ_L,0)/||v^∗_L,n||_w.
		
		This observation and the fact that ||u^∗_L,n||_w = 1, imply
		
    ℒ̂_L,n(ν)  ≥  o_𝐏(n^-1)  - { t^2 - 2t   (G(α_L,0)[u^∗_L,n])^TH_L^-1 E_P_n[g_J(Z,α_L,0)] + 2t (θ_L,0-ν)/||v^∗_L,n||_w}.

		By our choice of t,  it follows that
		
    ℒ̂_L,n(ν)  ≥     o_𝐏(n^-1) +  ( n^-1∑_i=1^n (G(α_L,0)[u^∗_L,n])^TH_L^-1 g_J(Z_i,α_L,0) )^2
        + 2(ν - θ_L,0)/||v^∗_L,n||_w( n^-1∑_i=1^n (G(α_L,0)[u^∗_L,n])^TH_L^-1 g_J(Z_i,α_L,0) ).

		
		Step 3. We now show that
		
    ℒ̂_L,n(ν)  ≤   (θ_L,0 - θ_0/||v^∗_L,n||_w - ( n^-1∑_i=1^n (G(α_L,0)[u^∗_L,n])^TH_L^-1 g_J(Z_i,α_L,0) ) )^2  +  o_𝐏(n^-1).

		
		To do this we proceed as in Step 2. 	Using the results in step 1,
		
    ℒ̂_L,n(ν) ≤   {(α)^TH_L^-1(α) - (α̂_L,n)^TH_L^-1(α̂_L,n)  }
         +  γ_K{Pen(α) - Pen(α̂_L,n) } + o_𝐏(n^-1),
	
		for any α∈𝒜_K such that θ = ν. Let α = α̂_L,n - t u^∗_L,n, where t = ⟨ u^∗_L,n , α̂_L,n - α_L,0⟩_w + θ_L,0 - θ_0/||v^∗_L,n||_w = θ̂_L,n - θ_0/||v^∗_L,n||_w. Note that
		
    ⟨ v^∗_L,n,  α⟩_w =  θ̂_L,n - t ||v^∗_L,n||_w = θ̂_L,n - (θ̂_L,n - θ_0) = θ_0,

		and under the null, θ_0 = ν. Thus α satisfies with the restriction ν = θ. Moreover, by Lemma <ref> and Assumption <ref>, it follows that t = O_𝐏(n^-1/2). Thus, by Lemma <ref> — under Assumption <ref> and the null —, α∈𝒩_L,n wpa1.
		
		This observation, Assumption <ref>, and analogous calculations to those in Step 1 imply
		
    ℒ̂_L,n(ν) ≤{  t^2 - 2  t (G(α_L,0)[u^∗_L,n])^TH_L^-1(α̂_L,n) }   +  o_𝐏(n^-1),

		and 
    (G(α_L,0)[u^∗_L,n])^TH_L^-1(α̂_L,n) =    ⟨ u^∗_L,n , α̂_L,n - α_L,0⟩_w + n^-1∑_i=1^n (G(α_L,0)[u^∗_L,n])^TH_L^-1 g_J(Z_i,α_L,0)
    ≡   θ̂_L,n - θ_0/||v^∗_L,n||_w - θ_L,0 - θ_0/||v^∗_L,n||_w+ F_n
    
    		=     t - θ_L,0 - θ_0/||v^∗_L,n||_w+ F_n.

		
		Therefore,
		
    ℒ̂_L,n(ν) ≤   { t^2 - 2  t (t - θ_L,0 - θ_0/||v^∗_L,n||_w+ F_n) }   +  o_𝐏(n^-1) 
    
    		=    { - t^2 + 2  t (θ_L,0 - θ_0/||v^∗_L,n||_w - F_n) }   +  o_𝐏(n^-1)
    
    		=    { - t^2 + 2  t (θ_L,0 - θ_0/||v^∗_L,n||_w - F_n) - (θ_L,0 - θ_0/||v^∗_L,n||_w - F_n)^2} + (θ_L,0 - θ_0/||v^∗_L,n||_w - F_n)^2  +  o_𝐏(n^-1)
    ≤   (θ_L,0 - θ_0/||v^∗_L,n||_w - F_n)^2  +  o_𝐏(n^-1).

		
		
		Step 4. The desired result follows from imposing the null hypothesis in Step 2.
	











	Supplemental Material





§ SUPPLEMENTAL MATERIAL FOR APPENDIX <REF>



The next lemma presents some properties of the
tangent space.


	 The tangent space of ℳ at P∈ℳ
	is included in the class of all g ∈ L^2_0(P) such that
	
    ∫ρ_2(y,w,h(P)) g(y,w,·) P_YW | X(dy,dw|· ) ∈
    	Range(𝐓_P).



 By our definition of ℳ any curve, t ↦ P[t], must satisfy 
    ∫ρ_2(y,w,h(P[t]))p[t]_YW|X (y,w| X) dydw = 0.

	Since the curve is in ℳ it has a pdf, which we denote as p[t]; also we write ρ_2 only as a function of h and not α to stress the fact that θ is not present. Since θ(P) ∈ int(Θ) the equation θ(P[t]) = - E_P[t][ℓ_P[t](W) h(P[t])(W)] does not impose — locally — any restrictions; so it can be ignored for the computation of the tangent space.
	
	Since p_X > 0 for any P in the model, the previous display implies that
	
    ∫ρ_2(y,w,h(P[t]))p_YW|X[t](y,w| X) dydw = ∫ (F_Y|WX[t](h(P[t])(w) | w,x) - τ) p_WX[t](w, x) dw,

	where F_Y|WX[t] is the conditional cdf of Y, given w,x associated to P[t]. By our assumption <ref>, taking derivative with respect to t implies
	
    ∫ρ_2(y,w,h(P)) g(y,w,·) p(y,w | x)dydw = 𝐓_P[ ḣ(P)[g]  ](x)

	for all x ∈𝕏. Hence, for any g in the tangent space, it must hold that the LHS function belongs to Range (𝐓_P).



	
	Part 1. By laws of differentiation, if the derivative exists, it has to satisfy
	
    d ∫ (ℓ_𝐏[t](g)(w) h_id(𝐏[t](g))(w)) dw /dt|_t=0 = d ∫ℓ_𝐏(w) h_id(𝐏[t](g))(w) dw /dt|_t=0 + d ∫ℓ_𝐏[t](g)(w) h_id(𝐏)(w) dw /dt|_t=0.

	
	Since for all P ∈ℳ, 
    ℓ_P(w) = μ^'(w)p_W(w) + μ(w) p^'_W(w) = μ^'(w)∫ p(y,w,x)dydx + μ(w) d∫ p(y,w,x) dydx/dw
,
	it follows that
	
    d ∫ℓ_𝐏[t](g)(w) h_id(𝐏)(w) dw /dt|_t=0 =    d ∫μ'(w) h_id(𝐏)(w) 𝐩[t](g)(w)  dw /dt|_t=0
        + d ∫μ(w) h_id(𝐏)(w) 𝐩^'[t](g)(w)  dw /dt|_t=0.

	
	Since 𝐩[t](g) belongs to the model, 𝐩^'[t](g) is continuous.  We now show that
    lim_w →∓∞μ(w) h(𝐏)(w) 𝐩[t](g)(w) dw = 0.

	Since 𝐩[t](g) is a density, it is enough to show that ||μ h(𝐏)||_L^∞(𝐏_W) < ∞, but this follows by the fact that μ and 𝐩_W are uniformly bounded and by definition of ℍ. 	By this result and integration by parts,  it follows that
	
    d ∫ℓ_𝐏[t](g)(w) h_id(𝐏)(w) dw /dt|_t=0 =    d ∫μ^'(w) h_id(𝐏)(w) 𝐩[t](g)(w)  dw /dt|_t=0
        - d ∫{μ^'(w) h_id(𝐏)(w) + μ(w) h^'_id(𝐏)(w) }𝐩[t](g)(w)  dw /dt|_t=0
    
    	=     -d ∫μ(w) h^'_id(𝐏)(w) 𝐩[t](g)(w)  dw /dt|_t=0.

	
	By definition of derivative of the curve t ↦𝐩[t](g) and the fact that ||μ h^'_id(𝐏) ||_L^2(Leb)≾  || h^'_id(𝐏) ||_L^2(Leb) < ∞, it follows by Dominated Convergence Theorem that 	
	
    d ∫ℓ_𝐏[t](g)(w) h_id(𝐏)(w) dw /dt|_t=0 =    d ∫μ(w) h^'_id(𝐏)(w) 𝐩[t](g)(w)  dw /dt|_t=0
    
    	=     -∫μ(w) h^'_id(𝐏)(w) ∫ g(y,w,x) 𝐩(y,w,x) dy dx  dw.

	
	
	Since ℓ_𝐏∈ L^2(𝐏_W), g ↦∫ℓ_𝐏(w) g(w) dw is a bounded linear functional; this, part 2 below and the Dominated Convergence Theorem, imply
	
    d ∫ℓ_𝐏(w) h(𝐏[t](g))(w) dw /dt|_t=0 = ∫ℓ_𝐏(w) ḣ_id(𝐏)[g](w) dw.

	
	Hence,
	
    θ̇(𝐏)[g] =  - ∫ℓ_𝐏(w) ḣ_id(𝐏)[g](w) dw  +  ∫μ(w) h^'_id(𝐏)(w)  ∫ g(y,w,x) 𝐩(y,w,x) dydx dw.

	
	Note that the last term in the RHS is, by definition, θ(g ·𝐏). Hence
	
    θ̇(𝐏)[g] = - ∫ℓ_𝐏(w) ḣ_id(𝐏)[g](w) dw + θ(g ·𝐏).

	
	Part 2. This part of the proof follows from applying the implicit function theorem to (h,P) ↦ G(h,P) ≡∫ (F_Y|WX(h(w) | w,x) - τ) p_W|X(w | x) dw, where F_Y|WX is the cdf associated to the probability measure P. The derivatives of the mapping G at (h(𝐏),𝐏), with direction (ζ,Q) where ζ∈ℍ and Q is a measure over ℤ, are given by
	
	
    dG(h(𝐏),𝐏)/dP[Q](x) = ∫ρ_2(y,w,h(𝐏)) Q_YW|X(dy,dw | x) dydw ∀ x,

	and
	
    dG(h(𝐏),𝐏)/dh[ζ](x) = 𝐓[ζ](x) = 𝐓[ζ_id](x) = 𝐓_id[ζ_id](x), ∀ x,

	where the second equality follows because any  ζ∈ℍ can be decomposed as the sum of ζ_id∈ Kernel(𝐓)^⊥ and an element in Kernel(𝐓); in the third expression 𝐓_id is defined as the restriction of 𝐓 to Kernel(𝐓)^⊥. We observe that 𝐓_id is invertible (in the sense that the inverse is a linear functional); its extension to the whole of L^2(Leb) is the generalized inverse and we denote it as 𝐓^+ (see <cit.> p. 33).
		
	Taking Q such that Q_YW|X = g ·𝐏_YW|X for g ∈𝒯, it follows that dG(h(𝐏),𝐏)/dP[Q] = A_𝐏[g]. Moreover, by Lemma <ref>, any g ∈𝒯 is such that  A_𝐏[g] ∈ Range(𝐓_id) = Range(𝐓). Thus, it follows that by the Implicit Function Theorem that
	
    ḣ_id(𝐏)[g] = (𝐓)^+ [A_𝐏[g]] =  (𝐓^∗𝐓)^+𝐓^∗ [A_𝐏[g]]

	where the second equality follows from the results by <cit.> p. 35.
	


	The calculations are analogous to those in <cit.> so they are omitted.




§ SUPPLEMENTAL MATERIAL FOR APPENDIX <REF>



Let Π_Kα_0 be the projection of α_0 onto 𝒜_K=Θ×ℋ_K; since ℋ_K is closed and convex (Assumption <ref>
), it is well-defined.


	 Suppose Assumption <ref> holds. Then
	
    ||E[g_J(Z,α_L,0)]||^2_e + γ_K Pen(α_L,0) ≾
    	||E[g_J(Z,Π_Kα_0)]||^2_e + γ_K
    	Pen(Π_Kα_0),

	and ||μ h^'_L,0||_L^∞(𝕎)≤γ_K^-1
	||E[g_J(Z,Π_Kα_0)]||^2_e + Pen(Π_Kα_0).




	By definition of the α_L,0 and the fact that Π_Kα_0∈𝒜_K, Q̅_J(α_L,0,𝐏) ≤Q̅_J(Π_Kα_0,𝐏). By Lemma <ref>, C^-1 I ≤ H^-1_J(α_0,𝐏) ≤ C I some C ≥ 1, thus
	
    C^-1 ||E[g_J(Z,α_L,0)]||^2_e + γ_K Pen(α_L,0) ≤ C ||E[g_J(Z,Π_Kα_0)]||^2_e + γ_K Pen(Π_Kα_0)

	and the first result follows.
	
	The second result follows from the first result and Assumption <ref>(iii).
		




 §.§ Supplementary Lemmas


We present and prove a sequence of lemmas used in the proofs of the Lemmas of Section <ref>.


	 For any α∈𝔸, any A ≡ A_y× A_w× A_x⊆ℤ Borel, and any L=(J,K) ∈ℕ^2,
	
    sup_z ∈ A ||g_J(z,α)||_e≤g̅_L(α,A) ≡θ + ||h^'||_L^∞(A_w,μ) + sup_x ∈ A_x
    	||q^J(x)||_e.





	The result follows from the fact that,
	
    sup_z ∈ A|| g_J(z,α)||_e≤   sup_z ∈ A|θ - μ(w)h'(w)| +  sup_z ∈ A ||q^J(x)||_e
    ≤   θ +  sup_w |μ(w) h'(w)| + sup_x ||q^J(x)||_e.



Throughout, for any δ>0, let B(δ) ≡{λ∈
	ℝ^J+1 ||λ||_e≤δ}.


	 Suppose Assumption <ref> holds. Then,
	there exists a η>0 such that for any ϵ>0, any (L=(J,K),n)
	and any δ > 0 for which
	
    δθ < η/3, and  b^ϱ_ϱ,J n
    	(3δ)^ϱ/(η)^ϱ < ϵ/2, and δ×℧_L,n < η/3, some ϱ>0,

	it follows that 𝐏( B(δ) ⊆Λ(α,P_n)
	 ∀α∈𝒜̅_L,n) ≥ 1-ϵ.




	Since 𝒮∋ 0, there exists a η>0 for which: For any (Z_i)_i=1^∞,  if
	
    sup_α∈𝒜̅_L,nsup_z ∈ supp(P_n)max_λ∈ B(δ) |λ^T g_J(z,α)| ≤η,

	then λ∈Λ(α,P_n), for all α∈𝒜̅_L,n.
	
	By the Cauchy-Schwarz inequality and Lemma <ref>,
	
    sup_α∈𝒜̅_ksup_z ∈ supp(P_n) ||λ||_e|| g_J(z,α)||_e≤δ×sup_α∈𝒜̅_L,ng̅_L(α,supp(P_n)).
Thus, it suffices to show that P( δ×sup_α∈𝒜̅_L,ng̅_L(α,supp(P_n)) ≤η ) ≥ 1-ϵ (for the ϵ in the statement of the lemma).
	
	To show this we bound each term of sup_α∈𝒜̅_L,ng̅_L(α,supp(P_n)). Note that δθ < η/3; also note that by the Markov inequality
	
    P( δmax_1 ≤ i ≤ n ||q^J(X_i)||_e≥η/3  ) ≤ n E_P[||q^J(X)||^ϱ_e]/(η)^ϱ× (3δ)^ϱ≤ b^ϱ_ϱ,J n (3δ)^ϱ/(η)^ϱ,

	it suffices that the RHS is less than ϵ/2, which it does by assumption.
	
	Finally, by definition of 𝒜̅_L,n and Assumption <ref>, || h^' ||_L^∞(𝕎,μ)≤℧_L,n, hence
	
    sup_α∈𝒜̅_L,n	|| h'||_L^∞(supp(P_n),μ)≤ l_nγ_K^-1Γ_L,n = ℧_L,n, a.s.

	Since by assumption δ×  l_nγ_K^-1 ( Γ_L,n ) < η/3,
	




	 Suppose Assumptions <ref> and <ref> hold. For any (L=(J,K),n) and any positive real-valued sequence, (δ_n)_n, such that
	
	
		
  * b^4_4,J/n = o(1).
		
		
  * δ_n = o(1) and b_ϱ,J^ϱ n
		δ_n^ϱ = o(1) and δ_n℧_L,n = o(1) for some ϱ > 0.
		
		
  * √(g̅^2_L,0/n +
			||E[g_J(Z,α_L,0)]||^2_e) = o(δ_n).
	
	
	it follows that the conditions of Lemma <ref> hold for (α_L,0,P_n) wpa1.




	
	To show condition (1) in Lemma <ref> it suffices to show that
	
    sup_λ∈ B(δ_n)sup_z ∈ℤλ^Tg_J(z,α_L,0)
 is bounded. But this follows from the proof of Lemma <ref>, the fact that α_L,0∈𝒜̅_L,n and the fact that all the conditions in that Lemma are satisfied.
	
	
	Condition (2) in Lemma <ref> holds wpa1 for some C by Lemma <ref> applied to α = α_L,0 and 𝐏, provided that  b^4_4,J/n = o(1) (note that |θ| + || h'_L,0||_L^∞(𝕎,μ) is bounded).
	
	By Lemma <ref>, for condition (3) in Lemma <ref> it is enough that
	
    √(θ + ||μ h'_L,0||_L^2(𝐏) + b_2,J^2/n + ||E[g_J(Z,α_L,0)]||^2_e)  = √(g̅^2_L,0/n  + ||E[g_J(Z,α_L,0)]||^2_e) = o(δ_n)
  which holds by assumption.
	
	Finally, δ_n = o(1) and b_ρ,J^ρ n δ_n^ρ = o(1) and δ_n l_nγ_K^-1Γ_L,n = δ_n℧_L,n = o(1), so condition (4) in Lemma <ref> holds for any ϵ>0.




	We note that the requirement of δ_n× l_nγ_K^-1Γ_L,n = δ_n×℧_L,n = o(1) (condition 2), does not
	contradict condition 3. Even though Γ_L,n contains √(g̅^2_L,0/n + ||E[g_J(Z,α_L,0)]||^2_e) in condition
	2, there is the l_nγ_K^-1 term which is large, at least for
	sufficiently large K. 



	 Suppose Assumption <ref> holds. Then, for
	any (L=(J,K),n) such that
	
    (1 + θ + ℧_L,n)^2√( 1 + b_4,J^4/n) =
    	o(1),

	it follows that
	
    𝐏( e_max( H_J(α̂_L,n,P_n) ) ≤
    	C_L,n) → 1

	where C_L,n≡ (1 + θ + ℧_L,n)^2.




	Step 1. We show that 	
	
    n^-1∑_i=1^n g_J(Z_i,α)g_J(Z_i,α)^T≤  n^-1∑_i=1^n ||ρ(Y_i,W_i,α)||^2_e M_J(X_i)M_J(X_i)^T,

	where
	
    M_J(x) =    [
    	[      1      0;      0 q^J(x) ]]

	0 is a J × 1 vector of zeros.	
	
	To do this, note that for each z ∈ℤ,
	
    g_J(z,α)g_J(z,α)^T =    [
    	[               (ρ_1(y,w,α))^2 ρ_1(y,w,α)ρ_2(y,w,α)q^J(x)^T;   ρ_1(y,w,α)ρ_2(y,w,α)q^J(x) (ρ_2(y,w,α))^2q^J(x)q^J(x)^T ]] 
    ≡    M_J(x) ρ(y,w,α)ρ(y,w,α)^T M_J(x)^T.

	
	It follows that ρ(y,w,α)ρ(y,w,α)^T≤ ||ρ(y,w,α)||^2_e I.
	Therefore
	
    n^-1∑_i=1^n g_J(Z_i,α)g_J(Z_i,α)^T≤    n^-1∑_i=1^n ||ρ(Y_i,W_i,α)||^2_e M_J(X_i)M_J(X_i)^T.

	
	Step 2. Let (y,w) ↦ R(y,w) ≡sup_α∈𝒜̅_L,n ||ρ(y,w,α)||^2_e. We now show that
	
    ‖ n^-1∑_i=1^n R(Y_i,W_i) M_J(X_i)M_J(X_i)^T - E[  R(Y,W) M_J(X)M_J(X)^T] ‖_e
    
    	=      O_𝐏( √( E[ E[(R(Y,W))^2|X] (1+||q^J(x)||^2_e)^2] /n)).

	
	By the Markov inequality it is enough to bound
	
    n^-1/2√(E [   ‖  R(Y,W) M_J(X)M_J(X)^T‖^2_e] )≤ n^-1/2√( E[ (R(Y,W))^2 (trace{ M_J(X)M_J(X)^T})^2]   ).

	Since
	
    M_J(x)M_J(x)^T =    [
    	[              1            0^T;              0 q^J(x)q^J(x)^T ]],

	the previous display implies that
	
    n^-1/2√(E [   ‖  R(Y,W) M_J(X)M_J(X)^T‖^2_e] )≤ n^-1/2√( E[ E[(R(Y,W))^2|X] (1+||q^J(x)||^2_e)^2]   ).
	
	
	Step 3. By Steps 1-2 and Assumption <ref>
	
    n^-1∑_i=1^n g_J(Z_i,α)g_J(Z_i,α)^T≤   sup_x E[R(Y,W) | X = x] × I 
        +  O_𝐏( √(sup_x  E[(R(Y,W))^2|X=x]   1 + E[||q^J(X)||^4_e] /n)).

	
	Also,
	
    R(y,w)  ≤sup_α∈𝒜̅_L,n |θ - μ(w)h'(w)|^2 + 1  	≤  1 + θ^2 + (l_nγ_K^-1Γ_L,n)^2

	where the last inequality follows from Assumption <ref> and definition of 𝒜̅_L,n. So, to the extent that
	
    (1 + θ + l_nγ_K^-1Γ_L,n)^4 1 + b_4,J^4/n = o(1),

	it follows that
    sup_α∈𝒜̅_L,n e_max( n^-1∑_i=1^n g_J(Z_i,α)g_J(Z_i,α)^T) ≤ (1 + θ + l_nγ_K^-1Γ_L,n)^2

	wpa1. Since α̂_L,n∈𝒜̅_L,n wpa1, this implies the result.




 §.§ Proofs of the Lemmas stated in Appendix <ref>


We now present the proofs of the Lemmas stated in Appendix <ref>.


	By the mean value theorem,
	
    S_J(α,λ,P)  =     s'(0) λ^T E_P[g_J(Z,α)] + 1/2λ^T{∫_0^1 E_P[ s”( t  λ^T g_J(Z,α)  ) g_J(Z,α) g_J(Z,α)^T] dt  }λ
    
    	=     - λ^T E_P[g_J(Z,α)] + 1/2λ^T{∫_0^1 E_P[ s”( t  λ^T g_J(Z,α)  ) g_J(Z,α) g_J(Z,α)^T] dt  }λ.

	Note that
	
    ∫_0^1 E_P[ s”( t  λ^T g_J(Z,α)  ) g_J(Z,α) g_J(Z,α)^T] dt ≤∫_0^1sup_z s”( t  λ^T g_J(z,α)  ) dt E_P[ g_J(Z,α) g_J(Z,α)^T].

	
	Under our assumptions, sup_z s”( t  λ^T g_J(z,α)  ) ≤ - √(C) for all t ∈ [0,1]. This and the fact that
    e_min(  E_P[ g_J(Z,α) g_J(Z,α)^T] ) ≥√(C)
imply that
	
    S_J(α,λ,P) ≤ - λ^T E_P[g_J(Z,α)] - C/2 ||λ||^2_e≤ ||λ||_e ||E_P[g_J(Z,α)]||_e - C/2 ||λ||^2_e.

	Hence, since S_J(α,0,P) = 0, by evaluating the previous expression in λ_1∈max_λ∈ B(δ) S_J(α,λ,P) (it exists by continuity of S_J(α,.,P) and compactness of the set), we obtain
	
    ||λ_1||_e≤ 2 C^-1 ||E_P[g_J(Z,α)]||_e.

	
	Since 2 C^-1 ||E_P[g_J(Z,α)]||_e  < δ by assumption, λ_1 is in the interior of B(δ). Since, by Lemma <ref> — all the hypothesis of the lemma are satisfied — λ_1∈Λ_J(α,P) with probability higher than 1-ϵ, it holds that λ_1∈max_λ∈Λ_J(α,P) S_J(α,λ,P) with probability higher than 1-ϵ.
	
	Finally, this implies that
	
    sup_λ∈Λ_J(α,P) S_J(α,λ,P) = S_J(α,λ_1,P) ≤ 2 C^-1 ||E_P[g_J(Z,α)]||^2_e,

	with probability higher than 1-ϵ.



	It suffices to show that
	
    ||n^-1∑_i=1^n g_J(Z_i,α) - E[g_J(Z,α)]||^2_e = O_𝐏( θ + ||μ h'||^2_L^2(𝐏) + E[||q^J(X)||^2_e] /n).

	By the Markov inequality, we can study
	
    E[||n^-1∑_i=1^n g_J(Z_i,α) - E[g_J(Z,α)]||^2_e] ≤  n^-1 E[||g_J(Z,α)||^2_e],

	and it follows by Lemma <ref>, E[||g_J(Z,α)||^2_e] ≤θ + E[|μ(W) h'(W)|^2] + E[||q^J(X)||^2_e].
	



	Part 1. By analogous calculations to those  in the proof of Lemma <ref> and Lemma A.6 in DIN, ||H_J(α,P_n) - H_J(α,𝐏)||_e = O_𝐏(√(sup_x E[||ρ(Y,W,α)||^4_e|X=x] b^4_4,J/n)). Note that,
	
    E[||ρ(Y,W,α)||^4_e|X=x] ≤  E[ (1 + |θ - μ(W)h'(W)|^2)^2   | X=x ] ≾   θ^4 +  || μ h'||^4_L^4(𝐏_W)
    ≾   θ^4 +  || h'||^4_L^∞(𝕎,μ)

	where the last inequality follows from Assumption <ref> and some trivial algebra.
	
	Part 2. From Part 1 and the fact that A ↦ e_min(A) is Lipschitz, is suffices to show the result for P=𝐏. Note that
	
    E_𝐏[g_J(Z,α)g_J(Z,α)^T] = E[M_J(X) E[ρ(Y,W,α)ρ(Y,W,α)^T|X]M_J(X)^T]

	where M_J is defined in the proof of Lemma <ref>.
	
	
	We now argue that C^-1 I ≤ E[ρ(Y,W,α)ρ(Y,W,α)^T|X] ≤ C I for any α in some neighborhood of α_0. First note that, under Assumption <ref>, E[(ρ_1(Y,W,α))^2| X] = ∫ (θ_0 - μ(w)h'(w))^2𝐩_W|X(w|x)dw ≥ C Var_𝐏(μ(W) h'(W)) > 0 some C>0. Hence the trace of E[ρ(Y,W,α)ρ(Y,W,α)^T|X] is positive uniformly in x. Since ρ_1(.,α) and ρ_2(.,α) are not linearly dependent, by the Cauchy-Schwarz (strict) inequality the determinant is also positive; thus C^-1 I ≤ E[ρ(Y,W,α)ρ(Y,W,α)^T|X] uniformly on X. The reverse inequality is obtained in a similar fashion.
	
	
	Hence under Assumption <ref> the desired result follows.   	







	By the mean value theorem, for any α∈𝒜, P and λ in a neighborhood of 0,
	
    S_J(α,λ,P)  =     s'(0) λ^T E_P[g_J(Z,α)] + 1/2λ^T{∫_0^1 E_P[ s”( t  λ^T g_J(Z,α)  ) g_J(Z,α) g_J(Z,α)^T] dt  }λ
    
    	=     - λ^T E_P[g_J(Z,α)] + 1/2λ^T{∫_0^1 E_P[ s”( t  λ^T g_J(Z,α)  ) g_J(Z,α) g_J(Z,α)^T] dt  }λ.

	
	By our assumptions over s (in particular, that it has Hölder continuous second derivative), it follows 	
	
    |s”(λ^Tg_J(z,α)) - s”(0)| ≤ C × |λ^Tg_J(z,α)|

	for some C>0.	Hence
	
    | λ^T{∫_0^1 E_P[ s”( t  λ^T g_J(Z,α)  ) g_J(Z,α) g_J(Z,α)^T] dt  }λ - λ^T{ E_P[ s”(0) g_J(Z,α) g_J(Z,α)^T]  }λ| 
    ≤   |  ∫_0^1 E_P[ (s”( t  λ^T g_J(Z,α)  - s”(0)  ) λ^Tg_J(Z,α) g_J(Z,α)^Tλ] dt |
    ≤    C × E_P[ |λ^Tg_J(Z,α)| λ^Tg_J(Z,α) g_J(Z,α)^Tλ] 
    ≤    C ||λ||_e^3  E_P[ ||g_J(Z,α)||^3_e].

	
	Therefore, for α = α̂_L,n and P=P_n,
	
    S_J(α,λ,P)  ≥     - λ^T E_P[g_J(Z,α)] - 1/2λ^T H_J(α,P) λ - C ||λ||_e^3× E_P[ ||g_J(Z,α)||^3_e] 
    ≥    - λ^T E_P[g_J(Z,α)] - C × C_L,n/2 ||λ||^2_e - C ||λ||_e^3× E_P[ ||g_J(Z,α)||^3_e]
	
	where the second inequality follows from Lemma <ref>.
	
	Let λ̅≡ - δ_n E_P_n[g_J(Z,α̂_L,n)]/||E_P_n[g_J(Z,α̂_L,n)]||_e with δ_n satisfying the assumptions in the statement of the Lemma. Since ||λ̅||_e = δ_n and α̂_L,n∈𝒜̅_L,n wpa1 (Lemma <ref>), by Lemma <ref>, this choice of δ_n ensures that  λ̅∈Λ_J(α̂_L,n,P_n) wpa1.
	
	The previous expression for S_J(α,λ,P) yields
	
    S_J(α̂_L,n,λ̅,P_n)  ≥δ_n ||E_P_n[g_J(Z,α̂_L,n)]||_e - C × C_L,n/2δ^2_n - C δ^3_n× E_P_n[ ||g_J(Z,α̂_L,n)||^3_e].

By definition of α̂_L,n, 
    S_J(α̂_L,n,λ̅,P_n) + γ_K Pen(α̂_L,n) ≤sup_λ∈Λ_J(α_L,0,P_n) S_J(α_L,0,λ,P_n) + γ_K Pen(α_L,0)   wpa1.

	
	By Lemma <ref>, all the conditions in Lemma <ref> hold for (α_L,0,P_n) wpa1. In this case, the previous inequality implies that S_J(α̂_L,n,λ̅,P_n) + γ_K Pen(α̂_L,n) ≾ ||E_P_n[g_J(Z,α_L,0)]||^2_e + γ_K Pen(α_L,0) wpa1. Therefore
	
    δ_n ||E_P_n[g_J(Z,α̂_L,n)]||_e - C × C_L,n/2δ^2_n - C δ^3_n×  E_P_n[ ||g_J(Z,α̂_L,n)||^3_e]
       ≾ ||E_P_n[g_J(Z,α_L,0)]||^2_e + γ_K Pen(α_L,0),  wpa1.

Since α̂_L,n∈𝒜̅_L,n wpa1 (Lemma <ref>), it follows by the proof of Lemma <ref> that ||g_J(Z,α̂_L,n)||^3_e≤ (θ + l_nγ_K^-1Γ_L,n + ||q^J(X)||_e )^3. We now show that δ^3_n{ (θ + l_nγ_K^-1Γ_L,n)^3 + E_P_n[||q^J(X)||_e^3]) } = o_𝐏(δ^2_nC_L,n), so that the term C δ^3_n×  E_P_n[ ||g_J(Z,α̂_L,n)||^3_e] can be ignored.
	
	To show this, note by definition of C_L,n, it suffices to show δ^3_n{ (C_L,n)^2 + E_P_n[||q^J(X)||_e^3]) } = o_𝐏(δ^2_nC_L,n). 	By the conditions in the lemma δ_nC_L,n = o(1) and taking C_L,n≥ 1 (if this is not the case, the solution is trivial), and thus δ^3_n  (C_L,n)^2 = o(δ^2_nC_L,n). By the Markov inequality, it remains to show that δ_n b_3,J^3 = o_𝐏(C_L,n). As we take C_L,n≥ 1 and δ_n b_3,J^3 = o(1), the desired equality holds. Hence
	
    ||E_P_n[g_J(Z,α̂_L,n)]||_e≾ C_L,nδ_n +  δ^-1_n{ ||E_P_n[g_J(Z,α_L,0)]||^2_e + γ_K Pen(α_L,0) },  wpa1.

	


	By definition of g_J and simple algebra, it follows that
	
    ||E_P[g_J(Z,α)]||_e≤θ + sup_w  |h^'(w) μ (w)|  +  E_P[||q^J(X)||_e].

	By the fact that α∈𝒜̅_L,n, Pen(α) ≤℧_L,n = l_nγ^-1_KΓ_L,n. So, by Assumption <ref>(iii) it follows that sup_w  |h^'(w) μ (w)|  ≤℧_L,n = l_nγ^-1_KΓ_L,n, uniformly on h.




§ SUPPLEMENTAL MATERIAL FOR APPENDIX <REF>




	Recall that || · ||^2_w≡ (G(α_L,0)[·])^T H_L^-1 (G(α_L,0)[·]). Since α∈ lin{𝒜_K}, we can cast α = (θ,φ^K(·)^Tπ) some π∈ℝ^K. Hence
	
	
    G(α_L,0)[α] =    [
    	[                        θ - E[ℓ(W)φ^K(W)^Tπ]; E[𝐩_Y|WX(h_L,0(W) | W,X) q^J(X)φ^K(W)^Tπ  ] ]]
    
    	=     M_L× (θ,π)^T.

	Hence
	
    || α ||^2_w = (θ,π) M_L^T H_L^-1 M_L  (θ,π)^T.

	Thus, it is sufficient to show that M_L^T H_L^-1 M_L is positive definite. By Lemma <ref> — since {θ^2_L,0 + ||μ h'_L,0||^2_L^∞(𝕎)}√(b^4_4,J/n) = o(1) — it follows that e_min(H_L^-1) ≥ c^2> 0, Hence,
	
    || α ||^2_w≥ c^2× e_min(M_L^T M_L) ||θ,π||^2_e.

	Note that the eigenvalues of M_L^T M_L are those of
	
    (E[𝐩_Y|WX(h_L,0(W) | W,X) q^J(X)φ^K(W)^T])^T(E[𝐩_Y|WX(h_L,0(W) | W,X) q^J(X)φ^K(W)^T])

	and 1. By Assumption <ref>(iv) E[𝐩_Y|WX(h_L,0(W) | W,X) q^J(X)φ^K(W)^T] has full rank and thus the eigenvalues are positive.



	Note that
	
    ‖ H_J(α,P_n)  - H_J‖_e≤   ‖ H_J(α,P_n) - H_J(α_L,0,P_n) - { H_J(α,𝐏) - H_J(α_L,0,𝐏) }‖_e
        + ‖ H_J(α_L,0,P_n) - H_J(α_L,0,𝐏)  + H_J(α,𝐏) - H_J‖_e
    ≤   ‖ H_J(α,P_n) - H_J(α_L,0,P_n) - { H_J(α,𝐏) - H_J(α_L,0,𝐏) }‖_e
        + ‖ H_J(α_L,0,P_n) - H_J(α_L,0,𝐏) ‖_e
        + ‖ H_J(α,𝐏) - H_J‖_e
    ≡    Term_1,n + Term_2,n + Term_3,n.

	
	The term Term_1,n is controlled by Assumption <ref>. By Lemma <ref> applied to α_L,0, Term_2,n = O_𝐏( {θ_L,0 + || h'_L,0||_L^∞(𝕎,μ)}^2√(b_4,J/n)).
	
	Note that
    H_J(α,𝐏) - H_J =     E_𝐏[ M_J(X) {ρ(Y,W,α)ρ(Y,W,α)^T - ρ(Y,W,α_L,0)ρ(Y,W,α_L,0)^T} M_J(X)^T] 
    
    	=     E_𝐏[ M_J(X) E[{ρ(Y,W,α)ρ(Y,W,α)^T - ρ(Y,W,α_L,0)ρ(Y,W,α_L,0)^T}|X] M_J(X)^T] 
    ≤   sup_x |trace{ E_𝐏[ρ(Y,W,α)ρ(Y,W,α)^T  -  ρ(Y,W,α_L,0)ρ(Y,W,α_L,0)^T   |X=x]  }| 
       × E_𝐏[ M_J(X) M_J(X)^T].

	Moreover,
	
    |trace{ E_𝐏[ρ(Y,W,α)ρ(Y,W,α)^T  -  ρ(Y,W,α_L,0)ρ(Y,W,α_L,0)^T   |X=x]  }| 
    
    	=     | E_𝐏[ρ_1(Y,W,α)^2  -  ρ_1(Y,W,α_L,0)^2   |X=x]  |  
        + | E_𝐏[ρ_2(Y,W,α)^2  -  ρ_2(Y,W,α_L,0)^2   |X=x]  |
    
    	=      | E_𝐏[(θ - μ(W)h'(W))^2  -  (θ_L,0 - μ(W)h_L,0'(W))^2   |X=x]  | 
        + | (1-2τ)| | E_𝐏[ ( 1{Y ≤ h(W)} - 1{Y ≤ h_L,0(W)} ) |X=x]  | 
    
    	=     | E_𝐏[θ^2 - θ_L,0^2 - 2θμ(W)h'(W) + (μ(W)h'(W))^2 - (μ(W)h'_L,0(W))^2  + 2 θ_L,0μ(W)h_L,0'(W)   |X=x]  | 
        + | (1-2τ)| | E_𝐏[ ( F_Y|W,X(h(W)|W,X) -  F_Y|W,X(h_L,0(W)|W,X) ) |X=x]  |.

	
	By assumption <ref>,
	
    | E_𝐏[ ( F_Y|W,X(h(W)|W,X) -  F_Y|W,X(h_L,0(W)|W,X) ) |X=x]  | ≾ ||h-h_L,0||_L^2(Leb).

	
	Since E[M_J(X)M_J(X)^T] = I by assumption <ref>, it follows that 
    ||H_J(α,𝐏) - H_J||_e = O( ℧_L,n× ||α - α_L,0||   ).
	



	Observe that
	
    ‖ n^-1∑_i=1^n g_J(Z_i,α) - g_J(Z_i,α_L,0)  - E[g_J(Z,α) - g_J(Z,α_L,0)] ‖_e
    ≤    | n^-1∑_i=1^nμ(W_i){h'(W_i) - h_L,0'(W_i)}  - E[μ(W){h'(W) - h_L,0'(W)}]  | 
        + ‖ n^-1∑_i=1^n (1{ Y_i≤ h(W_i)  } - 1{ Y_i≤ h_L,0(W_i)  })q^J(X_i)  - E[(1{ Y ≤ h(W)  } - 1{ Y ≤ h_L,0(W)  })q^J(X) ] ‖_e
    ≤   √(J/n)( sup_(θ,h) ∈𝒩_L,n𝔾_n[μ· (h' - h_L,0')] +  max_1 ≤ j ≤ Jsup_g ∈𝒢̅_L,n𝔾_n[g · q_j]   ).




	By the triangle inequality and definition of u^∗_L,n, it suffices to check that ||α - α_L,0||_w + t ≤η_w,L,n, which holds since η_w,L,n≿ l_nn^-1/2. Regarding the latter term, by the triangle inequality it suffices to check that ||α - α_L,0|| + t ||u^∗_L,n|| ≤η_L,n. By Lemma <ref> and Assumption <ref>, ||u^∗_L,n|| ≾ (e_min(M_L^TM_L))^-1. By Assumption <ref>(iii), l_nn^-1/2 (e_min(M_L^TM_L))^-1 = o(η_L,n).
	
	It follows that by Assumption <ref>, Pen(α +t u^∗_L,n) ≤ Pen(α) + n^-1/γ_K. Since α∈𝒜̅_L,n and Γ_L,n≿ n^-1, it follows that α + t u^∗_L,n∈𝒜̅_L,n.



	Throughout the proof, let α∈{α̂_L,n,  α̂^ν_L,n}.
	
	By the calculations in the proof of Lemma <ref> and the fact that α∈𝒜̅_L,n wpa1 (for α = α̂^ν_L,n, this follows from Assumption <ref>), ||λ||_e× ||E_P_n[g_J(Z,α)]||_e≾  ( ||λ||_eθ + ||λ||_e℧_L,n + b^ϱ_ϱ,J n ||λ||_e^ϱ ). Since λ∈ B(δ_n), under Assumption <ref> (with ϱ = 3), |λ^Tg_J(z,α)| = o_𝐏(1), since s” is continuous, this implies condition 1.
	
	Regarding Condition 3, note that
	
    ||E_P_n[g_J(Z,α)]||_e≤    ||E_P_n[g_J(Z,α)-g_J(Z,α_L,0)] - E_𝐏[g_J(Z,α)-g_J(Z,α_L,0)]||_e
        + ||E_P_n[g_J(Z,α_L,0)] ||_e + ||E_𝐏[g_J(Z,α)-g_J(Z,α_L,0)]||_e.

	By Lemma <ref> and Assumption <ref>, the first term in the RHS is of order √(J/n)Δ_2,J,n. By the proof of Lemma <ref>, the third term is of order η^2_L,n b_2,J + η_w,L,n. Finally, by Lemma <ref>, the second term if of order √(g̅_L,0^2/n + ||E_𝐏[g_J(Z,α_L,0)]||^2_e). These results and assumption <ref> imply the condition.
	
	By the triangle inequality,
	
    ||E_P_n[g_J(Z,α)]||_e≤    ||E_P_n[g_J(Z,α)] - E_P_n[g_J(Z,α_L,0)] - { E_𝐏[g_J(Z,α)] - E_𝐏[g_J(Z,α_L,0)] }||_e
        + ||E_P_n[g_J(Z,α_L,0)]||_e + ||E_𝐏[g_J(Z,α) - g_J(Z,α_L,0)] ||_e.

	The first term in the RHS is of order O_𝐏(Δ_2,L,n) by Assumption <ref>. By Lemma <ref>, the second term is of order O_𝐏( √(θ + ||μ h'_L,0 ||^2_L^2(𝐏) + b^2_2,J/n      +   ||E_𝐏[g_J(Z,α_L,0)]||^2_e)   ). Finally, by the proof of Lemma <ref> the third term is of order O_𝐏( ||α - α_L,0||_w + ||α - α_L,0||^2 b_2,J).
	
	Thus, since α∈𝒩_n wpa1 (by Lemma <ref>),
	
    ||E_P_n[g_J(Z,α)]||_e =      O_𝐏( Δ_2,L,n +√(θ + ||μ h'_L,0 ||^2_L^2(𝐏) + b^2_2,J/n      +   ||E_𝐏[g_J(Z,α_L,0)]||^2_e)   + η_w,L,n + η_L,n^2 b_2,J) 
    
    	=     O_𝐏( Δ_2,L,n +√(g̅_L,0^2/n      +   ||E_𝐏[g_J(Z,α_L,0)]||^2_e)   + η_w,L,n + η_L,n^2 b_2,J)

	By Assumption <ref>(ii), this result implies Condition 4. Condition 5 holds by assumption.	



	By Lemma <ref>(i) (applied to α=α_L,0) it suffices to bound ||(α)||_e. By Lemma <ref>, ||E_P_n[g_J(Z,α_L,0)]||_e = O_𝐏( √(g̅_L,0^2/n + ||E[g_J(Z,α_L,0)]||^2_e)).
	
	Also, by Lemma <ref> (applied to α=α_L,0),
	
    ||G(α_L,0)[α - α_L,0]||_e≾ ||α - α_L,0||_w.




	By Lemma <ref>, α̂_L,n - tu^∗_L,n∈𝒩_L,n for all t = O(l_n n^-1/2). By the same calculations of Step 1 in the proof of lemma <ref> and definition of α̂_L,n, for any t = O(l_n n^-1/2).
	
    0 ≤   { ((α̂_L,n + tu^∗_L,n))^TH_L^-1((α̂_L,n + tu^∗_L,n))   } - { ((α̂_L,n))^TH_L^-1((α̂_L,n))   } + rem_n
    
    	=    { t^2 - 2 t ((α̂_L,n)^TH_L^-1(G(α_L,0)[u^∗_L,n])  } + rem_n,

	with rem_n = o_𝐏(n^-1). Taking t = ±√(rem_n), it follows that
	
    |((α̂_L,n)^TH_L^-1(G(α_L,0)[u^∗_L,n])| ≾√(rem_n).

	Since (α̂_L,n)^TH_L^-1(G(α_L,0)[u^∗_L,n]) = ⟨ u^∗_L,n , α̂_L,n - α_L,0⟩_w + n^-1∑_i=1^n (G(α_L,0)[u^∗_L,n])^T H_L^-1 g_J(Z_i,α_L,0), the desired result follows.



	Let ζ_L,n≡ (G(α_L,0)[u^∗_n])^TH_L^-1{g_J(Z_i,α_L,0)  - E[g_J(Z,α_L,0)] }. It is clear that E[ζ_L,n] = 0 and Var(ζ_L,n) = 1, so to show asymptotic normality it suffices to show that the Lyapounov condition holds.
	
	By Cauchy-Swarchz inequality and definition of u^∗_n, for any ϱ>0,
	
    E_𝐏[ | ζ_L,n|^2+ϱ] ≤    E_𝐏[ ||u^∗_n||_w^2+ϱ  || g_J(Z,α_L,0)  - E[g_J(Z,α_L,0)]||_e^2+ϱ]
    
    	=     E_𝐏[ || g_J(Z,α_L,0)  - E[g_J(Z,α_L,0)]||_e^2+ϱ] 
    ≾    E_𝐏[ || g_J(Z,α_L,0) ||_e^2+ϱ].

	By the calculations in the proof of Lemmas <ref> and <ref>, || g_J(Z,α_L,0) ||_e≤θ + || h'_L,0||_L^∞(𝕎,μ) + ||q^J(X)||_e. Thus
	
    E_𝐏[ | ζ_L,n|^2+ϱ] ≾   (θ + || h'_L,0||_L^∞(𝕎,μ))^2+ϱ  + b^2+ϱ_2+ϱ,J.
	
	By Assumption <ref>, b^2+ϱ_2+ϱ,J/n = o(1) and (θ + || h'_L,0||_L^∞(𝕎,μ))^2+ϱ/n^2+ϱ = o(1) for some ϱ> 0. Thus, the Lyapounov condition holds.




