Score estimation in the monotone single index model

arXiv:1712.05593v1 [] 15 Dec 2017

Fadoua Balabdaoui
Université Paris-Dauphine, PSL Research University,
CNRS, CEREMADE, 75016 Paris, France
and
Seminar für Statistik, ETH Zürich,
8092, Zürich, Schweiz
e-mail: fadoua.balabdaoui@gmail.com

Piet Groeneboom
Delft University of Technology, Mekelweg 4, 2628 CD Delft, The Netherlands.
e-mail: P.Groeneboom@tudelft.nl

Kim Hendrickx
Hasselt University, I-BioStat, Agoralaan, B3590 Diepenbeek, Belgium.
e-mail: kim.hendrickx@uhasselt.be
Abstract: We consider estimation of the regression parameter in the single index model where the
link function ψ is monotone. For this model it has been proposed to estimate the link function nonparametrically by the monotone least square estimate ψ̂nα for a fixed regression
P parameter α and to
estimate the regression parameter by minimizing the sum of squared deviations i {Yi −ψ̂nα (αT Xi )}2
over α, where Yi are the observations and Xi the corresponding covariates. Although
√ it is natural to
propose this least squares procedure, it is still unknown whether it will produce n-consistent estimates of α. We show that the latter property will hold if we solve a score equation corresponding to
this minimization problem. We also compare our method√with other methods such as Han’s maximum
rank correlation estimate, which has been proved to be n-consistent.
AMS 2000 subject classifications: 62G05, 62G20, 62H12.
Keywords and phrases: monotone link functions, nonparametric least squares estimates, semiparametric model, single index regression model.

1. Introduction
Single index models are flexible models used in regression analysis of the type E(Y |X) = ψ0 (αT0 X), where
ψ0 is an unknown link function and α0 is an unknown regression parameter. By lowering the dimensionality
of the classical linear regression problem, determined by the number of covariates, to a univariate αT0 X
index, single index models do not suffer from the “curse of dimensionality”. They also provide an advantage
over the generalized linear regression models by overcoming the risk of misspecifying the link function ψ0 . To
ensure identifiability of the single index model, one typically assumes that the Euclidean norm kα0 k equals
one with the first non-zero element of α0 being positive.
Several estimation approaches have been considered in the literature of single index models. These methods
can be classified into two groups: M-estimators and direct estimators. In the first approach, one considers a
non-parametric regression estimate for the infinite dimensional link function ψ0 and then estimates α0 by
minimizing a certain criterion function, where ψ0 is replaced by its estimate. Examples of this type are the
semi-parametric least squares estimators of [16] and [13] and the pseudo-maximum likelihood estimator of
[8], all using kernel regression estimates for the unknown link functions. An example of an M-estimator that
does not depend on an estimate of the link function ψ0 is Han’s maximum rank correlation estimator [12].
Direct estimators, such as the average derivative estimator of [14] or the slicing regression method proposed in [9], avoid solving an optimization problem and are often computationally more attractive than
M-estimators.
In this paper we focus on estimating the regression parameter α0 under the constraint that ψ0 is monotone.
Shape constrained inference arises naturally in a variety of fields. For example in economics where a concavity
restriction is assumed in utility theory to indicate the exhibition of risk conversion in economic behavior.
Convex optimization problems also appear frequently and often allow for straightforward computation and
1
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

2

optimization. The single index model with convex link has been studied in [18] where the
√ authors consider
estimation of a penalized least squares estimator using smoothing splines and prove n-consistency and
asymptotic normality of their proposed estimator.
A special case of the monotone singe index model is the widely used econometric binary choice model
where interest is in estimating a choice probability based on a binary response variable Y and one or more
covariates X. Whether or not the outcome is zero or one depends on an underlying utility score, i.e. Y = 1
if αT0 X ≥ ε, where ε is an unobserved disturbance term with unknown distribution function F0 . The binary
choice model therefore belongs to the class of monotone single index models since E(Y |X) = F0 (αT0 X).
Estimation of the regression parameters in the binary choice model is among others considered in [6], [7] and
[17].
[1] considered a global least squares estimator for the pair (α0 , ψ0 ) in the general single index model
under monotonicity of the function ψ0 . They derived an n−1/3 convergence rate, but the asymptotic limiting
distribution for their estimator of α0 has not been derived. A conjecture is made in [20] that this rate is too
slow. In this paper, we will give simulation results on the asymptotic variance of the least squares estimator
and investigate its rate of convergence numerically.
Recently, [10] developed several score estimators for the current status linear regression model Y =
β0T Z + ε, where the distribution function F0 of ε is left unspecified. Instead of observing the response
Y a censoring variable T and censoring indicator ∆ = 1Y ≤T are observed. This model is a special case
of the monotone single index model and can be formulated as E(∆|T, Z) = F0 (T − β0T Z) = F0 (αT0 X)
where α0 = (1, −β0 )T and X = (T, Z)T . Their estimators are obtained by the root of a score function
involving
√ the maximum likelihood estimator (MLE) of the distribution function for fixed β. The authors
prove n-consistency and asymptotic normality of their estimators and show that under certain smoothness
assumptions the limiting variance of a score estimator is arbitrarily close to the efficient
√ variance. Their result
is remarkable since it is the first time in the current status regression model that a n-consistent estimate
for β0 is proposed based on the MLE for F0 which only converges at n−1/3 -rate to the true distribution
function F0 .
We consider extending the estimators of [10] to the more general single index regression problem and
propose two different score equations involving the least squares estimator (LSE) ψ̂nα minimizing
n
o2
1 Xn
Yi − ψ(αT Xi ) ,
n i=1

(1.1)

over all monotone increasing functions ψ for fixed α. We establish an n−1/3 log n-rate for the estimator ψ̂nα
and propose a single index score estimator of α0 that converges at the parametric rate n−1/2 to the true
regression parameter α0 .
2. The single-index model with monotone link
Consider the following regression model
Y = ψ0 (αT0 X) + ε,

(2.1)

where Y is a one-dimensional random variable, X = (X1 , . . . , Xd )T is a d-dimensional random vector with
distribution G and ε is a one-dimensional random variable such that E[ε|X] = 0 G-almost surely. The function
ψ0 is a monotone link function in M, where M is the set of monotone increasing functions defined on R and
α0 is a vector of regression parameters belonging to the d−1 dimensional sphere Sd−1 := {α ∈ Rd : kαk = 1},
where k · k denotes the Euclidean norm in Rd .
3. The least squares estimator (LSE) for the link function ψ
Let (X1 , Y1 ), . . . , (Xn , Yn ) denote n random variables which are i.i.d. like (X, Y ) in (2.1), i.e. E(Y |X) =
ψ0 (αT0 X) G-almost surely and consider the sum of squared errors
n

Sn (ψ, α) =

o2
1 Xn
Yi − ψ(αT Xi ) ,
n i=1

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

3

which can be computed for any pair (ψ, α) ∈ M × Sd−1 . For a fixed α, order the values αT X1 , . . . , αT Xn
in increasing order and arrange Y1 , . . . , Yn accordingly. Then, well-known results from monotone regression
theory imply that the functional ψ 7→ Sn (ψ, α) is minimized by the left derivative of the greatest convex
minorant of the cumulative sum diagram
i
n
 X

o
(0, 0), i,
Yiα , i = 1, . . . , n ,
j=1

where Yiα corresponds to the ith order statistic of the αT Xi . See for example Theorem 1.1 in [4].
By strict convexity of ψ 7→ Sn (ψ, α), the minimizer is unique at the distinct projections. We denote by
ψ̂nα the monotone function which takes the values of this minimizer at the distinct projections and is a
stepwise and right-continuous function outside the set of those projections.
We first list below the assumptions needed to prove the asymptotic results stated in the remainder of the
paper.

Assumptions A1-A6
A1. The space X is convex, with a nonempty interior. There exists also R > 0 such that X ⊂ B(0, R).
A2. There exists K0 > 0 such that the true link function ψ0 satisfies |ψ0 (u)| ≤ K0 for all u in {αT x, x ∈
X , α ∈ Sd−1 }.


A3. There exists δ0 > 0 such that the function u 7→ E ψ0 (αT0 X)|αT X = u is monotone increasing on
Iα := {αT x, x ∈ X } for all α ∈ B(α0 , δ0 ) := {α : kα − α0 k ≤ δ0 }.

A4. Let a0 and b0 denote the infimum and supremum of the interval Iα0 = αT0 x, x ∈ X . Then, the true
link function ψ0 is continuously differentiable on (a0 − δ0 R, b0 + δ0 R), where R is the same radius of
assumption A1 above, and there exists C > 0 such that ψ00 ≥ C on (a0 − δ0 R, b0 + δ0 R).
A5. The distribution of X admits a density g, which is differentiable on X . Also, there exist positive
constants c0 , c̄0 , c1 and c̄1 such thath c0 ≤ g ≤ c̄0 and
i c1 ≤ ∂g/∂xi ≤ c̄1 on X for all i = 1, . . . , d.

A6. There exist a0 , M0 > 0 such that E |Y |m |X = x ≤ m!M0m−2 a0 for all integers m ≥ 2 and x ∈ X
G-almost surely.

Assumption A1 ensures that the support of the linear predictor αT X is an interval for all α ∈ B(α0 , δ0 ).
Assumption A3 is made to enable deriving the explicit limit of the LSE ψ̂nα for all α ∈ B(α0 , δ0 ). We
will show, in the supplemental material, the plausibility of this Assumption
A3 by proving
that for α in


a neighborhood of α0 the derivative of the function u 7→ ψα := E ψ0 (αT0 X)|αT X = u is indeed strictly
positive if the derivative of the true link function stays away from zero. Assumption A6 is needed to show
that max1≤i≤n |Yi | = Op (log n). As noted in [1], such an assumption is satisfied if the conditional distribution
of Y |X = x belongs to an exponential family.
In Figure 1 we compare the true link function ψ0 with the function ψα for the model E(Y |X) =
√
√
i.i.d
ψ0 (α01 X1 + α02 X1 ), where X1 , X2 ∼ U [0, 1], ψ0 (x) = x3 and α01 = α02 = 1/ 2 for α1 = 1/2, α2 = 3/2.
Figure 1 illustrates the monotonicity of the function ψα introduced in Assumption A3.
We have the following results.
Proposition 3.1. Suppose that Assumptions A1-A2 hold and let the function ψα be defined by


ψα (u) = E ψ0 (αT X)|αT X = u .

(3.1)

Then, the functional Lα given by,
ψ 7→ Lα (ψ) :=

Z 

2
ψ0 (αT0 x) − ψ(αT x) g(x)dx,

(3.2)

X

admits a minimizer ψ̂ α , over the set of monotone increasing
on R, denoted by M, such
n functions defined
o
that ψ̂ α is uniquely given by the function in (3.1) on Iα = αT x : x ∈ X .
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

4

√
Fig 1: The real ψ0 (red, solid) and the function ψα (blue, dashed) for ψ0 (x) = x3 , α01 = α02 = 1/ 2 and
√
i.i.d
α1 = 1/2, α2 = 3/2, with X1 , X2 ∼ U [0, 1].

Proposition 3.2. Under Assumptions A1-A6, we have,
Z n
o2


sup
ψ̂nα (αT x) − ψα (αT x) dG(x) = Op (log n)2 n−2/3 .
α∈B(α0 ,δ0 )

The proofs of Proposition 3.1 and Proposition 3.2 are given in the Appendix.
4. The score estimator on the unit sphere
Consider the problem of minimizing
n
o2
1 Xn
Yi − ψ̂nα (αT Xi ) ,
n i=1

(4.1)

over all α ∈ Sd−1 , where ψ̂nα is the LSE of ψα .
Let S be a local parametrization mapping Rd−1 to the sphere Sd−1 , i.e., for each α ∈ B(α0 , δ0 ) on the
sphere Sd−1 , there exists a unique vector β ∈ Rd−1 such that
α = S (β) .
The minimization problem given in (4.1) is equivalent to minimizing
n

o2
1 Xn
Yi − ψ̂nα S(β)T Xi
,
n i=1

(4.2)

over all β where ψ̂nα is the LSE of ψα with α = S(β). Analogously to the treatment of the score approach
in the current status regression model proposed by [10], we consider the derivative of (4.2) w.r.t. β, where
we ignore the non-differentiability of the LSE ψ̂nα . This leads to the set of equations,
n

n
o
1X
T
(JS (β)) Xi Yi − ψ̂nα S(β)T Xi = 0,
n i=1

(4.3)

where JS is the Jacobian of the map S and where 0 ∈ Rd−1 is the vector of zeros. Just as in the analogous
case of the “simple score equation” in [10], we cannot hope to solve equation (4.3) exactly. Instead, we define
the solution in terms of a “zero-crossing” of the above equation. The following definition is taken from [10].
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

5

Definition 4.1 (zero-crossing). We say that β∗ is a crossing of zero of a real-valued function ζ : A 7→ R :
β 7→ ζ(β) if each open neighborhood of β∗ contains points β1 , β2 ∈ A such that ζ̄(β1 )ζ̄(β2 ) ≤ 0, where ζ̄ is
the closure of the image of the function (so contains its limit points). We say that an m-dimensional function
ζ : A 7→ Rm : β 7→ ζ(β) = (ζ1 (β), . . . ζm (β))0 has a crossing of zero at a point β∗ , if β∗ is a crossing of zero
of each component ζj : A 7→ R, j = 1 . . . , m.
Our index score estimator α̂n is defined by,
α̂n := S(β̂n ),

(4.4)

where β̂n is a zero crossing of the function
Z
n
o
T
φn (β) := (JS (β)) x y − ψ̂nα S(β)T x dPn (x, y),

(4.5)

and Pn denotes the empirical probability measure of (X1 , Y1 ), . . . , (Xn , Yn ). The probability measure of
(X, Y ) will be denoted by P0 in the remainder of the paper.
In addition to Assumptions A1-A6 above, the following assumptions will also be made.
Assumptions A7-A9
A7. For all β 6= β0 such that S (β) ∈ B(α0 , δ0 ), the random variable
h
i

Cov (β0 − β)T JS (β)T X, ψ0 S(β0 ) | S(β)T X ,
is not equal to 0 almost surely.
A8. The functions JSij (β), where JSij (β) denotes the i × j entry of JS (β) for i = 1, . . . , d and j = 1, . . . , d − 1
are d − 1 times continuously differentiable on C := {β ∈ Rd−1 : S (β) ∈ B(α0 , δ0 )} and there exists
M > 0 satisfying
max sup |Dk JSij (β)| ≤ M

(4.6)

k.≤d−1 β∈C

where k = (k1 , . . . , kd ) with kj an integer ∈ {0, . . . , d − 1}, k. =
Dk s(β) ≡

Pd−1
i=1

ki and

∂ k. s(β)
.
∂βk1 . . . ∂βkd

We also assume
that C is a convex andi bounded set in Rd−1 with a nonempty interior.
h
T
A9. (JS (β0 )) E ψ00 (αT0 X) Cov(X|αT0 X) (JS (β0 )) is non-singular.
Theorem 4.1. Let Assumptions A1-A9 be satisfied. Let also α̂n be defined by (4.4). Then:
(i) [Existence of a root] For all large n, a crossing of zero β̂n of φn (β) exists with probability tending to
one.
(ii) [Consistency]
p

α̂n → α0 ,

n → ∞.

(iii) [Asymptotic normality] Define the matrices,
h
i
def
A = E ψ00 (αT0 X) Cov(X|αT0 X) ,

(4.7)

and
def

Σ = E

h
Y − ψ0 (αT0 X)

2



X − E(X|αT0 X) X − E(X|αT0 X)

T

i

.

(4.8)

Then
√


n(α̂n − α0 ) →d Nd 0, A− ΣA− ,

where A− is the Moore-Penrose inverse of A.
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

6

Remark 4.1. Note that αT0 A = 0 and that the normal distribution Nd (0, A− ΣA− ) is concentrated on the
(d−1)-dimensional subspace, orthogonal to α0 and is therefore degenerate, as is also clear from its covariance
matrix A− ΣA− , which is a matrix of rank d − 1.
4.1. The asymptotic relation
To obtain the asymptotic normality result of the score estimator α̂n given in Theorem 4.1, we shall prove
the following asymptotic relationship for β̂n :

 Z



T 
B β̂n − β0 = (JS (β0 )) x − E(X|S(β0 )T X = S(β0 )T x) y − ψ0 S(β0 )T x d Pn − P0 (x, y)


+ op n−1/2 + kβ̂n − β0 k .
where
h
i
T
T
B = (JS (β0 )) E ψ00 (S(β0 )T X) Cov(X|S(β0 )T X) (JS (β0 )) = (JS (β0 )) A JS (β0 ),

(4.9)

in R(d−1)×(d−1) . We assume in Assumption A9 that B is invertible so that

√ 
n β̂n − β0
Z



√
T 
= nB −1 (JS (β0 )) x − E(X|S(β0 )T X = S(β0 )T x) y − ψ0 S(β0 )T x d Pn − P0 (x, y)


√
+ op 1 + nkβ̂n − β0 k
→d N (0, Π),
where
T

Π = B −1 (JS (β0 )) Σ JS (β0 ) B −1 ∈ R(d−1)×(d−1) .

(4.10)

The limit distribution of the single index score estimator α̂n defined in (4.4) now follows by an application
of the delta-method and we conclude that

√


√
√ 
√
n(α̂n − α0 ) = n S β̂n − S (β0 ) = JS (β0 ) n(β̂n − β0 ) + op
n(β̂n − β0 )



T
→d Nd 0, JS (β0 )Π (JS (β0 ))
= Nd 0, A− ΣA− ,
where the last equality follows from the following lemma.
Lemma 4.1. Let the matrix A be defined by (4.7) and let A− be the Moore-Penrose inverse of A. Then
n
o−1
T
T
T
A− = JS (β0 ) (JS (β0 )) AJS (β0 )
(JS (β0 )) = JS (β0 )B −1 (JS (β0 )) .
The proof of Lemma 4.1 is given in Supplement C of [2]. An example of the mapping S and corresponding
matrix JS is given in Section 6.
Remark 4.2. For each map S and each parameter vector β, we have
T

(S(β)) S(β) = 1.
Taking derivatives w.r.t. β, we get
T

(S(β)) JS (β) = 0T ,
so that the columns of JS (β) belong to the space
n
o

⊥
T
{α}⊥ ≡ {S(β)} ≡ z ∈ Rd : αT z = 0 ≡ z ∈ Rd : (S(β)) z = 0 .
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

7

By Lemma 4.1 it is now easy to see that also αT0 A− = 0. It is shown in Lemma 1 of [18] that it is possible
to construct a set of “local parametrization matrices” Hα for each α ∈ B(α0 , δ0 ) with kαk = 1 satisfying
α T Hα = 0 T

T

and

(Hα ) Hα = Id−1 .

T

Their matrix (Hα ) corresponds to the Moore-Penrose pseudo-inverse of the matrix Hα and is the analogue
T
of our matrix (JS (β)) in the proof of asymptotic normality of their estimator. We however show that the
orthonormality assumption is not needed in the proofs.
5. The efficient score estimator
In this section we extend the score approach of Section 4 by incorporating an estimate of the derivative of
the link function ψ0 to obtain an efficient estimator of α0 . Let ψ̂nα denote again the LSE of ψα defined in
0
Section 3 and define the estimate ψ̃nh,α
by
0
ψ̃nh,α
(u)

1
=
h



Z
K

u−x
h


dψ̂nα (x),

where h is a chosen bandwidth. Here dψ̂nα represents the jumps of the discrete function ψ̂nα and K is one
of the usual symmetric twice differentiable kernels with compact support [−1, 1], used in density estimation.
The estimator α̃n is given by
α̃n := S(β̃n ),

(5.1)

where β̃n is a zero crossing of ξnh (see Definition 4.1) defined by
Z
n
o
T
0
ξnh (β) := (JS (β)) x ψ̃nh,α
S(β)T x y − ψ̂nα S(β)T x dPn (x, y).

(5.2)

The function ξnh is inspired by representing the sum of squares
n

1 X
Yi − ψα (αT Xi )
n i=1

2

,

in a local coordinate system with d − 1 unknown parameters β = (β1 , . . . , βd−1 )T followed by differentiation
of the re-parametrized sum of squares w.r.t. β where we also consider differentiation of the function ψα .
We make the following additional assumptions for establishing the weak convergence of β̃n
Assumptions A10-A11
A10. The function
differentiable on Iα for all α.
h ψα is two times continuously
i
T
0
T
2
T
A11. (JS (β)) E ψ0 (α0 X) Cov(X|α0 X) JS (β) is non-singular.
Theorem 5.1. Let Assumptions A1-A8, A10-A11 be satisfied. Let α̃n be defined by (5.1) and suppose
h  n−1/7 . Then:
(i) [Existence of a root] For all large n, a crossing of zero β̃n of ξnh (β) exists with probability tending to
one.
(ii) [Consistency]
p

α̃n → α0 ,

n → ∞.

(iii) [Asymptotic normality] Define the matrices,
h
i
def
Ã = E ψ00 (αT0 X)2 Cov(X|αT0 X) ,

(5.3)

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

8

and
def

Σ̃ = E

h

Y − ψ0 (αT0 X)

2



ψ00 (αT0 X)2 X − E(X|αT0 X) X − E(X|αT0 X)

T

i

.

(5.4)

Then
√



n(α̃n − α0 ) →d Nd 0, Ã− Σ̃Ã− ,

where Ã− is the Moore-Penrose inverse of Ã.
Remark 5.1. The asymptotic variance of the estimator α̃n is similar to that obtained for the “efficient”
estimates proposed in [24] and in [18]. The efficient score function for the semi-parametric single index model
is
)
(


y − ψ αT0 x 0 T 
E σ −2 (X)X|αT0 X = αT0 x
˜

.
`α0 ,ψ0 (x, y) =
ψ α0 x x −
σ 2 (x)
E σ −2 (X)|αT0 X = αT0 x
More details on the efficiency calculations can be found in e.g. [22], chapter 25 for a general description of
the efficient score functions and in [8] or [18] for the efficient score in the single index model.
In a homoscedastic model with var(Y |X = x) = σ 2 , where σ 2 is independent of covariates x, the asymptotic variance equals σ 2 Ã− which is the same as the inverse of E(`˜α0 ,ψ0 (X, Y )`˜α0 ,ψ0 (X, Y )T ). This indeed
shows that our estimate defined in (5.1) is efficient in the homoscedastic model. As also explained in Remark 2 of [18], our estimator has also a high relative efficiency with respect to the optimal semi parametric
efficiency bound if the constant variance assumption provides a good approximation to the truth.
The asymptotic variance is obtained similarly to the derivations of the asymptotic limiting distribution
for the simple score estimator as shown in Section 4.1. First the asymptotic variance is expressed in terms of
the parametrization S as in (4.10) and next, similar to Lemma 4.1, equivalence to the expression Ã− Σ̃Ã−
given in Theorem 5.1 is proved.
6. Computation
In this section we describe how the score estimator α̂n defined in (4.4) can be obtained using a local coordinate
system representing the unit sphere in combination with a pattern search numerical optimization algorithm.
An example of such a parameterization is the spherical coordinate system S : [0, π](d−2) × [0, 2π] 7→ Sd−1 :
(β1 , β2 , . . . , βd−1 ) 7→(cos(β1 ), sin(β1 ) cos(β2 ), sin(β1 ) sin(β2 ) cos(β3 ), . . . ,
sin(β1 ) . . . sin(βd−2 ) cos(βd−1 ), sin(β1 ) . . . sin(βd−2 ) sin(βd−1 ))T .
The map parameterizing the positive half of the sphere S : {(β1 , β2 , . . . , βd−1 ) ∈ [0, 1](d−1) : kβk ≤ 1} 7→
Sd−1 :
q

T
2
(β1 , β2 , . . . , βd−1 ) 7→ β1 , β2 , . . . , βd−1 , 1 − β12 − . . . − βd−1
,
is another example that can be used provided αd is positive. Prior knowledge about the position of α0
can be derived from an initial estimate such as the LSE proposed in [1]. We illustrate the set of equations
corresponding to (4.3) for dimension d = 3 using the simulation model studied in Section 8. We consider the
model
√
i.i.d
X1 , X2 , X3 ∼ U [1, 2], ε ∼ N (0, 1),
Y = ψ0 (αT0 X) + ε, , ψ0 (x) = x3 , α01 = α02 = α03 = 1/ 3,
where ε is independent of the covariate vector X = (X1 , X2 , X3 )T . For this model, we have





2 −1 −1
2 −1 −1
2
17 
1 



A=
−1 2 −1 , Σ =
−1 2 −1 and Ã = Σ̃ = 11.898545 −1
15
36
−1 −1 2
−1 −1 2
−1

−1
2
−1


−1

−1 ,
2

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

9

where the matrices A, Σ, Ã and Σ̃ are defined in (4.7), (4.8), (5.3) and (5.4) respectively. Note that the rank
of the matrices is equal to d − 1 = 2. We consider the parametrization
S3 = {(α1 , α2 , α3 ) = (cos(β1 ) sin(β2 ), sin(β1 ) sin(β2 ), cos(β2 )) : 0 ≤ β1 ≤ 2π, 0 ≤ β2 ≤ π} ⊂ R2 ,

(6.1)

and we solve the problem

n
o
 s1 (β1 , β2 ) = 1 Pn (− sin(β1 ) sin(β2 )Xi1 + cos(β1 ) sin(β2 )Xi2 ) Yi − ψ̂nα (αT Xi ) = 0,
i=1
n
n
o
 s2 (β1 , β2 ) = 1 Pn (cos(β1 ) cos(β2 )Xi1 + sin(β1 ) cos(β2 )Xi2 − sin(β2 )Xi3 ) Yi − ψ̂nα (αT Xi ) = 0.
i=1
n
(6.2)
Note that
 √
√
√ T
T
S(β0 ) = (cos(β01 ) sin(β02 ), sin(β01 ) sin(β02 ), cos(β02 )) = 1/ 3, 1/ 3, 1/ 3 ,
and

where β01


 − √1
− sin(β01 ) sin(β02 ) cos(β01 ) cos(β02 )
3

  √1
JS (β0 ) =  cos(β01 ) sin(β02 ) sin(β01 ) cos(β02 )  = 
3

0
− sin(β02 )
0
√
= π/4 and β02 = arctan( 2). We also have,
S(β)T JS (β) = (0, 0) ,

√1
6
√1
q6

−



2
3


,


(6.3)

for all β. This implies that the columns of JS (β) are perpendicular to the vector α = S(β). Note moreover
that the columns are linearly independent and hence form a basis for {α}⊥ . The asymptotic variance of α̂n
resp. α̃n defined in Theorem 4.1 resp. Theorem 5.1 is equal to,




2 −1 −1
2 −1 −1
25 



A− ΣA− =
resp. Ã− Σ̃Ã− = Ã− = 0.009338 −1 2 −1 .
(6.4)
−1 2 −1
2601
−1 −1 2
−1 −1 2
By the discontinuous nature of the functions s1 and s2 in (6.2), it is not possible to solve equations (6.2)
exactly, we search the crossing of zero (see Definition 4.1), by minimizing the sum of squares s21 (β) + s22 (β)
over all possible values of β = (β1 , β2 )T . Note that the crossing of zero of the score function is equivalent to
the minimizer of the sum of squared component scores so that the minimization procedure is justified.
We use a derivative free optimization algorithm proposed by [15]. The method is a pattern-search optimization method that does not require the objective function to be continuous. The algorithm starts from
an initial estimate of the minimum and looks for a better nearby point using a set of 2d equal step sizes
along the coordinate axes in each direction, first making a step in the direction of the previous move. For
the object function we take the sum of the squared values of the component functions, which achieve a
minimum at a crossing of zero. If in no direction an improvement is found, the step size is halved, and a new
search for improvement is done, with the reduced step sizes. This is repeated until the step size has reached
a prespecified minimum. A very clear exposition of the method is given in [21], section 4.3. In this paper also
convergence proofs for the optimization algorithm are presented.
7. Lagrange approach
Instead of tackling the fact that our parameter space is essentially of dimension d − 1 by the parametrization
α = S(β) which locally maps Rd−1 into the sphere Sd−1 , one can introduce the restriction kαk = 1 via a
Lagrangian term. We then consider the problem of minimizing
n
o2

1 Xn
Yi − ψ̂nα (αT Xi ) + λ kαk2 − 1 ,
n i=1

(7.1)

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

10

where ψ̂nα is the LSE defined in Section 3 and λ is a Lagrange parameter which we add to the sum of
squared errors to deal with the identifiability of the single-index model. We consider a Lagrange penalty for
solving the optimization problem under the constraint that kαk = 1.
Analogously to the treatment given in Section 4, we consider the derivative of (7.1) w.r.t. α, where we
ignore the non-differentiability of the LSE ψ̂nα . This leads to the set of equations,
n

n
o
1X
Xij ψ̂nα (αT Xi ) − Yi + λαj = 0,
n i=1

j = 1, . . . , d.

(7.2)

Here λ has to satisfy
λ=λ

d
X
j=1

n

αj2 = −

n
o
1X T
α Xi Yi − ψ̂nα (αT Xi ) .
n i=1

(7.3)

Plugging in the above expression for λ in (7.2), we would consider the score equation
!
n
n
n
o
n
o
1X
1X
T
T
T
0=
Xi Yi − ψ̂nα (α Xi ) − α
Xi Yi − ψ̂nα (α Xi )
α
n i=1
n i=1
Z n

o
= I − ααT
x y − ψ̂nα αT x dPn (x, y),

(7.4)

where I is the d × d identity matrix.
A computer program was implemented to solve (7.4). It has the advantage that we do not have to deal
with the parametrization α = S(β), but has the disadvantage that we cannot assume that α̂n has exactly
norm 1 because we again have to deal with crossings of zero instead of exact equality to zero. One way
to circumvent this problem is to normalize the solution after each iteration by dividing by its norm. This
approach seems to provide reasonable solutions, although it is not entirely satisfactory from a theoretical
point of view. Also note that if the right-hand side of (7.3) equals zero, so λ = 0, the equation does not force
the norm of α to be one; it only does so if λ 6= 0. Indeed, in our computer experiments, λ was never zero, so
this problem did actually not occur, but λ will tend to zero with increasing sample sizes, so some numerical
instability is to be expected.
For reasons of space we do not further describe all details of this approach, but instead restrict ourselves
to showing a picture of the simple score estimate of ψ0 for n = 1000 and d = 10 for the simulation where
all the Xi variables
and √
the random error variable ε are standard normal and independent, ψ0 (x) = x3
√
and α0 = (1/ 10, . . . , 1/ 10)T . It is clear that the estimate of ψ0 will be rather accurate because of the
information provided by the 10 covariates Xi (instead of, say, just two covariates X1 , X2 ).
8. Simulations
In this section we illustrate the finite sample behavior of the single index score estimators proposed in Section
4 and Section 5. We consider the model
3
Y = ψ0 (αT0 X) + ε = αT0 X + ε,
(8.1)
√
where α0i = 1/ d, i = 1, . . . , d and ε ∼ N (0, 1), independent of X. We consider two different dimensions
i.i.d

i.i.d

d = 2 and d = 3 and two different distributions for the covariate vector X, Xi ∼ U [1, 2] and Xi ∼ N (0, 1)
for i = 1, . . . , d. This yields four different simulation set-ups.
For the two dimensional models, we consider the parametrization S(β0 ) = (cos(β0 ), sin(β0 ))T and we use
the parametrization given in (6.1) for the three dimensional models. In each simulation setting, we estimate
α0 by the simple score estimator (SSE) and the efficient score estimator (ESE) and compare the behavior of
our proposed estimates with the least squares estimate (LSE) minimizing (1.1) over all possible (ψ, α) and the
maximum rank correlation estimate (MRCE) proposed in [12]. For sample sizes n = 100, 500, 1000, 2000, 5000
and n = 10000 we generate N = 5000 datasets from Model (8.1) and show, in Tables 1-4, the mean and n
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

11

40

20

0

−20

−3

−2

−1

0

1

2

3

√
Fig 2: The real ψ0 (red, solid) and the function ψ̂nα̂n (black, step function) for ψ0 (x) = x3 , α0i = 1/ 10
i.i.d

and Xi ∼ N (0, 1), i = 1 . . . 10.

times the covariance of the estimates. Tables 1-4 also show the asymptotic values to which the results should
converge based on Theorem 4.1 and Theorem 5.1. The limiting distribution of the LSE is still unknown
and therefore no asymptotic results are provided for the LSE. For the MRCE, [19] has an expression for
the asymptotic covariance matrix in an (implicit) (d − 1)-dimensional representation in his Theorem 4 on
p. 133. If, in accordance with the methods of our paper, we turn this into an expression in terms of our
d-dimensional representation, we obtain as the asymptotic covariance matrix of the MRCE V − ΣV − where
h
i

T
Σ = E X − E(X|αT0 X) X − E(X|αT0 X) S(Y, αT0 X)2 g0 (αT0 X)2 ,
and V − is the Moore-Penrose inverse of
h

V = E X − E(X|αT0 X) X − E(X|αT0 X)

T

i
S2 (Y, αT0 X)g0 (αT0 X)2 ,

and g0 is the density of αT0 X and S and S2 are defined by


S(y, u) = E 1{y>Y } − 1{y<Y } αT0 X = u ,

S2 (y, u) =

∂
S(y, u).
∂u

It is clear from our simulations that the factor 2 in front of V in (20) of [19] cannot be correct and indeed [5]
have a note on p. 361 of their paper, attributed to Myoung-Jae Lee that this factor 2 should not be there.
For all simulation studies, the results shown in Tables 1-4 show convergence of n times the variancecovariance matrices towards its asymptotic values.
The performance of the ESE is slightly better than the performance of the SSE; the difference between
the asymptotic limiting variances is smaller in the model with uniform[1, 2] covariates Xi than the difference
in the model with standard normal covariates Xi . Tables 1-4 also illustrate that n times the variances of the
estimates decrease if the dimension d of the model increases in the model with Xi ∼ U [1, 2] but increases in
the model with Xi ∼ N (0, 1).
Although the model with standard normal covariates violates Assumptions A1, A2 and A4 given in Section
3, our proposed estimates perform reasonably well.
The performance of the MRCE is worse than the performances of our proposed score estimates in all
simulation settings. In the model with standard normal covariates, the variances of the MRCE are remarkable
larger than the variances of the score estimates and the LSE. Also, the asymptotic variances of the MRCE
in these models are considerably larger than the variances of the score estimators. This might be caused by
the fact that only the indicators 1{y>Yi } and 1{y>Yi } are used in the definition of the MRCE and not the
actual values of the Yi ’s.
The behavior of the LSE is rather remarkable. Tables 1 and 2 suggest an increase of n times the covariance
matrix, whereas Tables 3 and 4 suggest a decrease.
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

12

The results presented in Tables 3 and 4 show that the performance of the LSE is better than the performance of the SSE for small sample sizes when Xi ∼ N (0, 1). For the model with uniform covariates,
summarized in Tables 1 and 2 our proposed score estimates outperform the LSE. The variances for the LSE
presented in Tables 1-4 suggest that the rate of convergence for the LSE is faster than the cube-root n1/3 -rate
proved in [1]. The asymptotic distribution of the LSE falls beyond the scope of this paper and needs to be
addressed in further research.
Table 1
Simulation model (Xi ∼ U [1, 2], d = 2): The mean value (µ̂i = mean(α̂in ), i = 1, 2) and n times the variance-covariance
(σ̂ij = n·cov(α̂in , α̂jn ),i, j = 1, 2) of the simple score estimate (SSE), the efficient score estimate (ESE), the least squares
estimate (LSE) and the maximum rank correlation estimate (MRCE) for different sample sizes n with N = 5000 and
Xi ∼ U [1, 2]. The line, preceded by ∞, gives the asymptotic values.

Method
SSE

n
100
500
1000
2000
5000
10000
∞

µ̂1
0.707249
0.707224
0.707175
0.707173
0.707134
0.707109
0.707107

µ̂2
0.706389
0.706890
0.706992
0.707016
0.707070
0.707100
0.707107

σ̂11
0.040773
0.035018
0.033262
0.034017
0.034011
0.033344
0.032439

σ̂22
0.040866
0.035057
0.033273
0.034012
0.034012
0.033350
0.032439

σ̂12
-0.040787
-0.035033
-0.033265
-0.034014
-0.034011
-0.033347
-0.032439

ESE

100
500
1000
2000
5000
10000
∞

0.707293
0.707230
0.707185
0.707175
0.707130
0.707111
0.707107

0.706359
0.706888
0.706983
0.707015
0.707074
0.707098
0.707107

0.039631
0.033888
0.032302
0.032992
0.032925
0.032278
0.031516

0.039758
0.033922
0.032316
0.032989
0.032924
0.032283
0.031516

-0.039663
-0.033900
-0.032307
-0.032990
-0.032924
-0.032280
-0.031516

LSE

100
500
1000
2000
5000
10000
∞

0.706848
0.707060
0.707138
0.707122
0.707118
0.707128
0.707107

0.706624
0.707002
0.707000
0.707053
0.707079
0.707077
0.707107

0.052397
0.053547
0.053513
0.055502
0.059731
0.061843
?

0.052415
0.053570
0.053573
0.055519
0.059756
0.061868
?

-0.052321
-0.053542
-0.053535
-0.055506
-0.059741
-0.061854
?

MRCE

100
500
1000
2000
5000
10000
∞

0.707301
0.707178
0.707107
0.707145
0.707154
0.707125
0.707107

0.706188
0.706910
0.707047
0.707039
0.707048
0.707083
0.707107

0.051226
0.044397
0.041886
0.041231
0.040968
0.040129
0.035535

0.051272
0.044428
0.041904
0.041248
0.040976
0.040141
0.035535

-0.051182
-0.044402
-0.041891
-0.041237
-0.040971
-0.040135
-0.035535

9. Discussion
In this paper we propose a method for estimating the regression parameter in the monotone single index
model. Our estimators are defined as zero-crossings of an unsmooth score function which contains the LSE of
the monotone link function. The estimation approach extends the√results of [10] for the current status linear
regression model and as far as we know, it is the first time that n−consistent estimates of the regression
parameter are constructed based on the piecewise-constant LSE for the link function. Good performances,
both asymptotically and numerically, of our estimation approach are illustrated by the asymptotic normality
of our score estimators and by simulation studies that show comparable or even better behavior of our
score estimates compared to the maximum rank correlation estimate proposed by [12] and the least squares
estimate discussed in [1].
A limitation of our approach might be that we use a local parametrization of the sphere to obtain the
estimates. We however showed that the asymptotic properties of the resulting regression parameter estimators is independent of the parametrization used and we moreover developed a computationally interesting
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

13

Table 2
Simulation model (Xi ∼ U [1, 2], d = 3): The mean value (µ̂i = mean(α̂in ), i = 1, 2, 3) and n times the variance-covariance
(σ̂ij = n·cov(α̂in , α̂jn ),i, j = 1, 2, 3) of the simple score estimate (SSE), the efficient score estimate (ESE), the least squares
estimate (LSE) and the maximum rank correlation estimate (MRCE) for different sample sizes n with N = 5000 and
Xi ∼ U [1, 2]. The line, preceded by ∞, gives the asymptotic values.

Method
SSE

n
100
500
1000
2000
5000
10000
∞

µ̂1
0.577082
0.577066
0.577128
0.577232
0.577286
0.577324
0.577350

µ̂2
0.576760
0.577370
0.577364
0.577281
0.577343
0.577351
0.577350

µ̂3
0.577536
0.577505
0.577505
0.577512
0.577412
0.577370
0.577350

σ̂11
0.026000
0.020865
0.020471
0.020101
0.019549
0.019208
0.019223

σ̂22
0.026531
0.021461
0.020915
0.020454
0.019796
0.019732
0.019223

σ̂33
0.025168
0.020629
0.020576
0.020325
0.019966
0.019746
0.019223

σ̂12
-0.013676
-0.010840
-0.010394
-0.010107
-0.009688
-0.009597
-0.009612

σ̂13
-0.012352
-0.010012
-0.010054
-0.009987
-0.009855
-0.009609
-0.009612

σ̂23
-0.012797
-0.010623
-0.010531
-0.010346
-0.010112
-0.010138
-0.009612

ESE

100
500
1000
2000
5000
10000
∞

0.576080
0.576671
0.576861
0.577055
0.577175
0.577251
0.577350

0.577006
0.577423
0.577388
0.577293
0.577351
0.577359
0.577350

0.578298
0.577850
0.577750
0.577677
0.577516
0.577436
0.577350

0.025577
0.020379
0.019936
0.019503
0.019054
0.018714
0.018677

0.026523
0.020804
0.020265
0.019941
0.019269
0.019187
0.018677

0.024775
0.020022
0.019954
0.019749
0.019449
0.019205
0.018677

-0.013601
-0.010557
-0.010099
-0.009830
-0.009431
-0.009344
-0.009338

-0.011903
-0.009793
-0.009803
-0.009657
-0.009612
-0.009364
-0.009338

-0.012910
-0.010250
-0.010174
-0.010108
-0.009843
-0.009845
-0.009338

LSE

100
500
1000
2000
5000
10000
∞

0.576900
0.577271
0.577343
0.577440
0.577379
0.577335
0.577350

0.577212
0.577276
0.577285
0.577277
0.577338
0.577356
0.577350

0.576726
0.577256
0.577294
0.577267
0.577305
0.577345
0.577350

0.046654
0.047999
0.049601
0.050351
0.054862
0.058277
?

0.047393
0.047934
0.050000
0.051728
0.055341
0.057892
?

0.045991
0.047352
0.049575
0.051742
0.054135
0.058661
?

-0.023955
-0.024291
-0.024999
-0.025168
-0.028024
-0.028773
?

-0.022584
-0.023683
-0.024590
-0.025194
-0.026838
-0.029497
?

-0.023394
-0.023653
-0.024991
-0.026544
-0.027305
-0.029143
?

MRCE

100
500
1000
2000
5000
10000
∞

0.576976
0.577285
0.577285
0.577339
0.577381
0.577370
0.577350

0.576965
0.577414
0.577414
0.577325
0.577371
0.577355
0.577350

0.576916
0.577265
0.577265
0.577347
0.577285
0.577319
0.577350

0.046496
0.017158
0.034317
0.030230
0.028795
0.026680
0.021436

0.046523
0.016660
0.033319
0.030323
0.028768
0.027576
0.021436

0.044848
0.016967
0.033935
0.031574
0.029245
0.027700
0.021436

-0.024092
-0.008421
-0.016843
-0.014493
-0.014163
-0.013279
-0.010718

-0.022429
-0.008723
-0.017445
-0.015744
-0.014630
-0.013402
-0.010718

-0.022333
-0.008247
-0.016493
-0.015825
-0.014611
-0.014297
-0.010718

Lagrangian approach that avoids the use of a local parametrization and allows for easy estimation in high
dimensions. The theory for the Lagrangian procedure is complicated by the fact that we do no longer work
on the sphere, which is rather unusual in the theory on single index models where identifiability assumptions
are commonly used. Our simulations however suggest that its asymptotic properties are the same as those
presented in this paper.
The presence/absence of smoothing procedures involved with the estimation techniques is one of the main
difference between our two score estimates. Although the simple score estimator, avoiding any smoothing
parameters is not an efficient estimate of the regression parameter, its performance in finite samples is not
remarkably worse than the performance of the efficient score estimates. By avoiding the use of a smooth estimate for the derivative of the link function, the simple score approach has the advantage that no bandwidth
selection procedures are needed, which makes it computationally more attractive.
10. Appendix
In this Section, we give the proofs of the results stated in Sections 3 and 4. The results given in Section 5
together with additional technical lemmas needed for proving our main results are given in the supplementary
material [2]. We will write “Lemma X in Supplement Y” when we refer to these results in the remainder of
the Appendix. Entropy results are used in our proofs. Before we start the proofs, we first introduce some
notations and definitions used in the remainder of the Appendix.
We will denote the L2 -norm of a function f defined on X with respect to some probability measure P by

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

14

Table 3
Simulation model (Xi ∼ N (0, 1), d = 2): The mean value (µ̂i = mean(α̂in ), i = 1, 2) and n times the variance-covariance
(σ̂ij = n·cov(α̂in , α̂jn ),i, j = 1, 2) of the simple score estimate (SSE), the efficient score estimate (ESE), the least squares
estimate (LSE) and the maximum rank correlation estimate (MRCE) for different sample sizes n with N = 5000 and
Xi ∼ N (0, 1). The line, preceded by ∞, gives the asymptotic values.

Method
SSE

n
100
500
1000
2000
5000
10000
∞

µ̂1
0.705202
0.706395
0.706824
0.707100
0.707026
0.707091
0.707107

µ̂2
0.706163
0.707509
0.707259
0.707056
0.707167
0.707113
0.707107

σ̂11
0.201378
0.109547
0.092557
0.080981
0.072021
0.067685
0.055556

σ̂22
0.200975
0.109291
0.092464
0.080966
0.071947
0.067665
0.055556

σ̂12
-0.200327
-0.109371
-0.092494
-0.080967
-0.071982
-0.067674
-0.055556

ESE

100
500
1000
2000
5000
10000
∞

0.706800
0.706905
0.706978
0.707061
0.707079
0.707104
0.707107

0.706173
0.707200
0.707190
0.707133
0.707128
0.707106
0.707107

0.087513
0.038450
0.031701
0.027930
0.023914
0.022827
0.018519

0.087878
0.038468
0.031672
0.027924
0.023907
0.022828
0.018519

-0.087480
-0.038453
-0.031685
-0.027926
-0.023911
-0.022827
-0.018519

LSE

100
500
1000
2000
5000
10000
∞

0.706748
0.706710
0.706737
0.707009
0.707087
0.707074
0.707107

0.706135
0.707309
0.707389
0.707161
0.707110
0.707131
0.707107

0.093819
0.068561
0.061614
0.061123
0.060759
0.061708
?

0.094319
0.068260
0.061325
0.061109
0.060722
0.061692
?

-0.093715
-0.068383
-0.061459
-0.061111
-0.060738
-0.061699
?

MRCE

100
500
1000
2000
5000
10000
∞

0.701888
0.705470
0.706374
0.707090
0.706895
0.707062
0.707107

0.704656
0.707715
0.707364
0.706892
0.707229
0.707107
0.707107

0.542166
0.364976
0.336674
0.328658
0.317497
0.313625
0.268711

0.539405
0.360968
0.335793
0.328860
0.317000
0.313331
0.268711

-0.530421
-0.362295
-0.335971
-0.328632
-0.317202
-0.313456
-0.268711

k · kP ; i.e.,
2 1/2

kf kP = P(f )

Z
=

1/2
f (x)dP(x)
.
2

(10.1)

X

Also, we will denote by k · kB,P the Bernstein norm of a function f defined on X × R which is given by

kf kP,B =


1/2
|f |
2P e − |f | − 1
.

(10.2)

For both norms, P will be taken to be P0 , the true joint probability measure of the (X, Y ). Note that when
f is only a function of x ∈ X then
Z
kf kP0 =
X

1/2 Z
1/2
2
f (x)dG(x)
=
f (x)g(x)dx
,
2

X

by Assumption A5.
For a class of functions F on R equipped with a norm k · k, we let NB (ζ, F, k · k) denote the minimal
number N for which there exists pairs of functions {[gjL , gjU ], j = 1, . . . , N } such that kgjU − gjL k ≤ ζ for all
j = 1, . . . , N and such that for each g ∈ F there is a j ∈ {1, . . . , N } such that gjL ≤ g ≤ gjU . The ζ−entropy
with bracketing of F is defined as HB (ζ, F, k · k) = log(NB (ζ, F, k · k)).
Results on entropy calculations used in proving our main results are given in the supplemental material.
Our proofs use inequalities for empirical processes described in Lemma 3.4.2 and Lemma 3.4.3 of [23].
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

15

Table 4
Simulation model (Xi ∼ N (0, 1), d = 3): The mean value (µ̂i = mean(α̂in ), i = 1, 2, 3) and n times the variance-covariance
(σ̂ij = n·cov(α̂in , α̂jn ),i, j = 1, 2, 3) of the simple score estimate (SSE), the efficient score estimate (ESE), the least squares
estimate (LSE) and the maximum rank correlation estimate (MRCE) for different sample sizes n with N = 5000 and
Xi ∼ N (0, 1). The line, preceded by ∞, gives the asymptotic values.

Method
SSE

n
100
500
1000
2000
5000
10000
∞

µ̂1
0.570983
0.575676
0.576424
0.576826
0.577037
0.577107
0.577350

µ̂2
0.575590
0.577118
0.577186
0.577144
0.577330
0.577377
0.577350

µ̂3
0.578029
0.578470
0.578120
0.577945
0.577635
0.577543
0.577350

σ̂11
0.263776
0.141363
0.123481
0.104429
0.093606
0.087822
0.074074

σ̂22
0.309305
0.161205
0.124819
0.104871
0.097177
0.092609
0.074074

σ̂33
0.282773
0.149813
0.121335
0.103671
0.093909
0.089642
0.074074

σ̂12
-0.144522
-0.076137
-0.063075
-0.052625
-0.048362
-0.045356
-0.037037

σ̂13
-0.114110
-0.064089
-0.060038
-0.051672
-0.045185
-0.042420
-0.037037

σ̂23
-0.165663
-0.085601
-0.061653
-0.052170
-0.048793
-0.047258
-0.037037

ESE

100
500
1000
2000
5000
10000
∞

0.571762
0.575761
0.576414
0.576780
0.577012
0.577125
0.577350

0.577007
0.577456
0.577396
0.577305
0.577403
0.577401
0.577350

0.579906
0.578542
0.578128
0.577918
0.577620
0.577517
0.577350

0.123299
0.056520
0.043292
0.036568
0.030453
0.029584
0.024691

0.141033
0.059089
0.043200
0.036203
0.032063
0.029680
0.024691

0.121794
0.051253
0.041763
0.036525
0.032005
0.030257
0.024691

-0.070078
-0.032075
-0.022266
-0.018055
-0.015234
-0.014494
-0.012346

-0.049452
-0.023321
-0.020903
-0.018448
-0.015195
-0.015074
-0.012346

-0.071866
-0.027809
-0.020953
-0.018145
-0.016831
-0.015193
-0.012346

LSE

100
500
1000
2000
5000
10000
∞

0.575121
0.576833
0.577022
0.577337
0.577311
0.577258
0.577350

0.574814
0.577377
0.577354
0.577418
0.577469
0.577498
0.577350

0.577588
0.577289
0.577415
0.577177
0.577224
0.577272
0.577350

0.173704
0.107186
0.101079
0.092130
0.090412
0.088953
?

0.174857
0.104621
0.098209
0.091427
0.088701
0.085219
?

0.173169
0.106974
0.100486
0.089505
0.089865
0.089801
?

-0.085890
-0.052394
-0.049366
-0.047012
-0.044678
-0.042188
?

-0.086918
-0.054652
-0.051622
-0.045113
-0.045745
-0.046740
?

-0.086607
-0.052262
-0.048862
-0.044391
-0.044061
-0.043056
?

MRCE

100
500
1000
2000
5000
10000
∞

0.568656
0.576584
0.577444
0.577241
0.577312
0.577239
0.577350

0.570735
0.576612
0.576655
0.576968
0.577253
0.577371
0.577350

0.571815
0.576154
0.576699
0.577275
0.577265
0.577334
0.577350

0.794208
0.513244
0.487464
0.436314
0.416947
0.398551
0.358281

0.796010
0.520888
0.482635
0.434728
0.430323
0.418207
0.358281

0.802239
0.524345
0.475335
0.436539
0.426823
0.410919
0.358281

-0.389900
-0.253712
-0.246720
-0.217338
-0.210246
-0.202886
-0.179141

-0.384086
-0.257288
-0.240780
-0.218922
-0.206836
-0.195591
-0.179141

-0.387755
-0.265869
-0.234323
-0.217178
-0.219833
-0.215307
-0.179141

Lemma 3.4.2 ([23]) Let F be a class of measurable functions such that kf kP ≤ δ and kf k∞ ≤ M for
every f in F. Then


h
i
Jn (δ, F, k · kP )
√ 2
EP kGn kF . Jn (δ, F, k · kP ) 1 +
M ,
nδ
where
Z
Jn (δ, F, k · k) =

δ

p
1 + HB (, F, k · k)d

0

Lemma 3.4.3 ([23]) Let F be a class of measurable functions such that kf kP,B ≤ δ for every f in F.
Then


h
i
Jn (δ, F, k · kP,B )
√ 2
EP kGn kF . Jn (δ, F, k · kP,B ) 1 +
,
nδ
In the sequel, and whenever the -bracketing entropy of some class F with respect to some norm k · k is
bounded above by C−1 for some constant C > 0 (which may depend on n), we will write for all e > 
Z e
Jn (d) =
(1 + C/)1/2 d
(10.3)
0

Moreover, we will use the inequality
Jn (d) ≤ d + 2C 1/2 e1/2
√
√
√
which is an immediate consequence of the fact that x + y ≤ x + y for all x, y ≥ 0.

(10.4)

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

16

10.1. Appendix A: The least squares estimator (LSE) of the link function ψ
In this section we first prove Proposition 3.1. We next show in Lemma 10.1 that the LSE ψ̂nα is of order
Op (log n) uniformly in B(α0 , δ0 ). This result is used in the proof of Proposition 3.2, given at the end of this
section.
Proof of Proposition 3.1. Note that with X ∼ g we can write
h
2 i
Lα (ψ) = E ψ0 (αT0 X) − ψ(αT X)
.
Thus,
h
2 i
E ψ0 (αT0 X) − E(ψ0 (αT0 X) | αT X)
= min 0 Lα (ψ),
ψ∈M

with M0 the set of all bounded Borel-measurable function defined on Iα . Therefore, if the minimizing
function u 7→ E(ψ0 (αT0 X) | αT X = u) is monotone increasing on Iα , then this implies that it necessarily
minimizes Lα over M. Furthermore, such a minimizer is unique by strict convexity of Lα .
Lemma 10.1.
max

sup ψ̂nα (αT x) = Op (log n).

α∈B(α0 ,δ0 ) x∈X

Proof. The proof of this lemma is similar to that of Lemma 4.4 of [1]. For a fixed α it follows from the
min-max formula of an isotonic regression that we have for all x ∈ X
Pk
Pn
α
α
i=1 Yi
i=k Yi
min
≤ ψ̂nα (αT x) ≤ max
.
1≤k≤n
1≤k≤n n − k + 1
k
Hence,
min Yi ≤ ψ̂nα (αT x) ≤ max Yi

1≤i≤n

1≤i≤n

and this in turn implies that
max

sup ψ̂nα (αT x) ≤ max |Yi |.

α∈B(α0 ,δ0 ) x∈X

1≤i≤n

Using similar arguments as in [1], we use Assumption A7 to show that max1≤i≤n |Yi | = Op (log n), which
completes the proof.

Proof of Proposition 3.2. By the definition of the LSE of the unknown monotone link function, ψ̂nα maximizes the map ψ 7→ Mn over M where,
Z


Mn (ψ, α) =
2yψ(αT x) − ψ 2 (αT x) dPn (x, y).
(10.5)
X ×R

Moreover, ψα maximizes the map ψ 7→ M over M, where
Z


M(ψ, α) =
2yψ(αT x) − ψ 2 (αT x) dP0 (x, y).

(10.6)

X ×R

Define the function fψ,α by,
fψ,α (x, y) = 2yψ(αT x) − ψ 2 (αT x)
Note that by definition of the LSE as the maximizer of (10.5), we have
Z


fψ̂nα ,α (x, y) − fψα ,α (x, y) dPn (x, y) ≥ 0.
X ×R

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

17

Moreover for all α ∈ B(α0 , δ0 ) and ψ ∈ M, we have,
Z


fψ,α (x, y) − fψα ,α (x, y) dP0 (x, y) = −d2α (ψ, ψα ),
X ×R

where, for any α ∈ B(α0 , δ0 ) and for any two elements ψ1 and ψ2 in M, we define the squared distance
Z 
2
2
dα (ψ1 , ψ2 ) =
ψ2 (αT x) − ψ1 (αT x) g(x)dx.
X

This can be seen as follows:
Z


fψ,α (x, y) − fψα ,α (x, y) dP0 (x, y)
X ×R
Z


2
=
2ψα (αT x)(ψ(αT x) − ψα (αT x)) − ψ 2 (αT x) + ψα
(αT x) dP0 (x, y)
X ×R
Z 
2
=−
ψ(αT x) − ψα (αT x) g(x)dx = −d2α (ψ, ψα ),
X

where we use that E{Y |αT X = u} = ψα (u). This implies that, for all α ∈ B(α0 , δ0 ) and ψ ∈ M, we have,
Z


fψ,α (x, y) − fψα ,α (x, y) d(Pn − P0 )(x, y) ≥ d2α (ψ, ψα ).
X ×R

We write,
(

)
dα (ψ̂nα , ψα ) ≥ 

sup

P

α∈B(α0 ,δ0 )

(

Z

≤P



sup
α∈B(α0 ,δ0 ),dα (ψ̂nα ,ψα )≥

X ×R



fψ̂nα ,α (x, y) − fψα ,α (x, y) d(Pn − P0 )(x, y) − d2α (ψ̂nα , ψα ) ≥ 0,
)
dα (ψ̂nα , ψα ) ≥ 

sup
α∈B(α0 ,δ0 )

(

Z

≤P



sup
α∈B(α0 ,δ0 ),ψ∈MRK ,dα (ψ,ψα )≥

max



fψ,α (x, y) − fψα ,α (x, y) d(Pn − P0 )(x, y) − d2α (ψα , ψ) ≥ 0,

X ×R



T

sup ψ̂nα (α x) ≤ K

α∈B(α0 ,δ0 ) x∈X


+P

max

T



sup ψ̂nα (α x) > K .

α∈B(α0 ,δ0 ) x∈X

Fix ν > 0. Since
max

sup ψ̂nα (αT x) = Op (log n),

α∈B(α0 ,δ0 ) x∈X

by Lemma 10.1, we can find K1 > 0 large enough such that


T
P
max sup ψ̂nα (α x) > K1 log n < ν/2.
α∈B(α0 ,δ0 ) x∈X

Define
n
o
MRK = ψ monotone non-decreasing on [−R, R] and bounded by K ,

(10.7)

and consider the related class



FRK = f (x, y) = 2y ψ(αT x) − ψα (αT x) − ψ(αT x)2 + ψα (αT x)2 ,

(α, ψ) ∈ B(α0 , δ0 ) × MRK and (x, y) ∈ X × R ,

(10.8)

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

18

and for some v > 0

FRKv : =


f ∈ FRK : dα (ψ, ψα ) ≤ v for all α ∈ B(α0 , δ0 ) .

(10.9)

Note now that the class FRKv is included in the class HRCδ defined in Lemma 10.5 of Supplement B of [2]
with C = 2K 2 and δ = 2Kv. This holds true provided that K0 ≤ K, and K ≥ 1 which we can assume for n
large enough since K will be chosen to be of order log n. To see the claimed inclusion, it is enough to show
that if m is a nondecreasing function [−R, R] then m2 can be written as the difference of two monotone
functions. This is true because m2 = m2 Im≥0 − (−m2 )Im<0 , and m2 and −m2 are nondecreasing on the
subsets {m ≥ 0} and {m < 0} respectively. When restricting attention to the event that ψ̂nα is bounded
by K for n large enough, we can consider only monotone functions ψ ∈ MRK . Using the expression of ψα
the latter is bounded by K0 ≤ K. On the other hand, for any function f ∈ FRKv , there exist nondecreasing
2
monotone functions f1 and f2 such that ψ 2 − ψα
= f2 − f1 , such that kf1 k∞ , kf2 k∞ ≤ K 2 + K02 ≤ 2K 2 .
Using that K ≥ 1 implies that k2ψk∞ , k2ψα k∞ ≤ 2K ≤ 2K 2 . To finish, note that for any α we have that
2
R
ψ(αT x) − ψα (αT x) dG(x) ≤ v 2 we also have that
X
Z
Z
2
2
2
ψ 2 (αT x) − ψα
(αT x) dG(x) ≤ (2K)2
ψ(αT x) − ψα (αT x) dG(x)
X

X
2 2

≤ 4K v .
The calculation above implies that we can take δ = 2Kv. Using the result of Lemma 10.5 in Supplement B
of [2], it follows that the related class FeRKv = D̃−1 FRKv with D̃ = 16M0 C = 32M0 K 2 and a given v > 0
satisfies


eRKv , k · kB,P
HB , FeRKv , k · kB,P0
≤ HB , H
0
A
≤

for some constant A > 0 (depending only on d and the other parameters of the problem), and that for all
f˜ ∈ FeRKv we have kf˜kB,P0 ≤ D̃−1 δ = (32M0 K 2 )−1 2Kv = (16M0 )−1 K −1 v ≡ A0 K −1 v. It follows from
Lemma 3.4.3 of [23] that


h
i
Jn (A0 K −1 v)
E kGn kFeRKv
. Jn (A0 K −1 v), 1 + K 2 √ 2 2
, where Jn is defined in (10.3)
nA0 v
1/2

≤

A0 K −1 v + 2A0 K −1/2 v 1/2 A1/2 , using the inequality in (10.4)

≤

B0 K −1/2 (v + v 1/2 )

for some constant B0 > 0, where we used the fact that K −1/2 ≥ K −1 . Therefore,


h
i
B0 K −1/2 (v + v 1/2 )
√ 2 2
E kGn kFeRKv
. B0 K −1/2 (v + v 1/2 ) 1 + K 2
nA0 v


1 + v 1/2
≤ C0 K −1/2 (v + v 1/2 ) 1 + C0 K 3/2 √ 3/2 .
nv
Using the definition of the class FeRKv , the preceding display implies that


h
i
1/2
3/2
1/2
3/2 1 + v
√ 3/2 .
E kGn kFRKv
. C0 K (v + v ) 1 + C0 K
nv

(10.10)

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

19

Now with K = K1 log n, we have that
(
)
dα (ψ̂nα , ψα ) ≥ 

sup

P

α∈B(α0 ,δ0 )

≤P





Z

sup

α∈B(α0 ,δ0 ),ψ∈MRK ,





2
fψ,α (x, y) − fψα ,α (x, y) d(Pn − P0 )(x, y) − dα (ψ, ψα ) ≥ 0,

X ×R

dα (ψ,ψα )≥

max

sup ψ̂nα (αT x) ≤ K


+ ν/2

(10.11)

α∈B(α0 ,δ0 ) x∈X





Z
∞



X
√
√ 2s 2 
≤
P
sup
n
fψ,α (x, y) − fψα ,α (x, y) d(Pn − P0 )(x, y) ≥ n2  , + ν/2


X ×R

s=0 α∈B(α0 ,δ0 ),ψ∈MRK ,
2s ≤dα (ψ,ψα )≤2s+1 

≤

∞
X

(
sup

P

√

Z
h(x, y)d(Pn − P0 )(x, y) ≥

n

h∈FRK2s+1 

s=0

√

)
2s 2

n2 

+ ν/2.

X ×R

We now show that there exists a constant C > 0 such that with  = M (log n)n−1/3 ,
)
(
Z
√
h(x, y)d(Pn − P0 )(x, y) ≤ CM 1/2 (log n)2 n−1/6 2(s+1)/2
n
E
sup
h∈FRK2s+1 

X ×R

(10.12)
An application of Markov’s inequality, together with (10.11), then yields, with  = M (log n)n−1/3
(
)
∞
X
CM 1/2 (log n)2 n−1/6 2(s+1)/2
√ 2s 2
P
sup
dα (ψ̂nα , ψα ) ≥ 
≤
+ ν/2
n2 
α∈B(α0 ,δ0 )
s=0
=

∞
∞
X
C(log n)2 n−1/6 2(s+1)/2
21/2 C X 1
√ 2s 3/2
+
ν/2
=
+ ν/2 ≤ ν,
n2 M (log n)2 n−2/3
M 3/2 s=0 23s/2
s=0

for M sufficiently large. The result of Proposition 3.2 hence follows by showing (10.12). Using the obtained
bound in (10.10) with v = 2s+1  and using that 2s+1 ≥ 1, s ≥ 0 and  ≤ 1 for n large enough we get some
some constant D0 > 0
(
)
Z
√
E
sup
n
h(x, y)d(Pn − P0 )(x, y)
h∈FRK2s+1 

X ×R



1 + M 1/2 2(s+1)/2 (log n)1/2 n−1/6
. (log n)3/2 M 1/2 2(s+1)/2 (log n)1/2 n−1/6 1 + D0 (log n)3/2 √
nM 3/2 23(s+1)/2 (log n)3/2 n−1/2


1 + M 1/2 2(s+1)/2 (log n)1/2 n−1/6
= (log n)2 n−1/6 M 1/2 2(s+1)/2 1 + D00
, with D00 = D0 M −3/2
23(s+1)/2
≤ 2(log n)2 n−1/6 M 1/2 2(s+1)/2 , for s ≥ 0 and n large enough.
This proves the desired result:
sup
α∈B(α0 ,δ0 )

d2α (ψ̂nα , ψα ) =

sup

Z n
o2


ψ̂nα (αT x) − ψα (αT x) dG(x) = Op (log n)2 n−2/3 .

α∈B(α0 ,δ0 )

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

20

10.2. Appendix B: Asymptotic behavior of the simple score estimator
In this section we prove Theorem 4.1 given in Section 4. The proof is decomposed into three parts: In Section
10.2.1 we first prove the existence of a crossing of zero of φn defined in (4.5). The proof of consistency and
asymptotic normality of α̂n are given in Section 10.2.2 and Section 10.2.3.
10.2.1. Proof of existence of a crossing of zero
Let φ be the population version of φn defined by
Z


T
φ(β) := (JS (β)) x y − ψα S(β)T x dP0 (x, y),

(10.13)

where ψα is defined by






ψα (u) := E ψ0 αT X |αT X = u ≡ E ψ0 S(β)T X |S(β)T X = u .
We have the following result:
Proposition 10.1.
φn (β) = φ(β) + op (1),


uniformly in β ∈ C := β ∈ Rd−1 : S(β) ∈ B(α0 , δ0 ) .
Proof. For any β ∈ C, we write,
Z


T
φn (β) = (JS (β)) x y − ψα S(β)T x dPn (x, y)
Z
n

o
T
+ (JS (β)) x ψα S(β)T x − ψ̂nα S(β)T x dPn (x, y)
Z


T
= φ(β) + (JS (β)) x y − ψα S(β)T x d(Pn − P0 )(x, y)
Z
n

o
T
+ (JS (β)) x ψα S(β)T x − ψ̂nα S(β)T x d(Pn − P0 )(x, y)
Z
n

o
T
+ (JS (β)) x ψα S(β)T x − ψ̂nα S(β)T x dP0 (x, y)
= φ(β) + I + II + III.

(10.14)

To find the rate of convergence of the term I in (10.14) we will use Lemma 10.6 in Supplement B of
T
[2]. Note first that for 1 ≤ i ≤ d − 1 the i-th component of the vector (JS (β)) x can be written as
s(β)i1 x1 + . . . + s(β)id xd , where by Assumption A8 the functions sij are assumed to be uniformly bounded
with partial derivatives that are also uniformly bounded on the bounded convex set C to which β belongs. If
B1 is the same constant found in Lemma 10.6 in Supplement B of [2] then the -bracketing entropy is bounded
above by B1 K0 /. Applying Lemma 3.4.3 of [23], Markov’s inequality and Lemma 10.6 in Supplement B to
each of the empirical processes corresponding to the term s(β)ij xj for 1 ≤ j ≤ d yields (with D = 8M RK0 ,
and M is a constant bounding the sum of s(β)ij and their partial derivatives) for A > 0



 D
Jn (B2 )
−1/2
P |I| ≥ An
≤ Jn (B2 ) 1 + √ 2 , where Jn is defined in (10.3) and B2 is the same
A
nB2
constant of Lemma 10.6 in Supplement B


D
B3
1/2 1/2 1/2
≤ B3 1 + √ 2 , using the inequality in (10.4) with B3 = B2 + 2B1 K0 B2
A
nB2
1
 .
A

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

21

This implies that I = Op (n−1/2 ). For the last term III in (10.14) we get by an application of the CauchySchwarz inequality and by Proposition 3.2 that this term is Op (n−1/3 log n), i.e.
Z
III ≤
= Op (n

2

T

(JS (β)) x
−1/3

2

1/2 Z n
1/2

o2
T
T
ψα S(β) x − ψ̂nα S(β) x
dG(x)
dG(x)

log n),
T

where we also use that (JS (β)) x is bounded in L2 norm, a straightforward implication of Assumption A1
(boundedness of X and Assumption A8 (uniform boundedness of the components of the matrix JS ). The
result now follows by showing that the term II is op (1). Consider the class of functions




GjRKv =
g(x, y) = s(β)xj ψα αT x − ψ(αT x ,
such that (α, β, ψ) ∈ Sd−1 × C × MRK and (x, y) ∈ X × ∈ R,

and
sup
dα (ψα , ψ) ≤ v
α∈B(α0 ,δ0 )

with s a function satisfying (10.43). Then, GjRKv ⊂ QjRK − QjRK , where QjRC is the same class defined in
(10.44). Here, we choose K large enough such that K ≥ K0 . If follows from (10.46) in the proof of Lemma
10.6 in Supplement B, that (at the cost of increasing the constant L in (10.46))

 LK
,
HB , G̃jRKv , k · kP0 ≤

where G̃jRKv = (16M0 K)−1 GjRKv . Also, we have for all g ∈ GjRKv
kgkP0 ≤ M Rv.
Fix ν > 0 and let sij be the i × j entry of JS (β) for 1 ≤ i ≤ d − 1 and 1 ≤ j ≤ d. Also, let
Z
n

o
IIij =
sij (β)xj ψα S(β)T x − ψ̂nα S(β)T x d(Pn − P0 )(x, y).
X ×R

Using Lemma 10.1 and Proposition 3.2 there exists some constant K1 > 0 large enough (and independent
of n ) such that with K = K1 log n and v = K1 log n n−1/3 we have that for A > 0


P |IIij | ≥ An−1/2
!
=P

|IIij | ≥ An−1/2 ,

sup

sup ψ̂nα (αT x) ≤ K,

α∈B(α0 ,δ0 ) x∈X

sup

dα (ψα , ψ) ≤ v

+ ν/2

α∈B(α0 ,δ0 )



Jn (M Rv)
K
+ ν/2, where Jn is defined in (10.3)
. Jn (M Rv) 1 + √
A
nM 2 R2 v 2


K
M Rv + 2(M RL)1/2 K 1/2 v 1/2
√
≤
M Rv + 2(M RL)1/2 K 1/2 v 1/2
1+
+ ν/2, using the inequality in (10.4)
A
nM 2 R2 v 2


1
M̃
(log n)2 n−1/6 1 +
≤
+ ν/2, for some constant M̃ > 0
A
log nM 2 R2
≤ν
for n large enough. We conclude that IIij = op (n−1/2 ) which in turn implies that II = op (n−1/2 ).

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

22

Proof of Theorem 4.1 (Existence). Using Proposition 10.1 we get, analogously to the development in [10],
the relation
φn (α) = φ0 (β0 )(β − β0 ) + Rn (β),

(10.15)

where Rn (α) = op (1) + o(β − β0 ), uniformly in β ∈ C and where φ0 is the derivative of φ defined in (10.13).
Using Lemma 10.9 in Supplement C of [2], we get that the derivative of φ at β0 is given by the matrix



T
T
φ0 (β0 ) = (JS (β0 )) E ψ00 S(β0 )T x Cov(X|S(β0 )T X) JS (β0 ) = (JS (β0 )) AJS (β0 ) = B,
where A and B are defined in (4.7) and (4.9) respectively. We now define, for h > 0, the functions
Z
1
Kh (u1 − β01 ) . . . Kh (ud−1 − βd−1 ) Rn (u1 , . . . , ud−1 ) du1 . . . dud−1 ,
R̃n,h (β) = d−1
h
where
Kh (x) = h−1 K(x/h),

x ∈ R,

letting K be one of the usual smooth kernels with support [−1, 1].
Furthermore, we define:
φ̃n,h (β) = φ0 (β0 )(β − β0 ) + R̃nh (β).
Clearly:
lim φ̃n,h (β) = φn (β)
h↓0

and

lim R̃nh (β) = Rn (β),
h↓0

for each continuity point β of φn .
We now reparametrize, defining
γ = φ0 (β0 )β,

γ0 = φ0 (β0 )β0 .

This gives:

φ0 (β0 )(β − β0 ) + R̃nh (β) = γ − γ0 + R̃nh B −1 γ ,
By (10.15), the mapping

γ 7→ γ0 − Rn B −1 γ ,
maps, for each η > 0, the ball Bη (β0 ) = {β : kβ − β0 } ≤ η} into Bη/2 (β0 ) = {β : kβ − β0 } ≤ η/2}
for all large n, with probability tending to one, where k · k denotes the Euclidean norm, implying that the
continuous map

γ 7→ γ0 − R̃nh B −1 γ ,
maps Bη (γ0 ) = {γ : kγ − γ0 k2 ≤ η} into itself for all large n and small h. So for large n and small h there
is, by Brouwer’s fixed point theorem, a point γnh such that

γnh = γ0 − R̃nh B −1 γnh .
Defining βnh = B −1 γnh , we get:
φ̃n,h (βnh ) = φ0 (β0 )(βnh − β0 ) + R̃nh (βnh ) = 0.

(10.16)

By compactness, (βn,1/k )∞
k=1 must have a subsequence (βn,1/ki ) with a limit β̃n , as i → ∞.
Suppose that the jth component φnj of φn does not have a crossing of zero at β̃n . Since φnj only has finitely
many jump discontinuities, since there can only be discontinuities at a changing of ordering of the values
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

23

αT Xi , there must be a closed ball Bδ (β̃n ) = {β : kβ − β̃n k ≤ δ} of β̃n such that {φ̄nj (β) : β ∈ Bδ (β̃n )} has
a constant sign in the closed ball Bδ , say φ̄nj (β) > 0 for β ∈ B̄δ (β̃n ). Again using that φnj only has finitely
many jump discontinuities, this means that
φ̄n,j (β) ≥ c > 0,

for all β ∈ B̄δ (β̃n ).

This means that the jth component φ̃n,h,j of φ̃n,h satisfies
φ̃n,h,j (β) = [φ0 (β0 )(β − β0 )]j + R̃nh,j (β)
Z n
o
1
[φ0 (β0 )(β − β0 )]j + Rnj (u1 , . . . , ud−1 ) Kh (u1 − β01 ) . . . Kh (ud−1 − βd−1 ) du1 . . . dud−1
= d−1
h
Z n
o
1
[φ0 (β0 )(u − β0 )]j + Rnj (u1 , . . . , ud−1 ) Kh (u1 − β1 ) . . . Kh (ud−1 − βd−1 ) du1 . . . dud−1 − c/2
≥ d−1
h
Z
1
≥ c d−1
Kh (u1 − β1 ) . . . Kh (ud − βd ) du1 . . . dud − c/2
h
= c/2,
for β ∈ Bδ/2 (β̃n ) and sufficiently small h, contradicting (10.16), since βnh , for h = 1/ki , belongs to Bδ/2 (β̃n )
for large ki .
10.2.2. Proof of consistency of the simple score estimator
Proof. Since β̂n is contained in the compact set C, the sequence (β̂n ) has a subsequence (β̂nk = β̂nk (ω)),
converging to an element β∗ . Let αnk = S(β̂nk ). If β̂nk = β̂nk (ω) −→ β∗ , we get by continuity of the map
S that αnk → α∗ = S(β∗ ). By Proposition 3.2, we also have

ψ̂nk ,α̂nk S(βnk )T x −→ ψα∗ (S(β∗ )T x),
where ψα is defined in (3.1). By Proposition 10.1 and the fact that in the limit, the crossing of zero becomes
a root of the continuous limiting function, we get,
lim φnk (βnk ) = φ(β∗ ) = 0

k→∞

(10.17)

where,
Z
φ(β∗ ) =
Z
=



JS (β∗ )T x y − ψα∗ S(β∗ )T x dP0 (x, y)



JS (β∗ )T x ψ0 S(β0 )T x − ψα∗ S(β∗ )T x dG(x)

Z






JS (β∗ )T x ψ0 S(β0 )T x − E ψ0 S(β0 )T x |S(β∗ )T X = S(β∗ )T x dG(x)




= E Cov JS (β)T X, ψ0 S(β0 )T |S(β∗ )T X
=

We next conclude that,
0 = (β0 − β∗ )T φ(β∗ )




= E Cov (β0 − β∗ )T JS (β∗ )T X, ψ0 S(β∗ )T X + (S(β0 ) − S(β∗ ))T X |S(β∗ )T X
which can only happen if β0 = β∗ where we use the positivity of the random variable Cov((α0 − α)T X,
ψ0 (αT X)|αT X) shown in Lemma 10.10 in Supplement C of [2] and Assumption A6 which guarantees that
the random variable Cov[(β0 − β∗ )T JS (β)T X, ψ0 (S(β)T X + (S(β0 ) − S(β))T X)|S(β)T X] is not equal to
0 almost surely for all β 6= β0 . Note that,



Cov (β0 − β)T JS (β)T X, ψ0 S(β)T X + (S(β0 ) − S(β))T X |S(β)T X = u



= Cov (S(β0 ) − S(β) + o(β − β0 ))T X, ψ0 S(β)T X + (S(β0 ) − S(β))T X |S(β)T X = u


= Cov (α0 − α)T X, ψ0 (αT X + (α0 − α)T X)|αT X = u + o(β − β0 )
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

24

where the first term in the expression above is positive for all α ∈ B(α0 , δ0 ) by Lemma 10.10 in Supplement
C.

10.2.3. Proof of asymptotic normality of the simple score estimator
We define φn at β̂n by putting
φn (β̂n ) = 0.

(10.18)

Note that, with this definition, we use the representation of the components as a convex combination of the
left and right limit at β̂n :
φn,j (β̂n ) = γj φn,j (β̂n −) + (1 − γj )φn,j (β̂n +) = 0,

(10.19)

where φn,j denotes the jth component of φn and where we can choose γj ∈ [0, 1] in such a way that (10.19)
holds since we have a crossing of zero componentwise. Note that this does not change the location of the
crossing of zero. Since the following asymptotic representations are also valid for one-sided limits as used in
(10.19) we can use Definition (10.18) and assume φn (β̂n ) = 0. We show
Z




φn (β̂n ) = JS (β0 )T
x − E(X|S(β0 )T x) y − ψ0 S(β0 )T x d Pn − P0 (x, y)
Z



T
+ JS (β0 )
x − E(X|S(β0 )T x) y − ψα̂n S(βn )T x dP0 (x, y)


+ op n−1/2 + kβ̂n − β0 k ,
(10.20)
where from now on we will use the notation E(X|S(β)T x) to denote E(X|S(β)T X = S(β)T x) for all β ∈ C
and x ∈ X .
T
0
T
T
Since β̂n →p β0 and since
 the function β → ψS(β) (S(β) x) has derivative ψ0 (S(β0 ) x)JS (β0 ) x −
T
T
E(X|S(β0 ) X = S(β0 ) x) at β = β0 for all x ∈ X (See Lemma 10.9 in Supplement C), we get by
Definition (10.18) and a Taylor expansion at β = β0 that,
Z




JS (β0 )T
x − E(X|S(β0 )T x) y − ψ0 S(β0 )T x d Pn − P0 (x, y)




= B β̂n − β0 + op n−1/2 + kβ̂n − β0 k ,
(10.21)
where B is defined in (4.9). We conclude that,
√
n(β̂n − β0 ) →d Nd (0, Π) ,
where Π is defined in (4.10). The asymptotic normality of the estimator α̂n then follows by noting that,
√



√
√
T
n(α̂n − α0 ) = JS (β0 ) n(β̂n − β0 ) + op
n(β̂n − β0 ) →d Nd 0, JS (β0 )Π (JS (β0 )) .
To prove (10.20) we first define the piecewise constant function Ēn,β

 
if ψα (u) > ψ̂nα (τi ) for all u ∈ (τi , τi+1 ),
 E X|S(β)T X = τi,β

Ēn,β (u) =
E X|S(β)T X = s
if ψα (s) = ψ̂nα (s) for some s ∈ (τi , τi+1 ),

 
if ψα (u) < ψ̂nα (τi ) for all u ∈ (τi , τi+1 ),
E X|S(β)T X = τi+1,β
where the τi,β denote the sequence of jump points of the monotone LSE ψ̂nα = ψ̂nS(β) . We then have
Z

n

o
Ēn,β̂n S(β̂n )T x y − ψ̂nα̂n S(β̂n )T x
dPn (x, y) = 0.
(10.22)
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

25

2
R
This follows from the fact that ψ̂nα , i.e. the minimizer of the quadratic criterion X ×R y − ψ(αT x) dPn (x, y)
over monotone functions
of the greatest convex minorant of the cumulative
o
 Pψ ∈ M, is the left derivative
i
α
sum diagram {(0, 0), i, j=1 Yi , i = 1, . . . , n . (See also [11], p.332). By Lemma 10.14 in Supplement C
0
of [2] we also know that ψα
stays away from zero for all α = S(β) in a neighborhood of α0 = S(β0 ). Using
the same techniques as in [11], we can find a constant C > 0 such that for all i = 1, . . . , d and u ∈ Iα ,


E Xi |S(β)T X = u − Ēni,β (u) ≤ C ψα (u) − ψ̂nα (u)

(10.23)

where Ēni,β denotes the ith component of En,β . In the sequel, we will use JS (β̂n ) = Op (1), an immediate
consequence of consistency of α̂n and Assumption A8. Now, as a consequence of (10.22), we can write
Z n

o n

o
φn (β̂n ) = JS (β̂n )T
x − E X|S(β̂n )T x
y − ψ̂nα̂n S(β̂n )T x
dPn (x, y)
Z n 

o n

o

y − ψ̂nα̂n S(β̂n )T x
dPn (x, y)
E X|S(β̂n )T x − Ēn,β̂n S(β̂n )T x
+ JS (β̂n )T

= JS (β̂n )T I + II .
(10.24)
The term II can be written as
Z n 


o n

o
II =
E X|S(β̂n )T x − Ēn,β̂n S(β̂n )T x
y − ψ̂nα̂n S(β̂n )T x
d(Pn − P0 )(x, y)
Z n 

o


o n
dP0 (x, y)
+
E X|S(β̂n )T x − Ēn,β̂n S(β̂n )T x
y − ψα̂n S(β̂n )T x
Z n 


o n



o
+
E X|S(β̂n )T x − Ēn,β̂n S(β̂n )T x
ψ̂nα̂n S(β̂n )T x − ψα̂n S(β̂n )T x
dP0 (x, y)
= IIa + IIb + IIc .

(10.25)

We first note that by Lemma 10.12 in Supplement C, the functions u 7→ E(Xi |S(β)T X = u) are uniformly
bounded by R for all β ∈ C and i ∈ {1, . . . , d}. Also, they admit a bounded variation, with a total variation
that is uniformly bounded for all β ∈ C and i ∈ {1, . . . , d}. By definition of Ēn,β its i-th component, Ēni,β
is also uniformly bounded by R and has a finite total variation which cannot exceed the total variation of
u 7→ E(Xi |S(β)T X = u). Using Lemma 10.13 of Supplement C , we can find two monotone functions f1
and f2 such that u 7→ E X|S(β)T X = u − Ēn,β (u) = f2 (u) − f1 (u) with f1 , f2 ∈ MRC1 for some constant
C1 > 0. Also, we know that ψ̂nα̂n ∈ MRK with K = K1 log n with increasing probability as n → ∞ provided
that K1 > 0 is chosen large enough. Noting that for any bounded increasing functions f1 , f2 , f3 we have
that (f2 − f1 )f3 is again bounded and has a bounded variation, it follows that the class of functions, Fa
say, involved in term IIa is included in HRK 0 v defined in Lemma 10.5 in Supplement B . Here, the constant
K 0 = K2 log n for some large enough constant K2 > 0, and v = C2 (log n)2 n−1/3 for some constant C2 > 0
using (10.23) and Proposition 3.2. Using Lemma 10.5 in Supplement B, we can show that (when the event
ψ̂nα̂n ∈ MRK occurs)
 B1
, for some constant B1 > 0,
HB , Fea , k · kB,P0 ≤

with Fea = D̃−1 Fa with D̃  K 0 = K2 log n. Also, for any element f˜ = D̃−1 f ∈ Fe we have that
kf˜kB,P0 ≤ B2 D̃−1 v = C2 (log n)n−1/3 = δn , for some constant C2 > 0.
Let IIa,i be the term corresponding to i-th component of X. Using Markov’s inequality we have for a fixed

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

26

A > 0, ν > 0 and n large enough that




P |IIa,i | ≥ An−1/2 = P

!
|IIa,i | ≥ An−1/2 ,

sup

sup ψ̂nα (αT x) ≤ K

+ ν/2

α∈B(α0 ,δ0 ) x∈X



D̃
Jn (δn )
. Jn (δn ) 1 + √ 2 + ν/2, where Jn is defined in (10.3)
A
nδn
!
log n
B2
1/2
.
B2 δ n
1 + √ 3/2 + ν/2, for some constant B2 > 0, using the inequality in (10.4)
A
nδn


1
B3
−3/2
+ ν/2, with B3 = B2 C2
. (log n)3/2 n−1/6 1 +
A
(log n)3/2
≤ν
forn large enough. We conclude that IIa,i = op (n−1/2 ) which in turn implies that
IIa = op (n−1/2 ).

We turn now to IIb . Using Lemma 10.9 in Supplement C and a Taylor expansion of β 7→ ψα S(β)T x we
get,





ψα S(β)T x = ψ0 S(β0 )T x + (β − β0 )T JS (β0 )T x − E(X|S(β0 )T X = S(β0 )T x) ψ00 S(β0 )T x
+ o(β − β0 ),

(10.26)

so that
Z n 


o n

o

IIb = JS (β̂n )T
E X|S(β̂n )T x − Ēn,β̂n S(β̂n )T x
ψ0 S(β0 )T x − ψα̂n S(β̂n )T x
dP0 (x, y)


= op β̂n − β0
using consistency of β̂n . We next consider the term IIc . Using uniform boundedness of JS on C and the
inequality in (10.23) it follows that
Z n
o2

kIIc k .
dG(x)
ψα̂n α̂Tn x − ψ̂nα̂n α̂Tn x
= Op ((log n)2 n−2/3 ) = op (n−1/2 )
uniformly in β ∈ C. We conclude that (10.24) can be written as
Z n

o n

o
T
φn (β̂n ) = JS (β̂n )
x − E X|S(β̂n )T x
y − ψ̂nα̂n S(β̂n )T x
dPn (x, y)


+ op n−1/2 + (β̂n − β0 )
Z n

o n

o
T
= JS (β̂n )
x − E X|S(β̂n )T x
y − ψα̂n S(β̂n )T x
dPn (x, y)
Z n

o n



o
+ JS (β̂n )T
x − E X|S(β̂n )T x
ψα̂n S(β̂n )T x − ψ̂nα̂n S(β̂n )T x
dPn (x, y)


+ op n−1/2 + (β̂n − β0 )


= Ia + Ib + op n−1/2 + (β̂n − β0 ) .
(10.27)

We show below that Ib = op n−1/2 + (α̂n − α0 ) such that the limiting distribution of the score estimator
follows from the analysis of the term Ia which can be rewritten as
Z n

o n

o
T
Ia = JS (β̂n )
x − E X|S(β̂n )T x
y − ψα̂n S(β̂n )T x
d(Pn − P0 )(x, y)
Z n

o n

o
dP0 (x, y)
(10.28)
+ JS (β̂n )T
x − E X|S(β̂n )T x
y − ψα̂n S(β̂n )T x

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

27


where we recall that ψα (u) = E ψ0 (αT X|αT X = u . For the second term on the right-hand side of (10.28)
we have by (10.26)
Z n

o

o n
T
dP0 (x, y)
x − E X|S(β̂n )T x
y − ψα̂n S(β̂n )T x
JS (β̂n )
(
Z

 
 T
= − JS (β0 )T ψ00 S(β0 )T x x − E X|S(β0 )T x
x − E X|S(β0 )T x
dP0 (x, y)
)
× JS (β0 ) (β̂n − β0 )
+ op (β̂n − β0 ).

(10.29)

For the first term on the right-hand side of (10.28) we have that
Z n

o

o n
d(Pn − P0 )(x, y)
x − E X|S(β̂n )T x
y − ψα̂n S(β̂n )T x
JS (β̂n )T
Z

 

= JS (β0 )T
x − E X|S(β0 )T x
y − ψ0 S(β0 )T x d(Pn − P0 )(x, y) + op (n−1/2 ) + op (β̂n − β0 ).
(10.30)
Indeed, since this amounts to showing that

T Z n

o n

o
A = JS (β̂n ) − JS (β0 )
x − E X|S(β̂n )T x
y − ψα̂n S(β̂n )T x
d(Pn − P0 )(x, y)
= op (β̂n − β0 ),

Z  


T
T
B=
E X|S(β̂n ) x − E X|S(β0 ) x (y − ψ0 (αT0 x))d(Pn − P0 )(x, y) = op (n−1/2 )

(10.31)
(10.32)

and
Z 
C=



T

x − E X|S(β̂n ) x

 

ψ0 (αT0 x)



T

− ψα̂n S(β̂n ) x



d(Pn − P0 )(x, y) = op (n−1/2 ).

(10.33)



We start by proving (10.31). Using again that u 7→ E Xi |S(β̂n )T X = u is a bounded function with a
uniformly bounded total variation, and that xi is a fixed (and deterministic) function, we can show that
the class of functions involved in A, FA say, satisfies FA ⊂ xi HRC1 v + HRC1 v with v and C1 are some
constants that are independent of n (since ψα , X and u 7→ E[X|αT X = u] are all bounded by constants

eRC v , k · kB,P . 1/ with
independent of n). Now it follows by Lemma 10.5 in Supplement B that HB , H
1
0
eRC v = (16M0 C1 )−1 HRC v and kh̃kB,P . C2 for some constant C2 > 0 that is independent of n for all
H
1
1
0
eRC v . Hence, using arguments similar to those of the proof of IIa = op (n−1/2 ) we can show that
h̃ ∈ H
1
Z n

o n

o
x − E X|S(β̂n )T x
y − ψα̂n S(β̂n )T x
d(Pn − P0 )(x, y) = Op (n−1/2 ).
Using a Taylor expansion of JS (β) around β0 gives the desired rate in (10.31).
Now we turn to term B in (10.32). Fix ν > 0 and i ∈ {1, . . . , d}. Using consistency of β̂n and Lemma
10.11 in Supplement C, then for all η > 0 there exists n large enough such that



E Xi |S(β̂n )T x − E Xi |S(β0 )T x ≤ η,

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

28

with probability at least 1 − ν/2. Thus, for L > 0 we have that for n large enough
P (|Bi | > Ln−1/2 )
 Z  




T
T
T
−1/2
>
Ln
=P
E Xi |S(β̂n ) x − E Xi |S(β0 ) x (y − ψ0 (α0 x))d(Pn − P0 )(x, y)
1
E[kGn kF 0 ], where F 0 is defined in (10.49)
L
C1
η for some constant C1 > 0,
≤ ν/2 +
L
≤ ν/2 +

where Bi denotes the ith component of B defined in (10.32) and where we have used the result of Lemma
10.8. Choosing η such that η ≤ νLC1−1 /2 gives the claimed rate of convergence in (10.32).
To establish the convergence rate of C, we first note that, for i ∈ {1, . . . , d}, we have that x 7→
E(Xi |S(β̂n )T X = S(β̂n )T x)ψα̂n (S(β̂n )T x) belongs to the class GRC1 − GRC1 for some constant C1 > 0 where
GRK was defined in (10.40). This follows from using again the fact that the function u 7→ E Xi |S(β)T X = u
is uniformly bounded and has a uniform total variation for all β ∈ C, that ψα is a bounded monotone function, and the fact that (f1 − f2 )f3 is a bounded function with bounded total variation for any increasing and
bounded functions f1 , f2 and f3 , where we again use Lemma 10.13 in Supplement C to write the function
u 7→ E Xi |S(β)T X = u as the difference f1 − f2 . Note now that both x 7→ xi and x 7→ ψ0 (S(β0 )T x) are
fixed and bounded functions, and that the order bracketing entropy of a class does not get altered after multiplication its members by such functions (similarly for addition). It follows from Lemma 10.3 and Lemma
10.4 in Supplement B, that the -bracketing entropy of the class of functions involved in term C with respect
to k · kP0 is bounded above by B/ for some constant B.
Furthermore, using consistency of α̂n and Lemma 10.11 of Supplement C, we can find for any fixed ν > 0
an η > 0 such that supx |ψ0 (αT0 x) − ψα̂n (α̂Tn x)| ≤ η with probability at least 1 − ν/2 for n large enough.
Hence, at the cost of increasing the constant B, both the k · k∞ and k · kP0 norms of the functions of the
class involved in term C are bounded above by Bη. Using Markov’s inequality and Lemma 3.4.2 of [23] it
follows that for all L > 0
P (|Ci | > Ln−1/2 )

Z 

 


xi − E Xi |S(β̂n )T x
ψ0 (S(β0 )T x) − ψα̂n S(β̂n )T x d(Pn − P0 )(x, y) ≥ Ln−1/2
=P




1
Jn (Bη)
1
B1 1
√
≤ ν/2 + Jn (Bη) 1 + Bη √ 2 2 ≤ ν/2 +
B1 η 1/2 +
≤ν
L
L
B n
nB η
taking η small enough and n large enough. We conclude that C = op (n−1/2 ). Now we come back to term Ib
given by
Z n

o n



o
T
Ib = JS (β̂n )
x − E X|S(β̂n )T x
ψα̂n S(β̂n )T x − ψ̂nα̂n S(β̂n )T x
dPn (x, y).
Note first that
Ib = JS (β̂n )T

Z n

o n



o
x − E X|S(β̂n )T x
ψα̂n S(β̂n )T x − ψ̂nα̂n S(β̂n )T x
d(Pn − P0 )(x, y)

= JS (β̂n )T Ib0 ,

(10.34)

since
Z n

o n



o
x − E X|S(β̂n )T x
ψα̂n S(β̂n )T x − ψ̂nα̂n S(β̂n )T x dP0 (x, y)
h

 



i
= E X − E X|S(β̂n )T X
ψα̂n S(β̂n )T X − ψ̂nα̂n S(β̂n )T X



i
h h


i
= E E X − E X|S(β̂n )T X |S(β̂n )T X ψα̂n S(β̂n )T X − ψ̂nα̂n S(β̂n )T X
= 0.
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

29

Let Fb denote the class of functions involved in term Ib0 defined in (10.34), where in the definition of this class
we consider the event where ψ̂nα̂n is bounded. Given the arguments used recurrently above we can directly
state that the -bracketing entropy of this class is no larger than A1 log n/ for some constant A1 > 0 with
increasing probability. Also, the k · k∞ and k · kP0 norms of the members of the class Fb are respectively
bounded above with increasing probability by A1 log n and A1 log n n−1/3 = ηn at the cost of taking a larger
A1 . For a fixed ν > 0 and L > 0 we have for i ∈ {1, . . . , d},using Lemma 3.4.2 of [23],

Z n


o n


o
xi − E Xi |S(β̂n )T x
ψα̂n S(β̂n )T x − ψ̂nα̂n S(β̂n )T x
d(Pn − P0 )(x, y) > Ln−1/2
P
!
1/2
A2
A2 (log n)1/2 ηn
1/2 1/2
√ 2
≤ ν/2 +
(log n) , for some constant A2 > 0
(log n) ηn
1+
L
nηn
!
A2
A2 (log n)3/2
1/2 1/2
≤ ν/2 +
(log n) ηn
1 + √ 3/2
, for some constant A2 > 0
L
nηn
!
A2
A2
−1/6
(log n)n
. ν/2 +
1 + 3/2 ≤ ν,
L
A
1

for n large enough. This implies that Ib = op (n−1/2 ). We conclude by (10.29), (10.30), (10.31), (10.32),
(10.33) and Definition (10.18) that,

 Z



T 
B β̂n − β0 = (JS (β0 )) x − E(X|S(β0 )T x) y − ψ0 S(β0 )T x d Pn − P0 (x, y)


+ op n−1/2 + kβ̂n − β0 k ,
where
h
i
T
B = (JS (β0 )) E ψ00 (S(β0 )T X) Cov(X|S(β0 )T X) (JS (β0 )) .
We get,
Z
 √



√ 
T 
n β̂n − β0 = nB −1 (JS (β0 )) x − E(X|S(β0 )T x) y − ψ0 S(β0 )T x d Pn − P0 (x, y)


√
+ op 1 + nkβ̂n − β0 k
→d N (0, Π),
where
T

Π = B −1 (JS (β0 )) Σ JS (β0 ) B −1 ∈ R(d−1)×(d−1) .
The asymptotic limiting distribution of the single index score estimator α̂n now follows by an application of
the Delta method and we conclude that
√



√
√
T
n(α̂n − α0 ) = JS (β0 ) n(β̂n − β0 ) + op
n(β̂n − β0 ) →d Nd 0, JS (β0 )Π (JS (β0 )) .
Finally, the result of Theorem 4.1 follows by Lemma 4.1. This completes the proof.
Acknowledgements
The research of the third author was supported by the Research Foundation Flanders (FWO) [grant number
11W7315N]. Support from the IAP Research Network P7/06 of the Belgian State (Belgian Science Policy) is
gratefully acknowledged. For the simulations we used the infrastructure of the VSC - Flemish Supercomputer
Center, funded by the Hercules Foundation and the Flemish Government - department EWI.
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

30

References
[1] Balabdaoui, F., Durot, C. and Jankowski, H. (2016). Least squares estimation in the monotone
single index model. arXiv preprint arXiv:1610.06026.
[2] Balabdaoui, F., Groeneboom, P. and Hendrickx, K. (2017a). Score estimation in the monotone
single index model: supplement. working paper.
[3] Balabdaoui, F., Groeneboom, P. and Hendrickx, K. (2017b). Score estimation in the monotone
single index model. working paper.
[4] Barlow, R. E., Bartholomew, D. J., Bremner, J. M. and Brunk, H. D. (1972). Statistical
inference under order restrictions. The theory and application of isotonic regression. John Wiley &
Sons, London-New York-Sydney Wiley Series in Probability and Mathematical Statistics. MR0326887
(48 ##5229)
[5] Cavanagh, C. and Sherman, R. P. (1998). Rank estimators for monotonic index models. J. Econometrics 84 351–381. MR1630210
[6] Cosslett, S. R. (1987). Efficiency bounds for distribution-free estimators of the binary choice and the
censored regression models. Econometrica 55 559–585. MR890854 (88e:62087)
[7] Cosslett, S. R. (2007). Efficient estimation of semiparametric models by smoothed maximum likelihood. Internat. Econom. Rev. 48 1245–1272. MR2375624
[8] Delecroix, M., Härdle, W. and Hristache, M. (2003). Efficient estimation in conditional singleindex regression. Journal of Multivariate Analysis 86 213–226.
[9] Duan, N. and Li, K.-C. (1991). Slicing regression: a link-free regression method. The Annals of Statistics 505–530.
[10] Groeneboom, P. and Hendrickx, K. (2017). Current status linear regression. To appear in Annals
of Statistics, available at https://arxiv.org/abs/1601.00202.
[11] Groeneboom, P. and Jongbloed, G. (2014). Nonparametric Estimation under Shape Constraints.
Cambridge Univ. Press, Cambridge.
[12] Han, A. K. (1987). Non-parametric analysis of a generalized regression model: the maximum rank
correlation estimator. Journal of Econometrics 35 303–316.
[13] Härdle, W., Hall, P. and Ichimura, H. (1993). Optimal smoothing in single-index models. Ann.
Statist. 21 157–178. MR1212171 (94d:62134)
[14] Härdle, W. and Stoker, T. M. (1989). Investigating smooth multiple regression by the method of
average derivatives. Journal of the American statistical Association 84 986–995.
[15] Hooke, R. and Jeeves, T. A. (1961). “Direct Search”Solution of Numerical and Statistical Problems.
Journal of the ACM (JACM) 8 212–229.
[16] Ichimura, H. (1993). Semiparametric least squares (SLS) and weighted SLS estimation of single-index
models. Journal of Econometrics 58 71–120.
[17] Klein, R. W. and Spady, R. H. (1993). An efficient semiparametric estimator for binary response
models. Econometrica 61 387–421. MR1209737 (93k:62214)
[18] Kuchibhotla, A. K. and Patra, R. K. (2016). Efficient Estimation in Single Index Models through
Smoothing splines. available at https://arxiv.org/abs/1612.00068.
[19] Sherman, R. P. (1993). The limiting distribution of the maximum rank correlation estimator. Econometrica 61 123–137. MR1201705
[20] Tanaka, H. (2008). Semiparametric least squares estimation of monotone single index models and its
application to the iterative least squares estimation of binary choice models Technical Report.
[21] Torczon, V. (1997). On the convergence of pattern search algorithms. SIAM J. Optim. 7 1–25.
MR1430554
[22] van der Vaart, A. W. (1998). Asymptotic statistics. Cambridge Series in Statistical and Probabilistic
Mathematics 3. Cambridge University Press, Cambridge. MR1652247 (2000c:62003)
[23] van der Vaart, A. W. and Wellner, J. A. (1996). Weak convergence and empirical processes.
Springer Series in Statistics. Springer-Verlag, New York. With applications to statistics. MR1385671
(97g:60035)
[24] Xia, Y. and Härdle, W. (2006). Semi-parametric estimation of partially linear single-index models.
Journal of Multivariate Analysis 97 1162–1184.

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

31

Score estimation in the single index model: supplement
We give the proofs of the remaining results given in the main manuscript that were not considered in
Section 10 of [3] together with additional technical lemmas needed for proving our main results.
Supplement A: Asymptotic normality of the efficient score estimator
In this section we prove (iii) of Theorem 5.1 on the asymptotic normality of the efficient score estimator
α̃n . The proofs of existence and consistency of α̃n , given in (i) and (ii) of Theorem 5.1 follow the same lines
as the corresponding proofs for the simple score estimator α̂n given in Sections 10.2.1 and 10.2.1 and are
omitted.
Proof of asymptotic normality: Let τi denote the sequence of jump points of the monotone LSE ψ̂nα .
We introduce the piecewise constant function ρ̄n,β defined for u ∈ [τi , τi+1 ) as

0
(τi )
if ψα (u) > ψ̂nα (τi ) for all u ∈ (τi , τi+1 ),
 E[X|S(β)T X = τi ]ψα
T
0
ρ̄n,β (u) =
E[X|S(β) X = s]ψα (s)
if ψα (s) = ψ̂nα (s) for some s ∈ (τi , τi+1 ),

0
E[X|S(β)T X = τi+1 ]ψα
(τi+1 ) if ψα (u) < ψ̂nα (τi ) for all u ∈ (τi , τi+1 ).
We can write,
ξnh (β̃n )
Z n





o n

o
0
0
xψ̃nh,α
S(β̃n )T x − E X|S(β̃n )T x ψα̃
S(β̃n )T x
y − ψ̂nα̃n S(β̃n )T x
dPn (x, y)
n
Z n 




o n

o
0
T
T
T
+ JS (β̃n )T
E X|S(β̃n )T x ψα̃
S(
β̃
)
x
−
ρ̄
S(
β̃
)
x
y
−
ψ̂
S(
β̃
)
x
dPn (x, y)
n
n
n
α̃
n
n
n,β̃n
n
= JS (β̃n )T

= J + JJ,

(10.35)

using,
Z


n

o
ρ̄n,β̃n S(β̃n )T x y − ψ̂nα̃n S(β̃n )T x
dPn (x, y) = 0.

The term JJ can be written as
Z n 



o

T
T
0
S(
β̃
)
x
−
ρ̄
S(
β̃
)
x
JJ = JS (β̃n )T
E X|S(β̃n )T x ψα̃
n
n
n,β̃n
n
n

o
· y − ψ̂nα̃n S(β̃n )T x
d(Pn − P0 )(x, y)
Z n 




o
0
T
T
+ JS (β̃n )T
E X|S(β̃n )T x ψα̃
S(
β̃
)
x
−
ρ̄
S(
β̃
)
x
n
n
n,β̃n
n
n

o
· y − ψα̃n S(β̃n )T x
dP0 (x, y)
Z n 


o


0
T
T
+ JS (β̃n )T
E X|S(β̃n )T x ψα̃
S(
β̃
)
x
−
ρ̄
S(
β̃
)
x
n
n
n,β̃n
n
n



o
· ψα̃n S(β̃n )T x − ψ̂nα̃n S(β̃n )T x
dP0 (x, y)
= JJa + JJb + JJc ,

(10.36)

0
0
We first note that by Assumption A10, the functions u 7→ ψα
(u) := ψS(β)
(u) are uniformly bounded and
have a total variation that is uniformly
bounded
for
all
β
∈
C.
This
also
implies,
using Lemma 10.12, that the
 0
functions u 7→ E Xi |S(β)T X = u ψα
(u) have a bounded variation for all β ∈ C. Using the same arguments
as those for term IIa defined in (10.25) in the proof of Theorem 4.1, it easily follows that,

JJa = op (n−1/2 ).

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

32

0
We next consider the term JJb . By Lemma 10.14 we know that ψα
stays away from zero for all S(β) in a
neighborhood of S(β0 ). Using the same techniques as in [11], we can find a constant K > 0 such that for all
i = 1, . . . , d and u ∈ Iα ,
 0
(10.37)
E Xi |S(β)T X = u ψα
(u) − ρ̄ni,β (u) ≤ K ψα (u) − ψ̂nα (u) ,

0
where ρ̄ni,β denotes the ith component of ρn,β . This implies that the difference E Xi |S(β)T X = u ψα
(u) −
ρ̄ni,β (u) converges to zero for all u ∈ Iα . Using Lemma 10.9 and a Taylor expansion of β 7→ ψα S(β)T x
we get,





ψα S(β)T x = ψ0 S(β0 )T x + (β − β0 )T JS (β0 )T x − E(X|S(β0 )T X = S(β0 )T x) ψ00 S(β0 )T x

+ o(β − β0 ),
such that
Z n 



o

0
S(β̃n )T x − ρ̄n,β̃n S(β̃n )T x
E X|S(β̃n )T x ψα̃
n
n

o

· ψ0 S(β0 )T x − ψα̃n S(β̃n )T x
dP0 (x, y)


= op β̃n − β0 .

JJb = JS (β̃n )T

For the therm JJc , we get by an application of the Cauchy-Schwarz inequality together with the uniform
boundedness of JS , Proposition 3.2 and (10.37) that,
!1/2
Z n 




o2
T
T
0
T
T
JJc ≤ JS (β̃n )
E X|S(β̃n ) x ψα̃n S(β̃n ) x − ρ̄n,β̃n S(β̃n ) x
dP0 (x, y)
·

!1/2
Z n



o2
T
T
ψα̃n S(β̃n ) x − ψ̂nα̃n S(β̃n ) x
dP0 (x, y)

Z n



o2
.
ψα̃n α̃Tn x − ψ̂nα̃n α̃Tn x
dG(x) = Op (log n)2 n−2/3 = op (n−1/2 ).
We conclude that (10.35) can be written as
ξnh (β̃n )
Z n





o
0
T
T
0
T
= JS (β̃n )
xψ̃nh,
α̃n S(β̃n ) x − E X|S(β̃n ) x ψα̃n S(β̃n ) x
n

o
· y − ψ̂nα̃n S(β̃n )T x
dPn (x, y)


+ op n−1/2 + (β̃n − β0 )
T

Z n





o
0
T
T
0
T
xψ̃nh,
S(
β̃
)
x
−
E
X|S(
β̃
)
x
ψ
S(
β̃
)
x
n
n
n
α̃n
α̃n
n

o
· y − ψα̃n S(β̃n )T x
dPn (x, y)
Z n





o
0
T
T
0
T
+ JS (β̃n )T
xψ̃nh,
α̃n S(β̃n ) x − E X|S(β̃n ) x ψα̃n S(β̃n ) x
n



o
· ψα̃n S(β̃n )T x − ψ̂nα̃n S(β̃n )T x
d(Pn − P0 )(x, y)
Z n





o
0
T
T
0
T
+ JS (β̃n )T
xψ̃nh,
α̃n S(β̃n ) x − E X|S(β̃n ) x ψα̃n S(β̃n ) x
n



o
dP0 (x, y)
· ψα̃n S(β̃n )T x − ψ̂nα̃n S(β̃n )T x


+ op n−1/2 + (β̃n − β0 )


= Ja + Jb + Jc + op n−1/2 + (β̃n − β0 ) .
(10.38)
= JS (β̃n )T

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

33

We first consider the term
A10, Lemma 10.12 and Lemma 10.15 we get that the functions
 0Jb . By Assumption
0
u 7→ E X|S(β)T x = u ψα̃
(u)
and
u
→
7
ψ̃
nh,α̃n (u) have a uniformly bounded total variation for all β ∈ C.
n
Using similar arguments as for the term Ib defined in (10.27) we get for A > 0 and ν > 0 that
P (|Jb | ≥ An−1/2 ) ≤ ν,
for n large enough and we conclude that Jb = op (n−1/2 ). For the term Jc we get,
Z n


o

T
0
T
Jc = JS (β̃n )
x − E X|S(β̃n )T x ψ̃nh,
α̃n S(β̃n ) x
n



o
· ψα̃n S(β̃n )T x − ψ̂nα̃n S(β̃n )T x
dP0 (x, y)
Z n

o 



T
T
0
0
E X|S(β̃n )T x
ψ̃nh,
+ JS (β̃n )T
α̃n S(β̃n ) x − ψα̃n S(β̃n ) x
n



o
· ψα̃n S(β̃n )T x − ψ̂nα̃n S(β̃n )T x
dP0 (x, y)
Z n



o 

0
T
0
T
ψ̃nh,
= JS (β̃n )T
E X|S(β̃n )T x
α̃n S(β̃n ) x − ψα̃n S(β̃n ) x
n



o
· ψα̃n S(β̃n )T x − ψ̂nα̃n S(β̃n )T x
dP0 (x, y)
Furthermore, let Hβ be the distribution function of the random variable S(β)T X and let E(X|u) denote the
conditional expectation of X given S(β)T X = u, then
Z n
o
n
o
0
0
(u)
E
(X|u)
ψ
ψ̃nh,
(u)
−
ψ
(u)
−
ψ̂
(u)
dHβ̃n (u)
α̃
n
α̃
α̃n
α̃n
n
n

Z  Z
n
o
1
0
(u)
E
(X|u)
ψ
(u)
−
ψ̂
(u)
dHβ̃n (u)
=
K ({u − v}/h) dψ̂nα̃n (v) − ψα̃
α̃
n
α̃
n
n
n
h

Z  Z
o
o
n
n
1
0
(u)
−
ψ̂
(u)
dHβ̃n (u)
=
(v)
dv
E
(X|u)
ψ
K
({u
−
v}/h)
ψ̂
(v)
−
ψ
α̃
n
α̃
α̃
n
α̃
n
n
n
n
h2

Z  Z
n
o
1
0
0
+
K ({u − v}/h) ψα̃
(v)
dv
−
ψ
(u)
E
(X|u)
ψ
(u)
−
ψ̂
(u)
dHβ̃n (u).
α̃
n
α̃
α̃
n
n
n
n
h


The last term on the right hand side is Op n−2/7−1/3 = op n−1/2 . This follows by an application of the
Cauchy-Schwarz inequality since
)1/2
(Z  Z
2


1
0
0
K ({u − v}/h) ψα̃n (v) dv − ψα̃n (u)
dHβ̃n (u)
= Op n−2/7 ,
h
and
Z 

1/2
2


ψα̃n (u) − ψ̂nα̃n (u) dHβ̃n (u)
= Op n−1/3 .



The first term on the right hand side is Op n1/7−2/3 = op n−1/2 using that for small h
Z  Z
n
o 
n
o
1
0
K
({u
−
v}/h)
ψ̂
(v)
−
ψ
(v)
dv E (X|u) ψα̃n (u) − ψ̂nα̃n (u) dHβ̃n (u)
nα̃n
α̃n
2
h
Z
2
1 
.
ψα̃n (u) − ψ̂nα̃n (u) dHβ̃n (u).
h

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

34

We conclude that (10.38) can be written as,
Z n

o




T
T
T
0
0
T
xψ̃nh,
ξnh (β̃n ) = JS (β̃n )
α̃n S(β̃n ) x − E X|S(β̃n ) x ψα̃n S(β̃n ) x
n

o


· y − ψα̃n S(β̃n )T x
dPn (x, y) + op n−1/2 + (β̃n − β0 )
Z n



o n

o
T
0
T
0
T
= JS (β̃n )
x ψ̃nh,
y − ψα̃n S(β̃n )T x
d(Pn − P0 )(x, y)
α̃n S(β̃n ) x − ψα̃n S(β̃n ) x
Z n

o n

o


T
T
0
0
y − ψα̃n S(β̃n )T x
dP0 (x, y)
+ JS (β̃n )T x ψ̃nh,
α̃n S(β̃n ) x − ψα̃n S(β̃n ) x
Z n


o
n

o
0
T
T
x − E X|S(β̃n )T x ψα̃
S(
β̃
)
+ JS (β̃n )T
x
y
−
ψ
S(
β̃
)
x
d(Pn − P0 )(x, y)
n
α̃
n
n
n
Z n

n

o

o
0
S(β̃n )T x y − ψα̃n S(β̃n )T x
dP0 (x, y)
x − E X|S(β̃n )T x ψα̃
+ JS (β̃n )T
n


+ op n−1/2 + (β̃n − β0 )


= JJJa + JJJb + JJJc + JJJd + op n−1/2 + (β̃n − β0 ) .
(10.39)
0
0
and ψ̃nh,α
We consider JJJa first and note that by Assumption A10 and Lemma 10.15, the functions ψα
have a uniformly bounded total variation. By an application of Lemma 10.13 we can write the difference
0
0
ψ̃nh,α
− ψα
as the difference of two monotone functions, say f1 , f2 ∈ MRC1 for some constant C1 > 0. This
implies that the class of functions
(
)



0
T
0
T
T
F1 = f (x, y) := {ψ̃nh,α S(β) x − ψα S(β) x }{y − ψα S(β) x }, (x, y, β) ∈ X × R × C ,

is contained in the class HRC1 v where v  h−1 log nn−1/3 (See the proof of Lemma 10.15). By Lemma 10.5
and the fact that the order bracketing entropy of a class does not get altered after multiplication with the
fixed and bounded function x 7→ xi we get that the class of functions involved with the term JJJa , say Fa ,
satisfies
 1
HB , Fa , k · kB,P0 .


and

kf kB,P0 . v

Using again an application of Markov’s inequality, together with Lemma 3.4.3 of [23] we conclude that for
A>0
P (|JJJa | > An−1/2 ) . v 1/2 = h−1/2 (log n)1/2 n−1/6 ,
which can be made arbitrarily small for n large enough and h  n−1/7 . We conclude that
JJJa = op (n−1/2 ).
Using similar arguments as for the term JJb defined in (10.36) we also get,


JJJb = op β̃n − β0 .
The result of Theorem 5.1 follows by noting that, using the same techniques as for the term Ia in (10.30),
we get
Z




T
JJJc = (JS (β0 ))
x − E X|S(β0 )T x ψ00 S(β0 )T x y − ψ0 S(β0 )T x d(Pn − P0 )(x, y)
+ op (n−1/2 ) + op (β̂n − β0 ),

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model


and that by a Taylor expansion of β 7→ ψα S(β)T x we get,
(
Z
2 
 

T
JJJd = − (JS (β0 ))
ψ00 S(β0 )T x
· x − E X|S(β0 )T x
x − E X|S(β0 )T x

35

T

!
dP0 (x, y)

)
× JS (β0 ) (β̃n − β0 ) + op (β̃n − β0 ).
The rest of the proof follows the same line as the proof of asymptotic normality of the simple score estimator
defined in Theorem 4.1 and is omitted.
Supplement B: Entropy results
Lemma 10.2. Fix  > 0, and consider F1 a class of functions defined on X × R bounded by some constant
A > 0 and equipped by the L2 norm k · kP0 with respect to P0 . Also, let F2 be another class of continuous
functions defined on a bounded set C ⊂ Rd−1 such that F2 is equipped by the supremum norm k · k∞ , and
bounded by some constant B > 0. Moreover assume that HB (, F1 , k · kP0 ) < ∞ and HB (, F2 , k · k∞ ) < ∞.
Consider
n
o
F = F1 F2 = f (x) = fβ (x, y) = f1 (x, y)f2 (β) : (x, y, β) ∈ X × R × C .
Then there exists some constant B > 0 such that
HB (, F, k · kP0 ) ≤ HB (B, F1 , k · kP0 ) + HB (B, F2 , k · k∞ ).

Proof. Let f = f1 f2 ∈ F for some pair (f1 , f2 ) ∈ F1 × F2 . For  > 0 consider the (f1L , f1U ) and (f2L , f2U )
-brackets with respect to k · kP0 for f1 and f2 . Note that since F1 and F2 are bounded by M = max(A, B)
we can always assume that −M ≤ fiL ≤ fiU ≤ M for i ∈ {1, 2}. As we deal with a product of two functions,
construction of a bracket for f requires considering different sign cases for a given pair (x, β):
1.
2.
3.
4.
5.
6.
7.
8.
9.

0 ≤ f1L (x) and 0 ≤ f2L (β),
0 ≤ f1L (x), f2L (β) < 0 and f2U (β) ≥ 0,
f1L (x) ≤ 0, f1U (x) ≥ 0 and 0 ≤ f2L (β),
f U (x) ≤ 0, f L (β) ≥ 0,
f L (x) ≥ 0, f U (β) ≥ 0,
f1L (x) ≤ 0, f U (x) ≥ 0, f2L (β) ≤ 0 and f2U (β) ≥ 0,
f1L (x) ≤ 0, f U (x) ≥ 0 and f2U (β) ≤ 0,
f U (x) ≤ 0, f L (β) ≤ 0 and f U (β) ≥ 0,
f1U (x) ≤ 0 and f2U (β) ≤ 0.

We can assume without loss of generality that each one these cases occur for all x ∈ X and β ∈ C since the
general case can be handled by considering the 9 different subsets of X × C. In the proof, we will restrict
ourselves to making the calculations explicit for cases 1 and 2 since the remaining cases can be handled very
similarly. Then, f1L f2L ≤ f ≤ f1U f2U . Also, we have that




f1U f2U − f1L f2L = f1U − f1L f2U + f1L f2U − f2L .
Recall that M = max(A, B). Then, it follows that
Z 
2
Z 
2

f1U f2U − f1L f2L dP0 ≤ 2M
f1U − f1L dP0 (x) + kf2U − f2L k2∞
X

X

≤ 4M 2 .

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

36

This in turn implies that HB (, F, k · kP0 ) ≤ HB (C, F1 , k · kP0 ) + HB (C, F2 , k · k∞ ) with C = (2M )−1 . Now
we consider case 2. It is not difficult to show that
f2L f1U ≤ f ≤ f1U f2U .
Hence,
Z 

f1U f2U − f2L f1U

2

dP0 ≤ A2 kf2U − f2L k2∞ ≤ A2 2

X

and we can take C = A−1 .

Lemma 10.3. Let F be a class of functions satisfying HB (, F, k · kP0 ) < ∞ for every  ∈ (0, 0 ) for some
given 0 > 0. If D = F − F the class of all differences of elements of F, then
HB (, D, k · kP0 ) ≤ 2HB (/2, F, k · kP0 ).

Proof. Let  ∈ (0, 0 ) and d = f2 − f1 denote an element in D with (f1 , f2 ) ∈ F 2 . Also, let (f1L , f1U ) and
(f2L , f2U ) -brackets for f1 and f2 . Define dL = f2L − f1U and dU = f2U − f1L . It is clear that (dL , dU ) is a
bracket for d. Furthermore, we have that
Z
2
dU (x, y) − dL (x, y) dP0 (x, y)
X
Z

Z
2
2
≤2
f1U (x, y) − f1L (x, y) dP0 (x, y) +
f2U (x, y) − f2L (x, y) dP0 (x, y)
X

X

≤ 42 .
Thus,



2
exp HB (2, D, k · kP0 ) ≤ exp HB (, F, k · kP0 ) ,
which is equivalent to the statement of the lemma.
Consider the class GRK defined as
n
o
GRK = g : g(x) = gα (x) = ψ(αT x), x ∈ X , (ψ, α) ∈ MRK × B(α0 , δ0 ) .

(10.40)

where MRK is the same class defined in (10.7).
Lemma 10.4. There exists A > 0 such that for  ∈ (0, K) we have that
HB (, GRK , k · kP0 ) ≤

AK
.


Proof. See the proof of Lemma 4.9 in [1].

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

37

Lemma 10.5. For some constants C > 0 and δ > 0 consider the class of functions
n
o
2
DRCδ = d : d = f1,α − f2,α , (f1,α , f2,α ) ∈ GRC
, kd(αT ·)kP0 ≤ δ for all α ∈ B(α0 , δ0 ) .
Let HRCv be a class of functions such that
n
o
2
HRCv = h : h(x, y) = yd1 (αT x) − d2 (αT x), (x, y, α) ∈ X × R × B(α0 , δ0 ), (d1 , d2 ) ∈ DRCv
(10.41)
where C ≥ K0 ∨ 1. Then, for all  ∈ (0, C) we have that




e k · kB,P ≤ HB C̃ −1 , HRCv , k · kP ≤ C̃C  1 ,
HB , H,
0
0


and that
kh̃kB,P0 . D̃−1 v,
where
0



A = A 2(a0 M0 + 1)

−1/2

1
, D̃ = 16M0 C and C̃ =
8M0



−1
1
2a0 + e(2M0 )
2

1/2

1
,
C

(10.42)

e def
with a0 , M0 the same constants from Assumption A6, A the same constant in Lemma 10.4, and H
=
HRCv D̃−1 .
U
L U
T
T
Proof. Consider (dL
1 , d1 ) and (d2 , d2 ) to be -brackets of the functions x 7→ d1 (α x) and x 7→ d2 (α x)
and some α ∈ B(α0 , δ0 ). It follows from Lemma 4.9 of [1] and Lemma 10.3 that there exists some constant
A > 0 such that

 AC
HB , DRC , k · kP0 ≤
.


Define now
(
U
ydL
1 (x) − d2 (x), if y ≥ 0
hL (x, y) =
U
ydU
1 (x) − d2 (x), if y < 0
and
(
L
ydU
1 (x) − d2 (x), if y ≥ 0
h (x, y) =
L
ydL
1 (x) − d2 (x), if y < 0.
U

Note first that (hL , hU ) is a bracket for h(x, y) = yd1 (αT x) − d2 (αT x). Next we compute the size of this
bracket with respect to k · kP0 . We have that
Z
Z 

2
nZ

2
2
o
L
L
hU (x, y) − hL (x, y) dP0 (x, y) ≤ 2
y 2 dU
(x)
−
d
(x)
dP
(x,
y)
+
dU
0
1
1
2 (x) − d2 x) dG(x)
X ×R
X ×R
Z 
Z  X
n
2
2
o
U
L
L
= 2 2a0
d1 (x) − d1 (x) dG(x) +
dU
(x)
−
d
x)
dG(x)
2
2
X
X
 2
≤ 2 2a0 + 1  ,
where a0 is the same constant of Assumption A6. It follows that

 ÃC
HB , H, k · kP0 ≤
,

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

38


−1/2
with Ã = A 2(2a0 + 1)
and A is the same constant of Lemma 10.4. Let now D > 0 be some constant
to be determined later. For a given h ∈ HRK 2 v , we consider h̃ = D−1 h which admits [D−1 hL , D−1 hU ] as
bracket. We will compute the size of this bracket with respect to the Bernstein norm. By definition of the
latter we can write for any function h such that hk is P0 integrable that
khk2B,P0 = 2

∞
X
1 k
|h| dP0 .
k!

k=2

Thus, using this and convexity of the function x 7→ |x|k for all k ≥ 2 it follows that
Z
∞
X
k

1
L
U
L
y dU
kD−1 hU − D−1 hL k2B,P0 = 2
1 (x) − d1 (x) + d2 (x) − d2 (x) dP0 (x, y)
k
k!D X ×R
k=2


Z
Z
∞
X 2k−1
k
k
k U
L
U
L
≤2
|y|
d
(x)
−
d
(x)
dP
(x,
y)
+
d
(x)
−
d
(x)
dP
(x,
y)
.
0
0
1
1
2
2
k!Dk
X ×R
X ×R
k=2

2
U
Using Assumption A7 and the fact that |dL
i | ≤ K and |di | ≤ 2C for i ∈ {1, 2} (an assumption that one can
always make in constructing brackets for a bounded class) we can write
 k 
Z
∞
X
2
1 2
L
a0 M0k−2 k!(4C)k−2
dU
kD−1 hU − D−1 hL k2B,P0 ≤
1 (x) − d1 (x) dP0 (x, y)
k! D
X
k=2

Z
2
L
+(4C)k−2
dU
(x)
−
d
(x)
dP
(x,
y)
0
1
1
X


k−2 
∞
X
1 8C
=
a0
+
2
D
k! D
k=2
k=2
k

k 
 2  X
∞
∞ 
X
1 8C
8M0 C
1
2
a0
+
2 , using k! ≥ 2(k − 2)!.
≤
D
D
2
k! D


2
D

2 

k−2
∞ 
X
8M0 C

k=0

k=0

Taking D = D̃ = 16M0 C yields
kD̃−1 hU − D̃−1 hL k2B,P0 ≤



2
D̃

2 

−1
1
2a0 + e(2M0 )
2 ,
2

which in turn implies that
kD̃−1 hU − D̃−1 hL kB,P0 ≤

1
8M0


1/2
−1
1
1
.
2a0 + e(2M0 )
2
C

e with D̃ defined as above.
This completes the proof of the first claim about the entropy bound of the class H
e
Now for a given element h̃ ∈ H we calculate
Z
∞
X
1 1
k
2
kh̃kB,P0 = 2
yd1 (αT x) − d2 (αT x) dP0 (x, y)
k
k!
D̃
X ×R
k=2
Z
∞
n
o
k−1
X2
1
k
k
k
≤ 2
y d1 (αT x) + d2 (αT x) dP0 (x, y)
D̃k k! X ×R
k=2


Z
Z
∞
X
2k−1 1
2
2
≤ 2
(2C)k−2 a0 M0k−2 k!
d1 (αT x) dP0 (x, y) +
d2 (αT x) dP0 (x, y)
D̃k k!
X ×R
X ×R
k=2
k−2 X

k−2 )
 2 ( X
∞ 
∞
8M0 C
1 8C
2
≤
a0
+
v 2 using the definition of the class
k! D̃
D̃
D̃
k=2
k=2
 2 

−1
2
1
≤
2a0 + e(2M0 )
v 2 using arguments as above,
2
D̃
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

39

implying that

1/2
−1
1
1
kh̃kB,P0 ≤ 2 2a0 + e(2M0 )
v . D̃−1 v,
2
D̃
as claimed.

Recall that X is the support of the covariates Xi , i = 1, . . . , n. Let us denote by Xj , j = 1, . . . , d the set
of the j-th projection of x ∈ X . Also, consider some function s that d − 1 times continuously differentiable
on a convex and bounded set C ∈ Rd−1 with a nonempty interior such that there exists M > 0 satisfying
max sup |Dk s(β)| ≤ M

(10.43)

k.≤d−1 β∈C

where k = (k1 , . . . , kd ) with kj an integer ∈ {0, . . . , d − 1}, k. =
Dk ≡

Pd−1
i=1

ki and

∂ k. s(β)
.
∂βk1 . . . ∂βkd

Consider now the class
n
o
QjRC = qj (x, y) = s(β)xj (y − ψ(αT x)), (α, β, ψ) ∈ B(α0 , δ0 ) × C × MRC and (xj , y) ∈ Xj × ∈ R (10.44)
.
Define
n
o
ejRC = q̃j : q̃j = qj D̃−1 , qj ∈ QRC ,
Q
where D̃ > 0 is some appropriate constant.
1/2

Lemma 10.6. Let  ∈ (0, 1) and C ≥ max(1, 2M0 , M e−1/4 2−1/2 R−1 , 2a0 e−1/2 ). Then, there exist some
constant B1 > 0 and B2 depending on a0 , M0 , and R such that


ejRC , k · kB,P ≤ B1 C , kq̃j kB,P ≤ B2 ,
HB , Q
0
0

if D̃ = 8M RC where a0 and M0 are the same positive constants in Assumption A6, and M is from (10.43).

Proof. Fix j ∈ {1, . . . , d}. The proof of this lemma uses similar techniques as in showing Lemma 10.5. Let
(g L , g U ) be -brackets for the class GRC . Using the result of Lemma 10.4 we know that there are at most
N ≤ exp(AC/) such brackets covering GRC for some constant A > 0. Define
 


  xj (y − g L (x)), xj (y − g U (x)) , if xj ≥ 0


k L (x, y), k U (x, y) =
(10.45)
 xj (y − g U (x)), xj (y − g L (x)) , if xj < 0 .
Then, the collection of all possible pairs (q L , q U ) form brackets for the class of functions
n
o
KjRC ≡ kj (x, y) = xj (y − ψ(αT x)), (α, ψ) ∈ B(α0 , δ0 ) × MRC and (xj , x, y) ∈ Xj × X × R .
Furthermore we have that
U

kk −

k L k2P0

Z

2
x2j g U (x) − g L (x) dG(x)
X
Z
2
g U (x) − g L (x) dG(x) ≤ R2 2 .
≤ kxk22
=

X

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

40

This implies that

 ARC
HB , KjRC , k · kP0 ≤
,

where A is the same constant of Lemma 10.4. Furthermore, the assumption in (10.43) implies that the
d−1
function s belongs to CM̃
as defined in Section 2.7 in [23], with M̃ = 2M . Using now Theorem 2.7.1 of [23]
it follows that there exists some constant B > 0 such that
 d/(d−1)


B
1
d−1
log N , CM̃
≤ .
, k · k∞ ≤ B


This also implies that



 2B
d−1
d−1
HB , CM̃
, k · k∞ = log N /2, CM̃
.
, k · k∞ ≤


d−1
d−1
Indeed, for an arbitrary s ∈ CM̃
there exists si , i ∈ {1, . . . , N }, with N = N /2, CM̃
, k · k∞ , such that
d−1
ks − si k∞ ≤ /2. The claim follows from noting that (si − /2, si + /2) is an -bracket for CM̃
with respect
to k · k∞ . Using Lemma 10.2 it follows that there exists some constant L > 0 such that




2LC
1 C
+
,
(10.46)
≤
HB , QjRC , k · kP0 ≤ L



using that C ≥ 1, d − 1 ≥ 1 and  ∈ (0, 1). Consider now a constant D > 0, and (q L , q U ) and -bracket. From
the proof of Lemma 10.2 we know that we can restrict attention to the case for example to case 1 assumed to
occur for all (x, β) ∈ X × C. In such that we have q L = sL k L and q U = sU k U where (sL , sU ) is an -bracket
1
for CM̃
equipped with k · k∞ , where the expression of (k L , k U ) is given in (10.45). We can now write
kD−1 q U − D−1 q L k2B,P0

=
≤

Z
∞
X
1 1
k
sU k U − sL k L dP0
k! Dk X ×R
k=2


Z
∞
X


2k 1
L U
L k
U
U
L k
dP0
s k −k | + k s −s
k! Dk X ×R

2

k=2

with
Z

sU k U − k L

X ×R



k

dP0 ≤ M k (2RC)k−2

Z

2
k U − k L dP0 = M 2 (2M CR)k−2 2

X ×R

where we used the fact that |s| ≤ M by assumption of the lemma (implying that we can constructs brackets
(sL , sU ) satisfying the same property), and k U − k U = xj (g U − g L ) ≤ 2RC. Also, if we assume without loss
of generality that xj ≥ 0 is satisfied for all x ∈ X we have that
Z
Z

L U
L k
k−2
k s −s
dP0 ≤ (2M )
xj (y − g L (x))|k dP0 (x, y) × 2
X ×R
X ×R
Z
n
o
k
k−2 k k−1
≤ (2M )
R 2
|y|k + g L (x)
dP0 (x, y) × 2
X ×R


≤ (2M )k−2 Rk 2k−1 a0 M0k−2 k! + C k 2 .
Putting these inequalities together and after some algebra we get
kD−1 q U − D−1 q L k2B,P0
!

2

2

2
1 2M
2RC
2R
1
4M CR/D
8M CR/D
≤
e
+
e
+ 2a0
) 2 .
2
D
D
D
1 − 8M M0 R/D
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

41

Now let us choose D̃ = D ≥ max(16M M0 R, 8M RC). In particular, we can assume that C is large enough
so that max(16M M0 R, 8M RC) = 8M RC = D̃ (or equivalently C ≥ 2M0 ). Then, 4M CR/D̃ = 1/2,
8M CR/D̃ = 1/4, and 8M M0 R/D̃ = M0 /C ≤ 1/2. Therefore,

2

2

2 !
2RC
2R
1 2M
−1 U
−1 L 2
1/2
kD̃ q − D̃ q kB,P0 ≤
e +
e + 4a0
2
2
D̃
D̃
D̃
 1

2
=
2M 2 e1/2 + 4R2 e C 2 + 16a0 R2
D̃2
ÃC 2 2
Ã
 =
2 ,
≤
2
64M 2 R2
D̃
if C is large enough, where Ã = 2M 2 e1/2 + 4R2 e + 16a0 R2 . It follows that we can find some constant L̃ > 0
such that
kD̃−1 q U − D̃−1 q L kB,P0 ≤ L̃.
This in turn implies that


e jRC , k · kB,P ≤ HB , QjRC , k · kP . 2M C ,
HB L̃, Q
0
0

using (10.46). Hence, we can find a constant B1 > 0 such that

e jRC , k · kB,P ≤ B1 C .
HB , Q
0

Now we turn to computing an upper bound for kq̃j kB,P0 . We have
kq̃j k2B,P0

=
≤

Z
∞
X
k
1 −k
D
|s(β)|k xj y − ψ(αT x) dP0 (x, y)
k!
X ×R
k=2
Z
∞
n
o
X 1
k
k
y + ψ(αT x) dP0 (x, y)
2k D−k (RM )k
k!
X ×R

2

k=2
∞
X



1 k −k
2 D (RM )k a0 M0k−2 k! + C k
k!
k=2

2 ∞ 
k−2

2 X

k−2
∞
1
2M R X 2RM M0
1 2M RC
2RM C
≤ a0
+
D
D
2
D
(k − 2)!
D
k=2
k=2
2 X







∞
k
2 ∞
k
1
1
1 1 X 1 1
≤ a0
+
2C
4
2 2
k! 2
k=0
k=0

2
 2
1
3 1 1
≤ a0
+
e1/2
if D = 4M RC and C ≥ max(1, 2M0 ).
2C
4 2 2
≤

The proof of the lemma is complete if we write B2 = (3a0 /16 + e1/2 /8)1/2 .

In the next lemma, we consider a given a class of functions F which admits a bounded bracketing entropy
with respect to k · kP0 for  ∈ (0, 1]. Suppose also that there exists D > 0 such that kf k∞ ≤ D and δ > 0
such that kf kP0 ≤ δ for all f ∈ F. Then we can derive an upper bound for the bracketing entropy for the
class
n


o
Fe =
f˜ : f˜(x, y) = (4M0 D)−1 f (x) y − λψ0 (αT0 x) , (x, y) ∈ X × R and f ∈ F
(10.47)
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

42

with respect to the Bernstein norm. Here, M0 is the same constant from Assumption A6 and D̃ is a positive
constant that will be determined below.
Lemma 10.7. Let F be a class of functions satisfying the conditions above. Then,
e k · kB,P ) ≤ HB (D̃−1 , F, k · kP ), and kf˜kB,P ≤ D̃δ
HB (, F,
0
0
0
where

D̃ =

λ2 K02 λK0 (2M0 )−1
a0
+
e
2
2M0
8M02

1/2

D−1 ,

(10.48)

and a0 , M0 are the same constants from Assumption A6.

Proof. Let (L, U ) be an -bracket for F with respect to k · kP0 . Consider the class
n


o
F 0 = f 0 : f 0 (x, y) = f (x) y − λψ0 (αT0 x) , (x, y) ∈ X × R and f ∈ F .
Then for f 0 ∈ F 0 we have
L(x)(y − λψ0 (αT0 x)) ≤ f 0 (x, y) ≤ U (x)(y − λψ0 (αT0 x)), if y − λψ0 (αT0 x) ≥ 0 or
U (x)(y − λψ0 (αT0 x)) ≤ f 0 (x, y) ≤ L(x)(y − λψ0 (αT0 x)), if y − λψ0 (αT0 x) < 0 .
Let (L0 , U 0 ) denote the new bracket. Using the definition of the Bernstein norm, convexity of x 7→ xk , k ≥ 2
and kψ0 k∞ ≤ K0 we have that
2

(U 0 − L0 )(4M0 D)−1 B,P0
Z
∞
X
(4M0 D)−k
=2
(U (x) − L(x))k |y − λψ0 (αT0 x)|k dP0 (x, y)
k!
X ×R
k=2
∞


−k Z
X
(4M0 D)
≤2
(U (x) − L(x))k 2k−1 |y|k + λk |ψ0 (αT0 x)|k dP0 (x, y)
k!
X ×R
k=2
Z
∞
X

1
(U (x) − L(x))k |y|k + λk K0k dP0 (x, y)
≤
k
k
k
2 D M0 k! X ×R
k=2
Z
∞
X
1
≤
(U (x) − L(x))k (a0 k!M0k−2 + λk K0k )dP0 (x, y)
k D k M k k!
2
X ×R
0
k=2
k−2 Z
∞
∞ 
X 1 Z
a0
λ2 K02 X λK0
1
2
≤
(U (x) − L(x)) g(x)dx +
(U (x) − L(x))2 g(x)dx
2
2
2
k−2
2
4M0 D
2
4D M0
2M0
k! X ×R
X
×R
k=2
k=2

k−2
∞
2 2 X
a0
λ K0
1
λK0
≤
2 +
2
2M02 D2
8D2 M02
(k − 2)! 2M0
k=2


2 2
a0
λ K0 λK0 (2M0 )−1 2
+
e
 = D̃2 2 .
≤
2M02 D2
8D2 M02
This implies that




e k · kB,P ≤ HB , F, k · kP ,
HB D̃, F,
0
0
or equivalently




HB , F 0 , k · kB,P0 ≤ HB D̃−1 , F, k · kP0 .
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

43

Using similar calculations we can write
kf˜k2B,P

=
≤

≤

2

∞
X

1
1
k
(4M0 D) k!

Z

1
1
(2M0 )k k!

Z

k=2
∞
X

1
D2

k=2

a0
4M02 D2

∞
X
k=2

|f (x)|k |y − λψ0 (αT0 x)|k dP( x, y)

X ×R

1
2k−2

X ×R

f (x)2 (a0 k!M0k−2 + λk K0k )g(x)dx

k−2
∞ 
λ2 K02 X λK0
1
+
2
2
8D M0
2M0
(k − 2)!
k=2

!Z

f (x)2 g(x)dx

X ×R

≤ D̃2 δ 2 ,
which completes the proof.

In the next corollary, we consider the class
n
o
F = x 7→ fα (x) = Ei,α0 (αT0 x) − Ei,α (αT x), x ∈ X , α ∈ B(α0 , δ) ,

where Ei,α (u) = E Xi |αT X = u for i ∈ {1, . . . , d} and δ ∈ (0, δ0 ). Using the same arguments in the proof
of Lemma 10.11 with f (x) = xi it follows that for all x ∈ X and α, α0 ∈ B(α0 , δ)
|fα0 (x) − fα (x)| ≤ M kα0 − αk,
for the same constant M of that lemma. Now, we can apply Theorem 2.7.11 of [23] to conclude that


NB 2, F, k · kP0 ≤ N , B(α0 , δ), k · k ,

where N , B(α0 , δ), k · k is the -covering number for B(α0 , δ) with respect to the norm k · k which is of
order (δ/)d for  ∈ (0, δ). Hence, using the inequality log(x) ≤ x for x > 0 we can find a constant M 0 > 0
depending on d such that
 M 0δ
HB , F, k · kP0 ≤
.

Furthermore, there exists M̃ > 0 such that kf k∞ ≤ M̃ δ and kf kP0 ≤ M̃ δ.
Lemma 10.8. Let F be the class of functions as above and consider the related class
n


o
F 0 = f 0 : f 0 (x, y) = f (x) y − λψ0 (αT0 x) , (x, y) ∈ X × R, f ∈ F .

(10.49)

Then,
E[kGn kF 0 ] . δ.

Proof. Note that for any function f 0 ∈ F 0 and constant C > 0 we have that Gn (f 0 C −1 ) = C −1 Gn f 0 implying
that kGn kF 0 = 4M0 M̃ δkGn kFe , where
n
o
Fe = f˜ : f˜(x, y) = (4M0 M̃ δ)−1 f 0 (x, y), f 0 ∈ F 0 .
Note also that the constant D̃ in Lemma 10.7 is given by D̃  δ −1 , where D̃ depends on M̃ , a0 , M0 and K0 .
Also, using the entropy calculations along with Lemma 10.7 we can show easily that
e k · kB,P ) . 1 ,
HB (, F,
0

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

44

and that kf˜kB,P0 . 1. Using Lemma 3.4.3 of [23] it follows that there exists some constant B > 0 such that


Jn
E[kGn kFe ] . Jn 1 + √ 2 ,
nB
RBp
with Jn = 0
1 + B/d. Hence, E[kGn kFe ] . 1 and E[kGn kF 0 ] . δ as claimed.
Supplement C: Auxiliary results
Proof of Lemma 4.1. We have:
n
o−1
T
T
T
T
(JS (β0 )) AJS (β0 ) (JS (β0 )) AJS (β0 )
(JS (β0 )) AJS (β0 ) = (JS (β0 )) AJS (β0 ).
In the parametrization that we consider, the columns of JS (β0 ) are orthogonal to α0 . We can therefore
extend the matrix JS (β0 ) with a last column α0 to a square nonsingular matrix J¯S (β0 ). This leads to the
equality
n
o−1
T
T
T
T
J¯S (β0 ) AJS (β0 ) (JS (β0 )) AJS (β0 )
(JS (β0 )) AJ¯S (β0 ) = J¯S (β0 ) AJ¯S (β0 ).
Multiplying on the left by



T −1
J¯S (β0 )
and on the right by J¯S (β0 )−1 , we get:

n
o−1
T
T
(JS (β0 )) A = A.
AJS (β0 ) (JS (β0 )) AJS (β0 )

(10.50)

n
o−1
T
T
(JS (β0 )) is a generalized inverse of A.
This shows that JS (β0 ) (JS (β0 )) AJS (β0 )
To complete the proof and show that it is indeed the Moore-Penrose inverse of A, we first note that
n
o−1
n
o−1
T
T
T
T
JS (β0 ) (JS (β0 )) AJS (β0 )
(JS (β0 )) AJS (β0 ) (JS (β0 )) AJS (β0 )
(JS (β0 ))
n
o−1
T
T
= JS (β0 ) (JS (β0 )) AJS (β0 )
(JS (β0 )) .
(10.51)
Furthermore,


T
n
o−1
T
T
AJS (β0 ) (JS (β0 )) AJS (β0 )
(JS (β0 ))

n
o−1
T
T
= JS (β0 ) (JS (β0 )) AJS (β0 )
(JS (β0 )) AT
n
o−1
T
T
= JS (β0 ) (JS (β0 )) AJS (β0 )
(JS (β0 )) A,
where the last equality holds since A is symmetric, being a covariance matrix. We have to show that
o−1
n
T
T
JS (β0 ) (JS (β0 )) AJS (β0 )
(JS (β0 )) A
n
o−1
T
T
= AJS (β0 ) (JS (β0 )) AJS (β0 )
(JS (β0 )) .
(10.52)
Multiplying on the left by (JS (β0 ))T and on the right by JS (β0 ), we get:
n
o−1
T
T
(JS (β0 ))T JS (β0 ) (JS (β0 )) AJS (β0 )
(JS (β0 )) AJS (β0 )
= (JS (β0 ))T JS (β0 )
n
o−1
T
T
= (JS (β0 ))T AJS (β0 ) (JS (β0 )) AJS (β0 )
(JS (β0 )) JS (β0 ),
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

45

and (10.52) follows by the orthogonality relation of the columns of JS (β0 ) with α0 in the same way as before,
replacing the matrix JS (β0 ) by J¯S (β0 ) in the outer factors of the equality relation.
In a similar way we obtain:

T
n
o−1
T
T
JS (β0 ) (JS (β0 )) AJS (β0 )
(JS (β0 )) A
n
o−1
T
T
= JS (β0 ) (JS (β0 )) AJS (β0 )
(JS (β0 )) A.

(10.53)

n
o−1
T
T
Since the matrix JS (β0 ) (JS (β0 )) AJS (β0 )
(JS (β0 )) satisfies properties (10.50), (10.51),(10.52) and
(10.53), the matrix satisfies the four properties which define the Moore-Penrose pseudo-inverse matrix of A.
This completes the proof of Lemma 4.1.
Remark 10.1. The same proof holds for showing that the Moore-Penrose inverse Ã is given by
n
o−1
T
T
JS (β0 ) (JS (β0 )) ÃJS (β0 )
(JS (β0 )) .
Lemma 10.9 (Derivative α 7→ ψα (αT x)).
∂
ψα (αT x)
∂αj

α=α0


= xj − E(Xj |αT X = αT0 x) ψ00 (αT0 x),

and
∂
ψα (αT x)
∂βj

where JS (β0 )T


j

∂
ψS(β) (S(β)T x)
∂βj
β=β0



T
= JS (β0 ) j x − E(X|S(β)T X = S(β)T x) ψ00 S(β)T x ,

=
α=α0

denotes the jth row of JS (β0 )T .

Proof. We assume without loss of generality that the first component α1 of α is not equal to zero. Denote
the conditional density of (X2 , . . . , Xd )T given αT X = u by hα (·|u) Using the change of variables t1 = αT x,
tj = xj for j = 1, . . . , d, the function ψα can be written as
ψα (αT x) = E[ψ0 (αT0 X)|αT X = αT x]


Z
d
d
Y
X
α
01
T
T


dx̃j ,
= ψ0
α0j x̃j hα (x̃2 , . . . , x̃d |α x)
(α x − α2 x̃2 − . . . − αd x̃d ) +
α1
j=2
j=2
with partial derivatives w.r.t. αj for j = 2, . . . , d given by,
∂
∂
ψα (αT x) =
E[ψ0 (αT0 X)|αT X = αT x]
∂αj
∂αj


Z
d
d
X
Y
α01
0  α01
T
T

=
(xj − x̃j )ψ0
(α x − α2 x̃2 − . . . − αd x̃d ) +
α0j x̃j hα (x̃2 , . . . , x̃d |α x)
dx̃j
α1
α1
j=2
j=2


Z
d
d
X
Y
α01 T
∂
+ ψ0 
(α x − α2 x̃2 − . . . − αd x̃d ) +
α0j x̃j 
hα (x̃2 , . . . , x̃d |αT x)
dx̃j ,
α1
∂αj
j=2
j=2
which is at α = α0 equal to
∂
ψα (αT x)
∂aj

Z
=
α=α0

d
Y

(xj − x̃j )ψ0T αT0 x hα0 (x̃2 , . . . , x̃d |αT0 x)
dx̃j
j=2

=

ψ00 (αT0 x)



xj −

E(Xj |αT0 X

=

αT0 x)

.

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

46

For the partial derivatives w.r.t. α1 we have,
∂
ψα (αT x)
∂α1



Z 
d
X
α
α01
α01 T
01
=
x1 − 2 (α x − α2 x̃2 − . . . − αd x̃d ) ψ00 
(αT x − α2 x̃2 − . . . − αd x̃d ) +
α0j x̃j 
α1
α1
α1
j=2
hα (x̃2 , . . . , x̃d |αT x)

d
Y

dx̃j

j=2


d
d
T
X
Y
∂
α
x
−
α
x̃
−
.
.
.
−
α
x̃
2
2
d
d
+
h(x̃2 , . . . , x̃d |αT x)
(α0j − αj )x̃j 
dx̃j ,
ψ0 αT x + (α01 − α1 )
α1
∂α1
j=2
j=2


Z
+
and,

∂
ψα (αT x)
∂a1

α=α0


= ψ00 (αT0 x) x1 − E(X1 |αT0 X = αT0 x) .

This proves the first result of Lemma 10.9. The proof for the second results follows similarly and is omitted.
Lemma 10.10. Let φ̄ be defined by
Z
Z


φ̄(α) = x y − ψα (αT x) dP0 (x, y) = x ψ0 (αT0 x) − ψα (αT x) dG(x),

(10.54)

then we have for each α ∈ B(α0 , δ0 ),



φ̄(α) = E Cov X, ψ0 (αT X + (α0 − α)T X)|αT X .
Moreover,
αT φ̄(α) = 0,
and,



(α0 − α)T φ̄(α) = E Cov (α0 − α)T X, ψ0 (αT X + (α0 − α)T X)|αT X ≥ 0,
and α0 is the only value such that the above equation holds uniform in α ∈ B(α0 , δ0 ).
Proof. We have,
Z
φ̄(α) =


x y − ψα (αT x) dP0 (x, y) =

Z


x ψ0 (αT0 x) − ψα (αT x) dG(x)

Z




x ψ0 (αT0 x) − E ψ0 (αT0 X)|αT X = αT x dG(x)



= E Cov X, ψ0 (αT0 X)|αT X ,
=

(10.55)

and
α

T

Z







x ψ0 (αT0 x) − E ψ0 (αT0 X)|αT X = αT x dG(x) = E Cov αT X, ψ0 (αT0 X)|αT X = 0.

We next note that,



(α0 − α)T φ̄(α) = E Cov (α0 − α)T X, ψ0 (αT0 X)|αT X



= E Cov (α0 − α)T X, ψ0 (αT X + (α0 − α)T X)|αT X ,
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

47

which is positive by the monotonicity of ψ0 . This can be seen as follows. Using Fubini’s theorem, one can
prove that for any random variables X and Y such that XY, X and Y are integrable, we have
Z
Cov {X, Y } = EXY − EXEY = {P(X ≥ s, Y ≥ t) − P(X ≥ s)P(Y ≥ t)} ds dt.
Denote Z1 = (α0 − α)T X and Z2 = ψ0 (u + (α0 − α)T X) = ψ0 (u + Z1 ), then, using monotonicity of the
function ψ0 , we have
P(Z1 ≥ z1 , Z2 ≥ z2 ) = P(Z1 ≥ max{z1 , z̃2 }) ≥ P(Z1 ≥ max{z1 , z̃2 })P(Z1 ≥ min{z1 , z̃2 })
= P(Z1 ≥ z1 )P(Z2 ≥ z2 },
where
z̃2 = ψ0−1 (z2 ) − u = inf{t ∈ R : ψ0 (t) ≥ z2 } − u.
We conclude that,

Cov (α0 − α)T X, ψ0 (αT X + (α0 − α)0 X)|αT X = u
Z
= {P(Z1 ≥ z1 , Z2 ≥ z2 ) − P(Z1 ≥ z1 )P(Z2 ≥ z2 )} ds dt ≥ 0,
and hence the first part of the Lemma follows. We next prove the uniqueness of the parameter α0 . We start
by assuming that, on the contrary, there exists α1 6= α0 in B(α0 , δ0 ) such that
(α0 − α)T φ̄(α) ≥ 0

and (α1 − α)T φ̄(α) ≥ 0

for all α ∈ B(α0 , δ0 ),

and we consider the point α ∈ B(α0 , δ0 ) such that
|αj − αj0 | = |αj − αj1 |

for j = 1, . . . , d.

For this point, we have,
(α0 − α)T φ̄(α) = −(α1 − α)T φ̄(α)

for all α ∈ B(α0 , δ0 ),

which is not possible since both terms should be positive. This completes the proof of Lemma 10.10.

Lemma 10.11. Let f : X → Rk , k ≤ d be a differentiable function on X such that there exists a constant
M > 0 satisfying kf k∞ ≤ M . Then, under the assumptions A1 and A5 we can find a constant M̃ > 0 such
that for all α ∈ B(α0 , δ0 ) we have that
sup E[f (X)|αT X = αT x] − E[f (X)|αT0 X = αT0 x] ≤ M kα − α0 k.

x∈X

Proof. We can assume without loss of generality that α0,1 6= 0 where α0,1 is the first component of α0 . At
the cost of taking a smaller δ0 , we can further assume that α̃1 6= 0 for all α ∈ B(α0 , δ0 ). Consider the change
of variables t1 = αT X, ti = xi for i = 1, . . . , d. Then, the density of (αT X, X2 , . . . , Xd ) is given by
 

1
1
g(αT X,X2 ,...,Xd ) (t1 , . . . , td ) = g
t1 − α2 t2 − . . . − αd td ), t2 , . . . , td
.
α1
α1
Then, for i = 2, . . . , d, the conditional density hα (·|u) of the (d − 1)-dimensional vector (X2 , . . . , Xd ) given
that αT X = u is equal to


d xd
g u−α2 x2 −...−α
,
x
,
.
.
.
,
x
2
d
α1
Q
:= hα (x2 , . . . , xd |u),
(10.56)
R  u−α2 t2 −...−αd td
d
,
t
,
.
.
.
,
t
g
2
d
j=2 dtj
α1
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

48

where the domain of integration in the denominator is the set {(x2 , . . . , xd ) : (x, X )}. Note that X1 =
(αT X − α2 X2 − . . . − αd Xd )/α1 . Thus, for (x, X ) we have that
E[f (X)|αT X = αT x] = E[f (X1 , X2 , . . . , Xd )|αT X = αT x]
  T


α X − α2 X2 − . . . − αd Xd
T
T
=E f
, X2 , . . . , Xd | α X = α x
α1

Z  T
d
Y
α x − α2 x2 − . . . − αd xd
= f
, x2 , . . . , xd hα (x2 , . . . , xd |αT x)
dxj .
α1
j=2
Note now that function
g



αT x−α2 x2 −...−αd xd
, x2 , . . . , xd
α1



Q
,
α 7→ hα (x2 , . . . , xd |αT x) = R  T
d
g α x−α2 tα21−...−αd td , t2 , . . . , td
j=2 dtj
is continuously differentiable on B(α0 , δ0 ). This follows from assumptions A1 and A5 together with Lebesgue
dominated convergence theorem which allows us to differentiate the density g under the integral sign. With
some notation abuse we write ∂h/∂xi for the i-th partial derivative of α 7→ hα (x2 , . . . , xd |αT x). Straightforward calculations yield
 T

∂hα
α x − α2 x2 − . . . − αd xd
= g
, x2 , . . . , xd
∂α1
α1

Q
R Pd
d
∂g
αT x−α2 t2 −...−αd td
,
t
,
.
.
.
,
t
)
(x
−
t
)
2
d
i
i
j=2 dtj
i=2
∂x1
α1
,
×
R  T
Q
2
d
α12
g α x−α2 tα21−...−αd td , t2 , . . . , td
j=2 dtj
and for i = 2, . . . , d

αT x − α2 x2 − . . . − αd xd
= −g
, x2 , . . . , xd
α1
Q

R
d
∂g
αT x−α2 t2 −...−αd td
,
t
,
.
.
.
,
t
)
(xi − ti ) ∂x
2
d
j=2 dtj
α1
i
×
R  T
Q
2 .
d
α1
g α x−α2 tα21−...−αd td , t2 , . . . , td
j=2 dtj


∂hα
∂αi

Assumptions A1 and A5 allow us to find a constant D > 0 depending on R, c0 , c̄0 and c̄1 such that
∂hα
∂αi

∞

≤ D,

for i = 1, . . . , d. Consider now the function α 7→ E[f (X)|αT X = αT x]. Using the assumptions of the lemma
and applying again Lebesgue dominated convergence theorem we conclude for i ∈ {1, . . . , d} that we have
∂E[f (X)|αT X = αT x]
∂αi

Z  T
d
α x − α2 x2 − . . . − αd xd
∂hα (x2 , . . . , xd |αT x) Y
= f
, x2 , . . . , xd
dxj .
α1
∂αi
j=2
Furthermore, we have that
sup
(x,X

∂E[f (X)|αT X = αT x]
≤ MD
∂αi

Z Y
d

dxj = M 0 ,

j=2

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

49

for all i ∈ {1, . . . , d} and (x, X ) and α ∈ B(α0 , δ). The results now follow using a first order Taylor expansion
to obtain
E[f (X)|αT X = αT x] − E[f (X)|αT0 X = αT0 x] =

d
X
eT X = α
e T x]
∂E[f (X)|α
(αi − α0,i ) ,
∂αi
i=1

e ∈ Rd such that kα
e − α0 k ≤ kα − α0 k. Bounding the right side of the preceding display by
for some α
M̃ kα − α0 k with M̃ = dM 0 gives the result.

Lemma 10.12. Denote for i ∈ {1, . . . , d} the ith component of the function u 7→ E[X|αT X = u] by Ei,α .
Then Ei,α has a total bounded variation. Furthermore, there exists a constant B > 0 such that for all
α ∈ B(α0 , δ0 )
Z
0
kEi,α k∞ ≤ B, and
|Ei,α
(u)|du ≤ B,
Iα

where Iα = {αT x : x ∈ X }.
Proof. Since X ⊂ B(0, R), it is clear that kEi,α k∞ ≤ R. As above let us assume without loss of generality
that the first component of α0 is not equal to 0. At the cost of taking a smaller δ0 , we can further assume
that α̃1 6= 0 for all α ∈ B(α0 , δ0 ). We known that for i = 2, . . . , d
Z
Ei,α (u) = xi hα (x2 , . . . , xd |u)dx2 . . . dxd ,
where integration is done over the set {(x2 , . . . , xd ) : (x, X )} and u ∈ Iα ⊂ (a0 − δ0 R, b0 + δ0 R) and where
hα denotes conditional density of (X2 , . . . , Xd )0 given αT X = u, defined in (10.56). Using assumptions A1
and A5 along with the Lebesgue dominated convergence theorem we are allowed to write
Z
∂
0
Ei,α
(u) = xi hα (x2 , . . . , xd |u) dx2 . . . dxd .
∂u
Straightforward calculations yield that
∂
hα (x2 , . . . , xd |u)
∂u

=

∂g
∂x1

u−α2 x2 −...−αd xd
, x2 , . . . , xd
α1



R 
Q

d
d td
α1
,
t
,
.
.
.
,
t
dt
g u−α2 t2 −...−α
2
d
j
j=2
α1
 
R

Q
d
∂g
u−α2 t2 −...−αd td
1
g α1 u − α2 x2 − . . . − αd xd ), x2 , . . . , xd
, t2 , . . . , td
j=2 dtj
∂x1
α1
−
.
R 
2
Q
d
d td
α1
g u−α2 t2 −...−α
,
t
,
.
.
.
,
t
dt
2
d
j
j=2
α1

Thus, we can find constant C > 0 depending only on |α0,1 |, c0 , c1 , c̄1 and R such that
all α ∈ B(α0 , δ0 ). Now B = max(R, C) gives the claimed inequalities. If i = 1, then
E1,α (u) =

R

0
|Ei,α
(u)|du ≤ C for

d
d


X
X
1 
1 
u − αj
Ej,α (u) , and e01,α (u) =
1 − αj
e0j,α (u) .
α1
α1
j=2
j=2

for u ∈ Iα . We conclude again that the claimed inequalities are true at the cost of increasing the constant
B obtained above.

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

50

Lemma 10.13. Let f be a function defined on some interval [a, b] such that
kf k∞ ≤ M, V (f, [a, b]) :=

n
X

sup

|f (xj ) − f (xj−1 )| ≤ M,

a=x0 <x1 ...<xn =b j=1

for some finite constant M > 0. Then, there exist two non-decreasing functions f1 and f2 on [a, b] such that
kf1 k∞ , kf2 k∞ ≤ 2M and f = f2 − f1 .

Proof. The fact that f = f2 − f1 with f1 and f2 non-decreasing on [a, b] follows from the well-known
Jordan’s decomposition. Furthermore, we can take f1 (x) = V (f, [a, x]) and f2 (x) = f (x) − f1 (x)for (x, [a, b].
By assumption, kf1 k∞ ≤ M ≤ 2M and kf2 k ≤ kf k∞ + kf1 k∞ ≤ 2M .

Lemma 10.14. Under Assumptions A4-A5, we can find a constant C > 0 such that for all α close enough
to α0 we have that
0
ψα
(u) > C,

for all u ∈ Iα .
Proof. We assume again that a1 6= 0. By calculations similar to the calculations made in the proof of Lemma
10.9, we get


Z
d
d
X
Y
α01
α
01
ψα (u) =
ψ00 
α0j x̃j  hα (x̃2 , . . . , x̃d |u)
dx̃j
(u − α2 x̃2 − . . . − αd x̃d ) +
α1
α1
j=2
j=2


Z
d
d
Y
X
∂
α
01
dx̃j .
α0j x̃j 
h(x̃2 , . . . , x̃d |u)
+ ψ0 
(u − α2 x̃2 − . . . − αd x̃d ) +
α1
∂u
j=2
j=2
Now, a Taylor expansion of αi in the neighborhood of α0,i and using that α0,1 /α1 = 1 − 1 /α0,1 + o(1 ) yields


α0,1
ψ0
(u − α2 x2 − · · · − αd xd ) + α0,2 x2 + · · · + α0,d xd
α1


1
(u − 2 x2 − . . . − d xd ) + o(1 )
= ψ0 u −
α0,1
1
= ψ0 (u) −
(u − 2 x2 − . . . − d xd )ψ00 (u) + o(1 )
α0,1
1
= ψ0 (u) −
uψ 0 (u) + o(kα − α0 k).
α0,1 0
Using the Lebesgue dominated convergence theorem and the fact that hα (x̃2 , . . . , x̃d |u) is a conditional
density it follows that


Z
d
d
X
Y
∂
α
01
(u − α2 x̃2 − . . . − αd x̃d ) +
α0j x̃j 
h(x̃2 , . . . , x̃d |u)
dx̃j , = o(α − α0 ),
ψ0 
α1
∂u
j=2
j=2
such that
0
ψα
(u)



1
≥C 1−
α0,1


+ o(α − α0 ) ≥ C > 0,

provided that kα − α0 k is small enough.
imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

/single index model

51

Lemma 10.15. If h  n−1/7 , then there exists a constant B > 0 such that for all α ∈ B(α0 , δ)
Z
0
00
kψnh,α k∞ ≤ B
and
|ψnh,α
(u)|du ≤ B,
Iα

where Iα = {αT x : x ∈ X }
Proof. Using integration by
Z
1
0
ψnh,α
(u) =
h
Z
1
=
h
Z
1
=
h

parts and Proposition 3.2, we have for all u ∈ Iα


u−x
K
dψ̂nα (x)
h




Z
u−x
1
u−x
0
K
K0
ψα
(x)dx + 2
(ψ̂nα (x) − ψα (x))dx
h
h
h


Z
u−x
1
K
dψα (x) +
K 0 (w) (ψ̂nα (u + hw) − ψα (u + hw))dw
h
h

0
0
= ψα
(u) + O(h2 ) + Op (h−1 log nn−1/3 ) = ψα
(u) + op (1).

This proves the first part of Lemma 10.15. For the second part, we get by a similar calculation that,


Z
Z
u−x
1
1
00
00
K
ψα
(x)dx + 2
K 00 (w) (ψ̂nα (u + hw) − ψα (u + hw))dw
ψnh,α
(u) =
h
h
h


Z
1
u−x
00
=
K
ψα
(x)dx + Op (h−2 log nn−1/3 ).
h
h
Since h−2 log nn−1/3 = o(1) for h  n−1/7 , the second part follows by Assumption A10.

imsart-generic ver. 2014/10/16 file: BGH_SIM.tex date: December 18, 2017

