
\section{Introduction}
\label{section:intro}
\setcounter{equation}{0}
The monotone single index model tries to predict a response from the linear combination of a finite number of parameters and a function linking this linear combination to the response via a monotone {\it link function} $\psi_0$ which is  unknown. So, more formally, we have the model
\begin{align*}
Y=\psi_0(\bm\alpha_0^T\bm X)+\varepsilon,
\end{align*}
where $Y $ is a one-dimensional random variable, $\bm X = (X_1,\ldots, X_d)^T$ is a $d$-dimensional random vector with distribution function $G$,  $\psi_0$ is monotone  and $\varepsilon$ is a one-dimensional random variable such that ${\mathbb E}[\varepsilon | \bm X] = 0$ $G$-almost surely. For identifiability, the regression parameter $\bm\alpha_0$ is a vector  of norm $\|\bm\alpha_0\|_2=1$, where $\|\,  \cdot \|_2$ denotes the Euclidean norm in $\mathbb R^d$, so $\bm\alpha_0\in\mathcal{S}_{d-1}$, the unit $(d-1)$-dimensional sphere.

The ordinary profile least squares estimate of $\bm\alpha_0$ is an $M$-estimate in two senses: for fixed $\bm\alpha$ the least squares criterion
\begin{align}
\label{step1}
\psi\mapsto n^{-1}\sum_{i=1}^n\left\{Y_i-\psi(\bm\alpha^T\bm X_i)\right\}^2
\end{align}
is minimized for all monotone functions $\psi$ (either decreasing or increasing) which gives an $\bm\alpha$ dependent function $\hat\psi_{n,\bm\alpha}$, and the function
\begin{align}
\label{step2}
\bm\alpha\mapsto n^{-1}\sum_{i=1}^n\left\{Y_i-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm X_i)\right\}^2
\end{align}
is then minimized over $\bm\alpha$. This gives a profile least squares estimator $\hat\bm\alpha_n$ of $\bm\alpha_0$, which we will call LSE in the sequel.
Although this estimate of $\bm\alpha_0$ has been  known now for a very long time (more than 30 years probably), it is not known whether it is $\sqrt{n}$ convergent (under appropriate regularity conditions), let alone that we know its asymptotic distribution. Also, simulation studies are rather inconclusive. For example, it is conjectured in \cite{tanaka2008} on the basis of simulations that the rate of convergence of $\hat\bm\alpha_n$ is $n^{9/20}$. Other simulation studies, presented in \cite{BDJ:19}, are also inconclusive. In that paper, it was also proved that an ordinary least squares estimator (which ignores that the link function could be non-linear) is $\sqrt{n}$-convergent and asymptotically normal under elliptic symmetry of the distribution of the covariate $\bm X$. Another linear least squares estimator of this type, where the restriction on $\bm\alpha$ is $\bm\alpha^T\bm S_n\bm\alpha=1$, where $S_n$ is the usual estimate of the covariance matrix of the covariates, and where a renormalization at the end is not needed (as it is in the just mentioned linear least squares estimator) was studied in \cite{FGH:19} and there shown to have similar behavior.
If this suggests that the profile LSE should also be $\sqrt n$-consistent,  the extended simulation study in \cite{FGH:19} shows that it is possible to find other estimates which exhibit a better performance in these circumstances. 



An alternative way to estimate the regression vector is to minimize the criterion
\begin{align}
\label{SSE_criterion}
\bm\alpha\mapsto \left\|n^{-1}\sum_{i=1}^n\left\{Y_i-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm X_i)\right\}\bm X_i\right\|^2
\end{align}
over $\bm\alpha\in \mathcal{S}_{d-1}$, where $\|\cdot\|$ is the Euclidean norm. Note that this is the sum of $d$ squares. We prove in Section \ref{section:analysis_SSE} that this minimization procedure  leads to a $\sqrt{n}$ consistent and asymptotically normal estimator, which is a more precise and informative result compared to what we know now about  the LSE..  Using the well-known properties of isotonic estimators, it is easily seen that the function (\ref{SSE_criterion}) is piecewise constant as a function of $\bm\alpha$, with finitely many values, so the minimum exists and is equal to the infimum over $\bm\alpha\in \mathcal{S}_{d-1}$. Notice that this estimator does not use any tuning paramenters, just like the LSE.

In \cite{FGH:19}, a similar Simple Score Estimator (SSE) $\hat\bm\alpha_n$ was defined as a point $\bm\alpha\in\mathcal{S}_{d-1}$ where all components of the function
\begin{align*}
\bm\alpha\mapsto n^{-1}\sum_{i=1}^n\left\{Y_i-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm X_i)\right\}\bm X_i
\end{align*}
cross zero. If the criterion function were continuous in $\bm\alpha$, this estimator would have been the same as the least squares estimator, minimizing (\ref{SSE_criterion}), with a minimum equal to zero, but in the present case we cannot assume this because of the discontinuities of the criterion function.

The definition of an estimator as a crossing of the $d$-dimensional vector $\bm0$  makes it necessary to prove the existence of such an estimator, which we found to be a rather non-trivial task. Defining our estimator directly as the minimizer of (\ref{SSE_criterion}), so as a least squares estimator,  relieves us from the duty to prove its existence. Since our estimator is asymptotically equivalent to the SSE, we refer to it here under the same name.

A fundamental function in our treatment is the function $\psi_{\bm\alpha}$, defined as follows.

\begin{definition}
\label{def_psi_alpha}
{\rm
Let $\mathcal{S}_{d-1}$ denote again the boundary of the unit ball in $\mathbb R^d$. Then, for each $\bm\alpha\in \mathcal{S}_{d-1}$, the function $\psi_{\bm\alpha}:\mathbb R\to\mathbb R$ is defined as the nondecreasing function which minimizes
\begin{align*}
\psi\mapsto{\mathbb E}\{Y-\psi(\bm\alpha^T\bm X)\}^2
\end{align*}
over all nondecreasing functions $\psi:\mathbb R\to\mathbb R$. The existence and uniqueness of the function $\psi_{\bm\alpha}$ follows for example from the results in \cite{landers_rogge:81}.

The function $\psi_{\bm\alpha}$ coincides in a neighborhood of $\bm\alpha_0$ with the ordinary conditional expectation function $\tilde\psi_{\bm\alpha}$
\begin{align}
\label{conditional_exp}
\tilde\psi_{\bm\alpha}(u)={\mathbb E}\left\{\psi_0(\bm\alpha_0^T\bm X)|\bm\alpha^T\bm X=u\right\},\qquad u\in\mathbb R,
\end{align}
see \cite{FGH:19}, Proposition 1. The general definition of $\psi_{\bm\alpha}$ uses conditioning on a $\sigma$-lattice, and $\psi_{\bm\alpha}$ is also called a {\it conditional $2$-mean} (see \cite{landers_rogge:81}).
}
\end{definition}

\vspace{0.3cm}
The importance of the function $\psi_{\bm\alpha}$ arises from the fact that we can differentiate this function w.r.t.\ $\bm\alpha$, in contrast with the least squares estimate $\hat\psi_{n,\bm\alpha}$, and that $\psi_{\bm\alpha}$ represents the least squares estimate of $\psi_0$ in the underlying model for fixed $\bm\alpha$, if we use $\bm\alpha^T\bm x$ as the argument of the monotone link function.

It is also possible to introduce a tuning parameter and use an estimate of $\frac{d}{du}\psi_{\bm\alpha}(u)\bigr|_{u=\bm\alpha^T\bm X}$. This estimate is defined by:
\begin{align}
\label{estimate_derivative_psi}
\tilde\psi_{n,h,\bm\alpha}'(u)=\frac1h\int K\left(\frac{u-x}h\right)\,d \hat\psi_{n,\bm\alpha}(x),
\end{align}
where $K$ is one of the usual kernels, symmetric around zero and with support $[-1,1]$, and where $h$ is a bandwidth of order $n^{-1/7}$ for sample size $n$. For fixed $\bm\alpha$, the least squares estimate $\hat\psi_{n,\bm\alpha}$ is defined in the same way as above. 
Note that this estimate is rather different from the derivative of a Nadaraya-Watson estimate which is also used in this context and which is in fact the derivative of a ratio of two kernel estimates. If we use the Nadaraya-Watson estimate we need in principle two tuning parameters, one for the estimation of $\psi_0$ and another one for the estimation of the derivative $\psi'_0$.

Using the estimate (\ref{estimate_derivative_psi}) of the derivative we now minimize
\begin{align}
\label{ESE_criterion}
\bm\alpha\mapsto \left\|n^{-1}\sum_{i=1}^n\left\{Y_i-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm X_i)\right\}\bm X_i\,\tilde\psi_{n,h,\bm\alpha}'(\bm\alpha^T\bm X_i)\right\|^2
\end{align}
instead of (\ref{SSE_criterion}), where $\|\cdot\|$ is again the Euclidean norm. A variant of this estimator was defined in \cite{FGH:19} and called the Efficient Score Estimator (ESE) there, since, if the conditional variance $\text{var}(Y|\bm X=\bm x)=\sigma^2$, where $\sigma^2$ is independent of the covariate $\bm X$  (the homoscedastic model), the estimate is efficient. As in the case of the simple score estimator (SSE), the estimate was defined as a crossing of zero estimate in \cite{FGH:19} and not as a minimizer of (\ref{ESE_criterion}). But the definition as a minimizer of (\ref{ESE_criterion}) produces an asymptotically equivalent estimator. For reasons of space, we will only give a sketch of the proof of this statement below in Section \ref{sec:ESE_PLSE}.

The qualification ``efficient'' is somewhat dubious, since the estimator is no longer efficient if we do not have homoscedasticy. We give an example of that situation in Section \ref{section:simulations}, where, in fact, the SSE has a smaller asymptotic variance than the ESE.
Nevertheless, to be consistent with our treatment in \cite{FGH:19} we will call the estimate, $\hat\bm\alpha_n$, minimizing (\ref{ESE_criterion}), again the ESE. 

Dropping the monotonicity constraint, we can also use as our estimator of the link function a cubic spline $\hat\psi_{n,\bm\alpha}$, which is defined as the function minimizing
\begin{align}
\label{spline}
\sum_{i=1}^n\left\{\psi(\bm\alpha^T\bm X_i)-Y_i\right\}^2+\mu\int_a^b \psi''(x)^2\,dx,
\end{align}
over the class of functions ${\cal S}_2[a,b]$ of differentiable functions $\psi$ with an absolutely continuous first derivative, where
\begin{align*}
a=\min_i\bm\alpha^T\bm X_i,\qquad b=\max_i\bm\alpha^T\bm X_i,
\end{align*}
see \cite{green_silverman:94}, pp. 18 and 19, where $\mu>0$ is the penalty parameter. Using these estimators of the link function, the estimate $\hat\bm\alpha_n$ of $\bm\alpha_0$ is then found in \cite{arun_rohit:20} by using a $(d-1)$-dimensional parameterization $\beta$ and a transformation $S:\bm\beta\mapsto S(\bm\b)=\bm\alpha$, where $S(\bm\b)$ belongs to the surface of the unit sphere in $\mathbb R^d$, and minimizing the criterion
\begin{align*}
\bm\b\mapsto\sum_{i=1}^n\{Y_i-\hat\psi_{S(\bm\b),\mu}(S(\bm\b)^T\bm X_i)\}^2,
\end{align*}
over $\bm\b$, where $\hat\psi_{S(\bm\b),\mu}$ minimizes (\ref{spline}) for fixed $\bm\alpha=S(\bm\b)$.

Analogously to our approach above we can skip the reparameterization, and minimize instead:
\begin{align}
\label{spline_eq}
\left\|\frac1n\sum_{i=1}^n \bigl\{\hat\psi_{n,\bm\alpha,\mu}(\bm\alpha^T\bm X_i)-Y_i\bigr\}\bm X_i\,\tilde\psi_{n,\bm\alpha,\mu}'(u)\bigr|_{u=\bm\alpha^T\bm X_i}\right\|
\end{align}
where $\tilde\psi_{n,\bm\alpha,\mu}$ minimizes (\ref{spline}) for fixed $\bm\alpha$ and $\tilde\psi'_{n,\bm\alpha,\mu}$ is its derivative. We call this estimator the spline estimator.

We finally give simulation results for these different methods in Section \ref{section:simulations}, where, apart from the comparison with the spline estimator, we make a comparison with other estimators of $\bm\alpha_0$ not using the monotonicity constraint: the Effective Dimension Reduction (EDR) method, proposed in \cite{hristache01} and implemented in the {\tt R} package {\tt edr}, the (refined) MAVE (Mean Average conditional Variance Estimator) method, discussed in \cite{xia:06}, and implemented in the {\tt R} package {\tt MAVE}, and EFM (Estimation Function Method), discussed in \cite{cui2011}.


\section{General conditions and the functions $\hat\psi_{n,\hat\bm\alpha}$  and $\psi_{\hat\bm\alpha}$}
\label{section:psi_alpha}
\setcounter{equation}{0}
We  give general conditions that we assume to hold in the remainder of the paper here and give graphical comparisons of the functions $\hat\psi_{n,\bm\alpha}$  and $\psi_{\bm\alpha}$, where $\psi_{\bm\alpha}$ is defined in Definition \ref{def_psi_alpha}.

\vspace{0.3cm}
\begin{example}
\label{example_uniform_covariates}
{\rm
As an illustrative example we take $d=2$, $\psi_0(x)=x^3$, $\bm\alpha_0=(1/\sqrt{2},1/\sqrt{2})^T$, $Y_i=\psi_0(\bm\alpha_0^TX_i)+\varepsilon_i$, where the $\varepsilon_i$ are i.i.d.\ standard normal random variables, independent of the $\bm X_i$, which are i.i.d.\ random vectors, consisting of two independent Uniform$(0,1)$ random variables.
In this case the conditional expectation function (\ref{conditional_exp}) is a rather complicated function of $\bm\alpha$ which we shall not give here, but can be computed by a computer package such as Mathematica or Maple.
The loss functions:
\begin{align}
\label{loss_functions}
L^{\text{LSE}}:\alpha_1\mapsto{\mathbb E}\{Y-\psi_{\bm\alpha}(\bm\alpha^T\bm X)\}^2\qquad\text{and}\qquad
\widehat L^{\text{LSE}}_n:\alpha_1\mapsto n^{-1}\sum_{i=1}^n\bigl\{Y_i-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm X_i)\bigr\}^2
\end{align}
where the loss function $\widehat L^{\text{LSE}}_n$ is for sample sizes $n=10,000$ and $n=100,000$, and $\bm\alpha=(\alpha_1,\alpha_2)^T$. For $\alpha_1\in[0,1]$ and $\alpha_2$ equal to the positive root $\{1-\alpha_1^2\}^{1/2}$, we get Figure \ref{fig:empirical_vs_analytic}.
The function $L^{\text{LSE}}$ has a minimum equal to $1$ at $\alpha_1=1/\sqrt{2}$ and $\widehat L^{\text{LSE}}_n$ has minimum at a value very close to $1/\sqrt{2}$ (furnishing the profile LSE $\hat\bm\alpha_n$).
}
\end{example}

\begin{figure}[!h]
	\centering
	\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=0.95\textwidth]{Loss_LSE_10_4.pdf}
	\caption{$n = 10^4$}
	\end{subfigure}
	\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=0.95\textwidth]{Loss_LSE_10_5.pdf}
	\caption{$n =10^5$}
\end{subfigure}\\
\caption{The loss functions $L^{\text{LSE}}$ (red, dashed) and $\widehat L^{\text{LSE}}_n$ (solid), where $n=10^4$ and $n=10^5$.}
\label{fig:empirical_vs_analytic}
	\end{figure}

In order to show the $\sqrt{n}$- consistency and asymptotic normality of the estimators in the next sections, we now introduce some conditions, which correspond to those in \cite{FGH:19}. We note that we do not need conditions on reparameterization. 

\begin{enumerate}
\item[(A1)] $\bm X$ has a density w.r.t.\ Lebesgue measure on its support $\mathcal{X}$, which is a convex  set $\mathcal{X}$ with a nonempty interior,
and satisfies $\mathcal{X}\subset\{\bm x\in\mathbb R^d:\|\bm x\|\le R\}$ for some $R>0$.
\item[(A2)] The function $\psi_0$ is bounded on the set $\{u\in\mathbb R:u=\bm\alpha_0^T\bm x,\,\bm x\in \mathcal{X}\}$.
\item[(A3)] There exists $\delta>0$ such that the conditional expectation $\tilde\psi_{\bm\alpha}$, defined by (\ref{conditional_exp}) is nondecreasing on $I_{\bm\alpha}=\{u\in\mathbb R: u=\bm\alpha^T\bm x,\,x\in\mathcal{X}\}$ and satisfies $\tilde\psi_{\bm\alpha}=\psi_{\bm\alpha}$, so minimizes
\begin{align*}
\left\|{\mathbb E}\left\{Y-\psi(\bm\alpha^T\bm X)\right\}\bm X\right\|^2,
\end{align*}
over nondecreasing functions $\psi$, if $\|\bm\alpha-\bm\alpha_0\|\le\delta$.
\item[(A4)] Let $a_0$ and $b_0$ be the (finite) infimum and supremum of the interval $\{\bm\alpha_0^T\bm x,\,\bm x\in\mathcal{X}\}$. Then $\psi_0$ is continuously differentiable on $(a_0-\delta R,a_0+\delta R)$, where $R$ and $\delta$ are as in Assumption A1 and A3.
\item[(A5)] The density $g$ of $\bm X$ is differentiable and there exist strictly positive constants $c_1$ to $c_4$ such that $c_1\le g(\bm x)\le c_2$ and $c_3\le\frac{\partial}{\partial x_i}g(\bm x)\le c_4$ for $\bm x$ in the interior of $\mathcal{X}$.
\item[(A6)] There exists a $c_0>0$ and $M>0$ such that ${\mathbb E}\{|Y|^m|\bm X=\bm x\}\le m!M_0^{m-2}c_0$ for all integers $m\ge2$ and $\bm x\in\mathcal{X}$ almost surely w.r.t.\ $dG$.
\end{enumerate}

These conditions are rather natural, and are discussed in \cite{FGH:19}. The following lemma shows that, for the asymptotic distribution of $\hat\bm\alpha_n$, we can reduce the derivation to the analysis of $\psi_{\hat\bm\alpha_n}$. We have the following result (Proposition 4 in \cite{FGH:19}) on the distance between $\hat\psi_{n,\hat\bm\alpha}$  and $\psi_{\hat\bm\alpha}$.

\begin{lemma}
\label{lemma:closeness_psi_n_alpha_psi_alpha}
Let conditions (A1) to (A6) be satisfied and let $G$ be the distribution function of $\bm X$. Then we have, for $\bm\alpha$ in a neighborhood ${\cal B}(\bm\alpha_0,\delta)$ of $\bm\alpha_0$:
\begin{align*}
	\sup_{\bm\alpha \in {\cal B}(\bm\alpha_0,\delta)} \int\left\{\hat \psi_{n\bm\alpha}(\bm\alpha^T\bm x)-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}^2\,dG(\bm x)=O_p\left((\log n) ^2n^{-2/3}\right).	\end{align*}	
\end{lemma}



\section{A $\sqrt{n}$ convergent profile least squares estimator without tuning parameters}
\label{section:analysis_SSE}
\setcounter{equation}{0}
In this section we study a profile least squares estimator which is $\sqrt{n}$ convergent and asymptotically normal. It is asymptotically equivalent to the estimator SSE (Simple Score Estimator) in \cite{FGH:19} and  we give it the same name. A  crucial role is played by the function $\psi_{\bm\alpha}$ of Definition \ref{def_psi_alpha}. In this section we use the following assumptions, additional to (A1) to (A6).

\begin{enumerate}
\item[(A7)] There exists a $\delta>0$ such that for all $\bm\alpha\in({\cal B}(\bm\alpha_0,\delta)\cap\mathcal{S}_{d-1})\setminus\{\bm\alpha_0\}$ the random variable
\begin{align*}
\text{cov}\left(\bm\alpha_0-\bm\alpha)^T\bm X,\psi_0(\bm\alpha_0^T\bm X)\bigm|\bm\alpha^T\bm X\right)
\end{align*}
is not equal to $0$ almost surely.
\item[(A8)] The matrix
\begin{align*}
{\mathbb E}\left[\psi_0'(\bm\alpha_0^T\bm X)\,\text{cov}(\bm X|\bm\alpha_0^T\bm X)\right]
\end{align*}
has rank $d-1$.
\end{enumerate}




We start by comparing (\ref{SSE_criterion}) with the function
\begin{align}
\label{SSE_criterion_theoretical}
\bm\alpha\mapsto\left\|{\mathbb E}\left\{Y-\psi_{\bm\alpha}(\bm\alpha^T\bm X)\right\}\bm X\right\|^2.
\end{align}
As in Section \ref{section:intro}, the function $\hat\psi_{n,\bm\alpha}$ is just the (isotonic) least squares estimate for fixed $\bm\alpha$.

\vspace{0.3cm}
\begin{example}[Continuation of Example \ref{example_uniform_covariates}]
\label{example_uniform_covariates2}
{\rm We consider the loss function given by
\begin{align}
\label{loss_functions2a}
L^{\text{SSE}}:\alpha_1\mapsto\left\|{\mathbb E}\left\{Y-\psi_{\bm\alpha}(\bm\alpha^T\bm X)\right\}\bm X\right\|^2,
\end{align}
and compare this with the loss function
\begin{align}
\label{loss_functions2b}
\widehat L^{\text{SSE}}_n:\alpha_1\mapsto \left\|n^{-1}\sum_{i=1}^n\left\{Y_i-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm X_i)\right\}\bm X_i\right\|^2,
\end{align}
for the same data as in Example \ref{example_uniform_covariates} in Section \ref{section:psi_alpha}.
If we plot the loss functions $L^{\text{SSE}}$ and $\hat L^{\text{SSE}}_n$ for the model of Example  \ref{example_uniform_covariates},  where $\bm\alpha=(\alpha_1,\alpha_2)^T$, for $\alpha_1\in[0,1]$ and $\alpha_2$ the positive root $\sqrt{1-\alpha_1^2}$, we get Figure \ref{fig:empirical_vs_analytic_SSE}. The function $L^{\text{LSE}}$ has a minimum equal to $0$ at $\alpha_1=1/\sqrt{2}$.

\begin{figure}[!h]
	\centering
	\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=0.95\textwidth]{Loss_SSE_10_4.pdf}
	\caption{$n = 10^4$}
	\end{subfigure}
	\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=0.95\textwidth]{Loss_SSE_10_5.pdf}
	\caption{$n =10^5$}
\end{subfigure}\\
\caption{The loss functions $L^{\text{SSE}}$ (red, dashed) and $\widehat L^{\text{SSE}}_n$ (solid), where $n=10^4$ and $n=10^5$.}
\label{fig:empirical_vs_analytic_SSE}
	\end{figure}

In general, the curve $\widehat L^{\text{SSE}}_n$ will be smoother than the curve $\widehat L^{\text{LSE}}_n$. The rather striking difference in smoothness of the loss functions $\widehat L^{\text{LSE}}_n$ and $\widehat L^{\text{SSE}}_n$ can be seen in Figure \ref{fig:loss_on_0.65_0.8}, where we zoom in on the interval $[0.65,0.80]$ for $n=10,000$ and the examples of Figure \ref{fig:empirical_vs_analytic} and Figure \ref{fig:empirical_vs_analytic_SSE}.}
\end{example}

\begin{figure}[!h]
	\centering
	\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=0.95\textwidth]{LSE_10_4.pdf}
	\caption{$\widehat L^{\text{LSE}}_n$}
	\end{subfigure}
	\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=0.95\textwidth]{SSE_10_4.pdf}
	\caption{$\widehat L^{\text{SSE}}_n$}
\end{subfigure}\\
\caption{The loss functions $\widehat L^{\text{LSE}}_n$ and $\widehat L^{\text{SSE}}_n$ on $[0.65,0.80]$, for $n=10^4$.}
\label{fig:loss_on_0.65_0.8}
	\end{figure}

\vspace{0.3cm}
In the computation of the SSE, we have to take a starting point. For this we use the LSE, which is proved to be consistent in \cite{BDJ:19}.
The proof of the consistency of the SSE is a variation on the proof for corresponding crossing of zero estimator in \cite{FGH:19} in (D.2) of the supplementary material. We use the following lemma, which is a corollary to Proposition 2 in the supplementary material of \cite{FGH:19}.


\begin{lemma}
\label{lemma:phi_n-phi}
Let $\phi_n$ and $\phi$ be defined by
\begin{align*}
\phi_n(\bm\alpha)=\int\bm x\left\{y-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d{\mathbb P}_n(\bm x,y),
\end{align*}
and
\begin{align*}
\phi(\bm\alpha)=\int\bm x\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,dP(\bm x,y).
\end{align*}
Then, uniformly for $\bm\alpha$ in a neighborhood ${\cal B}(\bm\alpha_0,\delta)\cap\mathcal{S}_{d-1}$ of $\bm\alpha_0$:
\begin{align*}
\phi_n(\bm\alpha)=\phi(\bm\alpha)+o_p(1).
\end{align*}
\end{lemma}

\begin{remark}
{\rm
The proof in \cite{FGH:19} used reparameterization, but this is actually not needed in the proof.
}
\end{remark}



\begin{theorem}[Consistency of the SSE]
Let $\hat\bm\alpha_n\in\mathcal{S}_{d-1}$ be the SSE of $\bm\alpha_0$ and let conditions (A1) to (A8) be satisfied. Then
\begin{align*}
\hat\bm\alpha_n\stackrel{p}\longrightarrow\bm\alpha_0.
\end{align*}
\end{theorem}

\begin{proof}
By Lemma \ref{lemma:phi_n-phi}:
\begin{align*}
&\inf_{\bm\alpha \in {\cal B}(\bm\alpha_0,\delta)}\left\|n^{-1}\sum_{i=1}^n\left\{Y_i-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm X_i)\right\}\bm X_i\right\|^2\\
&=\inf_{\bm\alpha \in {\cal B}(\bm\alpha_0,\delta)}\left\|\int\bm x\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,dP(\bm x,y)\right\|^2+o_p(1).
\end{align*}
Since
\begin{align*}
\inf_{\bm\alpha \in {\cal B}(\bm\alpha_0,\delta)}\left\|\int\bm x\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,dP(\bm x,y)\right\|^2=
\left\|\int\bm x\left\{y-\psi_{\bm\alpha_0}(\bm\alpha_0^T\bm x)\right\}\,dP(\bm x,y)\right\|^2=0,
\end{align*}
we get:
\begin{align*}
\left\|n^{-1}\sum_{i=1}^n\left\{Y_i-\hat\psi_{n,\hat\bm\alpha_n}(\hat\bm\alpha_n^T\bm X_i)\right\}\bm X_i\right\|^2
=\inf_{\bm\alpha \in {\cal B}(\bm\alpha_0,\delta)}\left\|n^{-1}\sum_{i=1}^n\left\{Y_i-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm X_i)\right\}\bm X_i\right\|^2=o_p(1).
\end{align*}
Hence for a subsequence $(n_k)$ such that $\hat\bm\alpha_{n_k}\to\bm\alpha_*\in\mathcal{S}_{d-1}\cap{\cal B}(\bm\alpha_0,\delta)$,
\begin{align*}
\lim_{k\to\infty}\phi_{n_k,\hat\bm\alpha_{n_k}}\left(\hat\bm\alpha_{n_k}\right)
=\lim_{k\to\infty}\phi\left(\hat\bm\alpha_{n_k}\right)=\phi(\bm\alpha_*)=0.
\end{align*}
Note that we can assume the existence of such subsequences, since we may assume that $\hat\bm\alpha_n\in\mathcal{S}_{d-1}\cap\,{\cal B}(\bm\alpha_0,\delta)$. Also note that the continuity of $\phi$ is used to get $\phi(\bm\alpha_*)=0$.

So we find, using $\psi_{\bm\alpha}(u)=E\,\psi_0(\bm\alpha_0^T\bm X|\bm\alpha^T\bm X=u)$ for $\bm\alpha\in{\cal B}(\bm\alpha_0,\delta)$,
\begin{align*}
0&=(\bm\alpha_0-\bm\alpha_*)^T\phi(\bm\alpha_*)=\int(\bm\alpha_0-\bm\alpha_*)^T\bm x\left\{y-\psi_{\bm\alpha_*}(\bm\alpha_*^T\bm x)\right\}\,dP(\bm x,y)\\
&=\int(\bm\alpha_0-\bm\alpha_*)^T\bm x\left\{\psi_0(\bm\alpha_0^T\bm x)-\psi_{\bm\alpha_*}(\bm\alpha_*^T\bm x)\right\}\,dG(y)\\
&={\mathbb E}\left[\text{cov}\left(\bm\alpha_0-\bm\alpha_*)^T\bm X,\psi_0(\bm\alpha_0)\bigm|\bm\alpha_*^T\bm X\right)\right],
\end{align*}
which can only happen if $\bm\alpha_*=\bm\alpha_0$ by Assumption (A7).
\end{proof}

\begin{lemma}
\label{lemma:transition_psi_n-psi_alpha}
Let $\hat\bm\alpha_n\in\mathcal{S}_{d-1}$ be a minimizer of
\begin{align}
\label{crossing_of_zero}
\left\|n^{-1}\sum_{i=1}^n\left\{Y_i-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm X_i)\right\}\bm X_i\right\|^2,
\end{align}
for $\bm\alpha\in\mathcal{S}_{d-1}$, where $\|\cdot\|$ denotes the Euclidean norm. Then, under conditions (A1) to (A8) we have:
\begin{align}
\label{SSE_root_psi_alpha}
n^{-1}\sum_{i=1}^n\left\{Y_i-\hat\psi_{n,\hat\bm\alpha_n}(\hat\bm\alpha_n^T\bm X_i)\right\}\bm X_i
=n^{-1}\sum_{i=1}^n\left\{Y_i-\psi_{\hat\bm\alpha_n}(\hat\bm\alpha_n^T\bm X_i)\right\}\left\{\bm X_i-{\mathbb E}\left(\bm X|\hat\bm\alpha_n^T\bm X_i\right)\right\}+o_p\left(n^{-1/2}\right).
\end{align}
\end{lemma}


\begin{proof}
We introduce the function $\bar E_{n,\bm\alpha}$, defined by:
\begin{align}
\label{E_n-def}
\bar E_{n,\bm\alpha}(u)=\left\{\begin{array}{lll}
{\mathbb E}\left\{\bm X|\bm\alpha^T\bm X=s\right\}\qquad &,\text{ if } \psi_{\bm\alpha}(u)>\hat\psi_{n,\bm\alpha}(\tau_i)\\
{\mathbb E}\left\{\bm X|\bm\alpha^T\bm X=\tau_{i+1}\right\}\qquad &,\text{ if } \psi_{\bm\alpha}(u)=\hat\psi_{n,\bm\alpha}(s),\text{ for some } s\in[\tau_i,\tau_{i+1}),\\
{\mathbb E}\left\{\bm X|\bm\alpha^T\bm X=\tau_i\right\}\qquad &,\text{ if } \psi_{\bm\alpha}(u)<\hat\psi_{n,\bm\alpha}(\tau_i),
\end{array}
\right.
\end{align}
where the $\tau_i$'s are the points of jump of the function $\hat\psi_{n,\bm\alpha}$. For similar constructions, relying on smooth functional theory, see \cite{piet_geurt:14},  Chapter 10, \cite{kim_piet:18}, Supplementary Material (S2.15), and \cite{FGH:19}, Supplementary Material, Section D.3.  We get, by the definition of the least squares estimate $\hat\psi_{n,\bm\alpha}$,
\begin{align*}
\int \bar E_{n,\bm\alpha}(\bm\alpha^T\bm x)\left\{y-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d{\mathbb P}_n(\bm x,y)=\bm0,
\end{align*}
see also (D.10), Supplementary Material, Section D.3 of \cite{FGH:19}, where, however, a reparameterization is used.

Hence we can write:
\begin{align}
\label{SSE-decomp}
&\int \bm x\left\{y-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d{\mathbb P}_n(\bm x,y)\nonumber\\
&=\int\left\{\bm x-\bar E_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\left\{y-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d{\mathbb P}_n(\bm x,y)\nonumber\\
&=\int\left\{\bm x-\bar E_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d{\mathbb P}_n(\bm x,y)\nonumber\\
&\qquad\qquad+\int\left\{\bm x-\bar E_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\left\{\psi_{\bm\alpha}(\bm\alpha^T\bm x)-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d{\mathbb P}_n(\bm x,y).
\end{align}
For $\bm\alpha$ in a neighborhood of $\bm\alpha_0$, we can simplify the first term on the right-hand side in the following way:
\begin{align*}
&\int\left\{\bm x-\bar E_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d{\mathbb P}_n(\bm x,y)\\
&=\int\left\{\bm x-\bar E_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,dP(\bm x,y)\\
&\qquad\qquad+\int\left\{\bm x-\bar E_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d\left({\mathbb P}_n-P\right)(\bm x,y)\\
&=\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha^T\bm x\right\}\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d\left({\mathbb P}_n-P\right)(\bm x,y)
+o_p\left(n^{-1/2}\right).
\end{align*}
using that, for $\bm\alpha$ in a neighborhood of $\bm\alpha_0$,
\begin{align*}
&\int\left\{\bm x-\bar E_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,dP(\bm x,y)\\
&={\mathbb E}\left[{\mathbb E}\left\{\bm X-\bar E_{n,\bm\alpha}(\bm\alpha^T\bm X)\right\}\left\{\psi_0(\bm\alpha_0^T\bm X)-\psi_{\bm\alpha}(\bm\alpha^T\bm X)\right\}\Bigm|\bm\alpha^T\bm X\right]=\bm0,
\end{align*}
since, for $\bm\alpha$ in a neighborhood of $\bm\alpha_0$, ${\mathbb E}\{\psi_0(\bm\alpha_0^T\bm X)|\bm\alpha^T\bm X\}=\psi_{\bm\alpha}(\bm\alpha^T\bm X)$.

We also have:
\begin{align*}
&\int\left\{\bm x-\bar E_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d\left({\mathbb P}_n-P\right)(\bm x,y)\\
&=\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha^T\bm x\right\}\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d\left({\mathbb P}_n-P\right)(\bm x,y)\\
&\qquad\qquad+\int\left\{{\mathbb E}\left\{\bm X|\bm\alpha^T\bm x\right\}-\bar E_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d\left({\mathbb P}_n-P\right)(\bm x,y)\\
&=\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha^T\bm x\right\}\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d\left({\mathbb P}_n-P\right)(\bm x,y)
+o_p\left(n^{-1/2}\right).
\end{align*}
For the last expression on the right-hand side of (\ref{SSE-decomp}) we get:
\begin{align*}
&\int\left\{\bm x-\bar E_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\left\{\psi_{\bm\alpha}(\bm\alpha^T\bm x)-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d{\mathbb P}_n(\bm x,y)\\
&=\int\left\{\bm x-\bar E_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\left\{\psi_{\bm\alpha}(\bm\alpha^T\bm x)-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\,dP(\bm x,y)\\
&\qquad\qquad+\int\left\{\bm x-\bar E_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\left\{\psi_{\bm\alpha}(\bm\alpha^T\bm x)-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d\left({\mathbb P}_n-P\right)(\bm x,y)\\
&=\int\left\{\bm x-\bar E_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\left\{\psi_{\bm\alpha}(\bm\alpha^T\bm x)-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\,dP(\bm x,y)+o_p\left(n^{-1/2}\right)\\
&=\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha^T\bm x\right\}\right\}\left\{\psi_{\bm\alpha}(\bm\alpha^T\bm x)-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\,dP(\bm x,y)\\
&\qquad\qquad+\int\left\{{\mathbb E}\left\{\bm X|\bm\alpha^T\bm x\right\}-\bar E_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\left\{\psi_{\bm\alpha}(\bm\alpha^T\bm x)-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\,dP(\bm x,y)+o_p\left(n^{-1/2}\right)\\
&=\int\left\{{\mathbb E}\left\{\bm X|\bm\alpha^T\bm x\right\}-\bar E_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\left\{\psi_{\bm\alpha}(\bm\alpha^T\bm x)-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\,dP(\bm x,y)+o_p\left(n^{-1/2}\right),
\end{align*}
where
\begin{align*}
&\int\left\{{\mathbb E}\left\{\bm X|\bm\alpha^T\bm x\right\}-\bar E_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\left\{\psi_{\bm\alpha}(\bm\alpha^T\bm x)-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\,dP(\bm x,y)\\
&=O_p\left(\left\|\psi_{\bm\alpha}-\hat\psi_{n,\bm\alpha}\right\|^2\right)=O_p\left(n^{-2/3}(\log n)^2\right),
\end{align*}
uniformly for $\bm\alpha\in B(\bm\alpha_0,\delta)=\{\bm\alpha:\|\bm\alpha-\bm\alpha_0\|\le\delta\}$. This follows from
\begin{align*}
\left\|\left\{{\mathbb E}\left\{\bm X|\bm\alpha^T\bm X=u\right\}-\bar E_{n,\bm\alpha}(u)\right\}\right\|\le K\left|\hat\psi_{n,\bm\alpha}(u)-\psi_{\bm\alpha}(u)\right|,
\end{align*}
for a constant $K>0$, which is ensured by Definition (\ref{E_n-def}), condition (A5) and Lemma \ref{lemma:closeness_psi_n_alpha_psi_alpha}.

 We also use the Cauchy-Schwarz inequality
\begin{align*}
&\left\|\int\left\{{\mathbb E}\left\{\bm X|\bm\alpha^T\bm x\right\}-\bar E_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\left\{\psi_{\bm\alpha}(\bm\alpha^T\bm x)-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\,dP(\bm x,y)\right\|^2\\
&\le\sum_{j=1}^d\left\{\int\left\{{\mathbb E}\left\{X_j|\bm\alpha^T\bm x\right\}-\bar E_{n,\bm\alpha}(\bm\alpha^T\bm x)_j\right\}^2\,dP(\bm x,y)\right\}
\left\{\int\left\{\psi_{\bm\alpha}(\bm\alpha^T\bm x)-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}^2\,dP(\bm x,y)\right\}.
\end{align*}
For a similar argument, see pp.\ 307 and 308 of \cite{piet_geurt:14}.

So the conclusion is:
\begin{align}
\label{SSE-rewrite}
&\int \bm x\left\{y-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d{\mathbb P}_n(\bm x,y)\nonumber\\
&=\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha^T\bm x\right\}\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d\left({\mathbb P}_n-P\right)(\bm x,y)
+o_p\left(n^{-1/2}\right).
\end{align}
uniformly for $\bm\alpha\in B(\bm\alpha_0,\delta)=\{\bm\alpha:\|\bm\alpha-\bm\alpha_0\|\le\delta\}$. This implies for the SSE $\hat\bm\alpha_n$:
\begin{align}
\label{equiv-statement-SSE}
&\int \bm x\left\{y-\hat\psi_{n,\hat\bm\alpha_n}(\hat\bm\alpha_n^T\bm x)\right\}\,d{\mathbb P}_n(\bm x,y)\nonumber\\
&=\int\left\{\bm x-{\mathbb E}\left\{\bm X|\hat\bm\alpha_n^T\bm x\right\}\right\}\left\{y-\psi_{\hat\bm\alpha_n}(\hat\bm\alpha_n^T\bm x)\right\}\,d\left({\mathbb P}_n-P\right)(\bm x,y)
+o_p\left(n^{-1/2}\right).
\end{align}
\end{proof}

We now have the following limit result.

\begin{theorem}[Asymptotic normality of the SSE]
\label{th:asympt_normality_SSE}
Let $\hat\bm\alpha_n$ be the minimizer of
\begin{align}
\label{crossing_of_zero}
\left\|n^{-1}\sum_{i=1}^n\left\{Y_i-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm X_i)\right\}\bm X_i\right\|^2,
\end{align}
for $\bm\alpha\in\mathcal{S}_{d-1}$, where $\|\cdot\|$ denotes the Euclidean norm.
		Let the matrices $\bm A$ and $\bm\Sigma$ be defined by:
		\begin{align}
		\label{def_A}
		\bm A={\mathbb E}\Bigl[\psi_0'(\bm\alpha_0^T\bm X)\,\text{\rm Cov}(\bm X|\bm\alpha_0^T\bm X)\Bigr],
		\end{align}
		and
		\begin{align}
		\label{def_Sigma}
		\bm \Sigma={\mathbb E}\left[\left\{Y -\psi_0(\bm\alpha_0^T\bm X)\right\}^2\,\left\{\bm X -{\mathbb E}(\bm X|\bm\alpha_0^T\bm X) \right\}\left\{\bm X -{\mathbb E}(\bm X|\bm\alpha_0^T\bm X) \right\}^T\right].
		\end{align}
Then, under conditions $(A1)$ to $(A8)$ we have:	 
		\begin{align*}
		\sqrt n (\hat \bm\alpha_n - \bm\alpha_0) \to_d N\left(\bm 0,  \bm A^- \bm \Sigma \bm A^-\right),
		\end{align*}
		where $\bm A^{-}$ is the Moore-Penrose inverse of $\bm A$.
\end{theorem}

\begin{proof}
By the consistency of $\hat\bm\alpha_n$, we may assume that $\hat\bm\alpha_n$ belongs to a small neighborhood of $\bm\alpha_0$.
Moreover, for $\bm\alpha$ in a neighborhood of $\bm\alpha_0$ we have the expansion:
\begin{align}
\label{SSE_expansion}
&\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha^T\bm x\right\}\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d\left({\mathbb P}_n-P\right)(\bm x,y)\nonumber\\
&\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha^T\bm x\right\}\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d{\mathbb P}_n(\bm x,y)\nonumber\\
&=\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha_0^T\bm x\right\}\right\}\left\{y-\psi_0(\bm\alpha_0^T\bm x)\right\}\,d\left({\mathbb P}_n-P\right)(\bm x,y)\nonumber\\
&\qquad\qquad+\frac{\partial}{\partial\bm\alpha}\left[\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha^T\bm x\right\}\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,P(\bm x,y)\right]\Biggr|_{\bm\alpha=\bm\alpha_0}\left(\bm\alpha-\bm\alpha_0\right)\nonumber\\
&\qquad\qquad+o_p\left(n^{-1/2}\right)+o_p\left(\|\bm\alpha-\bm\alpha_0\|\right),
\end{align}
where the factor of $\bm\alpha-\bm\alpha_0$ in the second term after the last equality sign is given by the matrix with elements:
\begin{align}
\label{matrix_representation_derivative}
\left(\frac{\partial}{\partial\alpha_j}\left[\int\left\{x_i-{\mathbb E}\left\{X_i|\bm\alpha^T\bm x\right\}\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,P(\bm x,y)\right]\right)_{(i,j)}\Biggr|_{\bm\alpha=\bm\alpha_0},\qquad i,j=1,\dots,d.
\end{align}
So we obtain from (\ref{SSE-rewrite}),
\begin{align}
\label{equiv-statement-SSE2}
&\left\|\int \bm x\left\{y-\hat\psi_{n,\hat\bm\alpha_n}(\hat\bm\alpha_n^T\bm x)\right\}\,d{\mathbb P}_n(\bm x,y)\right\|\nonumber\\
&=\inf_{\bm\alpha\in\mathcal{S}_{d-1}}\left\|\int \bm x\left\{y-\hat\psi_{n,\bm\alpha}(\bm\alpha^T\bm x)\right\}\,d{\mathbb P}_n(\bm x,y)\right\|\nonumber\\
&=\inf_{\bm\alpha\in\mathcal{S}_{d-1}}\left\|\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha_0^T\bm x\right\}\right\}\left\{y-\psi_0(\bm\alpha_0^T\bm x)\right\}\,d\left({\mathbb P}_n-P\right)(\bm x,y)\right.\nonumber\\
&\qquad\qquad\qquad+\frac{\partial}{\partial\bm\alpha}\left[\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha^T\bm x\right\}\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,P(\bm x,y)\right]\Biggr|_{\bm\alpha=\bm\alpha_0}\left(\bm\alpha-\bm\alpha_0\right)\nonumber\\
&\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad+o_p\left(n^{-1/2}\right)+o_p\left(\|\bm\alpha-\bm\alpha_0\|\right)\biggr\|.
\end{align}
Since:
\begin{align*}
&\inf_{\bm\alpha\in\mathcal{S}_{d-1}}\Biggl\|\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha_0^T\bm x\right\}\right\}\left\{y-\psi_0(\bm\alpha_0^T\bm x)\right\}\,d\left({\mathbb P}_n-P\right)(\bm x,y)\nonumber\\
&\qquad\qquad\qquad+\frac{\partial}{\partial\bm\alpha}\left[\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha^T\bm x\right\}\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,P(\bm x,y)\right]\Biggr|_{\bm\alpha=\bm\alpha_0}\left(\bm\alpha-\bm\alpha_0\right)\Biggr\|\\
&=\bm0,
\end{align*}
which follows by taking $\bm\alpha$ a solution of the linear equation
\begin{align}
\label{asymptotic_linear_eq}
&\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha_0^T\bm x\right\}\right\}\left\{y-\psi_0(\bm\alpha_0^T\bm x)\right\}\,d\left({\mathbb P}_n-P\right)(\bm x,y)\nonumber\\
&\qquad\qquad\qquad+\frac{\partial}{\partial\bm\alpha}\left[\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha^T\bm x\right\}\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,P(\bm x,y)\right]\Biggr|_{\bm\alpha=\bm\alpha_0}\left(\bm\alpha-\bm\alpha_0\right)\nonumber\\
&=\bm0,
\end{align}
we  get from (\ref{equiv-statement-SSE2}):
\begin{align}
\label{score_eq_SSE}
\int \bm x\left\{y-\hat\psi_{n,\hat\bm\alpha_n}(\hat\bm\alpha_n^T\bm x)\right\}\,d{\mathbb P}_n(\bm x,y)=o_p\left(n^{-1/2}\right)+o_p\left(\|\hat\bm\alpha_n-\bm\alpha_0\|\right).
\end{align}
Note that the two integrals on the left-hand side of (\ref{asymptotic_linear_eq}) are perpendicular to the vector $\bm\alpha_0$ and that the equation is therefore essentially an equation in $\mathbb R^{d-1}$, which is also clear if one treats the system by reparameterization in $\mathbb R^{d-1}$.

Moreover, by Lemma \ref{lemma:transition_psi_n-psi_alpha},
\begin{align}
\label{second_score_eq_SSE}
&\int \bm x\left\{y-\hat\psi_{n,\hat\bm\alpha_n}(\hat\bm\alpha_n^T\bm x)\right\}\,d{\mathbb P}_n(\bm x,y)\nonumber\\
&=\int\left\{\bm x-{\mathbb E}\left\{\bm X|\hat\bm\alpha_n^T\bm x\right\}\right\}\left\{y-\psi_{\hat\bm\alpha_n}(\hat\bm\alpha_n^T\bm x)\right\}\,d\left({\mathbb P}_n-P\right)(\bm x,y)
+o_p\left(n^{-1/2}\right).
\end{align}
and by (\ref{SSE_expansion})
\begin{align*}
&\int\left\{\bm x-{\mathbb E}\left\{\bm X|\hat\bm\alpha_n^T\bm x\right\}\right\}\left\{y-\psi_{\hat\bm\alpha_n}(\hat\bm\alpha_n^T\bm x)\right\}\,d\left({\mathbb P}_n-P\right)(\bm x,y)\\
&=\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha_0^T\bm x\right\}\right\}\left\{y-\psi_0(\bm\alpha_0^T\bm x)\right\}\,d\left({\mathbb P}_n-P\right)(\bm x,y)\nonumber\\
&\qquad\qquad+\frac{\partial}{\partial\bm\alpha}\left[\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha^T\bm x\right\}\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,P(\bm x,y)\right]\Biggr|_{\bm\alpha=\bm\alpha_0}\left(\hat\bm\alpha_n-\bm\alpha_0\right)\nonumber\\
&\qquad\qquad+o_p\left(n^{-1/2}\right)+o_p\left(\|\hat\bm\alpha_n-\bm\alpha_0\|\right).
\end{align*}
Combining this with (\ref{score_eq_SSE}) and (\ref{second_score_eq_SSE}) we find
\begin{align*}
&\frac{\partial}{\partial\bm\alpha}\left[\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha^T\bm x\right\}\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,P(\bm x,y)\right]\Biggr|_{\bm\alpha=\bm\alpha_0}\left(\hat\bm\alpha_n-\bm\alpha_0\right)\\
&=-\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha_0^T\bm x\right\}\right\}\left\{y-\psi_0(\bm\alpha_0^T\bm x)\right\}\,d\left({\mathbb P}_n-P\right)(\bm x,y)(\bm x,y)\\
&\qquad\qquad+o_p\left(n^{-1/2}\right)+o_p\left(\|\hat\bm\alpha_n-\bm\alpha_0\|\right).
\end{align*}

Finally:
\begin{align*}
&\frac{\partial}{\partial\bm\alpha}\left[\int\left\{\bm x-{\mathbb E}\left\{\bm X|\bm\alpha^T\bm x\right\}\right\}\left\{y-\psi_{\bm\alpha}(\bm\alpha^T\bm x)\right\}\,P(\bm x,y)\right]\Biggr|_{\bm\alpha=\bm\alpha_0}=-{\mathbb E}\Bigl[\psi_0'(\bm\alpha_0^T\bm X)\,\text{\rm Cov}(\bm X|\bm\alpha_0^T\bm X)\Bigr],
\end{align*}
since, by the first part of Lemma 10 in the supplementary material of \cite{FGH:19},
\begin{align*}
&\frac{\partial}{\partial\bm\alpha}\psi_{\bm\alpha}(\bm\alpha^T\bm x)\biggr|_{\bm\alpha=\bm\alpha_0}=\left(\bm x-{\mathbb E}\left[\bm X|\bm\alpha_0^T\bm X=\bm\alpha_0\bm x\right]\right)\psi_0'(\bm\alpha_0^T\bm x),
\end{align*}
and, moreover,
\begin{align*}
&\int\left\{\bm x-\frac{\partial}{\partial\bm\alpha}{\mathbb E}\left[\bm X|\bm\alpha^T\bm x\right]\Bigr|_{\bm\alpha=\bm\alpha_0}\right\}\left\{y-\psi_{\bm\alpha_0}(\bm\alpha_0^T\bm x)\right\}\,P(\bm x,y)=\bm0.
\end{align*}

The statement of the theorem now follows, where we use the Moore-Penrose generalized inverse to preserve symmetry.
\end{proof}

\vspace{0.5cm}
\begin{example}[Continuation of Example \ref{example_uniform_covariates2}] 
{\rm We compute the asymptotic covariance matrix for Example \ref{example_uniform_covariates2}. In this case we get for matrix $\bm A$ in part (ii) of Theorem \ref{th:asympt_normality_SSE}:
\begin{align*}
\bm A&={\mathbb E}\Bigl[\psi_0'(\bm\alpha_0^T\bm X)\,\text{\rm Cov}(\bm X|\bm\alpha_0^T\bm X)\Bigr]\\
&=\frac34{\mathbb E}\left[\left(\frac{X_1+X_2}{\sqrt{2}}\right)^2\left(\bm X-{\mathbb E}(\bm X|\bm\alpha_0^T\bm X)\right)\left(\bm X-{\mathbb E}(\bm X|\bm\alpha_0^T\bm X)\right)^T\right]\\
&=\left(\begin{array}{rrr}
1/15 &-1/15\\
-1/15	&1/15
\end{array}
\right).
\end{align*}
The Moore-Penrose inverse of $\bm A$ is given by:
\begin{align*}
\bm A^-=\left(\begin{array}{rrr}
15/4 &-15/4\\
-15/4	&15/4
\end{array}
\right).
\end{align*}
Furthermore, we get:
\begin{align*}
\bm \Sigma&={\mathbb E}\left[\left\{Y -\psi_0(\bm\alpha_0^T\bm X)\right\}^2\,\left\{\bm X -{\mathbb E}(\bm X|\bm\alpha_0^T\bm X) \right\}\left\{\bm X -{\mathbb E}(\bm X|\bm\alpha_0^T\bm X) \right\}^T\right]\\
&={\mathbb E}\left\{\bm X -{\mathbb E}(\bm X|\bm\alpha_0^T\bm X) \right\}\left\{\bm X -{\mathbb E}(\bm X|\bm\alpha_0^T\bm X) \right\}^T\\
&=\left(\begin{array}{rrr}
1/24 &-1/24\\
-1/24	&1/24
\end{array}
\right).
\end{align*}
So the asymptotic covariance matrix is given by:
\begin{align*}
\bm A^-\bm\Sigma\bm A^-=\left(\begin{array}{rrr}
75/32 &-75/32\\
-75/32	&75/32
\end{array}
\right)
\approx
\left(\begin{array}{rrr}
2.34375 &-2.34375\\
-2.34375	&2.34375
\end{array}
\right).
\end{align*}
}
\end{example}

\vspace{0.3cm}
\begin{remark}
{\rm Theorem \ref{th:asympt_normality_SSE} corresponds to Theorem 3 in \cite{FGH:19}, but note that the estimator has a different definition. Reparameterization is also avoided.
}
\end{remark}


\section{Two profile least squares estimators using a tuning parameter}
\label{sec:ESE_PLSE}
\setcounter{equation}{0}


The proofs of the consistency and asymptotic normality of the ESE and spline estimator are highly similar to the proofs of these facts for the SSE in the preceding section. The only extra ingredient is occurrence of the estimate of the derivative of the link function. We only discuss the asymptotic normality.

In addition to the assumptions (A1) to (A7), we now assume:
\begin{enumerate}
\item[(A8')] $\psi_{\bm\alpha}$ is twice differentiable on $(\inf_{x\in{\cal X}}(\bm\alpha^T\bm x),\sup_{x\in{\cal X}}(\bm\alpha^T\bm x))$.
\item[(A9)] The matrix
\begin{align*}
{\mathbb E}\left[\psi_0'(\bm\alpha_0^T\bm X)^2\,\text{cov}(\bm X|\bm\alpha_0^T\bm X)\right]
\end{align*}
has rank $d-1$.
\end{enumerate}

An essential step is again to show that
\begin{align*}
&\int\bm x\left\{y-\hat\psi_{n,\hat\bm\alpha_n}(\hat\bm\alpha_n^T\bm x)\right\}\hat\psi'_{n\hat\bm\alpha_n}(\hat\bm\alpha_n^T\bm x)\,d{\mathbb P}_n(\bm x,y)\\
&=\int\left\{\bm x-{\mathbb E}(X|\hat\bm\alpha_n^T\bm X)\right\}\left\{y-\hat\psi_{n,\hat\bm\alpha_n}(\hat\bm\alpha_n^T\bm x)\right\}\hat\psi'_{n\hat\bm\alpha_n}(\hat\bm\alpha_n^T\bm x)\,d{\mathbb P}_n(\bm x,y)
+o_p(n^{-1/2})+o_p(\hat\bm\alpha_n-\bm\alpha_0),
\end{align*}
For the ESE this is done by defining the piecewise constant function $\bar{\rho}_{n, \bm\alpha}$ for $u$ in the interval between successive jumps $\tau_i$  and $\tau_{i+1})$  of $\hat\psi_{n\bm\alpha}$ by:
\begin{eqnarray*}
	\bar {\rho}_{n, \bm\alpha}(u)  =  \left \{
	\begin{array}{lll}
		{\mathbb E}[\bm X| \bm\alpha^T\bm X= \tau_i]\psi_{\bm\alpha}'(\tau_i) \ \  \ \ \ \ \ \ \textrm{ if $\psi_{\bm\alpha}(u)  > \hat\psi_{n\bm\alpha}(\tau_i)$  \ for all $u \in (\tau_i, \tau_{i+1})$}, \\
		{\mathbb E}[\bm X| \bm\alpha^T\bm X= s]\psi_{\bm\alpha}'(s) \ \ \ \  \ \  \  \ \ \ \textrm{ if $\psi_{\bm\alpha}(s)  = \hat\psi_{n\bm\alpha}(s)$ \ for some $s \in (\tau_i, \tau_{i+1})$}, \\
		{\mathbb E}[\bm X| \bm\alpha^T\bm X= \tau_{i+1}]\psi_{\bm\alpha}'(\tau_{i+1})\ \ \ \textrm{if $\psi_{\bm\alpha}(u) < \hat\psi_{n\bm\alpha}(\tau_i)$  \ for all $u \in (\tau_i, \tau_{i+1})$}. 
	\end{array}
	\right.
\end{eqnarray*}
where $\bar{\rho}_{n,\bm\alpha}$ replaces $\bar E_{n, \bm\alpha}$ in (\ref{E_n-def}), see Appendix E in the supplement of \cite{FGH:19}. The remaining part of the proof runs along the same lines as the proof for the SSE. For additional details, see Appendix E in the supplement of \cite{FGH:19}.

The corresponding step in the proof for the spline estimator is given by the following lemma.


\begin{lemma}
\label{lemma:spline_lemma}
Let the conditions of Theorem 5 in  \cite{arun_rohit:20} be satisfied. In particular, let the penalty parameter $\mu_n$ satisfy $\mu_n=o_p(n^{-1/2})$. Then we have for all $\bm\alpha$ in a neighborhood of $\bm\alpha_0$ and for the corresponding natural cubic spline  $\hat\psi_{n\bm\alpha}$:
\begin{align*}
\int{\mathbb E}(\bm X|\bm\alpha^T\bm X)\left\{y-\hat\psi_{n\bm\alpha}\left(\bm\alpha^T\bm x\right)\right\}\hat\psi_{n\bm\alpha}'\left(\bm\alpha^T\bm x\right)\,d{\mathbb P}_n(\bm x,y)=O_p(\mu_n)=o_p\left(n^{-1/2}\right).
\end{align*}
\end{lemma}

\begin{remark}
{\rm The result shows that we have as our  basic equation in $\bm\alpha$:
\begin{align*}
&\frac1n\sum_{i=1}^n \bigl\{\hat\psi_{n\bm\alpha}(\bm\alpha^T\bm X_i)-Y_i\bigr\}\hat\psi'_{n\bm\alpha}(\bm\alpha^T\bm X_i)\bm X_i\\
&=\frac1n\sum_{i=1}^n \bigl\{\hat\psi_{n\bm\alpha}(\bm\alpha^T\bm X_i)-Y_i\bigr\}\hat\psi'_{n\bm\alpha}(\bm\alpha^T\bm X_i)\left\{\bm X_i-{\mathbb E}(\bm X_i|\bm\alpha^T\bm X_i)\right\}+o_p\left(n^{-1/2}\right)\\
&=o_p\left(n^{-1/2}\right).
\end{align*}
}
\end{remark}

\begin{proof}[Proof of Lemma \ref{lemma:spline_lemma}]
Fix $\bm\alpha$ and let $\hat\psi_{n,\bm\alpha}$ be the natural cubic spline, minimizing
\begin{align*}
n^{-1}\sum_{i=1}^n\left\{Y_i-\psi(t_i)\right\}^2+\mu_n\int_a^b \left\{\psi''(t)\right\}^2\,dt,
\end{align*}
over functions $\psi\in{\cal S}_2[a,b]$, where the $t_i$ are the the ordered values $\bm\alpha^T\bm X_i$, and where $\mu_n=o_p(n^{-1/2})$, and  $a=\min_i\bm\alpha^T\bm X_i$ and $b=\max_i\bm\alpha^T\bm X_i$. We can write the minimum in the following form:
\begin{align*}
&\int\left\{y-\hat\psi_{n\bm\alpha}(\bm\alpha^T\bm x)\right\}^2\,d{\mathbb P}_n(\bm x,y)+\mu_n\int_a^b \left\{\hat\psi_{n\bm\alpha}''(t)\right\}^2\,dt.
\end{align*}
We extend the natural cubic spline $\hat\psi_{n,\bm\alpha}$ linearly to a function on $\mathbb R$, and define the function
\begin{align}
\label{spline_derivative}
\bm v\mapsto \phi(\bm v)=&\int\left\{y-\hat\psi_{n\bm\alpha}\left(\bm\alpha^T\bm x+\bm v^T\,{\mathbb E}(\bm X|\bm\alpha^T\bm X)\right)\right\}^2\,d{\mathbb P}_n(\bm x,y)\nonumber\\
&\qquad+\mu_n\int_a^b \left\{\hat\psi_{n\bm\alpha}''\left(t+\bm v^T\,{\mathbb E}(\bm X|\bm\alpha^T\bm X=t)\right)\right\}^2\,dt.
\end{align}
We have:
\begin{align}
\label{spline_derivative2}
&\frac{\partial}{\partial\bm v} \int_a^b \left\{\hat\psi_{n\bm\alpha}''\left(t+\bm v^T\,{\mathbb E}(\bm X|\bm\alpha^T\bm X=t)\right)\right\}^2\,dt\biggr|_{\bm v=\bm0}\nonumber\\
&=2\int_a^b \hat\psi_{n\bm\alpha}''(t){\mathbb E}(\bm X|\bm\alpha^T\bm X=t)\,d\hat\psi_{n\bm\alpha}''(t)
=-\int_a^b \hat\psi_{n\bm\alpha}''(t)^2\frac{\partial}{\partial t}{\mathbb E}(\bm X|\bm\alpha^T\bm X=t)\,dt.
\end{align}
This implies, assuming the boundedness of the derivative of the function $t\mapsto\frac{\partial}{\partial t}{\mathbb E}(\bm X|\bm\alpha^T\bm X=t)$ for $t\in[a,b]$,
\begin{align}
\label{spline_derivative3}
&\left|\frac{\partial}{\partial\bm v} \int_a^b \left\{\hat\psi_{n\bm\alpha}''\left(t+\bm v^T\,{\mathbb E}(\bm X|\bm\alpha^T\bm X=t)\right)\right\}^2\,dt\Bigr|_{\bm v=\bm0}\right|
\lesssim\int_a^b \left\{\hat\psi_{n\bm\alpha}''(t)\right\}^2\,dt.
\end{align}
Also assuming that
\begin{align*}
\int_a^b \left\{\hat\psi_{n\bm\alpha}''(t)\right\}^2\,dt=O_p(1),
\end{align*}
(see Theorem 2 in \cite{arun_rohit:20}), we obtain from (\ref{spline_derivative}) to (\ref{spline_derivative3}) :
\begin{align*}
\phi'(\bm0)=\bm0=\int{\mathbb E}(\bm X|\bm\alpha^T\bm X)\left\{y-\hat\psi_{n\bm\alpha}\left(\bm\alpha^T\bm x\right)\right\}\hat\psi_{n\bm\alpha}'\left(\bm\alpha^T\bm x\right)\,d{\mathbb P}_n(\bm x,y)+O_p\left(\mu_n\right),
\end{align*}
since the function $\phi$ attains its minimum at $\bm0$ by the definition of the (natural) cubic spline as a least squares estimate. It follows that
\begin{align}
\label{spline_conditional_expectation}
\int{\mathbb E}(\bm X|\bm\alpha^T\bm X)\left\{y-\hat\psi_{n\bm\alpha}\left(\bm\alpha^T\bm x\right)\right\}\hat\psi'_{n\bm\alpha}\left(\bm\alpha^T\bm x\right)\,d{\mathbb P}_n(\bm x,y)&=
O_p\left(\mu_n\right)=o_p\left(n^{-1/2}\right).
\end{align}
\end{proof}

The remaining part of the proof of the asymptotic normality can either run along the same lines as the proof for the corresponding fact for the SSE, using the function $u\mapsto\psi_{\bm\alpha}(u)={\mathbb E}\{\psi_0(\bm\alpha^T\bm x)|\bm\alpha^T\bm X=u\}$, or directly use the convergence of $\hat\psi_{n\hat\bm\alpha_n}$ to $\psi_0$ and of $\hat\psi'_{n\hat\bm\alpha_n}$ to $\psi_0'$ (see Theorem 3 in \cite{arun_rohit:20}). For the SSE and ESE we were forced to introduce the intermediate function $\psi_{\bm\alpha}$ to get to the derivatives, because for these estimators the derivative of $\hat\psi_{n\hat\bm\alpha_n}$ did not exist.

We get the following result.

\begin{theorem}
	\label{theorem:asymptotics-efficient}
	Let either $\hat\bm\alpha_n$ be the ESE of $\bm\alpha_0$ and  let Assumptions (A1) to (A7) and (A8') and (A9) of the present section be satisfied or let $\hat\bm\alpha_n$ be the spline estimator of $\bm\alpha_0$ and let Assumptions (A0) to (A6) and (B1) to (B3) of  \cite{arun_rohit:20}) be satisfied.
	Moreover, let the bandwidth $h \asymp n^{-1/7}$ in the estimate of the derivative of $\psi_{\bm\alpha}$ for the ESE.
			Define the matrices,
		\begin{align}
		\label{def:I1}
		\tilde {\bm A}:={\mathbb E}\Bigl[\psi_0'(\bm\alpha_0^T\bm X)^2\,\text{\rm Cov}(\bm X|\bm\alpha_0^T\bm X)\Bigr],
		\end{align}
		and
		\begin{align}
		\label{def:I2}
		\tilde {\bm \Sigma}:={\mathbb E}\left[\left\{Y -\psi_0(\bm\alpha_0^T\bm X)\right\}^2\psi_0'(\bm\alpha_0^T\bm X)^2\left\{\bm X -{\mathbb E}(\bm X|\bm\alpha_0^T\bm X) \right\}\left\{\bm X -{\mathbb E}(\bm X|\bm\alpha_0^T\bm X) \right\}^T\right].
		\end{align}
		Then 
		\begin{align*}
		\sqrt n (\tilde \bm\alpha_n - \bm\alpha_0) \to_d N_{d}\left(\bm 0,  \tilde {\bm A}^- \tilde {\bm \Sigma} \tilde {\bm A}^-\right),
		\end{align*}
		where $\tilde {\bm A}^{-}$ is the Moore-Penrose inverse of $\tilde {\bm A}$.	
\end{theorem}

This corresponds to Theorem 6 in \cite{FGH:19} and Theorem 5 in \cite{arun_rohit:20}), but note that the formulation of Theorem 5 in \cite{arun_rohit:20} still contains  the Jacobian connected with the lower dimensional parameterization.


\section{Simulation and comparisons with other estimators}
\label{section:simulations}
\setcounter{equation}{0}

	
In this section we compare the LSE with the Simple Score Estimator (SSE), the Efficient Score Estimator (ESE), the Effective Dimension Reduction (EDR) estimate, the spline estimate, the MAVE estimate and the EFM estimate. We take part of the simulation settings in \cite{BDJ:19}, which means that we take the dimension $d$ equal to $2$. Since the parameter belongs to the boundary of a circle in this case, we only have to determine a $1$-dimensional parameter. Using this fact, we use the parameterization $\bm\alpha=(\alpha_1,\alpha_2)=(cos(\beta),\sin(\beta))$ and determine the angle $\beta$ by a golden section search for the SSE, ESE and spline estimate.  For the EDR we used the {\tt R} package {\tt edr}; the method is discussed in \cite{hristache01}. The spline method is described in \cite{arun_rohit:20}, and there exists an {\tt R} package {\tt simest} for it, but we used our own implementation.
For the MAVE method we used the {\tt R} package {\tt MAVE}, for theory see \cite{xia:06}. For the EFM estimate (see \cite{cui2011}) we used an {\tt R} script, due to Xia Cui and kindly provided to us by her and Rohit Patra. All runs of our simulations can be reproduced by running the {\tt R} scripts in \cite{github:18}.

\begin{figure}[!h]
	\centering
	\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=0.95\textwidth]{Mave_original.pdf}
	\caption{$\hat\bm\alpha_n=(0.71055,0.70364)$}
	\end{subfigure}
	\begin{subfigure}{0.45\linewidth}
	\includegraphics[width=0.95\textwidth]{Mave_antipodal.pdf}
	\caption{$\hat\bm\alpha_n=-(0.71055,0.70364)$}
\end{subfigure}\\
\caption{Two MAVE estimates of $\bm\alpha_0=2^{-1/2}(1,1)^T$ for model 1 with sample size $n=1000$: (a) from starting the iterations at $\bm\alpha_0$, (b) from starting the iterations at $-\bm\alpha_0$; the blue solid curve is the estimate of the link function, based on $\hat\bm\alpha_n$; the blue dashed function is $t\mapsto t^3$ in (a) and $t\mapsto -t^3$ in (b). Note that in (b) also the sign of the first coordinates of the points $(\hat\bm\alpha_n^T\bm X_i,Y_i)$ in the scatterplot is reversed. Under the restriction that the link function is nondecreasing (b) cannot be a solution.}
\label{fig:Mave_example}
	\end{figure}


In simulation model 1 we take $\bm\alpha_0=(1/\sqrt{2},1/\sqrt{2})^T$ and $\bm X=(X_1,X_2)^T$, where $X_1$ and $X_2$ are independent Uniform$(0,1)$ variables. The model is now:
\begin{align*}
Y=\psi_0(\bm\alpha_0^T\bm X)+\varepsilon,
\end{align*}
where $\psi_0(u)=u^3$ and $\varepsilon$ is a standard normal random variable, independent of $\bm X$.


In simulation model 2 we also take $\bm\alpha_0=(1/\sqrt{2},1/\sqrt{2})^T$ and $\bm X=(X_1,X_2)^T$, where $X_1$ and $X_2$ are independent Uniform$(0,1)$ variables.  This time, however, the model is:
\begin{align*}
Y=\text{Bin}\left(10,\exp(\bm\alpha_0^T\bm X)/\left\{1+\exp(\bm\alpha_0^T\bm X)\right\}\right),
\end{align*}
see also Table 2 in \cite{BDJ:19}.
This means:
\begin{align*}
Y=\psi_0(\bm\alpha_0^T\bm X)+\varepsilon,
\end{align*}
where
\begin{align*}
\psi_0(\bm\alpha_0^T\bm X)=10\,\exp(\bm\alpha_0^T\bm X)/\{1+\exp(\bm\alpha_0^T\bm X)\},\qquad \varepsilon=N_n-\psi_0(\bm\alpha_0^T\bm X),
\end{align*}
and
\begin{align*}
N_n=\text{Bin}\left(10,\frac{\exp(\bm\alpha_0^T\bm X}{1+\exp(\bm\alpha_0^T\bm X)}\right).
\end{align*}
Note that indeed ${\mathbb E}\{\varepsilon|\bm X)=0$,
but that we do not have independence of $\varepsilon$ and $\bm X$, as in the previous example.

It was noticed in \cite{xia:06}, p.\ 1113, that, although it was shown in \cite{hristache01} that the $\sqrt{n}$ rate of convergence for the estimation of $\bm\alpha_0$ can be achieved, the asymptotic distribution of the method proposed in \cite{hristache01} was not derived, which makes it difficult to compare the limiting efficiency of the estimation method with other methods. In \cite{xia:06} the asymptotic distribution of the rMAVE estimate is derived (see Theorem 4.2 of \cite{xia:06}), which shows that this limit distribution is actually the same as that of the ESE and the spline estimate. Since Xia is one of the authors of the recent {\tt MAVE} {\tt R} package, we assume that the rMAVE method has been implemented in this package, so we will identify MAVE with rMAVE in the sequel.

The proof of the asymptotic normality result for the MAVE method uses the fact that the iteration steps, described on p.1117 of \cite{xia:06}, start in a neighborhood $\{\bm\alpha:\|\bm\alpha-\bm\alpha_0\|\le C n^{-1/2+c_0}\}$ of $\bm\alpha_0$, where $C>0$ and $c_0<1/20$, and indeed our original experiments with the {\tt R} package showed many outliers, probably due to starting values not sufficiently close to $\bm\alpha_0$. A further investigation revealed that there were many solutions in the neighborhood of the points $-\bm\alpha_0$. This phenomenon is illustrated in Figure \ref{fig:Mave_example}, generated by our own implementation of the algorithm in \cite{xia:06}. The link function is constructed from the values $a_j^{\hat\bm\alpha_n}$ in the algorithm in \cite{xia:06}, p.\ 1117, where the ordered values of $\hat\bm\alpha_n^T\bm X_j$ are the first coordinates.

Because of the difficulty we just discussed, we reversed in the results of the {\tt MAVE} {\tt R} package the sign of the solutions in the neighborhood of $-\bm\alpha_0$. By the parameterization with a positive first coordinate in \cite{cui2011} situation (b) in Figure \ref{fig:Mave_example} cannot occur for the EFM algorithm. We also tried a modification of the same type as our modification of the MAVE algorithm for the EDR algorithm, but this did not lead to a similar improvement of the results.

It follows from Theorem \ref{th:asympt_normality_SSE} that the variance of the asymptotic normal distribution for the SSE is equal to $2.727482$ and from Theorem \ref{theorem:asymptotics-efficient} that the variance of the asymptotic normal distribution for the ESE and spline estimator equals 2.737200. We already noticed in Section \ref{sec:ESE_PLSE} that the present models is not homoscedastic. In this case the asymptotic covariance matrix for the SSE of Theorem \ref{th:asympt_normality_SSE} is in fact given by
$\bm A^-=\bm A^-\bm\Sigma\bm A^-$.


\begin{figure}
	\centering
	\begin{subfigure}{0.4\linewidth}
		\includegraphics[width=1.0\textwidth]{BoxPlot_alpha_uniform100.pdf}
		\caption{$n = 100$}
	\end{subfigure}
	\begin{subfigure}{0.4\linewidth}
	\includegraphics[width=1.0\textwidth]{BoxPlot_alpha_uniform500.pdf}
	\caption{$n = 500$}
\end{subfigure}\\
	\begin{subfigure}{0.4\linewidth}
		\includegraphics[width=1.0\textwidth]{BoxPlot_alpha_uniform1000.pdf}
		\caption{$n=1000$}
	\end{subfigure}
	\begin{subfigure}{0.4\linewidth}
	\includegraphics[width=1.0\textwidth]{BoxPlot_alpha_uniform5000.pdf}
	\caption{$n = 5000$}
	\end{subfigure}
	\caption{Boxplots of $\sqrt{n/2}\,\|\hat\bm\alpha_n-\bm\alpha_0\|_2$ for model 1.  In (b) and  (c) the values of EDR were truncated at $0.6$ to show more clearly the differences between the other estimates.}
	\end{figure}


\begin{table}
	\label{table:simulation1}
	\caption{Simulation, model 1; $\varepsilon_i$ is standard normal and independent of $\bm X_i$, consisting of two independent Uniform$(0,1)$ random variables.  The mean value $\hat\mu_i$ =  mean($\hat \alpha_{in}),\, i=1,2$ and $n$ times the variance-covariance $\hat\sigma_{ij}$ $=n\cdot$cov$(\hat\alpha_{in},\hat\alpha_{jn}),\,i,\,j=1,2,$\\ of the Efficient Dimension Reduction Estimate EDR, computed by the {\tt R} package {\tt edr}, the Least Squares Estimate (LSE), the Simple Score Estimate (SSE), the Efficient Score Estimate (ESE), the spline estimate, the MAVE estimate and the EFM estimate for different sample sizes $n$. The line, preceded by $\infty$, gives the asymptotic values (unknown for EDR and LSE). The values are based on $1000$ replications.}
	\vspace{0.5cm}
	\scalebox{1.0}{
		\begin{tabular}{|lr|cc|rrr |}
			\hline
			Method&$n$ & $\hat\mu_1$ & $\hat\mu_2$  & $\hat \sigma_{11}$& $\hat\sigma_{22}$& $\hat\sigma_{12}$\\						\hline
			&&&&&&\\
			EDR&100 & 0.621877 & 0.361894 &11.409222&36.869184& 9.152389  \\
			&500 & 0.701217 & 0.686094 & 7.334756 & 11.468453 & -3.881349 \\
			&1000 & 0.701669 & 0.702244  & 6.437653 & 8.090771& -3.552562 \\
			&5000& 0.706021 & 0.706798  &7.344431& 7.276717 & -7.288047\\
			\hline
			&$\infty$&0.707107 & 0.707107  & ? & ? & ?\\
		\hline
		&&&&&&\\
			LSE&100 & 0.672698 & 0.697350  &3.148912& 2.975246& -2.915427  \\
			&500 & 0.702163 & 0.701718  & 3.620960 & 3.665710 & -3.588491 \\
			&1000 & 0.704706 &0.704320  & 3.665561 & 3.664711 & -3.637541  \\
			&5000& 0.707262  & 0.705690  & 4.435842 & 4.485168 & -4.453713\\
			\hline
			&$\infty$&0.707107 & 0.707107  & ? & ? & ?\\
		\hline
		&&&&&&\\
			SSE&100 & 0.673997 & 0.6919403  &3.338637& 3.362656& -3.141408  \\
			&500 & 0.699986 & 0.706198  & 2.849647 & 2.807978 & -2.793798\\
			&1000 & 0.706477 & 0.704191  & 2.501106 & 2.510047 & -2.494237  \\
			&5000& 0.707090 & 0.706423  & 2.473765 & 2.485884 & -2.477371 \\
			\hline
			&$\infty$&0.707107 & 0.707107  & 2.343750 & 2.343750 & -2.343750\\
		\hline
		&&&&&&\\
			ESE&100 & 0.682781 & 0.687949  &3.067802& 2.991976& -2.855176  \\
			&500 & 0.702940 & 0.702462  & 3.100843& 3.116337 & -3.064151 \\
			&1000 & 0.704055 & 0.706387  & 2.676388 & 2.653164 & -2.650667  \\
			&5000& 0.707130 & 0.706444  & 2.257541 & 2.265547 & -2.259443 \\
			\hline
			&$\infty$&0.707107 & 0.707107  & 1.885522 & 1.885522 & -1.885522\\
		\hline
			&&&&&&\\
			spline&100 & 0.690741 & 0.705485  &1.801235& 1.762567& -1.711552  \\
			&500 & 0.703670 & 0.702640  & 1.795384 & 1.778454 & -1.773560 \\
			&1000 & 0.705684 &0.706007  & 1.786589 & 1.781797 & -1.777691  \\
			&5000&  0.706404 & 0.707193  & 2.180466 & 2.181544 & -2.179269 \\
			\hline
			&$\infty$&0.707107 &  0.707165  & 1.885522 & 1.885522 & -1.885522\\
		\hline
			&&&&&&\\
			MAVE&100 & 0.686503 & 0.684887  &2.423618& 3.546768& -2.245708  \\
			&500 & 0.703333 & 0.705537  & 1.897806 & 1.876220 & -2.040677 \\
			&1000 & 0.705840 &0.705660  & 1.929966 & 1.907128 & -1.911452  \\
			&5000& 0.707328 &0.706299  & 2.071168 & 2.082169 & -2.074914 \\
			\hline
			&$\infty$&0.707107 & 0.707107  & 1.885522 & 1.885522 & -1.885522\\
		\hline
			&&&&&&\\
			EFM&100 & 0.686292 & 0.684274  &2.802308& 3.280956& -2.312445  \\
			&500 & 0.703236 & 0.705133  & 2.082162 & 2.045150 & -2.044960 \\
			&1000 & 0.705629 & 0.705950  & 1.866486 & 1.860184 & -1.856340  \\
			&5000& 0.707269 & 0.707251  & 1.953800 & 1.964081 & -1.957351 \\
			\hline
			&$\infty$&0.707107 & 0.707107  & 1.885522 & 1.885522 & -1.885522\\
		\hline
		\end{tabular}}
	\end{table}

	\begin{figure}
	\centering
	\begin{subfigure}{0.4\linewidth}
		\includegraphics[width=1.0\textwidth]{BoxPlot_alpha100_binomial.pdf}
		\caption{$n = 100$}
	\end{subfigure}
	\begin{subfigure}{0.4\linewidth}
	\includegraphics[width=1.0\textwidth]{BoxPlot_alpha500_binomial.pdf}
	\caption{$n = 500$}
\end{subfigure}\\
	\begin{subfigure}{0.4\linewidth}
		\includegraphics[width=1.0\textwidth]{BoxPlot_alpha1000_binomial.pdf}
		\caption{$n=1000$}
	\end{subfigure}
	\begin{subfigure}{0.4\linewidth}
	\includegraphics[width=1.0\textwidth]{BoxPlot_alpha5000_binomial.pdf}
	\caption{$n = 5000$}
	\end{subfigure}
	\caption{Boxplots of $\sqrt{n/2}\,\|\hat\bm\alpha_n-\bm\alpha_0\|_2$ for model 2.  In (b) and  (c) the values of EDR were truncated at $0.6$ to show more clearly the differences between the other estimates.}

\end{figure}


\begin{table}
	\label{table:simulation2}
	\caption{Simulation, model 2; $Y_i\sim\text{Bin}\left(10,\exp(\bm\alpha_0^T\bm X_i)/\left\{1+\exp(\bm\alpha_0^T\bm X_i)\right\}\right)$, where $\bm X_i$ consists of two independent Uniform$(0,1)$ random variables.  The mean value $\hat\mu_i =  \text{mean}(\hat \alpha_{in}),\, i=1,2$ and $n$ times the variance-covariance $n\text{cov}(\hat\alpha_{in},\hat\alpha_{jn}),\,i,\,j=1,2,$ of the Efficient Dimension Reduction Estimate EDR, computed by the {\tt R} package {\tt edr}, the Least Squares Estimate (LSE), the Simple Score Estimate (SSE), the Efficient Score Estimate (ESE), the spline estimate,  the MAVE estimate and the EFM estimate for different sample sizes $n$. The line, preceded by $\infty$, gives the asymptotic values (unknown for EDR and LSE). The values are based on $1000$ replications.}
	\vspace{0.5cm}
	\scalebox{1.0}{
		\begin{tabular}{|lr|cc|ccc |}
			\hline
			Method&$n$ & $\hat\mu_1$ & $\hat\mu_2$  & $\hat \sigma_{11}$& $\hat\sigma_{22}$& $\hat\sigma_{12}$\\
			\hline
			&&&&&&\\
			EDR&100 & 0.587264 & 0.202005  &13.33724&48.15572& 11.87625  \\
			&500 & 0.670702 & 0.602469 & 26.76111 & 66.92737 & 14.09701 \\
			&1000 & 0.696075 & 0.666591  & 21.89080 & 49.31544& 9.345753 \\
			&5000& 0.704424 & 0.706604 & 11.39598 & 11.11493 & -11.17376\\
			\hline
			&$\infty$&0.707107 & 0.707107  & ? & ? & ?\\
		\hline
		&&&&&&\\
			LSE&100 & 0.658631 & 0.699725  &4.069966& 3.596783& -3.609490  \\
			&500 &0.695541 & 0.703007  & 5.650618 & 5.362877 & -5.358190 \\
			&1000 &0.704497 & 0.701243 & 5.909494 & 6.043808 & -5.911246  \\
			&5000& 0.704805 & 0.707621  & 6.303320 & 6.321866 & -6.298515\\
			\hline
			&$\infty$&0.707107 & 0.707107  & ? & ? & ?\\
		\hline
		&&&&&&\\
			SSE&100 & 0.667908 & 0.694376 &3.760921& 3.420387&-3.356968  \\
			&500 & 0.698498 & 0.706423 & 3.358458 & 3.182044 & -3.223734\\
			&1000 & 0.707276 & 0.702390  & 3.179623 & 3.236283 & -3.184724  \\
			&5000& 0.706162 & 0.707286  & 2.718742 & 2.707549 & -2.709870 \\
			\hline
			&$\infty$&0.707107 & 0.707107  & 2.727482 &2.727482 & -2.727482\\
		\hline
		&&&&&&\\
			ESE&100 & 0.684804 & 0.688063  &2.892165& 2.874755& -2.744223  \\
			&500 & 0.698078& 0.706159  & 3.562625 & 3.457337 & -3.446605 \\
			&1000 & 0.707879 & 0.701445  & 3.420159& 3.470217 & -3.418606  \\
			&5000& 0.706321 & 0.707110  & 2.775092 & 2.760287 &-2.764230 \\
			\hline
			&$\infty$&0.707107 & 0.707107  & 2.737200 & 2.737200 & -2.737200\\
		\hline
			&&&&&&\\
			spline&100 &0.677287& 0.695301  &3.009781&2.779876 & -2.714928  \\
			&500 & 0.699117 & 0.706946  & 2.952928 & 2.784383 & -2.830415 \\
			&1000 &0.707890 & 0.702001  & 3.027712 & 3.064772 & -3.026082  \\
			&5000& 0.706200& 0.707312  & 2.764447 & 2.762986 & -2.760530 \\
			\hline
			&$\infty$&0.707107 & 0.707232  & 2.737200 & 2.737200 & -2.737200\\
		\hline
			&&&&&&\\
			MAVE&100 & 0.667849 & 0.654361  &3.891510& 8.700093& -2.325804  \\
			&500 & 0.699108 & 0.706377  & 3.155191 & 2.990569 & -3.031249 \\
			&1000 & 0.707520 & 0.702341  & 3.040201 & 3.097965 & -3.049075  \\
			&5000& 0.707657& 0.705827  & 2.572343 & 2.573418 & -2.570275 \\
			\hline
			&$\infty$&0.707107 & 0.707107  & 2.737200 & 2.737200 & -2.737200\\
		\hline
			&&&&&&\\
			EFM&100 & 0.663227 & 0.666070  &5.681573& 5.978194& -2.503058  \\
			&500 &0.698920 & 0.706295  & 3.279110 & 3.055940 & -3.118757 \\
			&1000 &0.707878 & 0.706275  & 3.102414 & 3.157143 &-3.108516  \\
			&5000& 0.706043&0.701894  & 2.669352 & 2.650343 & -2.656742 \\
			\hline
			&$\infty$&0.707107 & 0.707107  & 2.737200 & 2.737200 & -2.737200\\
		\hline

		\end{tabular}}
	\end{table}
	


It is clear that the  estimate EDR is inferior to the other methods for these models; even the LSE for which we do not know the rate of convergence has a better performance. In \cite{hristache01} it is assumed that the errors have a normal distribution, but also in model 1, where this condition is satisfied, the behavior is clearly inferior, in particular for the lower sample sizes.


\section{Concluding remarks}
We replaced the ``crossing of zero'' estimators in \cite{FGH:19} by profile least squares estimators. The asymptotic distribution of the estimators was determined and its behavior illustrated by a simulation study, using the same models
 as in \cite{BDJ:19}.
 
 In the first model the error is independent of the covariate and homoscedastic and in this case two of the estimators were efficient. In the other (binomial-logistic) model the error was dependent on the covariates and not homoscedastic. It was shown that the SSE (Simple Score Estimate) had in fact a smaller asymptotic variance in this model than the other estimators for which the asymptotic variance is known, although the difference is very small and does not really show up in the simulations.
 
There is no uniformly best estimate in our simulation, but the EDR estimate is clearly inferior to the other estimates, inluding the LSE, in particular for the lower sample sizes. On the other hand, the LSE is inferior to the other estimators except the EDR. All simulation results can be reproduced by running the {\tt R} scripts in \cite{github:18}.

\section*{Acknowledgement}
\label{section:acknowledgement}
We thank Vladimir Spokoiny for helpful discussions during the Oberwolfach meeting ``Statistics meets Machine Learning'', January 26 - February 1, 2020.               

\bibliographystyle{plainnat}
