Honest confidence sets in nonparametric IV
regression and other ill-posed models

arXiv:1611.03015v1 [] 9 Nov 2016

Andrii Babii*
Latest version: November 10, 2016

This paper provides novel methods for inference in a very general class of ill-posed
models in econometrics, encompassing the nonparametric instrumental regression,
different functional regressions, and the deconvolution. I focus on uniform confidence
sets for the parameter of interest estimated with Tikhonov regularization, as in
Darolles, Fan, Florens, and Renault (2011). I first show that it is not possible to
develop inferential methods directly based on the uniform central limit theorem. To
circumvent this difficulty I develop two approaches that lead to valid confidence
sets. I characterize expected diameters and coverage errors uniformly over a large
class of models (i.e. constructed confidence sets are honest). Finally, I illustrate
that introduced confidence sets have reasonable width and coverage properties in
samples commonly used in applications with Monte Carlo simulations and considering
application to Engel curves.
Keywords: nonparametric instrumental regression, functional linear regression, density deconvolution, honest uniform confidence sets, non-asymptotic inference, ill-posed models, Tikhonov
regularization
JEL classification: C14, C36

*

Ph.D. Candidate, Toulouse School of Economics, Université Toulouse 1 Capitole, babii.andrii@gmail.com. I’m
deeply indebted to my advisor Jean-Pierre Florens and to Eric Gautier and Ingrid Van Keilegom for many
helpful suggestions and insightful conversations. This paper also benefited from discussions with Rohit Kumar,
Elia Lapenta, Pascal Lavergne, and André Mas. I would also like to thank Christian Bontemps, Samuele
Centorrino, Jasmin Fliegner, Emanuele Guerre, Vitalijs Jascisens, Jihyun Kim, Thierry Magnac, Nour Meddahi,
Shruti Sinha, and other participants of TSE econometrics workshop for helpful comments and suggestions.

1

1 Introduction
This paper develops honest and uniform confidence sets for the structural function 𝜙 in a very
broad class of ill-posed models treated with the Tikhonov regularization. The leading example is
a nonparametric instrumental regression (NPIV) studied in Newey and Powell (2003), Florens
(2003), Darolles et al. (2011), Hall and Horowitz (2005), and Blundell, Chen, and Kristensen
(2007), and specified as
E[𝑈 |𝑊 ] = 0,

𝑌 = 𝜙(𝑍) + 𝑈,

(1)

where 𝑍 ∈ R𝑝 is a vector of explanatory variables, 𝑊 ∈ R𝑞 is a vector of instruments (both
continuously distributed), and 𝜙 is the function of interest. The NPIV model is ill-posed in the
sense that the map from the distribution of the data to the function 𝜙 is not continuous. To have
consistent estimator certain amount of regularization is needed to smooth out discontinuities.
In empirical studies the function 𝜙 represents a structural economic relation, such as Engel curve,
cost function, or demand curve. To infer the magnitude of economic effects, it is not sufficient
just to estimate this function. The magnitude of economic effects can only be inferred from
confidence sets. This paper is the first to provide inferential methods for Tikhonov-regularized
estimators. I focus on the uniform inference, which amounts to constructing a set containing
the entire function 𝜙 with a high probability. Uniform inference allows to asses global features
of the estimated function and to quantify the range of possible economic effects compatibly
with the data. Global features may include the evidence for non-linearities, the amount of
endogeneity bias comparing to the local polynomial estimator, or other shape properties. In
contrast, pointwise confidence intervals only contain the value 𝜙(𝑧0 ) at some particular point
𝑧0 with high probability and do not provide valid inference for the entire function 𝜙. Another
important feature of confidence sets constructed in this paper is honesty in the sense of Li (1989).
Honesty of confidence sets means that their coverage properties are valid uniformly over a large
class of specified models1 . Honesty is desirable since the underlying model is never known and
coverage properties of dishonest sets may vary from one model to another2 .
Building uniform confidence sets for function 𝜙 requires to approximate the distribution of the
supremum of the variance of the estimator. I show that for a broad class of ill-posed models
treated with Tikhonov regularization the variance of the estimator does not converge weakly in
the space of continuous functions. As a result, it is not possible to build the uniform confidence
set relying on the uniform central limit theorem. This calls for alternative approaches to inference.
1
2

See Eq. (6) in Section 3 for a formal definition.
Honest and dishonest sets are also called sets uniformly consistent in levels and pointwise consistent in levels.

2

In order to circumvent this problem I relax the requirement of asymptotically exact inference
and focus on inference that only ensures asymptotic coverage with probability at least 1 − 𝛾, for
𝛾 ∈ (0, 1). This relaxed coverage requirement is frequently used in econometrics, for instance,
for weakly identified models in Andrews and Mikusheva (2016), moment inequalities models in
Andrews and Soares (2010), and partial identification approach to NPIV model in Santos (2012).
To construct confidence sets with good coverage properties I rely on two different approaches.
The first approach is to focus on a suitably normalized stochastic upper bound on the supremum
of the variance of the estimator. This upper bound holds under quite general assumptions. For
inference I obtain Gaussian approximation to this upper bound and use a suitable quantile of
this Gaussian approximation to construct a confidence set for the function 𝜙. This approach is
flexible and valid for a large class of models.
The second inferential method developed in this paper relies on estimates of tail probabilities
with a suitable concentration inequality. To the best of my knowledge this approach to inference
for functions is new and this paper is the first to introduce it in econometrics. A simple illustration
of this approach is to build the uniform confidence band for the empirical distribution function
using the Dworetzky-Kiefer-Wolfowitz inequality. Given the empirical distribution function
∑︀
𝐹𝑛 (𝑥) = 𝑛1 𝑛𝑖=1 1{𝑋𝑖 ≤𝑥} based on the i.i.d. sample (𝑋𝑖 )𝑛𝑖=1 , the Dworetzky-Kiefer-Wolfowitz
tells us that for any finite sample size 𝑛 the probability of large deviations of 𝐹𝑛 from the true
distribution function 𝐹 in the supremum norm declines at the exponential rate
2

Pr (‖𝐹𝑛 − 𝐹 ‖∞ > 𝑥) ≤ 2𝑒−2𝑛𝑥 ,
Setting 𝑥 =

√︁

log(2/𝛾)
2𝑛

𝑥 > 0.

and 𝛾 ∈ (0, 1), the inequality becomes
(︃

√︂

Pr ‖𝐹𝑛 − 𝐹 ‖∞ ≤

log(2/𝛾)
2𝑛

)︃
≥ 1 − 𝛾.

This allows us to build the uniform confidence band for 𝐹 (𝑥) with guaranteed coverage probability
√︁
1 − 𝛾 taking 𝐹𝑛 (𝑥) ± log(2/𝛾)
2𝑛 . In this simple example there is no coverage error (the coverage is
√
at least 1 − 𝛾 for any finite sample size 𝑛) and the diameter of the set shrinks at the rate 1/ 𝑛.
This inferential approach should be contrasted with the alternative asymptotic approach based on
the Donsker central limit theorem, which also leads to confidence sets with diameters shrinking
√
at the rate 1/ 𝑛, but having the coverage level 1 − 𝛾 − 𝑜(1). The coverage error disappears
only as sample size goes to infinity. The empirical distribution function is an unbiased estimator,
while the majority of estimators of functions are biased. As a result, usually confidence sets for

3

such estimators, including ill-posed models considered in this paper, have coverage errors.
Concentration inequalities similar to the Dworetzky-Kiefer-Wolfowitz inequality for more
complex statistics than ‖𝐹𝑛 − 𝐹 ‖∞ are called Talagrand’s concentration inequalities, see for
instance Boucheron, Lugosi, and Massart (2013). In particular, I exploit the data-driven
Talagrand-type concentration inequality for the supremum of the variance of the estimator to
approximate quantiles of the unknown distribution. This approach does not rely on the existence
of the supremum of the Gaussian process approximating the supremum of the variance of the
estimator. As a result, it is valid for a wide class of data-generating processes and is especially
useful in settings where all other approaches to inference fail. Another remarkable feature of
confidence sets based on concentration inequality is that their coverage error will be driven by
the bias of the estimator and not by the accuracy of Gaussian approximation. Lastly, they are
significantly easier to implement and computationally faster than the approach based on the
Gaussian approximation.
Whenever possible, I characterize convergence rates for coverage errors and expected diameters.
It is especially important to know both rates, since consistency alone may not be very informative
for inference on function 𝜙. For instance, it may happen that the coverage error decreases slower
than the confidence set shrinks, implying that larger sample sizes will be needed to achieve good
coverage. I show that coverage errors of confidence sets are driven by the bias of the estimator.
Constructed confidence sets have excellent statistical properties. Their expected diameters
and coverage errors may decrease at polynomial rates. Convergence rates for coverage errors are
new, and have not been previously discussed neither for the NPIV model, nor for other ill-posed
models considered in this paper.
Though, the Tikhonov-regularized NPIV is the leading example, the inferential methods
developed in this paper are valid in other ill-posed models. This includes functional regression
models and the density deconvolution model. To the best of my knowledge, despite the existing
extensive literature on 𝐿2 results for the functional regression models, no uniform convergence
rates or uniform inferential methods are currently available. I provide more detailed comparison
to the existing literature below.
Related literature.

As was mentioned, this paper is the first to develop honest uniform inferential

methods in a general and unifying framework, encompassing different ill-posed models treated
with Tikhonov regularization. Previously, uniform confidence sets were studied only for sieve-type
estimators of the NPIV model and the density deconvolution model. Moreover, neither honesty,
nor rates for coverage errors were previously discussed. Horowitz and Lee (2012) develop uniform

4

confidence bands by first constructing pointwise confidence intervals at a finite grid of points and
then, letting the number of grid points to grow at a certain speed to achieve uniform coverage.
Chen and Christensen (2015) focus on uniform inference for functionals and notice that using
point evaluation functional may lead to uniform confidence sets under certain assumptions.
Lounici and Nickl (2011) obtain uniform confidence sets for the wavelet deconvolution estimator
relying on the Bousquet’s version of the Talagrand’s concentration inequality.
Kato and Sasaki (2016) consider more general case, where the density of the noise is not
known and is estimated from auxiliary sample. This paper studies the estimator based on Fourier
inversion and builds on coupling inequalities developed in Chernozhukov, Chetverikov, Kato,
et al. (2014) and Chernozhukov, Chetverikov, and Kato (2015).
Tikhonov-regularized estimators offer an appealing alternative to sieves. For instance, I show
that confidence sets developed in this paper may enjoy polynomial convergence rates of coverage
errors and expected diameters. At the same time, Tikhonov-regularized estimators do not require
to specify sieve bases and change smoothly with respect to tuning-parameters. It is also known
that even if the function 𝜙 is not identified, the Tikhonov-regularized estimator is still well-defined.
In this case it converges to the best approximation to the true function 𝜙 in the orthogonal
complement to the null space of the operator, see Florens, Johannes, and Van Bellegem (2011)
and Babii and Florens (2016). This is especially important in the light of the fact that the
identification condition in this class of models is not testable, see Canay, Santos, and Shaikh
(2012). The behavior of sieve estimators in the non-identified case and coverage properties
of confidence sets for sieve estimators is an open question. Lastly, the computation of sieve
estimators involves inversion of a random matrix, which may often be singular in applications
and needs to be additionally regularized, e.g. with Tikhonov regularization. On the other hand,
the Tikhonov-regularized estimator will always be well-defined.
It is also worth mentioning that to build a valid uniform confidence set, it is necessary to
know uniform convergence rates of the estimator. Gagliardini and Scaillet (2012) study the
Tikhonov-regularized minimum distance estimator, which is different from the one considered in
the present paper. Mbakop (2015) obtains convergence rates in probability for the sup-norm loss
function in the Tikhonov-regularized NPIV model. Convergence rates obtained in this paper
are of order of powers of log 𝑛/𝑛 and depend on the dimension of IV. To obtain good coverage
properties of confidence sets, I encountered the necessity of stronger result on convergence rates
of the risk of the estimator (that is the expected value of the sup-norm). I show that under the
mild restriction, uniform convergence rates of the Tikhonov regularized estimator are of order
of powers of 1/𝑛 and do not necessary depend on the dimension of the IV, which sharpens the

5

result of Mbakop (2015). These rates are valid for mildly ill-posed and some severely ill-posed
cases.3 .
Lastly, some pointwise inferential results are available for spectral cut-off estimators when
the ill-posed operator is known and is not estimated from the data, see Carrasco and Florens
(2011), Gautier and Kitamura (2013), and Florens, Horowitz, and Keilegom (2016). In this paper
I consider more realistic setting of the nonparametric IV and functional regressions, when the
operator is not known and is estimated from the data. Meanwhile Cardot, Mas, and Sarda (2007)
provide results on the asymptotic normality of linear functionals, which solves the prediction
problem for functional regression models. Lastly, Carrasco, Florens, and Renault (2013) study
asymptotic normality under the 𝐿2 norm for a fixed value of the regularization parameter 𝛼𝑛 .
The paper is organized as follows. I introduce notation in the remaining part of this section.
Section 2 introduces motivating examples for which we provide inferential methods. In Section 3 I
describe the problem of constructing honest uniform confidence sets and introduce two inferential
approaches. Under a general set of assumptions, that are verified later on in each particular
application, I establish convergence rates for coverage errors and diameters of constructed sets,
uniform over a general set of models. Section 4 introduces specific smoothness and regularity
conditions, while Section 5 applies results of Section 3 to the NPIV model, different functional
regression models, and density deconvolution. In Section 6 I show how to implement confidence
sets in practice for the NPIV estimator and explore their finite-sample properties with Monte
Carlo experiments. Section 7 considers the empirical application to Engel curves and Section 8
concludes.
Notation. Let 𝐿2 [𝑎, 𝑏]𝑝 denote the space of functions on some compact set [𝑎, 𝑏]𝑝 ⊂ R𝑝 , square
integrable with respect to the Lebesgue measure 𝜆. For 𝜙 ∈ 𝐿2 [𝑎, 𝑏]𝑝 , let ‖.‖ denote the usual 𝐿2
norm derived from the inner product ⟨., .⟩. Let 𝐶[𝑎, 𝑏]𝑝 denote the space of continuous functions
𝛽
endowed with supremum norm ‖.‖∞ . For some positive real number 𝛽, let 𝐶𝑀
[𝑎, 𝑏]𝑝 denote the

class of 𝛽-Hölder functions on (𝑎, 𝑏)𝑝 with 0 < 𝑀 < ∞
{︃
𝛽
𝐶𝑀
[𝑎, 𝑏]𝑝 =

3

𝜙 ∈ 𝐶[𝑎, 𝑏]𝑝 : max ‖𝑓 (𝑘) ‖∞ ≤ 𝑀, max sup
|𝑘|≤⌊𝛽⌋

|𝑘|=⌊𝛽⌋ 𝑧̸=𝑧 ′

⃒ (𝑘)
⃒
⃒𝜙 (𝑧) − 𝜙(𝑘) (𝑧 ′ )⃒
‖𝑧 − 𝑧 ′ ‖𝛽−⌊𝛽⌋

}︃
≤𝑀

,

It is known that minimax-optimal uniform convergence rates in the severely ill-posed case are of order of powers
of 1/ log 𝑛, see Chen and Christensen (2015). However, in the more restricted smoothness class, when the
estimated function can be described by the finite number of generalized Fourier coefficients (with respect to
the SVD basis), or more generally is analytic, it is possible to achieve faster uniform convergence rates of order
of powers of 1/𝑛. This information is especially helpful, since in many empirical settings estimated functions
are very smooth and regular.

6

where 𝑘 = (𝑘1 , . . . , 𝑘𝑝 ) ∈ N𝑝 is a multi-index, |𝑘| =

∑︀𝑝

𝑗=1 𝑘𝑗 ,

𝜙(𝑘) (𝑧) =

𝜕 |𝑘| 𝜙(𝑧)
𝑘

𝑘

𝜕𝑧1 1 ...𝜕𝑧𝑝 𝑝

, and ⌊𝛽⌋ is

the largest integer strictly smaller than 𝛽. Let ℒ2 , ℒ2,∞ and ℒ∞ be spaces of bounded linear
operators from 𝐿2 to 𝐿2 , from 𝐿2 to 𝐶, and from 𝐶 to 𝐶 respectively. Sets on which functions
are defined should be clear from the context. Spaces ℒ2 , ℒ2,∞ , and ℒ∞ are endowed with
standard operator norms, denoted by ‖𝐾‖ = sup‖𝜙‖≤1 ‖𝐾𝜙‖, ‖𝐾‖2,∞ = sup‖𝜙‖≤1 ‖𝐾𝜙‖∞ , and
‖𝐾‖∞ = sup‖𝜙‖∞ ≤1 ‖𝐾𝜙‖∞ , respectively. For 𝐾 ∈ ℒ2 , let 𝐾 * denote its Hilbert adjoint operator.
Let ℛ(𝑇 ) and 𝒟(𝑇 ) be respectively the range and the domain of the operator 𝑇 . Lastly, for two
real numbers 𝑎 and 𝑏, I denote 𝑎 ∧ 𝑏 = min{𝑎, 𝑏} and 𝑎 ∨ 𝑏 = max{𝑎, 𝑏}.

2 Motivating examples
2.1 Nonparametric IV
The NPIV model, described in Eq. (1) is
E[𝑈 |𝑊 ] = 0.

𝑌 = 𝜙(𝑍) + 𝑈,

The model leads to the following ill-posed equation
E[𝑌 |𝑊 = 𝑤] = E[𝜙(𝑍)|𝑊 = 𝑤].

(2)

Florens (2003) and Darolles et al. (2011) use Tikhonov regularization to build the estimator which
solves the regularized empirical counterpart to this equation. In this case both the conditional
mean-function 𝑟(𝑤) = E[𝑌 |𝑊 = 𝑤] and the conditional expectation operator 𝑇 are estimated
non-parametrically. In this paper I slightly modify this approach, multiplying Eq. (2) by the
density function 𝑓𝑊
∫︁
𝑟(𝑤) := E[𝑌 |𝑊 = 𝑤]𝑓𝑊 (𝑤) =

𝜙(𝑧)𝑓𝑍𝑊 (𝑧, 𝑤)d𝑧 =: (𝑇 𝜙)(𝑤).
[𝑎,𝑏]𝑝

The advantage of working with this equation is that it does not require handling random
denominators in the kernel estimation of conditional mean functions, which simplifies our
development of uniform asymptotic results.
For simplicity, I use product kernel estimator with equal bandwidth parameters ℎ𝑛 → 0 as
𝑛 → ∞ for all coordinates to estimate the joint density 𝑓𝑍𝑊 . Then, the estimators of different

7

components of the model are

𝑟^(𝑤) =

𝑛
(︀
)︀
1 ∑︁
𝑌𝑖 𝐾𝑤 ℎ−1
𝑞
𝑛 (𝑊𝑖 − 𝑤)
𝑛ℎ𝑛
𝑖=1

𝑛
(︀
)︀
(︀ −1
)︀
1 ∑︁
𝐾𝑧 ℎ−1
𝑛 (𝑍𝑖 − 𝑧) 𝐾𝑤 ℎ𝑛 (𝑊𝑖 − 𝑤)
𝑝+𝑞
𝑛ℎ𝑛 𝑖=1
∫︁
𝜙(𝑧)𝑓^𝑍𝑊 (𝑧, 𝑤)d𝑧.
(𝑇^𝜙)(𝑤) =

𝑓^𝑍𝑊 (𝑧, 𝑤) =

[𝑎,𝑏]𝑝

The adjoint operator 𝑇^* is defined as a solution to ⟨𝑇^𝜙, 𝜓⟩ = ⟨𝜙, 𝑇^* 𝜓⟩ and can be computed,
applying Fubini’s theorem to change the order of integration
(𝑇^* 𝜓)(𝑧) =

∫︁

𝜓(𝑤)𝑓^𝑍𝑊 (𝑧, 𝑤)d𝑤.

[𝑎,𝑏]𝑞

This operator is used in the expression of the Tikhonov-regularized estimator, see the following
Section 3.

2.2 Functional regressions
In the functional linear regression, the real dependent variable 𝑌 is explained by the continuoustime stochastic process 𝑍(𝑡), 𝑡 ∈ [0, 1]𝑝 . The distinctive feature of this class of models is that it
allows to handle high-dimensional data without relying on the sparsity assumption, which may
be restrictive in this setting. There is vast literature on estimation and testing for the functional
regression models in statistics, see Hall, Horowitz, et al. (2007), Fan, James, Radchenko, et al.
(2015), Comte, Johannes, et al. (2012), Cardot, Ferraty, Mas, and Sarda (2003), and some
growing literature in econometrics, see Florens and Van Bellegem (2015), Benatia, Carrasco, and
Florens (2015), Babii (2016). However all these paper focus on the estimation in the 𝐿2 norm
and to the best of my knowledge, there are no currently available uniform inferential methods
for functional regression models.
The functional IV regression model is described by
∫︁
𝑌 =

𝜙(𝑡)𝑍(𝑡)d𝑡 + 𝑈,
[0,1]𝑝

E[𝑈 𝑊 (𝑠)] = 0,

∀𝑠 ∈ [0, 1]𝑞 .

The slope parameter 𝜙 measures the strength of the impact of the process 𝑍 at different points
𝑡 ∈ [0, 1]𝑝 . If 𝑊 = 𝑍, we obtain the classical functional linear regression model without
endogeneity, see, for instance, Hall et al. (2007), while in the IV case of Florens and Van Bellegem
(2015), 𝑊 is some functional instrumental variable, uncorrelated with the error term.

8

The uncorelatedness assumption leads to the ill-posed equation
∫︁
𝑟(𝑠) := E[𝑌 𝑊 (𝑠)] =

𝜙(𝑡)E[𝑍(𝑡)𝑊 (𝑠)]d𝑡 =: (𝑇 𝜙)(𝑠).
[0,1]𝑝

The slope parameter 𝜙 is identified when the covariance operator is 1-1, which generalizes the
non-singularity condition for the covariance matrix in the finite-dimensional linear regression
model.
A variation of this model is studied in Babii (2016), where the identification is achieved
with real-valued instrumental variable through the conditional moment restriction E[𝑈 |𝑊 ] = 0.
The identifying restriction in this case is the linear completeness condition. Conditional meanindependence leads to the following ill-posed equation
∫︁
E[𝑌 |𝑊 ] =

𝜙(𝑡)E[𝑍(𝑡)|𝑊 ]d𝑡.

(3)

[0,1]𝑝

Consider now a function Ψ such that the map 𝜓(𝑊 ) ↦→ E[𝜓(𝑊 )Ψ(𝑠, 𝑊 )] is 1-1, see Stinchcombe
and White (1998). Applying this map to Eq. (3) leads to another ill-posed equation
∫︁
𝑟(𝑠) := E[𝑌 Ψ(𝑠, 𝑊 )] =

𝜙(𝑡)E[𝑍(𝑡)Ψ(𝑠, 𝑊 )] =: (𝑇 𝜙)(𝑠).
[0,1]𝑝

In what follows we denote by 𝑊 (𝑠) either the regressor 𝑍(𝑠), some functional IV 𝑊 (𝑠), or
the function Ψ(𝑠, 𝑊 ) of some real IV 𝑊 . This notation allows to encompass three functional
regression models: the model of Hall et al. (2007), the model of Florens and Van Bellegem (2015),
and the model studied in Babii (2016).
Unlike the NPIV, which requires non-parametric estimation of the joint density function, all
components of functional regression models can be estimated at the parametric rate using sample
analogs to population moments
𝑛

𝑛

1 ∑︁
𝑟^(𝑠) =
𝑌𝑖 𝑊𝑖 (𝑠),
𝑛

∑︁
^ 𝑠) = 1
𝑘(𝑡,
𝑍𝑖 (𝑡)𝑊𝑖 (𝑠).
𝑛

𝑖=1

𝑖=1

Operator 𝑇^ can be estimated as
(𝑇^𝜙)(𝑠) =

∫︁

^ 𝑠)d𝑡
𝜙(𝑡)𝑘(𝑡,

[0,1]𝑝

9

The adjoint operator 𝑇^* can be obtained using the Fubini’s theorem
(𝑇^* 𝜙)(𝑡) =

∫︁

^ 𝑠)d𝑠.
𝜓(𝑠)𝑘(𝑡,

[0,1]𝑞

2.3 Density deconvolution
Often economic data are not not measured precisely. Density deconvolution allows to estimate
the density of unobserved data from the data measured with errors. Density deconvolution is
encountered in a variety of econometric applications, e.g. to the earning dynamics in Bonhomme
and Robin (2010), to panel data in Evdokimov (2010), or to instrumental regression in Adusumilli,
Otsu, et al. (2015). In the simplest example of this model, we have some noisy scalar observations
𝑌 , of the latent variable 𝑍, contaminated by measurement errors 𝑈
𝑌 = 𝑍 + 𝑈,

𝑍 ⊥⊥ 𝑈.

Distributions of both 𝑍 and 𝑈 are assumed to be absolutely continuous with respect to the
Lebesgue measure with corresponding densities 𝜙 and 𝑓 . The density of measurement errors 𝑓 is
assumed to be known. The goal is to recover the density of the latent variable 𝑍 from observing
contaminated i.i.d. sample (𝑌𝑖 )𝑛𝑖=1 .
Independence and additivity of the noise imply that the density function 𝑟 of 𝑌 satisfies the
following convolution equation
∫︁
𝑟(𝑦) =

𝜙(𝑧)𝑓 (𝑦 − 𝑧)d𝑧 =: (𝑇 𝜙)(𝑦),

(4)

where the operator 𝑇 : 𝐿2 → 𝐿2 in Eq. (4) is compact, whenever the density 𝑓 is compactly
supported. Carrasco and Florens (2011) replace Lebesgue measure by measures which yield
compactness of 𝑇 and study Tikhonov regularization, obtaining 𝐿2 convergence rates and
pointwise asymptotic normality. For simplicity of presentation and tractability of our results,
I assume that all densities are continuous, bounded and compactly supported, with support
contained inside the interval [𝑎, 𝑏] ⊂ R. This assumption is not very restrictive, since most of
economic variables, e.g. reported earning, are bounded.
The adjoint operator to 𝑇 can be computed by Fubini’s theorem
*

(𝑇 𝜓)(𝑧) =

∫︁
𝜓(𝑦)𝑓 (𝑦 − 𝑧)d𝑦.

Notice hat operators 𝑇 and 𝑇 * are known in this problem because 𝑓 is known. Since 𝑟 is a

10

density function, we have (𝑇 * 𝑟)(𝑧) = E[𝑓 (𝑌 − 𝑧)]. The only component of the model that needs
to be estimated to obtain Tikhonov-regularized estimator is
𝑛

1 ∑︁
* 𝑟)(𝑧) =
(𝑇̂︂
𝑓 (𝑌𝑖 − 𝑧).
𝑛
𝑖=1

3 Honest confidence sets
In this section we first discuss two stochastic processes that will drive the distribution of Tikhonovregularized estimators and show that they do not converge weakly in the space of continuous
functions under the uniform topology. We describe the problem of building honest uniform
confidence sets and introduce two core approaches of this paper: the Gaussian approximation
to upper bounds and the concentration inequality approach. We introduce two sets of general
assumptions that allow us to characterize coverage errors and diameters of constructed confidence
sets and state main results of the paper.

3.1 Impossibility of weak convergence
We first discuss the impossibility of using the uniform central limit theorem to obtain the
distribution of the estimator. To fix notation through the rest of the paper, let Pr be a
probability measure corresponding to any of ill-posed model introduced in the previous section.
The model is described by the function equation 𝑟 = 𝑇 𝜙, where 𝜙 : [𝑎, 𝑏]𝑝 → R is a functional
parameter of interest. We aim to construct a random set 𝐶𝑛 such that it contains the function 𝜙
with probability close to 1 − 𝛾 for 𝛾 ∈ (0, 1) and that its expected diameter shrinks as the sample
size increases at a certain rate. We focus on confidence sets for Tikhonov-regularized estimator
defined as follows
𝜙^𝛼𝑛 = (𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 𝑇^* 𝑟^,
where 𝑇^, 𝑇^* , and 𝑟^ are appropriate estimators4 and 𝛼𝑛 is some positive sequence converging to
zero as 𝑛 → ∞. This estimator belongs to the general family of spectral regularization schemes,
see Carrasco, Florens, and Renault (2007).
It will be seen in the following section that each of the three motivating examples, the leading
stochastic component of the Tikhonov-regularized estimator is one of the following two stochastic

4

If some of operators are known, which is the case in the density deconvolution model, estimators are replaced by
known quantities.

11

processes driven by the i.i.d. centered random functions (𝑋1,𝑖 )𝑛𝑖=1 and (𝑋2,𝑖 )𝑛𝑖=1
𝑛

𝜈1,𝑛 =

1 ∑︁
(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝑇 * 𝑋1,𝑖 ,
𝑛

𝑛

or

𝑖=1

𝜈2,𝑛 =

1 ∑︁
(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝑋2,𝑖 ,
𝑛

(5)

𝑖=1

We assume that 𝑇 is some integral operator with continuous kernel function, so that 𝑇, 𝑇 * , and
𝑇 * 𝑇 map to the space of continuous functions. Thus, we can also think of the operator 𝑇 as
acting between (𝐶, ‖.‖∞ ) spaces. This will be useful to study uniform confidence sets. On the
other hand, considering 𝑇 as an operator between 𝐿2 spaces will allow to define the adjoint
operator 𝑇 * and to use effectively the spectral theory and functional calculus accessible in Hilbert
spaces. By Lemma 1 in the Appendix C, the operator (𝛼𝑛 𝐼 + 𝑇 * 𝑇 ) is invertible between spaces
of continuous functions. Therefore, both processes have trajectories in the space (𝐶, ‖.‖∞ ).
In order to build uniform confidence sets, we need to approximate the distribution of the
supremum of these two processes. The simplest possible route to achieve this, would be to
establish weak convergence of suitably normalized processes in Eq. (5) to some Gaussian processes
and then to rely on quantiles of corresponding Gaussian suprema to build uniform confidence
sets. Unfortunately, both processes do not converge weakly as random elements in (𝐶, ‖.‖∞ )
space, as illustrated in the following proposition.
Proposition 1. There does not exist a normalizing sequence 𝑟𝑛 such that 𝑟𝑛 𝜈1,𝑛 or 𝑟𝑛 𝜈2,𝑛 would
converge weakly in (𝐶, ‖.‖∞ ) to a non-degenerate random process.
Proposition 1 tells us that it is not possible to rely on the asymptotic approximation with
conventional central limit theorem. As a result, there does not exists a trivial way to build a
uniform confidence set for the structural function 𝜙 with asymptotically exact confidence level.
Given the difficulty of inference with exact coverage requirement, in this paper we relax this
requirement.

3.2 Honest confidence sets
We describe the problem of building the honest confidence sets. We focus on uniform confidence
sets, honest to some class of models ℱ 5 . For a given level 𝛾 ∈ (0, 1), the honest 1 − 𝛾 uniform
confidence set, denoted 𝐶𝑛 = {𝐶𝑛 (𝑧) = [𝐶𝑙 (𝑧), 𝐶𝑢 (𝑧)] , 𝑧 ∈ [𝑎, 𝑏]𝑝 }, should satisfy
inf
(𝜙,𝑇 )∈ℱ

5

Pr (𝜙(𝑧) ∈ 𝐶𝑛 (𝑧), ∀𝑧 ∈ [𝑎, 𝑏]𝑝 ) ≥ 1 − 𝛾 − 𝑂(𝑟𝑛 ),

The class of models will be introduced in Section 4.

12

(6)

for some sequence 𝑟𝑛 → 0. Honesty is necessary to produce confidence sets of practical value.
It ensures that for a sufficiently large sample size 𝑛, not depending on the model (𝜙, 𝑇 ), the
coverage of the set will be close to 1 − 𝛾. In contrast, a dishonest set requires a weaker condition
inf

lim inf Pr (𝜙(𝑧) ∈ 𝐶𝑛 (𝑧), ∀𝑧 ∈ [𝑎, 𝑏]𝑝 ) ≥ 1 − 𝛾,

(𝜙,𝑇 )∈ℱ 𝑛→∞

so that the sample size 𝑛 needed for coverage close to 1 − 𝛾 will depend on the unknown function6
𝜙 and the unknown operator 𝑇 .
The most interesting choice of confidence set is such that its expected diameter under the
supremum norm shrinks at the rate 𝑟¯𝑛 , which is the rate at which we estimate the function 𝜙
E |𝐶𝑛 |∞ ≡ E ‖𝐶𝑢 − 𝐶𝑙 ‖∞ = 𝑂(¯
𝑟𝑛 ).
Uniform confidence sets, unlike the one based on other metrics, such as the 𝐿2 distance, have
appealing visualization and are easy to implement numerically. For example, if 𝜙 is the function
on the real line, the confidence set becomes a band on the plane, which contains the whole graph
of the function with probability close to 1 − 𝛾.

3.3 Gaussian approximation and concentration
In this section we introduce two main approaches of this paper. Consider the following two
processes
𝑛

𝜈^1,𝑛 =

1 ∑︁
(𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 𝑇^* 𝑋1,𝑖 ,
𝑛

𝑛

or

𝜈^2,𝑛 =

𝑖=1

1 ∑︁
(𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 𝑋2,𝑖 .
𝑛

(7)

𝑖=1

The first idea suggested in this paper is to focus on some upper bound on suitably normalized
suprema ‖^
𝜈1,𝑛 ‖∞ or ‖^
𝜈2,𝑛 ‖∞ , whose distribution can be easily approximated with the distribution
of certain norms of Gaussian processes. It follows from the spectral theory, that for a bounded
linear operator 𝑇 and continuous function 𝑓 on [0, ‖𝑇 ‖2 ], we have 𝑓 (𝑇 * 𝑇 )𝑇 * = 𝑇 * 𝑓 (𝑇 𝑇 * ), see
Engl, Hanke, and Neubauer (1996). Using this fact, factorizing the operator norm, and noticing
that ‖(𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 ‖ ≤ 𝛼𝑛−1 , for any 𝑛 ∈ N we have
𝛼𝑛 𝑛1/2
𝛼𝑛
‖^
𝜈1,𝑛 ‖∞ =
‖𝑇^* ‖2,∞
‖𝑇^* ‖2,∞
6

⃦
⃦
𝑛
⃦
⃦
∑︁
1
⃦ ^*
⃦
𝑋1,𝑖 ⃦
⃦𝑇 (𝛼𝑛 𝐼 + 𝑇^𝑇^* )−1 √
⃦
⃦
𝑛
𝑖=1

∞

⃦
⃦
𝑛
⃦ 1 ∑︁
⃦
⃦
⃦
≤ ⃦√
𝑋1,𝑖 ⃦ .
⃦ 𝑛
⃦

(8)

𝑖=1

See also (Tsybakov, 2009, p.16-19) for the discussion why fixed (𝜙, 𝑇 ) results do not lead to a consistent notion of
optimality in non-parametric problems and why it is necessary to consider results uniform over some smoothness
class ℱ.

13

The latter distribution can be approximated invoking the CLT in the Hilbert space and continuous
mapping theorem
⃦
⃦
𝑛
⃦ 1 ∑︁
⃦
⃦
⃦ 𝑑
𝑋1,𝑖 ⃦ −
→ ‖G1 ‖ ,
⃦√
⃦ 𝑛
⃦
𝑖=1

where G1 is a zero-mean Gaussian process with covariance function (𝑤, 𝑤′ ) ↦→ E[𝑋1,𝑖 (𝑤)𝑋1,𝑖 (𝑤′ )]
and 𝑤, 𝑤′ ∈ [𝑎, 𝑏]𝑞 .
Similarly, by Lemma 1 in the Appendix C
3/2

𝛼𝑛 𝑛1/2
1/2

‖𝑇^* ‖2,∞ /2 + 𝛼𝑛

‖^
𝜈2,𝑛 ‖∞

⃦
⃦
𝑛
⃦ 1 ∑︁
⃦
⃦
⃦
≤ ⃦√
𝑋2,𝑖 ⃦ ,
⃦ 𝑛
⃦
𝑖=1

(9)

∞

and we can approximate the distribution in the right-hand side with uniform CLT
⃦
⃦
𝑛
⃦ 1 ∑︁
⃦
⃦
⃦
𝑋2,𝑖 ⃦
⃦√
⃦ 𝑛
⃦
𝑖=1

𝑑

−
→ ‖G2 ‖∞ ,

∞

where G2 is a zero-mean Gaussian process with covariance function (𝑧, 𝑧 ′ ) ↦→ E[𝑋2,𝑖 (𝑧)𝑋2,𝑖 (𝑧 ′ )]
and 𝑧, 𝑧 ′ ∈ [𝑎, 𝑏]𝑝 . To obtain explicit convergence rates for coverage error, instead of relying on
uniform central limit theorems, we will use suitable Berry-Esseen results. The distributions of
both ‖G1 ‖ and ‖G2 ‖∞ are not pivotal, but it is possible to estimate the covariance structure and
to simulate processes in order to obtain needed quantiles, see Section 6. Alternatively, we can
rely on estimates of tail probabilities based Gaussian concentration or to rely on the empirical or
exchangeable bootstrap7 , see (Van Der Vaart and Wellner, 2000, Chapter 3.6).
As an alternative to Gaussian approximation we study confidence sets based on non-asymptotic
estimates of tail probabilities of the distribution of ‖𝜈1,𝑛 ‖∞ and ‖𝜈2,𝑛 ‖∞ . To that end we will
rely on the data-driven Talagrand-type concentration inequality, to achieve the required coverage
level.
To describe confidence sets constructed with both approaches, we shall introduce additional
notation. We allow for the process 𝑋1,𝑖 to change with the sample size 𝑛. In particular, we assume
′ ) of zero-mean processes in 𝐶[𝑎, 𝑏]𝑞 not depending on 𝑛 and
that for some i.i.d. sequence (𝑋1,𝑖
′ . On the other
some known sequence of strictly positive real numbers 𝑢𝑛 , we have 𝑋1,𝑖 = 𝑢𝑛 𝑋1,𝑖
′ and 𝑋
hand, the process 𝑋2,𝑖 is assumed not to change with 𝑛. Since stochastic processes 𝑋1,𝑖
2,𝑖

^ ′ and 𝑋
^ 2,𝑖 their respective consistent estimators
may be not readily available, we denote by 𝑋
1,𝑖
^ ′ ‖ and 𝐹^2 = max1≤𝑖≤𝑛 ‖𝑋
^ 2,𝑖 ‖∞ the corresponding estimators of their
and by 𝐹^1 = max1≤𝑖≤𝑛 ‖𝑋
1,𝑖
envelops. Finally, for some i.i.d. sequence of Rademacher random variables (𝜀𝑖 )𝑛𝑖=1 , independent
7

The rigorous study of bootstrap properties is beyond the scope of this paper and is left for future research.

14

⃦ 𝜀 ⃦
⃦
⃦
⃦ and ⃦𝜈^𝜀 ⃦ the estimates of the suprema of symmetrized
from the data, we denote by ⃦𝜈^1,𝑛
2,𝑛
∞
∞
processes
⃦
⃦
𝑛
⃦
⃦ 1 ∑︁
⃦ 𝜀 ⃦
⃦
* ^ −1 ^ *
^
^
⃦𝜈^1,𝑛 ⃦ = ⃦
𝑋
𝑇
𝜀
𝑇
)
(𝛼
𝐼
+
𝑇
⃦
𝑖 1,𝑖 ⃦ ,
𝑛
∞
⃦
⃦𝑛
𝑖=1

⃦
⃦
𝑛
⃦ 1 ∑︁
⃦
⃦ 𝜀 ⃦
⃦
* ^ −1
^
^
⃦𝜈^2,𝑛 ⃦ = ⃦
𝑋
𝑇
)
𝜀
(𝛼
𝐼
+
𝑇
⃦
𝑖 2,𝑖 ⃦ .
𝑛
∞
⃦𝑛
⃦
𝑖=1

∞

∞

Confidence sets of level 1 − 𝛾 with 𝛾 ∈ (0, 1) are described as follows
𝑗,𝑠
𝐶1−𝛾,𝑛
(𝑧)

]︁
[︁
𝑗,𝑠
𝑗,𝑠
= 𝜙^𝛼𝑛 (𝑧) − 𝑞1−𝛾,𝑛 , 𝜙^𝛼𝑛 (𝑧) + 𝑞1−𝛾,𝑛 ,

𝑧 ∈ [𝑎, 𝑏]𝑝 ,

𝑗 ∈ {1, 2},

𝑠 ∈ {g, ci},
(10)

where s ∈ {g, ci} denotes sets based on the Gaussian approximation to the upper bound or sets
based on the concentration inequality, 𝑗 ∈ {1, 2} denotes sets for each of the two processes in
Eq. (5), and

1,g
𝑞1−𝛾,𝑛

=

2,g
=
𝑞1−𝛾,𝑛

1/2
𝑐1,1−𝛾 ‖𝑇^* ‖2,∞ + 𝑐0

,
𝛼𝑛 𝑛1/2
(︁
)︁
1/2
𝑐2,1−𝛾 ‖𝑇^* ‖2,∞ /2 + 𝛼𝑛
+ 𝑐0
3/2
𝛼𝑛 𝑛1/2

1,ci
𝑞1−𝛾,𝑛

2,ci
, 𝑞1−𝛾,𝑛

√︀
⃦ 𝜀 ⃦
3‖𝑇^* ‖2,∞ 𝐹^1 2 log(2/𝛾) + 𝑐0
⃦
⃦
= 2 𝜈^1,𝑛 ∞ +
𝑢𝑛 ,
𝛼𝑛 𝑛1/2
(︁
)︁
1/2 ^ √︀
3 ‖𝑇^* ‖2,∞ /2 + 𝛼𝑛
𝐹2 2 log(2/𝛾) + 𝑐0
⃦ 𝜀 ⃦
= 2 ⃦𝜈^2,𝑛 ⃦∞ +
.
3/2
𝛼𝑛 𝑛1/2

Here 𝑐1,1−𝛾 and 𝑐2,1−𝛾 are 1 − 𝛾 quantiles of norms of Gaussian processes ‖G1 ‖2 and ‖G2 ‖∞
respectively and 𝑐0 > 0 is some sufficiently small constant.

3.4 Coverage properties and diameters
In what follows, we consider two sets of regularity conditions for the main results of the paper.
These results will not be stated in the fullest possible generality, but rather at the level sufficient
to cover all examples of interest. The first set of regularity conditions imposes mild restrictions
on the data.
Assumption 1. Suppose that for all models in some class ℱ we have (i) (𝑋1,𝑖 )𝑛𝑖=1 and (𝑋2,𝑖 )𝑛𝑖=1
are sequences of i.i.d. centered random functions in 𝐶[𝑎, 𝑏]𝑞 or 𝐶[𝑎, 𝑏]𝑝 respectively, where
′ , where 𝑢−1 = 𝑂(1); (iii) trajectories of 𝑋
𝑋2,𝑖 does not depend on 𝑛; (ii) 𝑋1,𝑖 = 𝑢𝑛 𝑋1,𝑖
2,𝑖
𝑛
𝑠 [𝑎, 𝑏]𝑝 , 𝑠 ∈ (0, 1] a.s.; (iv) 𝑋
3
are in 𝐶𝑀
1,𝑖 does not depend on 𝑛 and E‖𝑋1,𝑖 ‖ ≤ 𝐶1 < ∞,

E‖𝑋2,𝑖 ‖3∞ ≤ 𝐶2 < ∞; (v) 𝑇 is 1-1 integral operator with continuous kernel function.
Assumption (i) could be relaxed to weakly dependent environments where the functional
Berry-Esseen and the data-driven concentration inequalities hold. Assumption (ii) is needed
to accommodate the NPIV model, in which case 𝑢𝑛 = ℎ−𝑞
𝑛 . Assumption (iii) reduces to the
mild smoothness restriction on certain densities in the NPIV and deconvolution models, or on

15

stochastic processes in functional regression models. We shall note that the existence of the
third moment in assumption (iv) is needed for functional Berry-Esseen theorems in the Gaussian
approximation approach. It could be relaxed to the finiteness of the second moment at costs
of not having explicit rates for coverage errors. On the other hand, for the construction with
concentration inequality, we need the stronger assumption (iv) of finite envelops. We should also
stress that if assumption (v) fails and 𝑇 is not injective, the Tikhonov-regularized estimator 𝜙^𝛼𝑛
is still well-defined and converges to the best approximation to the function 𝜙, see Florens et al.
(2011) and Babii and Florens (2016). In this case, the resulting confidence set will be valid for
this best approximation.
The second set of assumptions will be satisfied whenever the first set of assumptions holds.
We will illustrate this in each particular application in Section 5.
Assumption 2. There exist some sequences 𝑟𝑗,𝑛 → 0 as 𝑛 → ∞ for 𝑗 = 1, 2 such that
uniformly over some class of models ℱ, we have (i) ‖𝜙^𝛼𝑛 − 𝜙‖∞ ≤ ‖^
𝜈𝑗,𝑛 ‖∞ + 𝜂𝑗,𝑛 , where
(︀ )︀
processes 𝜈^𝑗,𝑛 , 𝑗 = 1, 2 are as in Eq. (7) and E𝜂𝑗,𝑛 = 𝑂(𝑟𝑗,𝑛 ); (ii) E‖𝑇^* − 𝑇 * ‖42,∞ = 𝑜 𝛼𝑛2 ; (iii)
^ ′ − 𝑋 ′ ‖2 = 𝑜 (1) and E max1≤𝑖≤𝑛 ‖𝑋
^ 2,𝑖 − 𝑋2,𝑖 ‖2∞ = 𝑜(1).
E max1≤𝑖≤𝑛 ‖𝑋
1,𝑖
1,𝑖
Assumption 2 (i) tells us that we can decompose the sup-norm loss function for the estimator
into the leading variance component ‖^
𝜈𝑗,𝑛 ‖∞ and the remainder 𝜂𝑗,𝑛 that converges to zero at the
rate 𝑟𝑗,𝑛 . Assumption 2 (ii) imposes certain requirement on the rate at which the regularization
parameter 𝛼𝑛 can converge to zero and, in particular, it ensures that the noise coming from
the estimation of the operator 𝑇 * is sufficiently small for inferential purposes. As regards
′ and 𝑋
Assumption 2 (iii), it ensures that envelops for processes 𝑋1,𝑖
2,𝑖 can be well-estimated.

Given the regularity conditions discussed above, we can state the first main result of the paper.
It describes convergence rates for coverage error of constructed confidence sets based on the
data-driven Talagrand-type concentration inequality (see also Appendix A for discussion of this
inequality).
Theorem 1. Suppose that Assumptions 1 (i), (v), and Assumptions 2 (i)-(iii) are satisfied, then
inf
(𝜙,𝑇 )∈ℱ

(︁
)︁
2,ci
Pr 𝜙 ∈ 𝐶1−𝛾,𝑛
≥ 1 − 𝛾 − 𝑂(𝜌2,𝑛 𝑟2,𝑛 ) − 𝑜(1)

3/2

with 𝜌2,𝑛 = 𝛼𝑛 𝑛1/2 . If additionally Assumption 1 (ii) is satisfied, the same inequality holds for
𝑗 = 1 with 𝜌1,𝑛 = 𝛼𝑛 𝑛1/2 𝑢−1
𝑛 .
Now we state the second main result of the paper, which is based on the Gaussian approximation.
This result relies on suitable Berry-Esseen type theorems in Hilbert and Banach spaces.

16

Theorem 2. Suppose that Assumptions 1 (i), (iii), (iv), (v), and Assumption 2 (i) are satisfied.
Then for
inf
(𝜙,𝑇 )∈ℱ

(︁
)︁
𝑗,g
Pr 𝜙 ∈ 𝐶1−𝛾,𝑛
≥ 1 − 𝛾 − 𝑂 (𝜌𝑗,𝑛 𝑟𝑗,𝑛 + 𝜀𝑗,𝑛 ) ,

𝑗 = 1, 2,

where 𝜀1,𝑛 = 𝑛−1/2 , 𝜀2,𝑛 = 𝑛−1/6 , and 𝜌𝑗,𝑛 , 𝑗 = 1, 2 are as in Theorem 1.
For the second set, we have 𝑛−1/6 instead of 𝑛−1/2 in the coverage error due to the fact that
approximation of the distribution of ‖^
𝜈2,𝑛 ‖∞ reduces to the approximation of the distribution of
⃦
⃦
∑︀
⃦
⃦ √1
𝑛
⃦ 𝑛 𝑖=1 𝑋2,𝑖 ⃦ . This is a non-smooth functional of the random functions taking values in the
∞

Banach space and the best currently available Berry-Esseen estimates are not as sharp as in the
case of the Hilbert space, see e.g. Bentkus, Götze, Paulauskas, and Račkauskas (2000).
For both sets we need to undersmooth in order to make the coverage error to decrease to
zero as sample size increases. The next result describes the convergence rate of the diameter of
constructed confidence sets.
Corollary 1. Suppose that assumptions of Theorem 1 are satisfied, then we have the following
rates for the expected diameters of confidence sets:
⃒
⃒
⃒ 1,ci ⃒
sup E ⃒𝐶1−𝛾,𝑛
⃒

∞

(𝜙,𝑇 )∈ℱ

(︂
=𝑂

)︂
𝑢𝑛
+
𝑟
1,𝑛 ,
𝛼𝑛 𝑛1/2

⃒
⃒
⃒ 2,ci ⃒
sup E ⃒𝐶1−𝛾,𝑛
⃒

(𝜙,𝑇 )∈ℱ

∞

(︃
=𝑂

1
3/2

𝛼𝑛 𝑛1/2

)︃
+ 𝑟2,𝑛

.

Suppose now that assumptions of Theorem 2 are satisfied, then
⃒
⃒
⃒ 1,g ⃒
sup E ⃒𝐶1−𝛾,𝑛
⃒

(𝜙,𝑇 )∈ℱ

(︂
∞

=𝑂

1
𝛼𝑛 𝑛1/2

)︂

⃒
⃒
⃒ 2,g ⃒
sup E ⃒𝐶1−𝛾,𝑛
⃒

,

(𝜙,𝑇 )∈ℱ

∞

(︃
=𝑂

1
3/2

𝛼𝑛 𝑛1/2

)︃
.

The bias-variance trade-off for the risk of the estimator 𝜙^𝛼𝑛 reduces to the trade-off between
rates at which coverage errors decrease and the diameters of confidence sets shrink. For instance,
𝑟𝑗,𝑛 can be roughly interpreted as the order of the bias. It will be seen in the following sections
that 𝑟𝑗,𝑛 contains the regularization bias which is of order 𝛼𝑛𝛽∧1 , and we need to set 𝛼𝑛 → 0 to
reduce this bias. On the other hand, the diameter of the confidence set is of the same order
as the variance of the estimator and depends on 𝛼𝑛−1 . As a result, the reduction of the bias
will inevitably lead to the increase of the variance and vice versa. The optimal choice of the
regularization parameter 𝛼𝑛 that balances the variance and the bias will in turn balance rates of
diameters and coverage errors.

17

4 Ill-posedness and regularization bias
In this section we discuss regularity condition on the structural function 𝜙 and the operator 𝑇 ,
needed to study honest and uniform confidence sets for Tikhonov-regularized estimators.
Assumption 3. The model belongs to the source class
{︁
}︁
𝑡
ℱ ≡ ℱ(𝑡, 𝛽, 𝐶) ≡ (𝜙, 𝑇 ) ∈ 𝐶𝑀
[𝑎, 𝑏]𝑝 × ℒ2,∞ : 𝜙 = (𝑇 * 𝑇 )𝛽 𝑇 * 𝜓, ‖𝑇 * ‖2,∞ ∨ ‖𝜓‖ ∨ ‖𝑇 ‖−1 ≤ 𝐶 ,
where 𝛽, 𝐶 > 0.
Source conditions are at the heart of spectral regularization theory, and describe effectively
the regularity of the problem by restricting how ill-posed the operator 𝑇 is, comparing to the
smoothness of the parameter of interest 𝜙 (see also Chen and Reiss (2011) for its comparability
with other assumptions used in the literature). The present source condition is different from the
one used to characterize 𝐿2 rates, where we would require 𝜙 = (𝑇 * 𝑇 )𝛽 𝜓 only, see Carrasco et al.
(2007).
We show in the next section that under this source condition, the uniform convergence
rates of Tikhonov-regularized estimators are polynomial. In the severely ill-posed case, when
singular values of the operator 𝑇 decay to zero exponentially fast, the source condition in
Assumption 3 requires that the function 𝜙 can be described by the finite number of generalized
Fourier coefficients (with respect to the SVD basis of 𝑇 ), or more generally is analytic (also
called supersmooth). Since many functions encountered in empirical work are smooth with
generalized Fourier coefficients decaying rapidly, the fact that we can still achieve polynomial
uniform convergence rates is a very good news for applications.
Under Assumption 3, the order of regularization bias can be established. Notice that in the
Proposition that follows, the bound is uniform over the source set, and this will be needed to
establish honesty of our confidence sets.
Proposition 2. Suppose that Assumption 3 is satisfied, then
⃦
⃦
sup ⃦(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝑇 * 𝑟 − 𝜙⃦∞ ≤ 𝑅𝛼𝑛𝛽∧1 ,

(𝜙,𝑇 )∈ℱ

[︀
]︀
where 𝑅 = 𝐶 2 𝛽 𝛽 (1 − 𝛽)1−𝛽 10<𝛽<1 + 𝐶 2(𝛽−1) 1𝛽≥1 .
(︁
)︁
The regularization bias is of order 𝑂 𝛼𝑛𝛽∧1 . It is well known that for the simple Tikhonov
regularization, it is not possible to characterize faster convergence rate for the bias in case of very

18

regular problems with 𝛽 > 1. This minor drawback can be fixed with iterated or extrapolated
Tikhonov regularization approach.

5 Applications
In this section, we discuss how high-level conditions of Theorems 1 and 2 map to specific
econometric models. We provide inferential results for three models considered in Section 2.

5.1 Nonparametric IV
The following assumptions are sufficient to characterize confidence sets for the NPIV model.
Assumption 4. (i) (𝑌𝑖 , 𝑍𝑖 , 𝑊𝑖 )𝑛𝑖=1 is an i.i.d. sample of (𝑌, 𝑍, 𝑊 ) such that |𝑈 | ≤ 𝐹0 < ∞
a.s.; (ii) the density 𝑓𝑍𝑊 (𝑧, 𝑤) is bounded away from 0 and there exists some real number 𝑠 > 0
such that 𝑓𝑍𝑊 ∈ 𝐶𝐿𝑠 [𝑎, 𝑏]𝑝+𝑞 ; (iii) 𝐾𝑧 and 𝐾𝑤 are continuous product kernel functions of order
∫︀
∫︀
⌊𝑡⌋ ∨ ⌊𝑠⌋ such that ‖𝑢‖𝑡 |𝐾𝑧 (𝑢)|d𝑢 < ∞ and ‖𝑢‖𝑠 |𝐾𝑤 (𝑢)|d𝑢 < ∞ with individual kernels of
bounded 𝑝-variation for some 𝑝 ≥ 1 and supported on the rectangle which is a subset of [𝑎, 𝑏]; (iv)
the integral operator 𝑇 : 𝐿2 [𝑎, 𝑏]𝑝 → 𝐿2 [𝑎, 𝑏]𝑞 is 1-1.
Assumption (i) rules out distributions with unbounded supports, such as the normal distribution,
but it allows, for example, for truncated normal. This should not be very restrictive since the vast
majority of economic variables are bounded due to scarcity and economic constraints. Assumption
(iii), in particular, allows to use boundary-corrected kernels, see Hall and Horowitz (2005) and
Darolles et al. (2011)8 for further discussion. Lastly, Assumption (v) is a completeness condition,
Newey and Powell (2003). In case of its failure, the estimator is still well-defined and converge
to the best approximation to function 𝜙 in the orthogonal complement of the null space of the
operator 𝑇 , see Babii and Florens (2016).
Theorem 3. Suppose that Assumptions 3 and 4 are satisfied and the sequences of bandwidth
𝑝+𝑞
parameters ℎ𝑛 → 0 is such that log ℎ−1
𝑛 /(𝑛ℎ𝑛 ) = 𝑂(1) as 𝑛 → ∞. Then for 𝑗 = 1, 2 we have

‖𝜙^𝛼𝑛 − 𝜙‖∞ ≤ ‖^
𝜈𝑗,𝑛 ‖∞ + 𝜂𝑗,𝑛 ,
(︀ −1
)︀
where 𝜈^𝑗,𝑛 , 𝑗 = 1, 2 are as in Eq. (7) with 𝑋1,𝑖 (𝑤) = 𝑈𝑖 ℎ−𝑞
𝑛 𝐾 ℎ𝑛 (𝑊𝑖 − 𝑤) , 𝑋2,𝑖 (𝑧) =

8

Otherwise, to avoid problems at end-points, we will make restriction to the interior of [𝑎, 𝑏]𝑝+𝑞 and all results
should be read as uniform over the interior of this set.

19

𝑓𝑍,𝑊 (𝑧, 𝑊𝑖 )𝑈𝑖 , sup(𝜙,𝑇 )∈ℱ E𝜂𝑗,𝑛 = 𝑂 (𝑟𝑗,𝑛 ), and

𝑟2,𝑛 =

𝑟1,𝑛 =

ℎ𝑠𝑛

+ ℎ𝑡𝑛
3/2
𝛼𝑛
ℎ𝑡𝑛
𝑞/2

𝛼𝑛 ℎ𝑛

1
√︀ 𝑞 + ℎ𝑛𝑡−𝑞/2 ⎝
𝑛ℎ𝑛
⎛√︃
⎞
−1
1
log ℎ𝑛
𝑠⎠
+ 𝛼𝑛𝛽∧1 + 1/2 ⎝
𝑝+𝑞 + ℎ𝑛 .
𝑛ℎ𝑛
𝛼𝑛
+ 𝛼𝑛𝛽∧1 +

1

)︃ ⎛√︃

(︃

3/2
𝛼𝑛

log ℎ−1
𝑛
𝑝+𝑞
𝑛ℎ𝑛

⎞
+ ℎ𝑠𝑛 ⎠ +

⎛√︃

1 ⎝

1/2
𝛼𝑛

log ℎ−1
𝑛
𝑝+𝑞
𝑛ℎ𝑛

⎞
+ ℎ𝑠𝑛 ⎠ ,

Moreover,
(︃
sup E‖^
𝜈1,𝑛 ‖∞ = 𝑂
(𝜙,𝑇 )∈ℱ

)︃

1
𝑞/2

𝛼𝑛 𝑛1/2 ℎ𝑛

(︃
sup E‖^
𝜈2,𝑛 ‖∞ = 𝑂

,

(𝜙,𝑇 )∈ℱ

1
3/2

𝛼𝑛 𝑛1/2

)︃
.

Rates 𝑟𝑗,𝑛 have three components. The first two are the bias of the nonparametric kernel
density estimators and the regularization bias respectively. The last component comes from the
fact that we need to estimate the operator 𝑇 . It is necessary to select tuning parameters, so
that the variance is balanced with respect to the two bias terms. It is also necessary to ensure
that third component, which represents the noise coming from the estimation of the operator 𝑇
converges to zero faster than the leading variance and the bias terms.
The following corollary achieves both requirements by imposing minimal conditions on the
smoothness of the joint density and the structural function 𝜙. Following Darolles et al. (2011),
we interpret these smoothness requirements as the ”strong instrument” condition9 . If 𝑝 = 𝑞 = 1
and the second-order derivatives of 𝜙 and 𝑓𝑍𝑊 are sufficiently smooth in the Hölder sense, then
this assumptions is not binding.
Corollary 2. Suppose that assumptions of Theorem 3 hold. Assume additionally that components
of the model are sufficiently smooth and tuning parameters converge to zero at certain speeds,
1

in the sense that (i) for 𝑗 = 1, we have 𝑡 > 𝑞/2 + 2𝑝, 𝑠 > 0.75(𝑡 − 𝑞/2), ℎ𝑛 ∼ 𝑛− 2𝑡 , and
𝑡−𝑞/2

𝛼𝑛 ∼ 𝑛

− 2𝑡(𝛽∧1+1)

; (ii) for 𝑗 = 2, we have 𝑠 ∧ 𝑡 > 5(𝑝 + 𝑞)/4, ℎ𝑛 ∼ 𝑛

1
− 2(𝑡∧𝑠)

−

1

2(𝛽∧1)+3 .
, and
(︂ 𝛼𝑛 ∼ 𝑛
)︂
(𝑡−𝑞/2)(𝛽∧1)
− 2𝑡(𝛽∧1+1)
Then we uniform convergence rates of the risk of the NPIV estimator are 𝑂 𝑛
for
(︂
)︂
− 𝛽∧1
𝑗 = 1 and 𝑂 𝑛 2(𝛽∧1)+3 for 𝑗 = 2.

Despite the fact that we have to estimate some parts of the model non-parametrically, rates in
the second decomposition with 𝜈2,𝑛 process are free from dimension of the IV, once appropriate
smoothness conditions are satisfied. This is similar to the 𝐿2 theory developed in Darolles et al.
(2011). On the other hand, convergence rates in the first decomposition depend on the dimension
9

Notice that this analytical characterization of strong instrument is different from the usual correlation requirement
for the linear IV model.

20

of the IV. Even though this ”curse of dimensionality” may result in worser performance from
the asymptotic point of view, frequently, in the empirical research, we only have one or two
instruments and the first decomposition may lead to better performance of confidence sets in
finite samples. Obtained polynomial uniform convergence rates are new and improve upon earlier
results by Mbakop (2015) who obtain rates of order of powers of log 𝑛/𝑛 that depend on the
dimension of both the regressors and the instrument.
The interesting feature of Tikhonov regularization is that convergence rates are of order of
powers of 1/𝑛 in mildly and some severely ill-posed settings.
In the next result we verify conditions of Theorem 1 and 2 for the NPIV estimator show that
our confidence sets provide a valid coverage. For confidence sets based on the concentration
inequality, we show that the bias of the estimator will have an impact on the coverage error.
On the other hand, for the confidence set based on the Gaussian approximation we characterize
completely convergence rates of coverage errors, which will be driven by the bias of the estimator
and the accuracy of Gaussian approximation.
Theorem 4. Suppose that assumptions of Corollary 2 are satisfied, then
inf
(𝜙,𝑇 )∈ℱ

(︁
)︁
𝑗,ci
Pr 𝜙 ∈ 𝐶1−𝛾,𝑛
≥ 1 − 𝛾 − 𝑂 (𝜌𝑗,𝑛 𝑟𝑗,𝑛 ) − 𝑜(1),

𝑗 = 1, 2

and
inf
(𝜙,𝑇 )∈ℱ

(︁
)︁
(︁
)︁
2,g
Pr 𝜙 ∈ 𝐶1−𝛾,𝑛
≥ 1 − 𝛾 − 𝑂 𝛼𝑛3/2 𝑛1/2 𝑟2,𝑛 + 𝑛−1/6 ,

where 𝑟𝑗,𝑛 , 𝑗 = 1, 2 are as in Theorem 3.
Notice, that Assumptions of Corollary 2 do not ensure that coverage errors of confidence sets
decrease to zero. To achieve this requirement, it is necessary to undersmooth, increasing the
speed at which tuning parameters converge to zero. In the next corollary we assume that tuning
parameters converge to zero, so that the rate at which the diameter of the confidence set shrink
and the rate at which the coverage error decreases are balanced.
Corollary 3. Suppose that Assumptions of Theorem 3 hold. Assume additionally that components
of the model are sufficiently smooth and tuning parameters converge to zero at certain speeds
𝛽∧1+1/2
in the sense that (i) for 𝑗 = 1, we have 𝛽 < 1, 𝑡 > 𝑞/2 + 𝑝 1+𝛽∧1
1−𝛽∧1 and 𝑠 > (𝑡 − 𝑞/2) 𝛽∧1+1 ,

ℎ𝑛 ∼ 𝑛

𝛽∧1+1
− (𝛽∧1)(𝑡+𝑞/2)+2𝑡

ℎ𝑛 ∼ 𝑛

2(𝛽∧1)+3
− 2(𝑠∧𝑡)(𝛽∧1+3)

𝑡−𝑞/2

, and 𝛼𝑛 ∼ 𝑛

, and 𝛼𝑛 ∼ 𝑛

− (𝛽∧1)(𝑡+𝑞/2)+2𝑡

1
− 𝛽∧1+3

; (ii) for 𝑗 = 2, we have 𝑠 ∧ 𝑡 > 2.5(𝑝 + 𝑞),

. Then we have the following convergence rates for

21

expected diameters
⃒
⃒
⃒ 1,ci ⃒
sup E ⃒𝐶1−𝛾,𝑛
⃒

∞

(𝜙,𝑇 )∈ℱ

(︂
=𝑂 𝑛

(𝛽∧1)(𝑡−𝑞/2)

− 2((𝛽∧1)(𝑡+𝑞/2)+2𝑡)

)︂

⃒
⃒
⃒ 2,ci ⃒
sup E ⃒𝐶1−𝛾,𝑛
⃒

and

(𝜙,𝑇 )∈ℱ

∞

(︂
)︂
𝛽∧1
− 2(𝛽∧1+3)
=𝑂 𝑛
.

⃒
⃒
⃒ 2,g ⃒
2,g
Moroever, we have the same rate for E ⃒𝐶1−𝛾,𝑛
⃒ and the coverage error of the set 𝐶1−𝛾,𝑛 as for
∞
⃒
⃒
⃒ 2,ci ⃒
E ⃒𝐶1−𝛾,𝑛
⃒ .
∞

5.2 Functional regressions
The following set of assumptions provides sufficient conditions for functional regression models.
Assumption 5. (i) (𝑌, 𝑍, 𝑊 ) is a random vector in (R×𝐶[0, 1]𝑝 ×𝐶[0, 1]𝑞 ) a.s., and (𝑌𝑖 , 𝑍𝑖 , 𝑊𝑖 )𝑛𝑖=1
𝜌
is an i.i.d. sample from (𝑌, 𝑍, 𝑊 ); (ii) (𝑡, 𝑠) ↦→ 𝑍(𝑡)𝑊 (𝑠) has trajectories in 𝐶𝑀
[0, 1]𝑝+𝑞 a.s. for

some 𝜌 ∈ (0, 1]; (iii) ‖𝑈 𝑊 ‖ ≤ 𝐹1 < ∞ a.s.; (iv) E‖𝑈 𝑊 ‖3 ≤ 𝐶1 < ∞; (v) the integral operator
𝑇 : 𝐿2 [0, 1]𝑝 → 𝐿2 [0, 1]𝑞 is injective.
Confidence sets for functional regression models will be based on the risk decomposition with
𝜈1,𝑛 process as can be seen from the following theorem.
Theorem 5. Suppose that Assumptions 3 and 5 are satisfied with 𝑡 = 0, then
‖𝜙^𝛼𝑛 − 𝜙‖∞ ≤ ‖^
𝜈1,𝑛 ‖∞ + 𝜂1,𝑛
(︁
with 𝑋1,𝑖 = 𝑈𝑖 𝑊𝑖 , E𝜂1,𝑛 = 𝑂 𝛼𝑛𝛽∧1 +

1/2

𝛼𝑛
𝛼𝑛 𝑛1/2

)︁

. Moreover,
(︂

sup E ‖𝜙^𝛼𝑛 − 𝜙‖∞ = 𝑂

(𝜙,𝑇 )∈ℱ

)︂
1
𝛽∧1
+ 𝛼𝑛
.
𝛼𝑛 𝑛1/2

It follows that when 𝛼𝑛 → 0 and 𝛼𝑛 𝑛1/2 → ∞, the term

1/2

𝛼𝑛
𝛼𝑛 𝑛1/2

converges to zero much faster

than both the leading variance term, ‖𝜈1,𝑛 ‖∞ , and the bias term, 𝛼𝑛𝛽∧1 . The optimal balancing
between the two is achieved when the regularization parameter is 𝛼𝑛 ∼ 𝑛

1
− 2(𝛽∧1+1)

. In this case,

the uniform convergence rate of the risk in the functional linear regression model is
sup E‖𝜙^𝛼𝑛 − 𝜙‖∞

(︂
)︂
𝛽∧1
− 2(𝛽∧1+1)
=𝑂 𝑛
.

(𝜙,𝑇 )∈ℱ

The next result justifies the validity of our confidence sets for functional regression models.
Theorem 6. Suppose that Assumptions 3, 5 are satisfied and the sequence of regularization

22

parameter is such that 𝛼𝑛 → 0 and 𝛼𝑛 𝑛1/2 → ∞ as 𝑛 → ∞. Then for any 𝛾 ∈ (0, 1)
inf
(𝜙,𝑇 )∈ℱ

(︁
)︁
(︁
)︁
1,ci
Pr 𝜙 ∈ 𝐶1−𝛾,𝑛
≥ 1 − 𝛾 − 𝑂 𝛼𝑛𝛽∧1+1 𝑛1/2 − 𝑜(1)

and
inf
(𝜙,𝑇 )∈ℱ

(︁
)︁
(︁
)︁
1,g
Pr 𝜙 ∈ 𝐶1−𝛾,𝑛
≥ 1 − 𝛾 − 𝑂 𝛼𝑛𝛽∧1+1 𝑛1/2 + 𝛼𝑛1/2 .

Once again, the choice of tuning parameters optimal for the risk, does not ensure that coverage
errors decrease to zero. To reduce the impact of the bias on the coverage properties we need to
undersmooth by setting tuning parameter 𝛼𝑛 to go to zero faster than optimal for the risk. This
will result in confidence sets with diameter shrinking at the slower rate as can be seen from the
following application of Corollary 1.
Corollary 4. Suppose that assumptions of Theorem 6 are satisfied, then
⃒
⃒
⃒ 1,ci ⃒
E ⃒𝐶1−𝛾,𝑛
⃒

∞

(︂
=𝑂

1
+ 𝛼𝑛𝛽∧1
𝛼𝑛 𝑛1/2

)︂
and

⃒
⃒
⃒ 1,g ⃒
E ⃒𝐶1−𝛾,𝑛
⃒

∞

(︂
=𝑂

1
𝛼𝑛 𝑛1/2

)︂
.

If we wish to balance the rate at which coverage errors decrease and
set shrinks,
(︂ the confidence
)︂
𝛽∧1
1
− 2(𝛽∧1+2)
− 𝛽∧1+2
, in which case both will be of order 𝑂 𝑛
we should set 𝛼𝑛 ∼ 𝑛
.

5.3 Density deconvolution
We introduce the following assumptions.
Assumption 6. (i) (𝑌𝑖 )𝑛𝑖=1 is an i.i.d. sample of 𝑌 ; (ii) 𝑓, 𝜙, 𝑟 are continuous and compactly
𝑠 [𝑎, 𝑏], for some 𝑠 ∈ (0, 1]; (iv)
supported on some subsets of [𝑎, 𝑏] with 𝑎 < 𝑏 < ∞; (iii) 𝑓 ∈ 𝐶𝑀

the integral operator 𝑇 : 𝐿2 [𝑎, 𝑏] → 𝐿2 [𝑎, 𝑏] is 1-1;
The next result gives uniform risk decomposition of the density deconvolution estimator.
Confidence sets for functional regression models will be based on the variance process of type
𝜈1,𝑛 .
Theorem 7. Suppose that Assumption 3 is satisfied with 𝑡 = 0, then
‖𝜙^𝛼𝑛 − 𝜙‖∞ ≤ ‖𝜈2,𝑛 ‖∞ + 𝑅𝛼𝑛𝛽∧1
with 𝑋2,𝑖 (𝑧) = 𝑓 (𝑌𝑖 − 𝑧) and 𝑅 as in Proposition 2. Suppose additionally that Assumption 6
holds, then
(︃
sup E ‖𝜙^𝛼𝑛 − 𝜙‖∞ = 𝑂

(𝜙,𝑇 )∈ℱ

23

1
3/2

𝛼𝑛 𝑛1/2

)︃
+ 𝛼𝑛𝛽∧1

.

To balance the variance and the bias, we should select 𝛼𝑛 ∼ 𝑛

1
− 2(𝛽∧1)+3

. In this case, the

uniform convergence rate of the deconvolution estimator is
sup E ‖𝜙^𝛼𝑛 − 𝜙‖∞

(︂
)︂
𝛽∧1
− 2(𝛽∧1)+3
.
=𝑂 𝑛

(𝜙,𝑇 )∈ℱ

Since operators 𝑇 and 𝑇 * are known, we can consider confidence sets as in Eq. (10) with
estimators replaced by the respective known quantities

⃦ 𝜀 ⃦
2,ci
⃦ +
𝑞1−𝛾,𝑛
= 2 ⃦𝜈2,𝑛
∞

(︁
)︁
1/2 √︀
3‖𝑓 ‖∞ ‖𝑇 * ‖2,∞ /2 + 𝛼𝑛
2 log(2/𝛾)
.

3/2

𝛼𝑛 𝑛1/2
1/2

2,g
𝑞1−𝛾,𝑛

= 𝑐2,1−𝛾

‖𝑇 * ‖2,∞ /2 + 𝛼𝑛
3/2

𝛼𝑛 𝑛1/2

,

where 𝑐2,1−𝛾 is 1 − 𝛾 quantile of centered Gaussian processes ‖G2 ‖∞ with covariance function
(𝑠, 𝑡) ↦→ E[𝑓 (𝑌 − 𝑠)𝑓 (𝑌 − 𝑡)], 𝑠, 𝑡 ∈ [𝑎, 𝑏].
The next result gives coverage errors for both sets.
3/2+𝛽∧1 1/2
𝑛

Theorem 8. Suppose that Assumptions 3 and 6 are satisfied. Then if 𝛼𝑛

→ 0 for any

𝛾 ∈ (0, 1)
inf
(𝜙,𝑇 )∈ℱ

inf
(𝜙,𝑇 )∈ℱ

(︁
)︁
2,ci
Pr 𝜙 ∈ 𝐶1−𝛾,𝑛
≥ 1 − 𝛾 − 𝑜(1),
(︁
)︁
(︁
)︁
2,g
Pr 𝜙 ∈ 𝐶1−𝛾,𝑛
≥ 1 − 𝛾 − 𝑂 𝑛−1/6 − 𝑜(1).

The coverage error is effectively driven by the bias for sets built with concentration inequality.
On the other hand, for sets built with Gaussian approximation we have additional 𝑛−1/6 coming
from the Berry-Esseen estimate. The following result can be obtained from a trivial application
of Corollary 1.
Corollary 5. Suppose that assumptions of Theorem 8 are satisfied, then expected diameters
⃒
⃒
⃒
⃒
(︁
)︁
⃒ 2,g ⃒
⃒ 2,ci ⃒
E ⃒𝐶1−𝛾,𝑛
⃒ and E ⃒𝐶1−𝛾,𝑛 ⃒ are of 𝑂 3/21 1/2 .
∞

∞

𝛼𝑛 𝑛

24

6 Monte Carlo experiments in the NPIV model
This section reports results of Monte Carlo experiments for our confidence sets. We focus on the
NPIV estimator. Samples of size 𝑛 ∈ {1000, 5000} are generated as follows
2

𝑌 = 𝜙(𝑍) + 𝑈,
𝜙(𝑧) = 𝑒−𝑧 /0.8 ,
⎛ ⎞
⎛⎛ ⎞ ⎛
⎞⎞
𝑍
0
𝜎𝑧2
𝜌𝜎𝑧 𝜎𝑤 𝜎𝑧𝑢
⎜ ⎟
⎜⎜ ⎟ ⎜
⎟⎟
⎜ ⎟
⎜⎜ ⎟ ⎜
⎟⎟
2
⎜𝑊 ⎟ ∼ 𝑁 ⎜⎜0⎟ , ⎜𝜌𝜎𝑧 𝜎𝑤
𝜎𝑤
0 ⎟⎟ ,
⎝ ⎠
⎝⎝ ⎠ ⎝
⎠⎠
𝑈
0
𝜎𝑧𝑢
0
𝜎𝑢2
where 𝜎𝑧 = 𝜎𝑤 = 0.3, 𝜎𝑢 =

√

0.03, 𝜎𝑧𝑢 = 0.04, and 𝜌 = 0.3. To be consistent with the theory, we

keep only observations inside sufficiently large compact set. Since the joint density of (𝑍, 𝑊 ) is
bivariate normal, this choice of functional forms resembles up to a constant the convolution of
Gaussian densities. Therefore, with appropriate modifications, we can easily adapt simulation
design not only to functional regression, but also to the deconvolution model. Given that the
structure behind all three models would be the same and given that the NPIV is the most
complex among ill-posed model, we report results of MC experiments only for it. We also
remark that the normal density is analytic, leading to singular values decaying exponentially
fast. Therefore, this design corresponds to the most difficult severelly ill-posed setting. There
were 5000 replications of each experiment. The estimate of the density function 𝑓𝑍𝑊 is obtained
using kernels. For simplicity of implementation, we do not optimize the performance with
higher-order boundary kernels and simply take the product of second-order Epanechnikov kernel
𝐾(𝑥) = 0.75(1 − 𝑥2 )1{|𝑥|≤1} .
The estimator is discretized using simple Riemann sum on the grid of 100 equidistant points.
In our setting, this number of grid points ensures that the numerical errors is negligible comparing
to the statistical noise in our setting. Higher number of grid points or numerical cubatures can
give better approximation if needed. The discretized estimator has closed-form expressions
(︁
)︁−1
^ = 𝛼𝑛 I + K⊤ K
𝜙
K⊤ r,
where I is the 𝑇 × 𝑇 identity matrix, r =

(︀ 1 ∑︀𝑛
𝑛

−1
𝑖=1 𝑌𝑖 ℎ𝑛 𝐾

(︀

ℎ−1
𝑛 (𝑊𝑖 − 𝑤)

)︀)︀
1≤𝑗≤𝑇

, K = 𝑓 Δ with

𝑓 = (𝑓^𝑍𝑊 (𝑧𝑘 , 𝑤𝑗 ))1≤𝑗,𝑘≤𝑇 , and Δ is the grid step. By Lemma 4, the operator norm ‖𝑇^* ‖2,∞ can
be computed using the mixed norm of the kernel function of 𝑇 .
In Figure 1 we plot estimates with 95% confidence band based on Gaussian approximation,
averaged over 5000 Monte Carlo experiments. Figure 2 represents the same plot for confidence

25

1.2

1.2

1

1

0.8

0.8

0.6
0.6
0.4
0.4
0.2
0.2

0
95% confidence band
Estimate
Truth

-0.2
-0.4
-1.5

-1

-0.5

0

0.5

95% confidence band
Estimate
Truth

0

1

-0.2
-1.5

1.5

-1

-0.5

0

0.5

1

1.5

z

z

(a) 𝑛 = 1000, 𝛼𝑛 = 0.14, ℎ𝑛 = 1

(b) 𝑛 = 5000, 𝛼𝑛 = 0.1, ℎ𝑛 = 1

Figure 1: Estimates and confidence bands based on Gaussian approximation, average over 5000
experiments.
bands based on concentration inequality. In our Monte Carlo experiments, Gaussian approximation leads to better performance. Resulting confidence bands are narrower and the estimator is
centered closer to the population value of the parameter of interest.
As was discussed in the Section 3, for confidence sets, the bias-variance trade-off for the risk of
the estimator reduces to the trade-off between coverage errors and the diameter of confidence
sets. In both figures we calibrated tuning parameters, so that the empirical coverage probability
is as close to the nominal 95% as possible. For larger values of tuning parameters, the band
becomes too narrow and the bias starts to dominate, reducing uniform coverage. On the other
side, even thought smaller values of tuning parameters reduce bias and the estimator becomes
closer to its population value, they also increase the size of the variance and lead to wider bands.
The optimal choice of tuning parameters balances the two.
The question of the data-driven choice of tuning parameters is extremely important in applications. Recently Centorrino (2014) developed a cross-validation approach to the choice of the
regularization parameter, optimal with respect to the 𝐿2 loss for the NPIV model of Darolles et al.
(2011). Rate-adaptive tuning parameter selection procedures in the sieve approach are discussed
in Horowitz (2014) for the 𝐿2 loss and in Chen and Christensen (2015) for the supremum loss.
In this paper we study a modification of the model of Darolles et al. (2011) and we need the
method to be optimal with respect to the sup-norm risk. Methods existing in the literature could
be used, but their theoretical optimality for confidence sets should be studied separately. It also
an open question, both for the sieve and the Tikhonov-regularized estimators, whether under
some restrictions on the class of models it is possible to obtain adaptive confidence sets.

26

1.2

1.2

1

1

0.8

0.8

0.6
0.6
0.4
0.4
0.2
0.2
0
0

-0.2

95% confidence band
Estimate
Truth

-0.4
-0.6
-1.5

-1

-0.5

0

0.5

95% confidence band
Estimate
Truth

-0.2

1

-0.4
-1.5

1.5

-1

z

-0.5

0

0.5

1

1.5

z

(a) 𝑛 = 1000, 𝛼𝑛 = 0.24, ℎ𝑛 = 1

(b) 𝑛 = 5000, 𝛼𝑛 = 0.17, ℎ𝑛 = 0.6

Figure 2: Estimates and confidence bands based on concentration inequality, average over 5000
experiments.
In the DGP used in MC experiments, Gaussian approximation demonstrates better performance
leading to more narrow bands and smaller bias. For example, Figure 1 (b) gives an excellent
confidence band with reasonably small amount of the bias. It is clear that we need to tolerate
some amount of the regularization bias in small samples to have confidence bands of reasonable
width and with good coverage properties. The amount of the bias can be reduced once the
sample size becomes sufficiently large. It is well-known that undersmoothing and bias correction
can decrease the bias at the cost of wider confidence sets, see also Florens et al. (2016) for a
discussion of bias-corrected pointwise confidence intervals.

7 Confidence sets for Engel curves
In this section we estimate Engel curves using the NPIV approach and construct confidence sets
using Gaussian approximation. Engel curves describe how the demand for commodity changes
while the household’s budget increases. Estimation of Engel curves is fundamental for the analysis
of consumer behavior and has implications in different fields of empirical research. Interesting
applications include the measurement of welfare losses associated with tax distortions in Banks,
Blundell, and Lewbel (1997), estimation of growth and inflation in Nakamura, Steinsson, and
Liu (2014), or estimation of income inequality across countries in Almås (2012).
Previously Blundell et al. (2007) estimated the shape-invariant system of Engel curves on the
UK data with sieve approach. For simplicity, we focus on the non-parametric specification of the
Engel curve. Partially linear specification can be easily estimated in two steps. First, we estimate

27

0.4

0.4
95% confidence band
NPIV
Local Linear

0.35

95% confidence band
NPIV
Local Linear

0.35

0.3

0.3

0.25

0.25

0.2

0.2

0.15

0.15

0.1

0.1

0.05

0.05

0

0
8.5

9

9.5

10

10.5

8.5

(a) With children

9

9.5

10

10.5

(b) Without children

Figure 3: Engel curves for food: 𝛼𝑛 = 0.02, ℎ𝑛 = 4.
the parametric component with differencing, see Yatchew (2003). At the second stage we get rid
of this parametric component and estimate the non-parametric model with IV approach.
Our dataset is drawn from the 2015 Consumer Expenditure Survey data in US and we
estimate Engel curves for food and alcohol. In our subsample, we have married couples with
positive income during the past 12 month, including households with and without children. The
dependent variable is the share of expenditures on the particular commodity in total non-durable
expenditures. The log of total expenditures on non-durable goods is used as an independent
variable. As in Blundell et al. (2007) we instrument the log of total expenditures with income
before tax.
NPIV estimates with 95% confidence bands are plotted in Fig. 3 and 4. For comparison, we
also include the local linear estimator, which does not correct for the endogeneity bias. We find
that the Engel curve obtained with NPIV estimator is steeper and that simple local polynomial
estimates are often outside of our confidence sets. For the local linear estimator we use the
cross-validation to obtain the bandwidth parameter. For the NPIV model, we select tuning
parameters empirically, comparing the plot for different values of tuning parameters and selecting
the one at the middle of the two extreme points, after which the estimator becomes extremely
wiggly or flat10 . To illustrate the impact of tuning parameters on estimates and confidence sets,
in Fig. 5 we plot several different choices of the regularization and the bandwidth parameters.

10

This empirical selection of tuning parameters is sometimes advocated as alternative to other approaches, see,
Cameron and Trivedi (2005). Notice that this methods is only appropriate for kernel or Tikhonov-regularized
estimators.

28

0.035

0.035
95% confidence band
NPIV
Local Linear

0.03

95% confidence band
NPIV
Local Linear

0.03

0.025

0.025

0.02

0.02

0.015

0.015

0.01

0.01

0.005

0.005

0

0
8.5

9

9.5

10

10.5

8.5

9

(a) With children

9.5

10

10.5

(b) Without children

Figure 4: Engel curves for alcohol: 𝛼𝑛 = 0.05, ℎ𝑛 = 4.

0.04

0.04
95% confidence band
NPIV
Local Linear

0.035
0.03

0.03

0.025

0.025

0.02

0.02

0.015

0.015

0.01

0.01

0.005

0.005

0
8.5

9

9.5

10

10.5

95% confidence band
NPIV
Local Linear

0.035

0
8.5

11

9

(a) 𝛼𝑛 = 0.02, ℎ𝑛 = 3

10

10.5

11

(b) 𝛼𝑛 = 0.08, ℎ𝑛 = 3

0.04

0.04
95% confidence band
NPIV
Local Linear

0.035

0.03

0.025

0.025

0.02

0.02

0.015

0.015

0.01

0.01

0.005

0.005

9

9.5

10

10.5

95% confidence band
NPIV
Local Linear

0.035

0.03

0
8.5

9.5

0
8.5

11

(c) 𝛼𝑛 = 0.05, ℎ𝑛 = 1

9

9.5

10

10.5

11

(d) 𝛼𝑛 = 0.05, ℎ𝑛 = 5

Figure 5: Engel curves for alcohol (with children). Sensitivity to tuning parameters.

29

8 Conclusions
This paper studies uniform inferential methods in ill-posed models treated with Tikhonov
regularization. Building uniform confidence sets in this setting is a difficult problem and requires
to approximate the distribution of the supremum of a complex empirical process. I show that it is
not possible to establish the weak convergence of this process in the space of continuous function
under the uniform topology for a very general class of ill-posed models known in econometrics
and statistics. Nonetheless, I demonstrate that it is possible to obtain uniform confidence sets
with relaxed coverage requirement.
In this paper I develop two alternative approaches to uniform inference that lead to honest
confidence sets. Honest confidence sets are of practical interest for several reasons. They allow to
ensure that there exists a certain sample size after which the coverage level will be not significantly
smaller then the nominal coverage level, regardless of how complicated the estimated function is
withing a given smoothness class. Moreover, as it is widely recognized in non-parametric statistics,
results for a fixed model can lead to inconsistent notions of optimality. Honest confidence sets,
on the other hand, have uniform validity.
The first approach developed in this paper relies on the Gaussian approximation to a certain
upper bound of the supremum of the statistics of interest. The second approach is based on the
assessment of tail probabilities with data-driven concentration inequalities. It is shown that the
coverage error is driven by the bias. Obtained rates may be polynomial for mildly ill-posed and
some severely ill-posed problems. These rates are new and have not been previously discussed in
the literature.
Confidence sets based on the concentration inequality are easier to compute. On the other
hand, the Gaussian approximation seems to be less conservative in Monte Carlo experiments and
leads to narrower bands with the same coverage level. As a matter of practical advice, it seems
reasonable to use confidence sets based on the concentration inequality whenever the sample size
is sufficiently large to produce small and informative sets and to rely on Gaussian approximation
otherwise.
Two important extensions left for future research are the investigation of the bootstrap and
adaptive tuning parameter selection as well as their impact on properties of constructed confidence
sets.

30

References
Karun Adusumilli, Taisuke Otsu, et al. Nonparametric instrumental regression with errors in
variables. Technical report, Suntory and Toyota International Centres for Economics and
Related Disciplines, LSE, 2015.
Ingvild Almås. International income inequality: Measuring ppp bias by estimating engel curves
for food. The American Economic Review, 102(2):1093–1117, 2012.
Donald WK Andrews and Gustavo Soares. Inference for parameters defined by moment inequalities
using generalized moment selection. Econometrica, 78(1):119–157, 2010.
Isaiah Andrews and Anna Mikusheva. A geometric approach to nonlinear econometric models.
Econometrica, 84(3):1249–1264, 2016.
Andrii Babii. Identification and estimation in the functional linear instrumental regression.
Working paper, 2016.
Andrii Babii and Jean-Pierre Florens. Is completeness necessary? penalized estimation in
non-identified models. Working paper, 2016.
James Banks, Richard Blundell, and Arthur Lewbel. Quadratic engel curves and consumer
demand. Review of Economics and statistics, 79(4):527–539, 1997.
David Benatia, Marine Carrasco, and Jean-Pierre Florens. Functional linear regression with
functional response. Working paper, 2015.
V Bentkus, F Götze, V Paulauskas, and A Račkauskas. The accuracy of gaussian approximation
in banach spaces. In Limit theorems of probability theory, pages 25–111. Springer, 2000.
R. Blundell, X. Chen, and D. Kristensen. Semi-nonparametric iv estimation of shape-invariant
engel curves. Econometrica, 75(6):1613–1669, 2007.
Stéphane Bonhomme and Jean-Marc Robin. Generalized non-parametric deconvolution with an
application to earnings dynamics. The Review of Economic Studies, 77(2):491–533, 2010.
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic theory of independence. OUP Oxford, 2013.
A Colin Cameron and Pravin K Trivedi. Microeconometrics: methods and applications. Cambridge
university press, 2005.

31

Ivan Canay, Andres Santos, and Azeem Shaikh. On the testability of identification in some
nonparametric models with endogeneity. Technical report, cemmap working paper, 2012.
Hervé Cardot, Frédéric Ferraty, André Mas, and Pascal Sarda. Testing hypotheses in the
functional linear model. Scandinavian Journal of Statistics, 30(1):241–255, 2003.
Hervé Cardot, André Mas, and Pascal Sarda. Clt in functional linear regression models. Probability
Theory and Related Fields, 138(3-4):325–361, 2007.
M. Carrasco, J.P. Florens, and E. Renault. Linear inverse problems in structural econometrics
estimation based on spectral decomposition and regularization. Handbook of econometrics, 6:
5633–5751, 2007.
Marine Carrasco and Jean-Pierre Florens. A spectral method for deconvolving a density. Econometric Theory, 27(03):546–581, 2011.
Marine Carrasco, Jean-Pierre Florens, and Eric Renault. Asymptotic normal inference in linear
inverse problems. J. S. Racine, A. Ullah and L. Su, eds, ‘Handbook of Applied Nonparametric
and Semiparametric Econometrics and Statistics’., 2013.
S Centorrino. Data driven selection of the regularization parameter in additive nonparametric
instrumental regressions. Technical report, Working paper, Stony Brook, 2014.
Xiaohong Chen and Timothy Christensen. Optimal sup-norm rates, adaptivity and inference in
nonparametric instrumental variables estimation. 2015.
Xiaohong Chen and Markus Reiss. On rate optimality for ill-posed inverse problems in econometrics. Econometric Theory, 27(03):497–521, 2011.
Victor Chernozhukov, Denis Chetverikov, Kengo Kato, et al. Gaussian approximation of suprema
of empirical processes. The Annals of Statistics, 42(4):1564–1597, 2014.
Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Coupling inequalities for suprema of
non-centered empirical and bootstrap processes. arXiv preprint arXiv:1502.00352, 2015.
Fabienne Comte, Jan Johannes, et al. Adaptive functional linear regression. The Annals of
Statistics, 40(6):2765–2797, 2012.
S. Darolles, Y. Fan, J.P. Florens, and E. Renault. Nonparametric instrumental regression.
Econometrica, 79(5):1541–1565, 2011.

32

Heinz Werner Engl, Martin Hanke, and Andreas Neubauer. Regularization of inverse problems,
volume 375. Kluwer Academic Pub, 1996.
Kirill Evdokimov. Identification and estimation of a nonparametric panel data model with
unobserved heterogeneity. Department of Economics, Princeton University, 2010.
Yingying Fan, Gareth M James, Peter Radchenko, et al. Functional additive regression. The
Annals of Statistics, 43(5):2296–2325, 2015.
Jean-Pierre Florens. Inverse problems and structural econometrics. In Advances in Economics
and Econometrics: Theory and Applications, Eighth World Congress, volume 2, page 284.
Cambridge University Press, 2003.
Jean-Pierre Florens and Sébastien Van Bellegem. Instrumental variable estimation in functional
linear models. Journal of Econometrics, 186(2):465–476, 2015.
Jean-Pierre Florens, Jan Johannes, and Sébastien Van Bellegem. Identification and estimation by
penalization in nonparametric instrumental regression. Econometric Theory, 27(03):472–496,
2011.
Jean-Pierre Florens, Joel Horowitz, and Ingrid Keilegom. Bias-corrected confidence intervals in a
class of linear inverse problems. Working paper, 2016.
Patrick Gagliardini and Olivier Scaillet. Tikhonov regularization for nonparametric instrumental
variable estimators. Journal of Econometrics, 167(1):61–75, 2012.
Eric Gautier and Yuichi Kitamura. Nonparametric estimation in random coefficients binary
choice models. Econometrica, 81(2):581–607, 2013.
Evarist Giné and Richard Nickl. Mathematical foundations of infinite-dimensional statistical
models. Cambridge Series in Statistical and Probabilistic Mathematics, 2015.
Peter Hall and Joel L Horowitz. Nonparametric methods for inference in the presence of
instrumental variables. The Annals of Statistics, 33(6):2904–2929, 2005.
Peter Hall, Joel L Horowitz, et al. Methodology and convergence rates for functional linear
regression. The Annals of Statistics, 35(1):70–91, 2007.
Joel L Horowitz. Adaptive nonparametric instrumental variables estimation: Empirical choice of
the regularization parameter. Journal of Econometrics, 180(2):158–173, 2014.

33

Joel L Horowitz and Sokbae Lee. Uniform confidence bands for functions estimated nonparametrically with instrumental variables. Journal of Econometrics, 168(2):175–188, 2012.
Kengo Kato and Yuya Sasaki. Uniform confidence bands in deconvolution with unknown error
distribution. arXiv preprint arXiv:1608.02251, 2016.
Vladimir Koltchinskii. Rademacher penalties and structural risk minimization. Information
Theory, IEEE Transactions on, 47(5):1902–1914, 2001.
Vladimir Koltchinskii et al. Local rademacher complexities and oracle inequalities in risk
minimization. The Annals of Statistics, 34(6):2593–2656, 2006.
Ker-Chau Li. Honest confidence regions for nonparametric regression. The Annals of Statistics,
pages 1001–1008, 1989.
Karim Lounici and Richard Nickl. Global uniform risk bounds for wavelet deconvolution
estimators. The Annals of Statistics, 39(1):201–231, 2011.
Eric Mbakop. Tikhonov regularization for nonparametric iv regression in sup-norm. Working
paper, 2015.
M Thamban Nair. Linear operator equations: approximation and regularization. World Scientific,
2009.
Emi Nakamura, Jón Steinsson, and Miao Liu. Are chinese growth and inflation too smooth?
evidence from engel curves. Technical report, National Bureau of Economic Research, 2014.
Whitney K Newey and James L Powell. Instrumental variable estimation of nonparametric
models. Econometrica, 71(5):1565–1578, 2003.
Vygantas Paulauskas and Alfredas Rackauskas. Approximation Theory in the Central Limit
Theorem: Exact Results in Banach Spaces. Springer Science & Business Media, 1989.
MP Rajan. Convergence analysis of a regularized approximation for solving fredholm integral
equations of the first kind. Journal of mathematical analysis and applications, 279(2):522–530,
2003.
Sidney I Resnick. Extreme values, regular variation and point processes. Springer, 1987.
A. Santos. Inference in nonparametric instrumental variables with partial identification. Econometrica, 80(1):213–275, 2012.

34

Maxwell B Stinchcombe and Halbert White. Consistent specification testing with nuisance
parameters present only under the alternative. Econometric theory, 14(03):295–325, 1998.
Alexandre Tsybakov. Introduction to nonparametric estimation, volume 11. Springer, 2009.
Aad W Van Der Vaart and Jon A Wellner. Weak Convergence. Springer, 2000.
Adonis Yatchew. Semiparametric regression for the applied econometrician. Cambridge University
Press, 2003.
VV Yurinskii. On the accuracy of normal approximation of the probability of hitting a ball.
Theory of Probability & Its Applications, 27(2):280–289, 1982.

35

Appendix A: Concentration of the supremum of empirical processes
Let (𝑋𝑖 )𝑖∈N be an i.i.d. sequence of random variables taking values in some measure space (𝑆, 𝒮)
and let 𝒢 be a countable class of real functions defined on 𝑆. Consider the empirical process
𝑛

1 ∑︁
𝑔(𝑋𝑖 ) − E𝑔(𝑋𝑖 ),
𝜈𝑛 (𝑔) ≡
𝑛

𝑔∈𝒢

𝑖=1

and denote the symmetrized version of this process by
𝑛

𝜈𝑛𝜀 (𝑔)

1 ∑︁
=
𝜀𝑖 𝑔(𝑋𝑖 ),
𝑛
𝑖=1

where (𝜀𝑖 )𝑖∈N is an i.i.d. sequence of Rademacher random variables, independent from (𝑋𝑖 )𝑖∈N .
We use ‖.‖𝒢 to denote the supremum norm over the class of functions 𝒢.
Proposition 3. Let (𝑋𝑖 )𝑛𝑖=1 be a sequence of i.i.d. random variables in some measure space
(𝑆, 𝒮), (𝜀𝑖 )𝑛𝑖=1 be a sequences of i.i.d. Radomacher random variables independent from (𝑋𝑖 )𝑛𝑖=1 ,
and 𝒢 be a countable class of functions. Then
2−1 E ‖𝜈𝑛𝜀 ‖𝒢 − 2−1 𝑛−1/2 ‖𝑃 𝑔‖𝒢 ≤ E‖𝜈𝑛 ‖𝒢 ≤ 2E ‖𝜈𝑛𝜀 ‖𝒢 .

(11)

The second inequality in Eq. (11), is a symmetrization inequality, while the first is the
desymmetrization inequality. These inequalities have important applications in the empirical
process theory, see Van Der Vaart and Wellner (2000). We refer to (Koltchinskii et al., 2006, p.7)
and references therein for a proof of this modification of the symmetrization inequality.
Our construction of confidence sets in this paper relies on the Talagrand-type concentration
inequality. More precisely, it can be derived from the concentration inequality for functions of
bounded difference, known also as McDiarmid’s inequality, see excellent treatment of concentration
inequalities in Boucheron et al. (2013). Talagrand’s concentration inequalities only describe
exponential decline of tails probabilities for the deviation of the supremum of empirical process
‖𝜈𝑛 ‖𝒢 from its expected value E‖𝜈𝑛 ‖𝒢 . This expected value is not known in practice. However,
it can be estimated using the multiplier bootstrap based on Rademacher random variables. The
symmetrization inequality allows then to compare the expected value of this estimate ‖𝜈𝑛𝜀 ‖𝒢
to the original expected value, which leads to the data-driven concentration inequality. This
insightful idea comes from the machine learning literature, see e.g. Koltchinskii (2001).
The next proposition states the precise version of the data-driven Talagrand-type concentration
inequality used in the present paper.

36

Proposition 4. Suppose that (𝑋𝑖 )𝑛𝑖=1 , (𝜀𝑖 )𝑛𝑖=1 , and 𝒢 as in Proposition 3. Suppose also that the
absolute value of all 𝑔 ∈ 𝒢 is uniformly bounded by some constant 𝐹 < ∞. Then for all 𝑛 ∈ N
(︃
Pr ‖𝜈𝑛 ‖𝒢 >

√︂
2 ‖𝜈𝑛𝜀 ‖𝒢

+ 3𝐹

2𝑥
𝑛

)︃
≤ 2𝑒−𝑥 .

See (Giné and Nickl, 2015, Theorem 3.4.5.) for the proof of this result.

37

Appendix B: Proofs of main results
Impossibility of weak convergence
Proof of Proposition 1. The proof is inspired by Cardot et al. (2007) who show impossibility
of weak convergence in 𝐿2 in the special case of functional linear regression model without
endogeneity. For simplicity we focus on the process 𝜈1,𝑛 , since the proof for the process 𝜈2,𝑛 is
similar. Recall that for some normalizing sequence 𝑟𝑛 , the weak convergence of 𝑟𝑛 𝜈1𝑛 in 𝐿2 to
some random element requires ⟨𝑟𝑛 𝜈1𝑛 , 𝛿⟩ to converge weakly in R for all 𝛿 ∈ 𝐿2 , (Van Der Vaart
and Wellner, 2000, Theorem 1.8.4). Remark also that (𝑇 * 𝑇 )−1 is unbounded densely defined
operator with 𝒟[(𝑇 * 𝑇 )−1 ] ⊂ 𝐿2 [𝑎, 𝑏]𝑝 . Then, if 𝛿 ∈ 𝒟[(𝑇 * 𝑇 )−1 ], we need to set 𝑟𝑛 = 𝑛1/2 , since
𝑛

(︁
⟩︀ 𝑑
⟨︀
⟩︀2 )︁
1 ∑︁ ⟨︀ *
𝑇 𝑋𝑖 , (𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝛿 −
→ 𝑁 0, E 𝑇 * 𝑋1 , (𝑇 * 𝑇 )−1 𝛿
.
⟨𝑟𝑛 𝜈1𝑛 , 𝛿⟩ = √
𝑛
𝑖=1

On the other hand, if 𝛿 ̸∈ 𝒟[(𝑇 * 𝑇 )−1 ], ‖(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝛿‖2 → ∞, making E⟨𝑇 * 𝑋1 , (𝛼𝑛 𝐼 +
𝑇 * 𝑇 )−1 𝛿⟩2 → ∞, and so ⟨𝑛1/2 𝜈1𝑛 , 𝛿⟩ can’t converge in distribution. This shows that it is not
possible to converge weakly in 𝐿2 . Since bounded and continuous functionals on 𝐿2 are bounded
and continuous on (𝐶, ‖.‖∞ ) for finite measure spaces, it follows from the definition of weak
convergence that 𝑟𝑛 𝜈1𝑛 does not converge weakly in (𝐶, ‖.‖∞ ) for any choice of normalizing
sequence 𝑟𝑛 .

Proofs of main results
Proof of Theorem 1. Using Assumption 2 (i) and triangle inequality, we obtain, for 𝑗 ∈ {1, 2}
(︁
)︁
(︁
)︁
𝑗,ci
𝑗,ci
Pr 𝜙 ∈ 𝐶1−𝛾,𝑛
= Pr ‖𝜙^𝛼𝑛 − 𝜙‖∞ ≤ 𝑞1−𝛾,𝑛
(︁
)︁
𝑗,ci
≥ Pr ‖^
𝜈𝑗,𝑛 ‖∞ + 𝜂𝑗,𝑛 ≤ 𝑞1−𝛾,𝑛
(︁
)︁
𝑗,ci
≥ Pr ‖𝜈𝑗,𝑛 ‖∞ + ‖^
𝜈𝑗,𝑛 − 𝜈𝑗,𝑛 ‖∞ + 𝜂𝑗,𝑛 ≤ 𝑞1−𝛾,𝑛
.

(12)

Now ‖𝜈𝑗,𝑛 ‖∞ , 𝑗 = 1, 2 are suprema of empirical processes indexed by the following classes of
functions changing with the sample size 𝑛
{︀
[︀
]︀
}︀
𝒢1,𝑛 = 𝑔 : 𝑥 ∈ 𝐶[𝑎, 𝑏]𝑞 → (𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝑇 * 𝑥 (𝑧), 𝑧 ∈ [𝑎, 𝑏]𝑝
{︀
[︀
]︀
}︀
𝒢2,𝑛 = 𝑔 : 𝑥 ∈ 𝐶[𝑎, 𝑏]𝑝 → (𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝑥 (𝑧), 𝑧 ∈ [𝑎, 𝑏]𝑝 .
Notice that for any 𝑔1 ∈ 𝒢1,𝑛 and 𝑔2 ∈ 𝒢2,𝑛 , under Assumptions 1 (ii) and (iv) by Lemma 1, we

38

have the following envelops
‖𝑔1 (𝑋1,𝑖 )‖∞ ≤ ‖𝑇 * ‖2,∞ ‖(𝛼𝑛 𝐼 + 𝑇 𝑇 * )−1 ‖‖𝑋1,𝑖 ‖ ≤

‖𝑇 * ‖2,∞
𝑢𝑛 𝐹1
𝛼𝑛

𝑎.𝑠.

1/2

‖𝑔2 (𝑋2,𝑖 )‖∞ ≤ ‖(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 ‖∞ ‖𝑋2,𝑖 ‖∞ ≤

‖𝑇 * ‖2,∞ /2 + 𝛼𝑛
3/2

𝛼𝑛

𝐹2

𝑎.𝑠.,

′ ‖ and ‖𝑋 ‖
where 𝐹1 and 𝐹2 are the smallest envelops for ‖𝑋1,𝑖
2,𝑖 ∞ respectively.

It follows from Eq. (12), reverse triangle inequality, inequality Pr(𝑋 + 𝑌 ≤ 𝑥 + 𝑦) ≥ Pr(𝑋 ≤
𝑥) − Pr(𝑌 > 𝑦), Markov’s inequality, and11 Proposition 4
(︁

1,ci
Pr 𝜙 ∈ 𝐶1−𝛾,𝑛

)︁

)︃
√︂
^* ‖2,∞
⃦ 𝜀 ⃦
2
log(2/𝛾)
𝑢
‖
𝑇
𝑛
≥ Pr ‖𝜈1,𝑛 ‖∞ + ‖^
𝜈1,𝑛 − 𝜈1,𝑛 ‖∞ + 𝜂1,𝑛 ≤ 2 ⃦𝜈^1,𝑛 ⃦∞ + 3
𝑢𝑛 𝐹^1
+
𝛼𝑛
𝑛
𝛼𝑛 𝑛1/2
(︃
)︃
√︂
(︂
)︂
*
1/2
⃦ 𝜀 ⃦
⃦ + 3 ‖𝑇 ‖2,∞ 𝑢𝑛 𝐹1 2 log(2/𝛾) − 𝑂 𝛼𝑛 𝑛 E𝑄1,𝑛
≥ Pr ‖𝜈1,𝑛 ‖∞ ≤ 2 ⃦𝜈1,𝑛
∞
𝛼𝑛
𝑛
𝑢𝑛
(︂
)︂
𝛼𝑛 𝑛1/2
≥1−𝛾−𝑂
E𝑄1,𝑛 ,
𝑢𝑛
(13)
(︃

where
E𝑄1,𝑛 =

⃒
⃒
⃦
⃦
𝑢𝑛
⃒ ^*
^1 − ‖𝑇 * ‖2,∞ 𝐹1 ⃒⃒+E ⃦𝜈^𝜀 − 𝜈 𝜀 ⃦ +E ‖^
E
‖
𝑇
‖
𝐹
𝜈1,𝑛 − 𝜈1,𝑛 ‖∞ +E𝜂1,𝑛 . (14)
⃒
2,∞
1,𝑛
1,𝑛 ∞
1/2
𝛼𝑛 𝑛

Under Assumptions 1 (i), (ii), (v) and Assumptions 2 (ii)-(iii) by Lemma 3
(︂
sup E ‖^
𝜈1,𝑛 − 𝜈1,𝑛 ‖∞ = 𝑜

(𝜙,𝑇 )∈ℱ

𝑢𝑛
𝛼𝑛 𝑛1/2

)︂
and

⃦
⃦ 𝜀
𝜀 ⃦
=𝑜
− 𝜈1,𝑛
sup E ⃦𝜈^1,𝑛
∞

(𝜙,𝑇 )∈ℱ

(︂

𝑢𝑛
𝛼𝑛 𝑛1/2

)︂
. (15)

On the other hand, E𝜂1,𝑛 = 𝑂(𝑟1,𝑛 ) under Assumption 2 (i).
To asses the order of the first term in Eq. (14), first observe that
⃒
⃒2
⃦
⃒
⃒2
⃦2
⃒
⃒
⃒^
⃒
⃦ ^′
′
′ ⃦
⃒
⃒
E ⃒𝐹1 − 𝐹1 ⃒ ≤ 2E ⃒ max ‖𝑋1,𝑖 ‖ − 𝐹1 ⃒ + 2E max ⃦𝑋1,𝑖 − 𝑋1,𝑖 ⃦ ,
1≤𝑖≤𝑛

(16)

1≤𝑖≤𝑛

𝑎.𝑠.

′ ‖ −
where the first term is 𝑜(1), since under Assumptions 1 (i), we have max1≤𝑖≤𝑛 ‖𝑋1,𝑖
−→ 𝐹1 ,

e.g. see (Resnick, 1987, p.8-9), while the second term is 𝑜(1) under Assumption 2 (iii).
Since 𝛼𝑛 → 0, Assumption 2 (ii) implies
‖𝑇^* − 𝑇 * ‖22,∞ = 𝑜(1),

11

By continuity of trajectories of 𝜈𝑗,𝑛 , 𝑗 = 1, 2 processes, their suprema can be restricted to countable sets of
rational numbers.

39

and so combining Eq. (13)-(15), we obtain
inf
(𝜙,𝑇 )∈ℱ

(︁
)︁
1,ci
Pr 𝜙 ∈ 𝐶1−𝛾,𝑛
≥ 1 − 𝛾 − 𝑂(𝜌1,𝑛 𝑟1,𝑛 ) − 𝑜 (1) .

This finishes the proof for the case 𝑗 = 1.
The proof of the second statement is similar and we omit it.
Proof of Theorem 2. Under Assumption 2 (i) by Markov’s inequality and computations in Eq. (8),
we have uniformly in ℱ
(︁
)︁
(︁
)︁
1,g
1,g
Pr 𝜙 ∈ 𝐶1−𝛾,𝑛
= Pr ‖𝜙^ − 𝜙‖∞ ≤ 𝑞1−𝛾,𝑛
⎛⃦
⎞
⃦2
𝑛
⃦
⃦ 1 ∑︁
⃦
⃦
≥ Pr ⎝⃦ √
𝑋1,𝑖 ⃦ ≤ 𝑐1,1−𝛾 ⎠ − 𝑂(𝜌1,𝑛 𝑟1,𝑛 )
⃦ 𝑛
⃦
𝑖=1
(︁
)︁
(︀
)︀
≥ Pr ‖G1 ‖2 ≤ 𝑐1,1−𝛾 − 𝑂 𝑛−1/2 + 𝜌1,𝑛 𝑟1,𝑛
(︁
)︁
= 1 − 𝛾 − 𝑂 𝑛−1/2 + 𝜌1,𝑛 𝑟1,𝑛 ,
where the second inequality follows under Assumptions 1 (i) and (iv) by the Berry-Esseen theorem
in Hilbert space of Yurinskii (1982).
Similarly,
(︁

Pr 𝜙 ∈

2,g
𝐶1−𝛾,𝑛

)︁

⃦
(︃⃦
𝑛
⃦ 1 ∑︁
⃦
⃦
⃦
≥ Pr ⃦ √
𝑋2,𝑖 ⃦
⃦ 𝑛
⃦
𝑖=1

)︃
≤ 𝑐2,1−𝛾

− 𝑂(𝜌2,𝑛 𝑟2,𝑛 ).

∞

Under Assumptions 1 (i), (iii), and (iv), the process inside the second probability is Donsker and
by the Berry-Esseen theorem in Banach space, see Paulauskas and Rackauskas (1989)
⃦
(︃⃦
𝑛
⃦ 1 ∑︁
⃦
⃦
⃦
inf Pr ⃦ √
𝑋2,𝑖 ⃦
⃦ 𝑛
⃦
(𝜙,𝑇 )∈ℱ
𝑖=1

)︃
≤ 𝑐2,1−𝛾

)︁
(︁
≥ 1 − 𝛾 − 𝑂 𝑛−1/6 ,

∞

giving the second result.
⃒
⃒
⃒ 𝑗,s ⃒
Proof of Corollary 1. Since E ⃒𝐶1−𝛾,𝑛
⃒

∞

⃒
⃒
⃒ 𝑗,s ⃒
= 2E ⃒𝑞1−𝛾,𝑛
⃒ for 𝑗 = 1, 2 and s ∈ {g, ci}, under As-

sumption 2 (ii), we have
⃒
⃒
⃒ 1,g ⃒
E ⃒𝐶1−𝛾,𝑛
⃒

∞

(︂
=𝑂

1
𝛼𝑛 𝑛1/2

)︂

⃒
⃒
⃒ 2,g ⃒
E ⃒𝐶1−𝛾,𝑛
⃒

,

∞

40

(︃
=𝑂

1
3/2

𝛼𝑛 𝑛1/2

)︃
.

On the other hand, under Assumption 2 (ii)
⃒
⃒
⃒ 1,ci ⃒
E ⃒𝐶1−𝛾,𝑛
⃒

(︃
∞

=𝑂

𝜀
E‖𝜈1,𝑛
‖∞

𝑢𝑛
𝛼𝑛 𝑛1/2

⃦
⃦ 𝜀
𝜀 ⃦
+
− 𝜈1,𝑛
+ E ⃦𝜈^1,𝑛
∞

(︃√︂

⃒2 √︁
⃒
⃒
⃒
E ⃒𝐹^1 − 𝐹1 ⃒ + E‖𝑇^* − 𝑇 * ‖22,∞

Under Assumption 1 (i), by the desymmetrization inequality in Proposition 3
𝜀
E‖𝜈1,𝑛
‖∞

(︂
= E‖𝜈1,𝑛 ‖∞ + 𝑂

)︂

1
𝑛1/2

(︂
=𝑂

𝑢𝑛
𝛼𝑛 𝑛1/2

)︂
.

𝜀 − 𝜈 𝜀 ‖.
By Lemma 3, we can have the same estimate for E‖^
𝜈1,𝑛
1,𝑛

Therefore, using the same reasoning as the one used to obtain estimates in Eq (16), we have
⃒
⃒
⃒ 1,ci ⃒
E ⃒𝐶1−𝛾,𝑛
⃒

(︂
∞

=𝑂

)︂
𝑢𝑛
+ 𝑟1,𝑛 .
𝛼𝑛 𝑛1/2

The proof that
⃒
⃒
⃒ 2,ci ⃒
E ⃒𝐶1−𝛾,𝑛 ⃒

(︃
∞

=𝑂

1
3/2

𝛼𝑛 𝑛1/2

)︃
+ 𝑟2,𝑛

is similar and so we omit it.
Proof of Proposition 2. By Lemma 1, (𝛼𝑛 𝐼 + 𝑇 * 𝑇 ) is invertible operator between (𝐶, ‖.‖∞ )
spaces. Using 𝑓 (𝑇 * 𝑇 )𝑇 * = 𝑇 * 𝑓 (𝑇 𝑇 * ) with 𝑓 (𝑥) = (𝛼𝑛 + 𝑥)−1 , and factorizing the operator norm
‖𝑇 * 𝑔(𝑇 𝑇 * )𝜑‖∞ ≤ ‖𝑇 * ‖2,∞ ‖𝑔(𝑇 𝑇 * )‖‖𝜑‖ with 𝑔(𝑥) = 𝛼𝑛 (𝛼𝑛 + 𝑥)−1 𝑥𝛽 , under Assumption 3 for
any (𝜙, 𝑇 ) ∈ ℱ
⃦
⃦
⃦[︀
]︀ ⃦
⃦(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝑇 * 𝑟 − 𝜙⃦ = ⃦ (𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝑇 * 𝑇 − 𝐼 𝜙⃦
∞
∞
⃦
⃦
*
−1
= ⃦𝛼𝑛 (𝛼𝑛 𝐼 + 𝑇 𝑇 ) 𝜙⃦∞
⃦
⃦
⃦
⃦
= ⃦𝛼𝑛 (𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 (𝑇 * 𝑇 )𝛽 𝑇 * 𝜓 ⃦
∞
⃦
⃦
⃦ *
⃦
= ⃦𝑇 𝛼𝑛 (𝛼𝑛 𝐼 + 𝑇 𝑇 * )−1 (𝑇 𝑇 * )𝛽 𝜓 ⃦
∞
⃦
⃦
⃦
⃦
*
* −1
≤ ‖𝑇 ‖2,∞ ⃦𝛼𝑛 (𝛼𝑛 𝐼 + 𝑇 𝑇 ) (𝑇 𝑇 * )𝛽 ⃦ ‖𝜓‖
Notice that for 𝑏 ∈ (0, 1), the function 𝜆 ↦→
maximum at 𝜆 =

𝑏
1−𝑏 𝛼𝑛 .

𝜆𝑏
𝛼𝑛 +𝜆

(17)

is strictly concave on (0, ∞) admitting its

On the other hand, for 𝑏 ∈ [1, ∞), this function is strictly increasing on

[0, ‖𝑇 ‖2 ], reaching its maximum at the end of this interval. Therefore, by isometry of functional
calculus
⃦
⃦
⃦
* −1
* 𝛽⃦
𝛼
(𝛼
𝐼
+
𝑇
𝑇
)
(𝑇
𝑇
)
⃦ 𝑛 𝑛
⃦ = 𝛼𝑛

41

sup
𝜆∈[0,‖𝑇 ‖2 ]

⃒
⃒
⃒ 𝜆𝛽 ⃒
˜ 𝛽∧1
⃒
⃒
⃒ 𝛼𝑛 + 𝜆 ⃒ ≤ 𝑅𝛼𝑛

)︃)︃
.

˜ = 𝛽 𝛽 (1 − 𝛽)1−𝛽 10<𝛽<1 + 𝐶 2(𝛽−1) 1𝛽≥1 . Therefore,
with 𝑅
⃦
⃦
sup ⃦(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝑇 * 𝑟 − 𝜙⃦∞ ≤ 𝑅𝛼𝑛𝛽∧1

(𝜙,𝑇 )∈ℱ

˜
with 𝑅 = 𝐶 2 𝑅.

Proofs for the NPIV model
Proposition 5. Suppose that Assumptions 4 (i)-(iii) are satisfied and the sequence of bandwidth
parameters is such that 1/(𝑛ℎ𝑝+𝑞
𝑛 ) = 𝑂(1), ∀𝑛 ∈ N, then for all 1 < 𝑟 < ∞
(︂ ⃦
⃦𝑟
⃦
⃦
E ⃦𝑇^* − 𝑇 * ⃦

⎛√︃

)︂1/𝑟

= 𝑂⎝

2,∞

log ℎ−1
𝑛
𝑝+𝑞
𝑛ℎ𝑛

⎞
+ ℎ𝑠𝑛 ⎠ .

Proof. By Lemma 4,
⎡
⃦
⃦𝑟
⃦
⃦
E ⃦𝑇^* − 𝑇 * ⃦

2,∞

(︃∫︁

= E ⎣ sup

𝑧∈(𝑎,𝑏)𝑝

[𝑎,𝑏]𝑞

⃒
⃒2
⃒^
⃒
𝑓
(𝑧,
𝑤)
−
𝑓
(𝑧,
𝑤)
⃒ 𝑍𝑊
⃒ d𝑤
𝑍𝑊

)︃1/2 ⎤𝑟
⎦

⃦
⃦𝑟
⃦
⃦
≤ (𝑏 − 𝑎)𝑞𝑟/2 E ⃦𝑓^𝑍𝑊 − 𝑓𝑍𝑊 ⃦ .
∞

By triangle inequality,
⃦𝑟 )︁1/𝑟 (︁ ⃦
⃦𝑟 )︁1/𝑟 (︁⃦
⃦𝑟 )︁1/𝑟
(︁ ⃦
⃦^
⃦
⃦^
⃦
⃦ ^
⃦
^
E ⃦𝑓𝑍𝑊 − 𝑓𝑍𝑊 ⃦
≤ E ⃦𝑓𝑍𝑊 − E𝑓𝑍𝑊 ⃦
+ ⃦E𝑓𝑍𝑊 − 𝑓𝑍𝑊 ⃦
.
∞

∞

∞

The order of the bias follows by standard computations under the assumption 𝑓𝑍𝑊 ∈ 𝐶𝐿𝑠 [𝑎, 𝑏]𝑝+𝑞
and Assumption 4 (iii), see Tsybakov (2009)
⃦
⃦
⃦ ^
⃦
⃦E𝑓𝑍𝑊 − 𝑓𝑍𝑊 ⃦

∞

= 𝑂 (ℎ𝑠𝑛 ) .

For the variance term, we apply the moment inequality in (Giné and Nickl, 2015, Theorem 5.1.5
and Theorem 5.1.15), which gives
⎞
⎛√︃
√︃
⃦𝑟 )︁1/𝑟
(︁ ⃦
−1
log ℎ𝑛
1
1 ⎠
⃦
⃦
E ⃦𝑓^𝑍𝑊 − E𝑓^𝑍𝑊 ⃦
= 𝑂⎝
+
+
.
𝑝+𝑞
𝑝+𝑞
∞
𝑛ℎ𝑛
𝑛ℎ𝑛
𝑛ℎ𝑝+𝑞
𝑛
Combining all estimates, the result follows.
Proof of the Theorem 3. We focus on 𝑗 = 1 first. The proof is based on the following decomposi-

42

tion
⃦
⃦
⃦
⃦
‖𝜙^𝛼𝑛 − 𝜙‖∞ = ⃦(𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 𝑇^* 𝑟^ − 𝜙⃦

∞

≤ ‖^
𝜈1,𝑛 ‖∞ + 𝜉𝑛 + 𝐵𝑛 ,
where
⃦
⃦
[︃ 𝑛
]︃
⃦
⃦
∑︁
(︀
)︀
1
⃦
*
−1 * ⃦
𝜉𝑛 = ⃦(𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 𝑇^*
𝜙(𝑍𝑖 )ℎ−𝑞 𝐾 ℎ−1
(𝑊
−
𝑤)
−
(𝛼
𝐼
+
𝑇
𝑇
)
𝑇
𝑟
⃦ ,
𝑖
𝑛
𝑛
⃦
⃦
𝑛
𝑖=1

∞

⃦
⃦
𝐵𝑛 = ⃦(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝑇 * 𝑟 − 𝜙⃦∞ .
(︀ −1
)︀
and the variance ‖^
𝜈1,𝑛 ‖∞ is as in Eq. (7) with 𝑋1,𝑖 (𝑤) = 𝑈𝑖 ℎ−𝑞
𝑛 𝐾𝑤 ℎ𝑛 (𝑊𝑖 − 𝑤) . Under
(︁
)︁
the Assumption 3 by Proposition 2, the regularization bias is 𝐵𝑛 = 𝑂 𝛼𝑛𝛽∧1 , uniformly over
(𝜙, 𝑇 ) ∈ ℱ.
Decompose the remaining term 𝜉𝑛 ≤ 𝜉1,𝑛 + 𝜉2,𝑛 with

𝜉1,𝑛
𝜉2,𝑛

⃦
[︃ 𝑛
]︃⃦
⃦
⃦
∑︁ 1
(︀
)︀
1
⃦
⃦
−1
= ⃦(𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 𝑇^*
𝑞 𝐾𝑤 ℎ𝑛 (𝑊𝑖 − 𝑤) {𝜙(𝑍𝑖 ) − [𝐾ℎ𝑛 * 𝜙](𝑍𝑖 )} ⃦ ,
⃦
⃦
𝑛
ℎ𝑛
𝑖=1
∞
⃦[︁
]︁ ⃦
⃦
⃦
= ⃦ (𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 𝑇^* 𝑇^ − (𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝑇 * 𝑇 𝜙⃦ ,

where [𝐾ℎ𝑛 *𝜙](𝑧0 ) =

∞

∫︀

[𝑎,𝑏]𝑝

(︀ −1
)︀
𝜙(𝑧)ℎ−𝑝
𝑛 𝐾𝑧 ℎ𝑛 (𝑧0 − 𝑧) d𝑧. Under Assumption 3 and Assumptions 4

(ii)-(iii), by Lemma 2 and Proposition 5, for the second term, we have
⎛
E𝜉2,𝑛 = 𝑂 ⎝

⎛√︃

1 ⎝

1/2
𝛼𝑛

log ℎ−1
𝑛
𝑝+𝑞
𝑛ℎ𝑛

⎞⎞
+ ℎ𝑠𝑛 ⎠⎠ .

Coming back to the first term, under Assumption 4, we have

𝜉1,𝑛

⃦
⃦
𝑛
⃦
⃦
⃦ ⃦ 1 ∑︁
(︀
)︀
1
⃦
⃦
⃦
⃦
−1
≤ ‖𝑇^* ‖2,∞ ⃦(𝛼𝑛 𝐼 + 𝑇^𝑇^* )−1 ⃦ ⃦
𝑞 𝐾𝑤 ℎ𝑛 (𝑊𝑖 − .) {𝜙(𝑍𝑖 ) − [𝐾ℎ𝑛 * 𝜙](𝑍𝑖 )}⃦ .
⃦
⃦𝑛
ℎ𝑛
𝑖=1

𝑡 [𝑎, 𝑏]𝑝 , so that using standard bias computations under AssumpUnder Assumption 3, 𝜙 ∈ 𝐶𝑀

tion 4 (iii), we have
(︀ )︀
‖[𝐾ℎ𝑛 * 𝜙] − 𝜙‖∞ = 𝑂 ℎ𝑡𝑛 ,

43

On the other hand, the second term is 𝛼𝑛−1 . Lastly, under Assumption 4 (i)
⃦2
⃦ 𝑛
⃦ 1 ∑︁ ⃒
⃦
(︀ −1
)︀⃒⃦
(︀
)︀⃦2
⃦
⃒𝐾𝑤 ℎ𝑛 (𝑊𝑖 − .) ⃒⃦
⃦
E⃦
⃦ ≤ E ⃦𝐾𝑤 ℎ−1
𝑛 (𝑊𝑖 − .)
⃦
⃦𝑛
𝑖=1
∫︁ ∫︁
(︀
)︀
=
𝐾 2 ℎ−1
˜ d𝑤𝑓
˜ 𝑊 (𝑤)d𝑤
𝑛 (𝑤 − 𝑤)
= 𝑂(ℎ𝑞𝑛 ).
Therefore, by the Cauchy-Schwartz inequality
(︁
)︁
E𝜉1,𝑛 = 𝑂 𝛼𝑛−1 ℎ𝑡−𝑞/2
.
𝑛
Now, we focus on 𝑗 = 2. Decompose
‖𝜙^𝛼𝑛 − 𝜙‖∞ ≤ ‖^
𝜈2,𝑛 ‖∞ + 𝜁𝑛 + 𝐵𝑛 ,
where
⃦
⃦
)︃
(︃
𝑛
⃦
⃦
∑︁
1
⃦
*
−1 * ⃦
* ^ −1
*
^
^
𝑈𝑖 𝑓𝑍𝑊 (., 𝑊𝑖 ) − (𝛼𝑛 𝐼 + 𝑇 𝑇 ) 𝑇 𝑟⃦
𝜁𝑛 = ⃦(𝛼𝑛 𝐼 + 𝑇 𝑇 )
𝑇 𝑟^ −
⃦
⃦
𝑛
𝑖=1

∞

𝐵𝑛 = ‖(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )𝑇 * 𝑟 − 𝜙‖∞ .
(︁
)︁
Similarly, to 𝑗 = 1 case, the bias is of order 𝑂 𝛼𝑛𝛽∧1 . Likewise, decompose 𝜁𝑛 ≤ 𝜁1,𝑛 + 𝜉2,𝑛 ,
where 𝜉2,𝑛 is the same as in the 𝑗 = 1 case, and

𝜁1,𝑛

⃦
(︃
)︃⃦
𝑛
⃦
⃦
∑︁
1
⃦
⃦
= ⃦(𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 𝑇^* 𝑟^ − 𝑇^* 𝑇^𝜙 −
𝑈𝑖 𝑓𝑍𝑊 (., 𝑊𝑖 ) ⃦
⃦
⃦
𝑛
𝑖=1
⃦
⃦∞
𝑛
⃦
⃦ ⃦
⃦
1 ∑︁
⃦
⃦ ⃦
⃦
≤ ⃦(𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 ⃦ ⃦𝑇^* (^
𝑟 − 𝑇^𝜙) −
𝑈𝑖 𝑓𝑍𝑊 (., 𝑊𝑖 )⃦
⃦
𝑛
∞⃦
𝑖=1

∞

=: 𝐼𝑛 × 𝐼𝐼𝑛 .
(︁
)︁
−3/2
By Lemma 1, E𝐼𝑛 = 𝑂 𝛼𝑛
. Decompose the second term as
⃦
⃦
𝑛
⃦
⃦
∑︁
1
⃦ *
⃦
𝐼𝐼𝑛 ≤ ⃦𝑇 (^
𝑟 − 𝑇^𝜙) −
𝑈𝑖 𝑓𝑍𝑊 (., 𝑊𝑖 )⃦
⃦
⃦
𝑛
𝑖=1

∞

≤ 𝐼𝐼𝐼𝑛 + 𝐼𝑉𝑛 + 𝑉𝑛 ,

44

⃦
⃦
⃦
⃦
+ ⃦(𝑇^* − 𝑇 * )(^
𝑟 − 𝑇^𝜙)⃦

∞

where

⃦ 𝑛
⃦
⃦ 1 ∑︁
⃦
⃦
⃦
𝐼𝐼𝐼𝑛 = ⃦
𝑈𝑖 {[𝑓𝑍𝑊 * 𝐾ℎ𝑛 ](., 𝑊𝑖 ) − 𝑓𝑍𝑊 (., 𝑊𝑖 )}⃦
⃦𝑛
⃦
𝑖=1
∞
⃦
⃦
𝑛
⃦
⃦ 1 ∑︁
⃦
⃦
{𝜙(𝑍𝑖 ) − [𝜙 * 𝐾ℎ𝑛 ](𝑍𝑖 )} [𝑓𝑍𝑊 * 𝐾ℎ𝑛 ](., 𝑊𝑖 )⃦
𝐼𝑉𝑛 = ⃦
⃦
⃦𝑛
𝑖=1
∞
⃦
⃦
⃦ ^*
⃦
*
^
𝑉𝑛 = ⃦(𝑇 − 𝑇 )(^
𝑟 − 𝑇 𝜙)⃦ ,
∞

and [𝑓𝑍𝑊 * 𝐾ℎ𝑛 ](𝑧, 𝑊𝑖 ) =

∫︀

[𝑎,𝑏]𝑞

(︀ −1
)︀
𝑓𝑍𝑊 (𝑧, 𝑤)ℎ−𝑞
𝑛 𝐾𝑤 ℎ𝑛 (𝑊𝑖 − 𝑤) d𝑤. Using standard bias com-

putations, under Assumption 4 (i)-(iii), we have a.s. 𝐼𝐼𝐼𝑛 = 𝑂(ℎ𝑠𝑛 ) and 𝐼𝑉𝑛 = 𝑂(ℎ𝑡𝑛 ). Lastly,
√︂ ⃦
⃦2
⃦ ^*
⃦
*
≤ E ⃦𝑇 − 𝑇 ⃦

⃦
⃦
⃦ ^*
⃦
*
^
E𝑉𝑛 = E ⃦(𝑇 − 𝑇 )(^
𝑟 − 𝑇 𝜙)⃦

∞

2,∞

⃦
⃦2
⃦
⃦
^
E ⃦𝑟^ − 𝑇 𝜙⃦ .

The order of the first term follows by Lemma 5, while
⃦
⃦2
⃦
⃦2
𝑛
𝑛
⃦
⃦2
⃦ 1 ∑︁
⃦ 1 ∑︁
(︀ −1
)︀⃦
(︀ −1
)︀⃦
1
1
⃦
⃦
⃦
⃦
⃦
⃦
^
E ⃦^
𝑟 − 𝑇 𝜙⃦ ≤ 2E ⃦
𝑈𝑖 𝑞 𝐾𝑤 ℎ𝑛 (𝑊𝑖 − .) ⃦ + 2E ⃦
{𝜙(𝑍𝑖 ) − [𝐾ℎ * 𝜙](𝑍𝑖 )} 𝑞 𝐾𝑤 ℎ𝑛 (𝑊𝑖 − .) ⃦
⃦𝑛
⃦
⃦𝑛
⃦
ℎ
ℎ
𝑛
𝑛
𝑖=1
𝑖=1
(︂
)︂
1
=𝑂
+ ℎ2𝑡−𝑞
.
𝑛
𝑛ℎ𝑞𝑛

Therefore, collecting all estimates, we obtain
⎛

⎞⎞
⎞⎞
⎛
⎛√︃
)︃ ⎛√︃
(︃
−1
−1
1
log
ℎ
1
log
ℎ
1
𝑛
𝑛
𝑠 ⎠⎠
𝑠 ⎠⎠
⎝
E𝜁𝑛 = 𝑂 ⎝ 3/2 ⎝ℎ𝑠𝑛 + ℎ𝑡𝑛 + √︀ 𝑞 + ℎ𝑡−𝑞/2
+ 1/2 ⎝
.
𝑛
𝑝+𝑞 + ℎ𝑛
𝑝+𝑞 + ℎ𝑛
𝑛ℎ
𝑛ℎ
𝑛ℎ𝑛
𝑛
𝑛
𝛼𝑛
𝛼𝑛
It remains to asses the order of variance terms
E ‖^
𝜈1,𝑛 ‖∞

⃦
⃦
𝑛
⃦ ⃦ 1 ∑︁
⃦
⃦
(︀
)︀
⃦
⃦
⃦
⃦
−1
≤ E‖𝑇^* ‖2,∞ ⃦(𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 ⃦ ⃦
𝑈𝑖 ℎ−𝑞
𝑛 𝐾𝑤 ℎ𝑛 (𝑊𝑖 − .) ⃦
⃦𝑛
⃦
𝑖=1
⎞
⎛ ⎯
⃦2
⎸ ⃦
𝑛
⃦ 1 ∑︁
⃦
(︀
)︀
⃦
⎜ 1 ⎸
⃦ ⎟
−𝑞
= 𝑂 ⎝ ⎷E ⃦
𝑈𝑖 ℎ𝑛 𝐾𝑤 ℎ−1
𝑛 (𝑊𝑖 − 𝑤) ⃦ ⎠ ,
⃦
⃦
𝛼𝑛
𝑛
𝑖=1

where
⃦2
⃦
(︂
)︂
𝑛
⃦
⃦ 1 ∑︁
⃦
(︀
)︀
(︀ −1
)︀⃦2
1
1
⃦
⃦
−𝑞
−1
⃦
⃦
E⃦
𝑈𝑖 ℎ𝑛 𝐾𝑤 ℎ𝑛 (𝑊𝑖 − 𝑤) ⃦ =
E 𝑈𝑖 𝐾𝑤 ℎ𝑛 (𝑊𝑖 − 𝑤)
=𝑂
.
⃦
⃦𝑛
𝑛ℎ𝑞𝑛
𝑛ℎ2𝑞
𝑛
𝑖=1
In the similar way, applying Lemma 1, we obtain
(︃
E‖^
𝜈2,𝑛 ‖∞ = 𝑂

45

1
3/2

𝛼𝑛 𝑛1/2

)︃
.

1

Proof of Corollary 2. Notice that in the 𝑗 = 1 case, it is necessary to set ℎ𝑛 ∼ 𝑛− 2𝑡 to balance the
(︀
)︀
variance with ℎ𝑡𝑛 /𝛼𝑛 ℎ𝑞/2 term. In this case the order of the variance will be 𝑂 (𝛼𝑛 𝑛1/2−𝑞/4𝑡 )−1
and we need 𝑡 > 𝑞/2 to make sure that it goes to zero. To balance this variance with re−

2𝑡−𝑞

𝛽∧1
maining
to set 𝛼𝑛 ∼ 𝑛 4𝑡(𝛽∧1+1) . It remains to make sure that
(︂√︂ bias term 𝛼)︂𝑛 , it(︁ is necessary
)︁
log ℎ−1
1
1
𝑠
𝑛
=𝑜
for the choice of tuning parameters specified above to en𝑝+𝑞 + ℎ𝑛
1/2
1/2 𝑞/2
𝛼𝑛

𝑛ℎ𝑛

𝛼𝑛 𝑛

ℎ𝑛

sure that this term converges to zero faster than the bias and the variance terms, or in other words,
2𝑠+𝑞
we need 𝛼𝑛 ℎ−𝑝
→ 0. The first requirement reduces to 𝑡 > 𝑞/2 + 𝑝(𝛽 ∧ 1 + 1),
𝑛 → 0 and 𝛼𝑛 𝑛ℎ𝑛

while the second requirement holds whenever 𝑠 > 0.75(𝑡 − 𝑞/2).
Now we turn to the case 𝑗 = 2. First, it is necessary to set ℎ𝑛 ∼ 𝑛

1
− 2(𝑠∧𝑡)

to balance bias coming
−

1

from non-parametric estimation with the variance term. Second, we need to (︂√︂
set 𝛼𝑛 ∼ 𝑛 2(𝛽∧1)+3
)︂
log ℎ−1
1
𝑠
𝑛
to balance the regularization bias with the variance term. To ensure that 1/2
+
ℎ
𝑝+𝑞
𝑛 =
𝑛ℎ𝑛
𝛼𝑛
(︁
)︁
− 𝑝+𝑞
𝑜 3/21 1/2 , we need 𝛼𝑛 ℎ𝑛 2 → 0 and 𝛼𝑛 𝑛1/2 ℎ𝑠𝑛 → 0. The first condition reduces to 𝑠 ∧ 𝑡 >
𝛼𝑛 𝑛
𝑝+𝑞
4 (2(𝛽

∧ 1) + 3), while the second requirement is satisfied for any 𝑠, 𝑡 > 0. It remains to ensure

2𝑡−𝑞
that 𝛼𝑛 𝑛−1 ℎ−𝑞
→ 0. The former holds whenever 𝑠 ∧ 𝑡 >
𝑛 → 0 and 𝛼𝑛 ℎ𝑛

5
12 𝑞,

while for the latter

it is sufficient to have 𝑡 > 𝑞/2. Obviously, 𝑠 ∧ 𝑡 > 54 (𝑝 + 𝑞) is the minimal binding condition for
)︂ (︂√︂
)︂
(︂
(︁
)︁
log ℎ−1
1
1
𝑠
𝑛
√ 1 𝑞 + ℎ𝑡−𝑞/2
+
ℎ
any 𝛽 > 0. Lastly, we show that 3/2
=
𝑜
, which
𝑛
𝑝+𝑞
3/2
𝑛
𝛼𝑛

reduces to verifying that

−𝑝/2−𝑞
𝑛−1/2 ℎ𝑛

𝑛ℎ𝑛

→ 0,

𝑡−𝑝/2−𝑞
ℎ𝑛

→ 0,

𝑛ℎ𝑛
𝑠−𝑞/2
ℎ𝑛

𝛼𝑛 𝑛1/2

→ 0, and

𝑡+𝑠−𝑞/2
𝑛1/2 ℎ𝑛

→ 0.

Under stated rates for tuning parameters, these four conditions in turn are satisfied whenever
𝑠 ∧ 𝑡 > 𝑝/2 + 𝑞, 𝑡 > 𝑝/2 + 𝑞, 𝑠 > 𝑞/2, and 𝑠 ∧ 𝑡 > 𝑞/2 respectively, so that we need 𝑠 ∧ 𝑡 > 𝑝/2 + 𝑞.
Therefore, in light of the previous estimates, the condition 𝑠 ∧ 𝑡 > 54 (𝑝 + 𝑞) is sufficient to ensure
that the variance and the bias component dominate in the risk decomposition.
Proof of Theorem 4. We check that under Assumptions 3 and 4, all relevant conditions of
Theorems 1 and 2 are satisfied. First, under Assumptions 4, we obtain Assumptions 1 with
(︀ −1
)︀
′
𝑢𝑛 = ℎ−𝑞
𝑛 and 𝑋1,𝑖 (𝑤) = 𝑈𝑖 𝐾𝑤 ℎ𝑛 (𝑤𝑖 − 𝑤) . In particular, boundedness and compact support
of the density function 𝑓𝑍𝑊 and the kernel function 𝐾 ensure both Assumptions 1 (iv) and (v).
Theorem 3 gives the risk decomposition and so Assumption 2 (i) is satisfied. Assumption 2 (ii)
follows, since by Proposition 5
(︂ ⃦
⃦4
1
⃦ ^*
*⃦
E
𝑇
−
𝑇
⃦
⃦
1/2

𝛼𝑛

2,∞

⎛√︃

)︂1/4
=

1 ⎝

1/2
𝛼𝑛

log ℎ−1
𝑛
𝑝+𝑞
𝑛ℎ𝑛

⎞
+ ℎ𝑠𝑛 ⎠ ,

which goes to zero under the smoothness conditions and orders on tuning parameters imposed in
Corollary 2. It remains to verify Assumption 2 (iii). To that end, notice that the first condition

46

follows by the uniform convergence of the estimator, since
⃦
⃦
⃦ ^′
′ ⃦
⃦𝑋1,𝑖 − 𝑋1,𝑖
⃦ ≤ ‖𝐾‖∞ ‖𝜙^𝛼𝑛 − 𝜙‖∞ .
For the second condition we also need uniform consistency of the kernel density estimator, since
⃦
⃦
⃦^
⃦
⃦𝑋2,𝑖 − 𝑋2,𝑖 ⃦

∞

⃦
⃦
⃦
⃦
≤ ‖𝜙^ − 𝜙‖∞ ‖𝑓𝑍𝑊 ‖∞ + 𝐹0 ⃦𝑓^𝑍𝑊 − 𝑓𝑍𝑊 ⃦

∞

𝑞/2

Proof of Corollary 3. For 𝑗 = 1, 𝜌𝑗,𝑛 = 𝑛1/2 𝛼𝑛 ℎ𝑛
need

−𝑞/2
𝛼𝑛−1 𝑛−1/2 ℎ𝑛

∼ 𝑛1/2 ℎ𝑡𝑛 ∼

𝑞/2
𝑛1/2 𝛼𝑛𝛽∧1+1 ℎ𝑛 .

⃦
⃦
⃦
⃦
+ ‖𝜙^ − 𝜙‖∞ ⃦𝑓^𝑍𝑊 − 𝑓𝑍𝑊 ⃦ .
∞

and to balance the variance and the bias we
This is achieved when ℎ𝑛 ∼ 𝑛

𝛽∧1+1
− (𝛽∧1)(𝑡+𝑞/2)+2𝑡

𝑡−𝑞/2

−

and 𝛼𝑛 ∼√︂𝑛 (𝛽∧1)(𝑡+𝑞/2)+2𝑡 . For this choice of tuning parameters, it remains to verify that
−𝛽∧1−1/2
−𝛽∧1−1/2 𝑠
log ℎ−1
𝑛
𝛼𝑛
ℎ𝑛 → 0 to ensure that the noise coming from the estimation
𝑝+𝑞 → 0 and 𝛼𝑛
𝑛ℎ𝑛

of the operator has smaller effect on the coverage error than the bias. The first requirement reduces
𝛽∧1+1/2
to 𝑡 > 𝑞/2 + 𝑝 1+𝛽∧1
1−𝛽∧1 while the second requirement is satisfied whenever 𝑠 > (𝑡 − 𝑞/2) 𝛽∧1+1 .

Now we turn to the case 𝑗 = 2. Here to balance the variance and the bias we need
−3/2 −1/2
𝑛

3/2+𝛽∧1 1/2
𝑛

𝛼𝑛

∼ 𝛼𝑛

and ℎ𝑛 ∼ 𝑛

− 2(𝑠∧𝑡)(𝛽∧1+3)

∼ 𝑛1/2 (ℎ𝑠𝑛 + ℎ𝑡𝑛 ). This requirement is achieved when 𝛼𝑛 ∼ 𝑛

1
− 𝛽∧1+3

2(𝛽∧1)+3

. It remains to show that additional noise has negligible impact on cov−𝑠∧𝑡−𝑞/2−(𝑝+𝑞)/2 −1
𝑛

erage errors, comparing to the bias. To that end, we need to verify that ℎ𝑛
−𝑠∧𝑡+𝑡−𝑞/2−(𝑝+𝑞)/2 −1/2
𝑛

ℎ𝑛

−𝑠∧𝑡−𝑞/2+𝑠 −1/2
𝑛

→ 0, ℎ𝑛

−𝑠∧𝑡+𝑡−𝑞/2+𝑠

→ 0, ℎ𝑛

→ 0,

−𝑠∧𝑡−(𝑝+𝑞)/2 −1/2
𝑛

→ 0, 𝛼𝑛 ℎ𝑛

→

0, and 𝛼𝑛 ℎ−𝑠∧𝑡+𝑠
→ 0. These requirements are satisfied when 𝑠 ∧ 𝑡 > (𝑝 + 2𝑞) 2(𝛽∧1)+3
,
𝑛
6
2(𝛽∧1)+3
2(𝛽∧1)+3
𝑠 ∧ 𝑡 > (𝑝 + 2𝑞) 2(𝛽∧1)+3
2(𝛽∧1+3) , 𝑠 ∧ 𝑡 > 𝑞 2(𝛽∧1+3) , 𝑠 ∧ 𝑡 > 𝑞/2, and 𝑠 ∧ 𝑡 > (𝑝 + 𝑞) 2(2−𝛽∧1) respectively

(the last requirement is trivially satisfied). All requirements hold whenever 𝑠 ∧ 𝑡 > 2.5(𝑝 + 𝑞).
A trivial application of the Corollary 1 describes convergence rates for expected diameters.

Proofs for functional regressions
Proof of Theorem 5. Continuity of trajectories of processes 𝑍 and 𝑊 ensures several useful
properties. First, by the Dominated convergence theorem, the covariance function E[𝑍(𝑡)𝑊 (𝑠)]
is continuous on [0, 1]𝑝+𝑞 . This in turn implies that covariance operators 𝑇 and 𝑇 * are in ℒ2,∞ .
Second, it also ensures that 𝑇^, 𝑇^* ∈ ℒ2,∞ a.s.
Consider the following decomposition
𝜈1,𝑛 ‖∞ + 𝜉𝑛 + 𝐵𝑛 ,
‖𝜙^𝛼𝑛 − 𝜙‖∞ ≤ ‖^

47

where
⃦
⃦
⃦
⃦
𝜉𝑛 = ⃦(𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 𝑇^* 𝑇^𝜙 − (𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝑇 * 𝑇 𝜙⃦
∞
⃦
⃦
⃦
⃦
‖^
𝜈1,𝑛 ‖∞ = ⃦(𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 𝑇^* (^
𝑟 − 𝑇^𝜙)⃦
∞

⃦
⃦
𝐵𝑛 = ⃦(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝑇 * 𝑟 − 𝜙⃦∞
The order of the bias term 𝐵𝑛 follows under the source condition (Assumption 3) by Proposition 2.
Notice that
⃦
⃦2
⃦
⃦
E ⃦𝑇^* − 𝑇 * ⃦

2,∞

⃦
⃦2
𝑛
⃦ 1 ∑︁
⃦
⃦
⃦
≤ E⃦
𝑍𝑖 𝑊𝑖 − E[𝑍𝑖 𝑊𝑖 ]⃦
⃦𝑛
⃦
𝑖=1

∞

Using Hoffman-Jørgensen’s inequality, e.g. (Giné and Nickl, 2015, Theorem 3.1.16)
⎛ ⃦
⃦2 ⎞1/2
⃦
(︃ ⃦ 𝑛
𝑛
⃦ 1 ∑︁
⃦
⃦ 1 ∑︁
⃦
⃦
⃦
⃦
⃦
⎝E ⃦
𝑍𝑖 𝑊𝑖 − E[𝑍𝑖 𝑊𝑖 ]⃦ ⎠ = 𝑂 E ⃦
𝑍𝑖 𝑊𝑖 − E[𝑍𝑖 𝑊𝑖 ]⃦
⃦𝑛
⃦
⃦𝑛
⃦
𝑖=1

𝑖=1

∞

(︂ [︂
]︂)︂1/2 )︃
+ 𝑛−1 E max ‖𝑍𝑖 𝑊𝑖 ‖2∞
.
1≤𝑖≤𝑛

∞

Under Assumption 5 (ii), the second term is 𝑂(𝑛−1 ), while the first term is the expected value of
the supremum of the empirical process. Since the process (𝑡, 𝑠) ↦→ 𝑍(𝑡)𝑊 (𝑠) has Hölder smooth
trajectories, by (Giné and Nickl, 2015, Theorem 3.5.13 and 4.3.36)
⃦
⃦
𝑛
⃦ 1 ∑︁
⃦
⃦
⃦
E⃦
𝑍𝑖 𝑊𝑖 − E[𝑍𝑖 𝑊𝑖 ]⃦
⃦𝑛
⃦
𝑖=1

(︂
=𝑂

∞

1
√
𝑛

)︂
.

Therefore, by Lemma 2
(︃
sup E𝜉𝑛 = 𝑂
(𝜙,𝑇 )∈ℱ

1
1/2

𝛼𝑛 𝑛1/2

)︃
,

which established the first claim of the theorem.
Now we consider the variance term
⃦
⃦
⃦
⃦
E‖^
𝜈1,𝑛 ‖∞ = E ⃦𝑇^* (𝛼𝑛 𝐼 + 𝑇^𝑇^* )−1 (^
𝑟 − 𝑇^𝜙)⃦
∞
⃦
⃦
⃦⃦
⃦
⃦
⃦
⃦
*
*
−1
^
^
^
^
≤ E‖𝑇 ‖2,∞ ⃦(𝛼𝑛 𝐼 + 𝑇 𝑇 ) ⃦ ⃦𝑟^ − 𝑇 𝜙⃦
√︂ ⃦
√︂
⃦2
⃦
⃦2
‖𝑇 * ‖2,∞
1
⃦
⃦
⃦
⃦
^
≤
E ⃦𝑟^ − 𝑇 𝜙⃦ +
E‖𝑇^* − 𝑇 * ‖22,∞ E ⃦𝑟^ − 𝑇^𝜙⃦ ,
𝛼𝑛
𝛼𝑛
where we applied the Cauchy-Schwartz inequality to obtain the last line.
Under Assumption 5 (i)
(︂ )︂
⃦
⃦2
1
⃦
⃦
^
E ⃦𝑟^ − 𝑇 𝜙⃦ = 𝑂
.
𝑛

(18)

Therefore, taking previous estimates into account, we obtain the desired order of the variance

48

and the second claim of theorem follows.
Proof of Theorem 6. We apply Theorem 1 and 2. Notice that we can set 𝑢𝑛 = 1 so that
′ and that under Assumptions 5 (i) and (iii)-(v), the Assumption 1 is satisfied.
𝑋1,𝑖 = 𝑋1,𝑖

Theorem 5 ensures Assumption 2 (i). To verify Assumption 2 (ii), notice that
⃦
⃦4
⃦
⃦
E ⃦𝑇^* − 𝑇 * ⃦

2,∞

⃦ 𝑛
⃦4
⃦ 1 ∑︁
⃦
⃦
⃦
≤ E⃦
𝑍𝑖 𝑊𝑖 − E[𝑍𝑖 𝑊𝑖 ]⃦ .
⃦𝑛
⃦
𝑖=1

∞

Under Assumption 5 (ii), applying the Hoffman-Jørgensen’s inequality, see (Giné and Nickl,
2015, Theorem 3.1.16) and (Giné and Nickl, 2015, Theorem 3.5.13 and 4.3.36)
⎛ ⃦
⃦4 ⎞1/4
𝑛
⃦ 1 ∑︁
⃦
(︁
)︁
⃦
⎝E ⃦
𝑍𝑖 𝑊𝑖 − E[𝑍𝑖 𝑊𝑖 ]⃦ ⎠ = 𝑂 𝑛−1/2 .
⃦
⃦𝑛
⃦
𝑖=1

∞

This gives
⃦
⃦4
⃦
⃦
E ⃦𝑇^* − 𝑇 * ⃦

2,∞

(︀
)︀
= 𝑂 𝑛−2 ,

which is 𝑜(𝛼𝑛2 ), since 𝛼𝑛 → 0 and 𝛼𝑛 𝑛1/2 → ∞ as 𝑛 → ∞.
Also notice that by Cauchy-Schwartz inequality under Assumption 5 (ii)
⃦
⃦
⃦^
⃦
E max ⃦𝑋
^𝛼𝑛 − 𝜙, 𝑍𝑖 ⟩ 𝑊𝑖 ‖ ≤ E ‖𝜙^ − 𝜙‖∞ ,
1,𝑖 − 𝑋1,𝑖 ⃦ = E ‖⟨𝜙
1≤𝑖≤𝑛

which goes to zero whenever 𝛼𝑛 → 0 and 𝛼𝑛 𝑛1/2 → ∞ by Theorem 5. This verifies Assumption 2
(iii) and finishes the proof.

Proofs for the deconvolution model
Proof of Theorem 7. The result follows from the following variance-bias decomposition
⃦
⃦
⃦
* 𝑟 − 𝑇 * 𝑟)⃦
‖𝜙^𝛼𝑛 − 𝜙‖∞ ≤ ⃦(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 (𝑇̂︂
⃦

∞

+ ‖(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝑇 * 𝑟 − 𝜙‖∞

=: ‖𝜈2,𝑛 ‖∞ + 𝐵𝑛 .
By Proposition 2, the bias can be bounded as 𝐵𝑛 ≤ 𝑅𝛼𝑛𝛽∧1 proving the first result. For the
second result, we bound the variance as

‖𝜈2,𝑛 ‖∞

⃦
⃦
𝑛
⃦
∑︁
⃦
⃦ ⃦
1
⃦
⃦
*
−1
⃦
⃦
≤ (𝛼𝑛 𝐼 + 𝑇 𝑇 )
𝑓 (𝑌𝑖 − .) − E𝑓 (𝑌 − .)⃦ ,
∞⃦
⃦𝑛
⃦
𝑖=1

49

∞

where the second term is the supremum of empirical process indexed by the class of functions
𝒢˜𝑛 = {𝑘 : [𝑎, 𝑏] → R, 𝑦 ↦→ 𝑓 (𝑦 − 𝑧), 𝑧 ∈ [𝑎, 𝑏]} .
By Assumption 6 (iii), 𝒢˜𝑛 is class of 𝑠-Hölder functions. Then by (Giné and Nickl, 2015, Corollary
3.5.8)
⃦
⃦
𝑛
⃦ 1 ∑︁
⃦
⃦
⃦
E⃦
𝑓 (𝑌𝑖 − .) − E𝑓 (𝑌 − .)⃦
⃦𝑛
⃦
𝑖=1

(︂
=𝑂

∞

1
√
𝑛

)︂
.

⃦
⃦
Applying Lemma 1 to ⃦(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 ⃦∞ , the result follows.
Proof of Theorem 8. We verify Assumptions 1 and 2. Recall that in the risk the composition
given in Theorem 7, the variance process is driven by 𝑋2,𝑖 (𝑧) = 𝑓 (𝑌𝑖 − 𝑧), which is bounded
by 𝐹2 = ‖𝑓 ‖∞ . Therefore, under Assumptions 6, Assumption 1 (i) and (iii)-(vi) are satisfied.
Assumption 2 (i) follows from Theorem 7 with 𝑟2,𝑛 = 𝛼𝑛𝛽∧1 , while (ii) and (iii) are trivially
satisfied due to the fact that the operator 𝑇 * and the process 𝑋2,𝑖 are known.

50

Appendix C: Additional results
We need the following inequality, known in the numerical ill-posed inverse problems literature,
e.g. see (Rajan, 2003, Proposition 2.1), (Nair, 2009, Chapter 5, problem (8)). For completeness
we provide a short proof here.
Lemma 1. Suppose that 𝑇 : 𝐿2 [𝑎, 𝑏]𝑝 → 𝐿2 [𝑎, 𝑏]𝑞 is an integral operator with continuous kernel
function. Then for any 𝛼 > 0 (𝛼𝐼 + 𝑇 * 𝑇 ) is invertible as an operator from ℛ(𝑇 * 𝑇 ) ⊂ (𝐶, ‖.‖∞ )
to (𝐶, ‖.‖∞ ) space and

*
1/2
⃦
⃦
⃦(𝛼𝐼 + 𝑇 * 𝑇 )−1 ⃦ ≤ ‖𝑇 ‖2,∞ /2 + 𝛼 .
∞
𝛼3/2

Proof. Continuity of the kernel function ensures that 𝑇 * 𝑇 maps to 𝐶[𝑎, 𝑏]𝑝 . Since for any
𝛼 > 0, 𝛼𝐼 + 𝑇 * 𝑇 is invertible as an operator on the 𝐿2 space, (Nair, 2009, Lemma 4.1), and
(𝐶, ‖.‖∞ ) ⊂ 𝐿2 , it is also invertible as an operator between (𝐶, ‖.‖∞ ).
Notice that for any 𝜙 ∈ 𝐶[𝑎, 𝑏]𝑝
[︀

]︀
(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝑇 * 𝑇 − 𝐼 𝜙 = −𝛼𝑛 (𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝜙.

Using this identity
⃦
⃦
⃦(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 ⃦ =
∞

⃦
⃦
‖(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝑇 * 𝑇 ‖∞ + 1
sup ⃦(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝜙⃦∞ ≤
.
𝛼𝑛
‖𝜙‖∞ =1

(19)

Factoring the norm as ‖𝑇 * 𝜓‖∞ ≤ ‖𝑇 * ‖2,∞ ‖𝜓‖, and using the isometry of functional calculus
⃦
⃦
⃦
⃦
⃦(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝑇 * 𝑇 ⃦ = ⃦𝑇 * (𝛼𝑛 𝐼 + 𝑇 𝑇 * )−1 𝑇 ⃦ ≤ ‖𝑇 * ‖2,∞
∞
∞

⃒
⃒
⃒ 𝜆1/2 ⃒ ‖𝑇 * ‖
⃒
⃒
2,∞
.
sup ⃒
⃒=
1/2
𝜆∈[0,‖𝑇 ‖2 ] ⃒ 𝛼𝑛 + 𝜆 ⃒
2𝛼𝑛
(20)

Combining this with Eq. (19) gives the result.
Lemma 2. Suppose that 𝑇 : 𝐿2 [𝑎, 𝑏]𝑝 → 𝐿2 [𝑎, 𝑏]𝑞 is an integral operator in ℒ2 with continuous
kernel function. Suppose further that Assumption 3 is satisfied, then if the risk E‖𝑇^* − 𝑇 * ‖22,∞
is bounded uniformly over ℱ, for all 𝑛 ∈ N and all (𝜙, 𝑇 ) ∈ ℱ, we have
⃦[︁
]︁ ⃦
⃦
⃦
E ⃦ (𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 𝑇^* 𝑇^ − (𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝑇 * 𝑇 𝜙⃦

∞

=𝑂

(︂√︁
)︂
𝛼𝑛−1 E‖𝑇^* − 𝑇 * ‖22,∞ .

^ < ∞ a.s. ensure that 𝑇^𝑇^* is a self-adjoint operator in ℒ2 a.s., so that the
Proof. ‖𝑇^‖ ≤ ‖𝑘‖
spectral theory for bounded self-adjoint operators in Hilbert spaces (in particular functional
calculus) applies to it similarly as to 𝑇 𝑇 * and 𝑇 * 𝑇 .

51

Decompose
[︁

]︁
[︁
]︁
(𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 𝑇^* 𝑇^ − (𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝑇 * 𝑇 𝜙 = 𝛼𝑛 (𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 − 𝛼𝑛 (𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝜙
[︁
]︁
= (𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 𝑇^* 𝑇^ − 𝑇 * 𝑇 𝛼𝑛 (𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝜙
≤ (𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 𝑇^* (𝑇^ − 𝑇 )𝛼𝑛 (𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝜙
+ (𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 (𝑇^* − 𝑇 * )𝛼𝑛 𝑇 (𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝜙
≡ 𝐼𝑛 + 𝐼𝐼𝑛 .

Both terms involve the error from the estimation of the operators and regularization bias. For
the first term, we factor the operator norm ‖𝑇^* 𝜓‖∞ ≤ ‖𝑇^* ‖2,∞ ‖𝜓‖
‖𝐼𝑛 ‖∞

⃦
⃦
⃦
⃦
⃦
*
* −1 ⃦ ^
^
^
^
≤ ‖𝑇 ‖2,∞ ⃦(𝛼𝑛 𝐼 + 𝑇 𝑇 ) ⃦ ‖𝑇 − 𝑇 ‖ ⃦𝛼𝑛 (𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝜙⃦

By isometry of functional calculus
⃦
⃦
⃦
⃦
⃦(𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 ⃦ =

⃒
⃒
⃒ 1 ⃒
⃒≤ 1 ,
⃒
sup ⃒
𝛼𝑛 + 𝜆 ⃒ 𝛼𝑛
𝜆∈[0,‖𝑇^‖2 ]

a.s.

Under Assumption 3, the last term is
⃦
⃦
⃦
⃦
⃦𝛼𝑛 (𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 (𝑇 * 𝑇 )𝛽 𝑇 * 𝜓 ⃦ ≤ 𝛼𝑛

⃒
⃒
⃒ 𝜆𝛽+1/2 ⃒
(︁
)︁
⃒
⃒
sup ⃒
⃒ 𝐶 = 𝑂 𝛼𝑛(𝛽+1/2)∧1 .
𝜆∈[0,‖𝑇 ‖2 ] ⃒ 𝛼𝑛 + 𝜆 ⃒

Combining these findings with Cauchy-Schwartz inequality and using the bound
‖𝑇^ − 𝑇 ‖ = ‖𝑇^* − 𝑇 * ‖ ≤ (𝑏 − 𝑎)𝑝/2 ‖𝑇^* − 𝑇 * ‖2,∞ ,
we have uniformly in (𝜙, 𝑇 ) ∈ ℱ
(︃
E‖𝐼𝑛 ‖∞ = 𝑂

)︃

(𝛽+1/2)∧1 √︁

𝛼𝑛

E‖𝑇^* − 𝑇 * ‖22,∞

𝛼𝑛

.

For the second term, we factor the operator norm in the following way
⃦
⃦ ⃦
⃦
⃦
⃦ ⃦
⃦
‖𝐼𝐼𝑛 ‖∞ ≤ E ⃦(𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 ⃦ ⃦𝑇^* − 𝑇 * ⃦
∞

2,∞

52

⃦
⃦
⃦𝛼𝑛 𝑇 (𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1 𝜙⃦

By Lemma 1 and triangle inequality
⃦
⃦
⃦
⃦
⃦(𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 ⃦

∞

1/2

‖𝑇^* − 𝑇 * ‖2,∞ /2 + ‖𝑇 * ‖2,∞ /2 + 𝛼𝑛

≤

3/2

𝛼𝑛

Under Assumption 3,
⃦
⃦
⃦
*
−1
*
𝛽 * ⃦
⃦𝛼𝑛 𝑇 (𝛼𝑛 𝐼 + 𝑇 𝑇 ) (𝑇 𝑇 ) 𝑇 𝜓 ⃦ ≤ 𝛼𝑛

sup
𝜆∈[0,‖𝑇 ‖2 ]

⃒ 𝛽+1 ⃒
⃒
⃒ 𝜆
⃒
⃒
⃒ 𝛼𝑛 + 𝜆 ⃒ 𝐶 = 𝑂(𝛼𝑛 ).

Combining all above findings, we have uniformly in (𝜙, 𝑇 )
(︃
E‖𝐼𝐼𝑛 ‖∞ = 𝑂

E‖𝑇^* − 𝑇 * ‖22,∞ + E‖𝑇^* − 𝑇 * ‖2,∞

)︃

1/2

,

𝛼𝑛

and the conclusion follows from collecting all estimates and using Jensen’s inequality.
Lemma 3. Suppose that 𝑇 : 𝐿2 [𝑎, 𝑏]𝑝 → 𝐿2 [𝑎, 𝑏]𝑞 is an integral operator in ℒ2 with continuous
kernel function for which we have continuous estimator. Consider the variance processes as in
Eq. (5) and Eq. (7), where 𝑋1,𝑖 and 𝑋2,𝑖 , 𝑖 = 1, . . . , 𝑛 are i.i.d. centered processes with continuous
′ ‖2 < ∞ and E‖𝑋 ‖2 < ∞ uniformly over the class of models ℱ.
trajectories such that E‖𝑋1,𝑖
2,𝑖
(︀ 2 )︀
*
*
4
Suppose additionally that E‖𝑇^ − 𝑇 ‖2,∞ = 𝑜 𝛼𝑛 , then for any (𝜙, 𝑇 ) ∈ ℱ, we have

(︂
E ‖^
𝜈1,𝑛 − 𝜈1,𝑛 ‖∞ = 𝑜

𝑢𝑛
𝛼𝑛 𝑛1/2

(︃

)︂
E ‖^
𝜈2,𝑛 − 𝜈2,𝑛 ‖∞ = 𝑜

,

1
3/2

𝛼𝑛 𝑛1/2

)︃
.

Moreover, for symmetrized processes, for any (𝜙, 𝑇 ) ∈ ℱ, we have
⃦ 𝜀
⃦
𝜀 ⃦
− 𝜈1,𝑛
=𝑜
E ⃦𝜈^1,𝑛
∞
and

(︂

(︃
⃦ 𝜀
⃦
𝜀 ⃦
E ⃦𝜈^2,𝑛
− 𝜈2,𝑛
=𝑜
∞

𝑢𝑛
𝛼𝑛 𝑛1/2

1
3/2

𝛼𝑛 𝑛1/2

(︂

)︂)︂
√︁
′
′
2
^
1 + E‖𝑋1,𝑖 − 𝑋1,𝑖 ‖

(︂
)︂)︃
√︁
^ 2,𝑖 − 𝑋2,𝑖 ‖2
1 + E‖𝑋
.

Proof. Similarly as in the proof of Lemma 2, decompose

E ‖^
𝜈2,𝑛 − 𝜈2,𝑛 ‖∞

⃦
⃦
𝑛
⃦[︁
⃦
]︁ 1 ∑︁
⃦
⃦
*
−1
*
−1
^
^
= E ⃦ (𝛼𝑛 𝐼 + 𝑇 𝑇 ) − (𝛼𝑛 𝐼 + 𝑇 𝑇 )
𝑋2,𝑖 ⃦
⃦
⃦
𝑛
𝑖=1
∞
⃦
⃦
𝑛
⃦
⃦
∑︁
1
⃦
⃦
= E ⃦(𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 (𝑇 * 𝑇 − 𝑇^* 𝑇^)(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1
𝑋2,𝑖 ⃦
⃦
⃦
𝑛
𝑖=1

≤ 𝑅1,𝑛 + 𝑅2,𝑛 ,

53

∞

where we can show using similar computations that

𝑅1,𝑛

𝑅2,𝑛

⃦
⃦
𝑛
⃦
⃦
∑︁
1
⃦
⃦
= E ⃦(𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 (𝑇^* − 𝑇 * )𝑇 (𝛼𝑛 𝐼 + 𝑇 𝑇 * )−1
𝑋2,𝑖 ⃦
⃦
⃦
𝑛
𝑖=1
⃦∞ 𝑛
⃦
⃦ 1 ∑︁
⃦
⃦
⃦ ⃦
⃦
⃦
⃦
⃦
⃦
⃦
⃦
⃦
⃦
*
−1
*
−1
*
*
^
^
^
⃦
⃦
𝑇 (𝛼𝑛 𝐼 + 𝑇 𝑇 )
𝑋2,𝑖 ⃦
≤ E ⃦(𝛼𝑛 𝐼 + 𝑇 𝑇 ) ⃦ ⃦𝑇 − 𝑇 ⃦
⃦
⃦𝑛
⃦
∞
2,∞
𝑖=1
)︃
(︃
(︂ ⃦
⃦4 )︂1/4 (︀
)︀1/2
1
⃦ ^*
*⃦
E‖𝑋2,𝑖 ‖2
E ⃦𝑇 − 𝑇 ⃦
=𝑂
1/2
2
2,∞
𝛼𝑛 𝑛
⃦
⃦
𝑛
⃦
⃦
∑︁
1
⃦
⃦
= E ⃦(𝛼𝑛 𝐼 + 𝑇^* 𝑇^)−1 𝑇^* (𝑇^ − 𝑇 )(𝛼𝑛 𝐼 + 𝑇 * 𝑇 )−1
𝑋2,𝑖 ⃦
⃦
⃦
𝑛
𝑖=1
⃦ 𝑛
⃦
⃦
⃦
⃦⃦
⃦⃦
∑︁
⃦⃦
1
⃦
⃦
⃦
⃦
⃦
⃦
*
−1
*
*
−1
𝑋2,𝑖 ⃦
≤ E‖𝑇^ ‖2,∞ ⃦(𝛼𝑛 𝐼 + 𝑇^𝑇^ ) ⃦ ⃦𝑇^ − 𝑇 ⃦ ⃦(𝛼𝑛 𝐼 + 𝑇 𝑇 ) ⃦ ⃦
⃦𝑛
⃦
𝑖=1
)︃
(︃
(︂ ⃦
⃦4 )︂1/4 (︀
)︀1/2
1
⃦ ^*
*⃦
E‖𝑋2,𝑖 ‖2
=𝑂
E ⃦𝑇 − 𝑇 ⃦
1/2
2
2,∞
𝛼𝑛 𝑛

Likewise,

E ‖^
𝜈1,𝑛 − 𝜈1,𝑛 ‖∞

⃦
⃦
𝑛
⃦[︁
⃦
]︁ 1 ∑︁
⃦
⃦
= E ⃦ 𝑇^* (𝛼𝑛 𝐼 + 𝑇^𝑇^* )−1 − 𝑇 * (𝛼𝑛 𝐼 + 𝑇 𝑇 * )−1
𝑋1,𝑖 ⃦
⃦
⃦
𝑛
𝑖=1

∞

≤ 𝑆1,𝑛 + 𝑆2,𝑛 ,
and we can show that
𝑆1,𝑛

𝑆2,𝑛

⃦
⃦
𝑛
⃦ [︁
⃦
]︁ 1 ∑︁
⃦
⃦
𝑋1,𝑖 ⃦
= E ⃦𝑇 * (𝛼𝑛 𝐼 + 𝑇^𝑇^* )−1 − (𝛼𝑛 𝐼 + 𝑇 𝑇 * )−1
⃦
⃦
𝑛
𝑖=1
∞
(︃
)︃
(︂ ⃦
)︂1/4
⃦
4
(︀
)︀
1
1/2
⃦
⃦
=𝑂
E ⃦𝑇^* − 𝑇 * ⃦
E‖𝑋1,𝑖 ‖2
.
3/2 1/2
𝛼𝑛 𝑛
⃦
⃦
𝑛
⃦
⃦
∑︁
1
⃦ ^*
⃦
= E ⃦(𝑇 − 𝑇 * )(𝛼𝑛 𝐼 + 𝑇^𝑇^* )−1
𝑋1,𝑖 ⃦
⃦
⃦
𝑛
𝑖=1
∞
√︂ ⃦
)︂
(︂
⃦
2
1
⃦
⃦
=𝑂
E ⃦𝑇^* − 𝑇 * ⃦
E‖𝑋1,𝑖 ‖2 .
1/2
2,∞
𝛼𝑛 𝑛

For the first symmetrized processes, using the fact that (𝜀𝑖 )𝑛𝑖=1 are i.i.d. Rademacher random
^ 1,𝑖 )𝑛 , we obtain for 𝑗 = 1, 2
variables, independent from (𝑋1,𝑖 , 𝑋
𝑖=1
⃦
⃦2
𝑛
⃦ 1 ∑︁
⃦
^ 1,𝑖 ‖2
^ 1,𝑖 − 𝑋1,𝑖 ‖2
E‖𝑋
2E‖𝑋1,𝑖 ‖2 2E‖𝑋
⃦
^ 1,𝑖 ⃦
E⃦
𝜀𝑖 𝑋
≤
+
⃦ =
⃦𝑛
⃦
𝑛
𝑛
𝑛
𝑖=1

and the result follows from previous computations. Similarly, we obtain result for the second
process.

54

Lemma 4. Suppose that 𝑇 * ∈ ℒ2,∞ is integral operator with continuous kernel function 𝑘, then

‖𝑇 * ‖2,∞ = sup
𝑧∈[𝑎,𝑏]𝑝

(︃∫︁

)︃1/2
|𝑘(𝑧, 𝑤)|2 d𝑤

≡ ‖𝑘‖2,∞ ,

[𝑎,𝑏]𝑞

where ‖𝑘‖2,∞ is a mixed norm on the iterated space 𝐿∞ [𝑎, 𝑏]𝑝 (𝐿2 [𝑎, 𝑏]𝑞 ).
Proof. By Cauchy-Schwartz inequality, we have ‖𝑇 * ‖2,∞ ≤ ‖𝑘‖2,∞ . On the other side, continuity
(︁∫︀
)︁1/2
𝑘(𝑧0 ,𝑤)
of 𝑘 implies that ∃𝑧0 ∈ [𝑎, 𝑏]𝑝 such that ‖𝑘‖2,∞ = [𝑎,𝑏]𝑞 |𝑘(𝑧0 , 𝑤)|2 d𝑤
. Take 𝜓(𝑤) = ‖𝑘(𝑧
.
0 ,.)‖
Then ‖𝜓‖ = 1, and
‖𝑇 * ‖2,∞ ≥ ‖𝑇 * 𝜓‖∞ ≥ |(𝑇 * 𝜓)(𝑧0 )| = ‖𝑘(𝑧0 , .)‖ = ‖𝑘‖2,∞ .

55

