arXiv:1801.09138v1 [] 27 Jan 2018

Cross-Fitting and Fast Remainder Rates for
Semiparametric Estimation
Whitney K. Newey∗
MIT

James M. Robins†
Harvard

January 2018

Abstract
There are many interesting and widely used estimators of a functional with finite semiparametric variance bound that depend on nonparametric estimators of nuisance functions.
We use cross-fitting (i.e. sample splitting) to construct novel estimators with fast remainder rates. We give cross-fit doubly robust estimators that use separate subsamples to
estimate different nuisance functions. We obtain general, precise results for regression
spline estimation of average linear functionals of conditional expectations with a finite
semiparametric variance bound. We show that a cross-fit doubly robust spline regression
estimator of the expected conditional covariance is semiparametric efficient under minimal conditions. Cross-fit doubly robust estimators of other average linear functionals of a
conditional expectation are shown to have the fastest known remainder rates for the Haar
basis or under certain smoothness conditions. Surprisingly, the cross-fit plug-in estimator
also has nearly the fastest known remainder rate, but the remainder converges to zero
slower than the cross-fit doubly robust estimator. As specific examples we consider the expected conditional covariance, mean with randomly missing data, and a weighted average
derivative.

Keywords: Semiparametric estimation, semiparametric efficieny, bias, smoothness.
∗

Department of Economics, MIT, Cambridge, MA 02139, U.S.A E-mail: wnewey@mit.edu.
Harvard School of Public Health, 677 Huntington Ave, Kresge Building Room 823, Boston, MA 02115 U.S.A
E-mail: robins@hsph.harvard.edu.
†

1

1

Introduction

There are many interesting and widely used estimators of a functional with finite semi-parametric
variance bound that depend on the estimation, in a first step, of nuisance functions, such
as conditional expectations or densities. Examples include estimators of the mean with data
missing at random, the average treatment effect, the expected conditional covariance, partially
linear models, and weighted average derivatives. Because the nuisance functions can often be
high dimensional it is desirable to minimize the impact of estimating these functions. By using
cross-fitting (i.e. sample splitting) to estimate the nuisance functions we obtain novel estimators
whose second order remainders converge to zero as fast as known possible. In particular, such
estimators are often root-n consistent under minimal smoothness conditions. Furthermore, such
estimators may have higher order mean square error that converges to zero as fast as known
possible.
Bias reduction is key to constructing semiparametric estimators with fast remainder rates.
The rates at which the variance of remainders goes to zero are quite similar for different semiparametric estimators but the bias rates differ greatly. We use cross-fitting for bias reduction.
We show how fast remainder rates can be attained by using different parts of an i.i.d. sample
to estimate different components of an estimator.
In this paper we consider regression spline estimation of average linear functionals of conditional expectations with a finite semiparametric variance bound, as we have been able to
obtain general, precise results for functionals in this class. The class includes the five examples
mentioned above.
We define a cross fit (CR) plug-in estimator to be one where we estimate the functional
by simply replacing the unknown conditional expectation by a nonparametric estimator from a
separate part of the sample. Cross-fitting eliminates an ”own observation” bias term, thereby
decreasing the size of the remainder. Functionals in our class have doubly robust influence functions that depend on two unknown functions. This implies there exists an estimator depending
on both unknown functions that has exact bias zero if the unknown functions are replaced by
fixed functions, at least one of which is equal to the truth. Here we use double cross-fitting
where the two unknown functions are themselves estimated from separate subsamples, so that
the final estimator depends on three separate subsamples. Surprisingly, single cross fitting in
which both unknown functions are estimated from the same subsample has a remainder that
can converge even slower than CF plug-in estimators. In contrast, doubly robust estimators
with double cross fitting improve on cross-fit plug-in estimators in the sense that remainder
terms can converge at faster rates. We also show how multiple cross-fitting could be used to
reduce bias for any semiparametric estimator that is a polynomial in first step spline estmators
of unknown functions.
We construct cross-fit (CF) plug-in and doubly cross-fit doubly robust (DCDR) estimators
2

that are semiparametrically efficient under minimal conditions when the nuisance functions
are in a Holder class of order less than or equal to one. When a nuisance function is Holder
of order exceeding one, we propose DCDR estimators that have remainders that converge no
slower and often faster than the CF plug-in estimator. In the special case of the expected
conditional covariance functional, the DCDR estimator is always semiparametric efficient under
minimal conditions. For other functionals in our class the CF plug-in and DCDR estimator
are semiparametric efficient under minimal conditions, provided the conditional expectation is
Holder of order greater than or equal to one-half the regressor dimension; furthermore, in this
case, the remainder goes to zero as fast as known possible for both CF plug-in and DCDR
estimators. When the conditional expectation is Holder of order less than or equal to one-half
the regressor dimension but greater than or equal to one, the remainder for the DCDR has a
remainder that converges faster than the CF plug-in estimator.
In the case where the conditional expectation is Holder of order no less than one but less than
one-half the regressor dimension, we show semiparametric efficiency under minimal conditions
for the expected conditional covariance, but not for other functionals. The higher order influence
function (HOIF) estimators of Robins et al. (2008, 2017) and Mukherjee, Newey, and Robins
(2017) will be semiparametric efficient under minimal conditions for these other functionals,
including the mean with data missing at random and the average treatment effect.
CF plug-in estimators have been considered by Bickel (1982) in the context of adaptive
semiparametric efficient estimation, Powell, Stock, and Stoker (1989) for density weighted average derivatives, and by many others. Kernel and series CF plug-in estimators of the integrated
squared density and certain other functionals of a density have been shown to be semiparametric
efficient under minimal conditions by Bickel and Ritov (1988), Laurent (1996), Newey, Hsieh,
and Robins (2004), and Gine and Nickl (2008). Our DCDR estimator appears to be novel as
does the fact that a CF plug-in estimator can be semiparametric efficient under minimal conditions. Ayyagari (2010), Robins et al. (2013), Kandasamy et. al. (2015), Firpo and Rothe
(2016), and Chernozhukov et al.(2017) have considered doubly robust estimators that eliminate
own observation terms. Double cross-fitting in double robust estimation appears not to have
been analyzed before.
Our results for splines make use of the Rudelson (1999) law of large numbers for matrices
similarly to Belloni et al.(2015). The results for the CF plug-in estimator for general splines
extend those of Ichimura and Newey (2017) to sample averages of functionals. The double
robustness of the influence function for the functionals we consider is shown in Chernozhukov
et al.(2016), where the doubly robust estimators of Scharfstein, Rotnitzky, and Robins (1999),
Robins, Rotnitzky, and van der Laan (2000), Robins et. al. (2008), and Firpo and Rothe (2016)
are extended to a wide class of average linear functionals of expectations.
The DCDR estimator for the mean with missing data and average treatment effect uses

3

a spline approximation to the reciprocal of the propensity score rather than the reciprocal of
a propensity score estimator. The reciprocal of a propensity score estimator has been used
in much of the previous literature on plug in and doubly robust estimation, including Robins
and Rotnitzky (1995), Rotnitzky and Robins (1995), Hahn (1998), and Hirano, Imbens, and
Ridder (2003). Estimators based on approximating the reciprocal of the propensity score have
been considered by Robins et al. (2007), Athey, Imbens, and Wager (2017), and recently in
independent work by Hirschberg and Wager (2017).
Other approaches to bias reduction for semiparametric estimators have been proposed.
Robins et al.(2008, 2017) and Mukherjee, Newey, and Robins (2017) develop higher order influence function (HOIF) estimators with smaller bias. In Section 2 we will discuss the relationship
of this paper to HOIF. Cattaneo and Jansson (2017) propose promising bootstrap confidence
intervals for plug-in kernel estimators that include bias corrections. Also, Cattaneo, Jansson,
and Ma (2017) show that the jackknife can be used to reduce bias of plug-in series estimators.
For the class of functionals in this paper cross-fitting removes bias so that there is no need for
bootstrap or jackknife bias corrections in order to attain the fastest remainder rates.
In Section 2 we will describe the cross-fitting approach to bias reduction and show how it
relates to HOIF. Section 3 describes the linear functionals and regression spline estimators we
consider. Sections 4, 5, and 6 give results for the CF plug-in estimator, the DCDR expected
conditional covariance estimator, and DCDR estimators of other linear functionals, respectively.
Before explaining the results of this paper it is helpful to be more specific about our goal. We
will consider i.i.d. data z1 , ..., zn . We are interested in an asymptotically linear semiparametric
estimator β̂ satisfying
n

√ 
1 X
n β̂ − β0 = √
ψ (zi ) + Op (∆n ) , ∆n −→ 0,
n i=1

(1.1)

where ψ (z) is the influence function of β̂ and ∆n characterizes the size of the remainder. Our
goal is to find estimators where ∆n converges to zero at the fastest known rate.
For the integrated squared density, Bickel and Ritov (1988) gave a kernel based estimator
where the rate for ∆n is fast enough that β̂ is semiparametric efficient under minimal conditions.
To motivate our candidate for the optimal rate the remainder can converge to zero for series
estimators of an average linear functional of a conditional expectation with positive information
bound, we consider the series estimator of the coefficients of a partially linear regression in
Donald and Newey (1994). The model there is E[yi |ai , xi ] = aTi β0 + λ0 (xi ) where λ0 (xi ) is an
unknown function of an r × 1 vector xi . Consider the estimator β̂ obtained from regressing yi
on ai and a K × 1 vector p(xi ) of power series or regression splines in an i.i.d. sample of size n.
Assume that the functions λ0 (x) and α0 (x) = E[ai |xi = x] are each members of a Holder class

4

of order sλ and sα respectively. Define
∆∗n

=

√

nK

−(sγ +sα )/r

+K

−sγ /r

+K

−sα /r

+

r

K
.
n

Donald and Newey (1994) showed that under regularity conditions, including K/n −→ 0, equa√
tion (1.1) is satisfied with ∆n = ∆∗n . Here nK −(sγ +sα )/r gives the rate at which the bias of
√
n(β̂ − β0 ) goes to zero. Also, K −sγ /r and K −sα /r are stochastic equicontinuity bias terms, and
p
K/n. that accounts for stochastic equicontinuity and degenerate U-statistic variance terms.
Furthermore, there exists K = Kn satisfying Kn /n −→ 0 such that ∆∗n −→ 0 if and only if
sγ + sα > r/2. However the Donald and Newey (1994) result used the fact that the partially
linear model implies yi − aTi β0 is mean independent of ai given xi and thus is not a locally
nonparametric model. A model is said to be locally nonparametric if, at each law P in the
model, the tangent space is all of L2 (P ) .Henceforth in this paper, we shall always assume a
locally nonparametric model.
Robins et al. (2009) showed that the condition sγ + sα > r/2 is necessary and sufficient for
the existence of a semparametric efficient estimator of
ξ0 = E [cov (ai , yi |xi )] /E [var(ai |xi )] ,
Note ξ0 is the probability limit of the Donald and Newey (1994) estimator regardless of whether
the partially linear model holds. That is, ξ0 is the coefficient b in the population linear projection
of yi on all functions of the form ai b + λ(xi ). Robins et al. (2008) proved sufficiency using a
higher order influence function estimator of ξ0 , which is a U-statistic whose order increases as
ln (n) . In contrast, the aforementioned estimator of Donald and Newey (1994), although much
simpler, is not semiparametric efficient for ξ0 in a locally nonparametric model under the minimal
condition sγ + sα > r/2. The current paper was thus motivated by the question whether one
could construct a simple efficient estimator of ξ0 whose remainder ∆n will go to zero as fast as
∆∗n , the fastest rate known to be possible. In summary, our goal is to construct estimators that
are much simpler than the HOIF estimators and yet satisfy equation (1.1) with ∆n = ∆∗n .

2

Cross-Fitting and Fast Remainder Rates

To explain how cross-fitting can help achieve fast remainder rates we consider estimation of the
expected conditional covariance
β0 = E[Cov(ai , yi |xi )] = E[ai {yi − γ0 (xi )}],
where γ0 (xi ) = E[yi |xi ]. This object is useful in the estimation of weighted average treatment
effects as further explained below. We assume that the functions γ0 (x) and α0 (x) = E[ai |xi = x]
are each members of a Holder class of order sγ and sα respectively.
5

One way to construct an estimator of β0 is the “plug-in” method where a nonparametric
estimator γ̂ is substituted for γ0 and a sample average for the expectation to form
n

1X
ai {yi − γ̂(xi )}.
β̄ =
n i=1

√
This estimator generally suffers from an ”own observation” bias that is of order K/ n when
γ̂ is a spline regression estimator, which converges to zero slower than ∆∗n . This bias can be
eliminated by replacing γ̂(x) with an estimator γ̂−i (x) that does not use zi in its construction.
The resulting estimator of β0 is
n

1X
ai {yi − γ̂−i (xi )}.
β̂ =
n i=1
This estimator is a cross-fit (CF) plug-in estimator in the sense that γ̂−i uses a subsample that
does not include i. The cross-fitting eliminates the own observation bias. The remainder rate
∆n for β̂ will be often be faster than for β̄, sometimes as fast as ∆∗n as explained below. This
approach to eliminating own observation bias when the first step is a density estimator has
been used by Bickel (1982), Bickel and Ritov (1988), Powell, Stock, and Stoker (1989), Laurent
(1996), and others. Here we obtain the novel result that, for a spline regression first step, a CF
plug-in estimator can have the fastest rate ∆∗n even when the usual plug-in estimator does not.
Doubly robust estimators have another source of bias that can also be eliminated by double
cross-fitting. To explain we consider a single cross-fit doubly robust estimator of the expected
conditional covariance. Let γ̂−i (x) and α̂−i (x) be nonparametric estimators of γ0 (xi ) = E[yi |xi ]
and α0 (xi ) = E[ai |xi ] that do not depend on the ith observation. Consider the estimator
n

1X
β̌ =
[ai − α̂−i (xi )][yi − γ̂−i (xi )].
n i=1
This estimator is doubly robust in the sense of Scharfstein, Rotnitzky, and Robins (1999) and
Robins, Rotnitzky, and van der Laan (2000), being consistent if either α̂−i or γ̂−i are consistent.
It uses cross-fitting to eliminate own observation bias. This estimator does have a nonlinearity
bias since α̂−i (xi ) and γ̂−i (xi ) are constructed from the same data in single crossfitting. That bias
√
is of the same order K/ n as the own observation bias for a spline regression plug-in estimator.
This bias can be thought of as arising from nonlinearity of β̌ in the two nonparametric estimators
α̂−i (xi ) and γ̂−i (xi ).
One can remove the nonlinearity bias in the doubly robust estimator by using different parts
of the data to construct the two nonparametric estimators. Let γ̂−i (xi ) be constructed from a
subset of the observations that does not include observation i and let α̃−i (xi ) be constructed
from a subset of the observations that does not include i or any observations used to form γ̂−i .
6

A doubly cross-fit doubly robust estimator (DCDR) is
n

1X
β̃ =
[ai − α̃−i (xi )][yi − γ̂−i (xi )].
n i=1
This estimator uses cross-fitting to remove both the own observation and the nonlinearity biases.
We will show that ∆∗n = ∆n when α̃−i (xi ) and γ̂−i (xi ) are spline regression estimators for
a K × 1 vector of multivariate splines of at least order max{sγ , sa } − 1 with evenly spaced
knots. Consequently, this estimator will be root-n consistent and semiparametric efficient when
sγ + sα > r/2 and K is chosen appropriately, which is the minimal condition of Robins et al.
(2009).
Remarkably, the doubly robust estimator β̌ where α̂−i (xi ) and γ̂−i (xi ) use the same data
may have a slower remainder rate than the CF plug-in estimator β̂. The use of the same data
√
for α̂−i (xi ) and γ̂−i (xi ) introduces a bias term of size K/ n. Such a term is not present in the
√
CF plug-in estimator. The K/ n term is eliminated for the doubly robust estimator by forming
α̃−i (xi ) and γ̂−i (xi ) from different samples. We find that the DCDR estimator β̃ improves on
the CF plug in estimator by increasing the rate at which a certain part of ∆n goes to zero.
Specifics will be given below.
We note that the own observation bias can also be thought of as nonlinearity bias. The
parameter β0 has the form
Z
β0 =

a{y − γ0 (x)}F0 (dz),

where F0 denotes the distribution of z = (y, a, x). This object is quadratic in γ0 and F0 jointly.
The own observation bias can be thought of as a quadratic bias resulting from using all the data
to simultaneously estimate γ0 and the distribution F0 of a single observation. The CF plug in
estimator β̂ eliminates this nonlinearity bias. Also, the doubly robust estimator can be thought
R
of as estimating [a − α0 (x)][y − γ0 (x)]F0 (dz), which is cubic in α0 , γ0 , and F0 jointly. The
DCDR estimator can be thought of as eliminating the cubic bias by estimating each of α0 (x),
γ0 (x), and F0 from distinct groups of observations.
One potential concern about DCDR estimators is that each of the nonparametric components
γ̂ and α̃ only use a fraction of the data because they are each based on subsamples that the
other does not use. For example, they only use less than half the data if they are based on
approximately the same subsample size. This does not affect remainder rates but could affect
small sample efficiency. One might be able to improve small sample efficiency by averaging over
DCDR estimators that use different sample splits to construct γ̂ and α̃, though that is beyond
the scope of this paper. Our concern in this paper is remainder rates for asymptotically efficient
estimation.
Cross-fitting can be applied to eliminate bias terms for any estimator that depends on powers of nonparametric estimators. Such cross-fitting would replace each power by a product of
7

nonparametric estimators that are computed from distinct subsamples of the data, analogously
to the DCDR estimators above.
We now provide a more quantitative version of our results. Let p(x) be a vector of multivariate regression splines of dimension K with evenly spaced knots. We will always take K = Kn
to satisfy K ln (K) /n → 0. Suppose that γ̂−i (x) = p(x)T [Σj∈Iℓ p(xj )p(xj )T ]−1 Σj∈Iℓ p(xj )yj is a
series estimator from regressing yj on p(xj ) in a subsample of observations indexed by Iℓ , where
/ Iℓ , L is fixed and the number of elements of each Iℓ is of
{Iℓ }Lℓ=1 is a partition of {1, ..., n}, i ∈
order n. Suppose that for the doubly robust estimator α̃(xi ) is constructed analogously from a
separate subsample.
When sγ ≤ 1 and sα ≤ 1 and p(x) is a Haar basis of dummy variables that are indicator
functions of cubes partitioning the support of xi we show that the CF plug-in estimator has
∆n = ∆∗n +ln(n)K −sγ /r and the DCDR doubly robust estimator has ∆n = ∆∗n . Hence the DCDR
estimator has the fast remainder rate. Further the CF plug-in estimator has the fast remainder
∆∗n , except at those laws where K −sγ /r is the dominating term in ∆∗n . At such laws, the DCDR
estimator improves on the CF plug-in but only by a factor of ln(n). We also show that these
results extend to the entire class of average linear functionals of a conditional expectation with
finite semiparametric variance bound.
When sγ and sα are any positive numbers and p(x) is a spline basis of order at least
max{sγ , sα } − 1 we show that the CF plug in estimator of the expected conditional covarip
ance has ∆n = ∆∗n + K ln(K)/nK 1/2−sγ /r and the DCDR estimator has ∆n = ∆∗n .Here the
plug-in estimator has the fast remainder ∆n = ∆∗n for sγ > r/2 and the doubly robust estimator
has ∆n = ∆∗n for all sγ . For other functionals in our class we show that the DCDR estimator has
p
∆n = ∆∗n + K 3 ln(K)2 /n3 K 1/2−sγ /r , which has ∆n = ∆∗n when [K ln(K)/n]K 1/2−sγ /r −→ 0 .
Thus the DCDR estimator has remainder that can converge to zero at a faster rate that of the
CF plug-in estimator.
We note that the source of the term in ∆n that is added to ∆∗n in each case can be attributed
to estimators of the second moment matrix Σ = E[p(xi )p(xi )T ] of the regression splines. If each
Σ̂ℓ were replaced by Σ in the estimators then the resulting objects would all have ∆n = ∆∗n .
For brevity, we demonstrate this only for plug-in estimator. Consider the plug-in object β̇
P
having the same formula as β̂ except that γ̂−i (x) is replaced by γ̇−i (x) = p(x)T Σ−1 j∈Iℓ p(xi )yi /nℓ .
Let ᾱ(x) = p(x)T Σ−1 E[p(xi )α0 (xi )]. Standard approximation properties of splines give the
approximation rates {E[{γ0 (xi ) − γ̄(xi )}2 ]}1/2 = O(K −sγ /r ) and {E[{α0 (xi ) − ᾱ(xi )}2 ]}1/2 =
O(K −sα/r ). By the Cauchy-Schwartz inequality
√

√

n{E[{α0 (xi ) − ᾱ(xi )}2 ]}1/2 {E[{γ0 (xi ) − γ̄(xi )}2 ]}1/2
√
= O( nK −(sγ +sα )/r ).

nE[{α0 (xi ) − ᾱ(xi )}{γ0 (xi ) − γ̄(xi )}] ≤

Note also that E[γ̇−i (x)] = γ̄(x) = p(x)T Σ−1 E[p(xi )γ0 (xi )]. Then the root-n normalized bias of
8

β̇ is
h√ 
i √ Z
E
n β̇ − β0 = n a{y − E [γ̇−i (x)]}F0 (dz) − E[ai {yi − γ0 (xi )}]
√
√
(2.1)
= nE[ai {γ0 (xi ) − γ̄(xi )}] = nE[α0 (xi ){γ0(xi ) − γ̄(xi )}]
√ −(sγ +sα)/r
√
),
= nE[{α0 (xi ) − ᾱ(xi )}{γ0(xi ) − γ̄(xi )}] = O( nK
which has our desired ∆∗n rate. Also, there will be stochastic equicontinuity bias terms of order
K −sγ /r and K −sα/r and stochastic equicontinuity variance and degenerate U-statistic variance
p
terms of order K/n. Overall the remainder for β̇ will satisfy ∆n = ∆∗n . Thus, a CF plug-in
object β̇ where Σ replaces each Σ̂ℓ will have the fast remainder rate.
We note that the bias in equation (2.1) depends on the product K −(sγ +sα)/r of the approximation rate K −sγ /r for γ0 (x) and the approximation rate K −sα /r for α0 (x), rather than just the
bias rate K −sγ /r for the nonparametric estimator being plugged-in. This product form results
from the fact that the parameter of interest β0 has a finite semiparametric variance bound. The
product bias form in equation (2.1) for plug-in series estimators was shown in Newey (1994).
It is interesting to compare our estimators with HOIF estimators. We continue to focus
on the average conditional covariance. The HOIF estimator of that β0 can depend on initial
estimators γ̂(x) and α̂(x) of γ0 (x) and α0 (x) obtained from a training subsample. For a vector
of spline regressors p(x) let Σ̂ be the sample second moment matrix of p(x) from the training
sample. Let B̂(x) = Σ̂−1 [p(x)p(x)T − Σ̂] and
n

X
1
1X
[ai − α̂(xi )]p(xi )T Σ̂−1 p(xj )[yj − γ̂(xj )]
[ai − α̂(xi )][yi − γ̂(xi )] −
n i=1
n(n − 1) i6=j


Q
q+1
X
X
X
(−1) (n − 2 − q)!
+
Πqr=1 B̂(xℓr ) Σ̂−1 p(xj )[yj − γ̂(xj )],
[ai − α̂(xi )]p(xi )T 
n!
q=1

β̂H =

i6=j

ℓ1 6=···6=ℓq 6=i6=j

where all the sums are over an estimation subsample that does not overlap with the training
sample. This β̂H is the empirical HOIF estimator of Mukherjee, Newey, and Robins (2017) of
√
order Q + 2. By Theorem 3 of Mukherjee, Newey, and Robins (2017) the bias of n(β̂H − β0 )
conditional on the training sample has order
Q/2
(Q−1)/2


√
K ln(K)
K ln(K)
n kα̂ − α0 k2 kγ̂ − γ0 k2
= kα̂ − α0 k2 kγ̂ − γ0 k2 K ln(K)
.
n
n
p
where kδk2 = {E[δ(xi )2 ]}1/2 . The order of this bias will be smaller than K/n as long as
K grows no faster than n1−ε for some ε > 0, although that is not needed for semiparametric
p
efficiency. As shown in Mukherjee, Newey, and Robins (2017), if Q grows like ln(n), K like
n/ ln(n)3 , and other regularity conditions are satisfied then β̂H will be semiparametric efficient
under the minimal condition sγ + sα > r/2 of Robins et al.(2009).
9

We can explain the different properties of HOIF and series estimators by comparing the CF
plug-in estimator with the HOIF when the training sample estimators γ̂ and α̂ are set equal to
zero. In that case the HOIF estimator is
n

X
1X
1
β̂H =
ai yi −
ai p(xi )T Σ̂−1 p(xj )yj
n i=1
n(n − 1) i6=j


Q
q+1
X
X (−1) (n − 2 − q)! X
Πqr=1 B̂(xℓr ) Σ̂−1 p(xj )yj .
ai p(xi )T 
+
n!
q=1
ℓ 6=···6=ℓ 6=i6=j
i6=j
1

q

P
Consider γ̌−i (x) = p(x)T Σ̂−1 j6=i p(xj )yj /(n − 1). This is an estimator of γ0 (x) that is like a
series estimator except the inverse second moment matrix Σ̂−1 comes from the training sample
P
and the cross-moments j6=i p(xj )yj /(n−1) from the estimation subsample. The first two terms
of the HOIF estimator can then be written as
n

1X
β̌ =
ai [yi − γ̌−i (xi )].
n i=1
Let T denote the training sample. Then we have
E[β̌ − β0 |T ] = E[α0 (xi ){γ0 (xi ) − γ̌−i (xi )}] = E[α0 (xi )γ0 (xi )] − E[α0 (xi )p(xi )T ]Σ̂−1 E[p(xi )γ0 (xi )]
= E[α0 (xi )γ0 (xi ) − ᾱ(xi )γ̄(xi )] + E[α0 (xi )p(xi )T ](Σ−1 − Σ̂−1 )E[p(xi )γ0 (xi )]

= O(K −(sγ +sα )/r ) + Λ(Σ̂, Σ), Λ(Σ̂, Σ) = E[α0 (xi )p(xi )T ](Σ−1 − Σ̂−1 )E[p(xi )γ0 (xi )].

Thus the bias of β̌ is the sum of the approximation bias K −(sγ +sα)/r and Λ(Σ̂, Σ). The rest of
the HOIF estimator, i.e. β̂H − β̌, can be thought of as a bias correction for Λ(Σ̂, Σ). Note that
E[β̂H − β̌|T ] =

Q
X
(−1)q+1 (n − 2 − q)!
q=1

n!

h
iq
−1
E[α0 (xi )p(xi )] Σ̂ (Σ − Σ̂) Σ̂−1 E[p(xi )γ0 (xi )].
T

Here we see that E[β̂H − β̌|T ] is the negative of a Taylor expansion to order Q of Λ(Σ̂, Σ) in Σ̂
around Σ. Therefore, it will follow that

Q/2
Q
K ln(K)
−(sγ +sα )/r
−(sγ +sα )/r
) + O(
E[β̂H − β0 |T ] = O(K
) + O( Σ̂ − Σ ) = O(K
),
op
n
where k·kop is the operator norm for a matrix and the second equality follows by the Rudelson
(1999) matrix law of large numbers. This equation is similar to the conclusion of Theorem 3 of
Mukherjee, Newey, and Robins (2017).
In comparison with the HOIF estimator the CF plug-in series estimator has a remainder rate
p
from estimating Σ that is ln(n)K −sγ /r for sγ , sα ≤ 1 and Haar splines and K ln(K)/nK 1/2−sγ /r
more generally, without any higher order U-statistic correction for the presence of Σ̂−1 . The
10

DCDR estimator has ∆n = ∆∗n , also without the need to rely on any higher-order U-statistics.
The key difference between the HOIF and these other estimators is that the plug-in and doubly
robust estimators use spline regression in their construction and the HOIF estimator uses Σ̂−1
from a training subsample.
Previously the HOIF estimator was the only known method of obtaining an semiparametric
efficient estimator of the expected conditional covariance under the minimal conditions of Robins
et al.(2009). We find here that the CF plug-in estimator with a Haar basis can do this for
sγ , sα ≤ 1 and for a general spline basis with sγ ≥ r/2. We also find that the DCDR estimator
can do this for all sγ and sα . These estimators are simpler than the HOIF estimator in not
requiring the higher order U-statistic terms. It would be interesting to compare the size of
constants in respective remainder terms where HOIF could have an advantage by virtue of its
higher order influence function interpretation. That comparison is beyond the scope of this
paper.
The HOIF estimator remains the only known estimator that is semiparametric efficient under
the Robins et al.(2009) minimal conditions for the mean with missing data over all sγ and sα . We
expect that property of HOIF to extend to all the linear average functionals we are considering
in this paper.
In summary, cross-fitting can be used to reduce bias of estimators and obtain faster remainder
rates. If cross fitting is not used for either the plug-in or the doubly robust estimator there would
√
be an additional K/ n bias term in the remainder. This extra term can increase the bias of the
estimator significantly for large K. It is well known to be very important in some settings, such
as instrumental variables estimation as shown by Blomquist and Dahlberg (1999) and Imbens,
Angrist, and Krueger (1999). Also, its presence prevents the plug-in estimator from attaining
root-n consistency under minimal conditions. Cross-fitting eliminates this large remainder for
the linear functionals we consider and results in plug-in and doubly robust estimators with
remainders that converge to zero as fast as known possible for sγ , sα ≤ 1, for sγ > r/2, and for
any sα and sγ for a doubly robust estimator of the expected conditional covariance.

3

Estimators of Average Linear Functionals

We will analyze estimators of functionals of a conditional expectation
γ0 (x) = E[yi |xi = x],
where yi is a scalar component and xi a subvector of zi . Let γ represent a possible conditional
expectation function and m(z, γ) denote a function of γ and a possible realization z of a data
observation. We consider
β0 = E [m(zi , γ0 )] ,
11

where m(z, γ) is an affine functional of γ for every z, meaning m(z, γ) − m(z, 0) is linear in γ.
There are many important examples of such an object. One of these is the expected conditional covariance we consider in Section 2. There m(z, γ) = a[y − γ(x)]. This object shows up
in different forms in the numerator and denominator of
ξ0 =

E[Cov(ai , yi |xi )]
.
E[V ar(ai |xi )]

Here δ0 is the coefficient of ai in the population least squares projection of yi on functions of
the form ai δ + g(xi ). Under an ignorability assumption this object δ0 can be interpreted as
a weighted average of conditional average treatment effects when ai is a binary indicator for
treatment and xi are covariates.
Another important example is the mean when data are missing at random. The object
of interest is β0 = E[Yi ] where Yi is a latent variable that is not always observed. Let ai be
an observed binary indicator where ai = 1 if Yi is observed. Suppose that there are observed
covariates wi such that Yi is mean independent of ai conditional on wi , i.e. E[Yi |ai = 1, wi ] =
E[Yi |wi ]. Then for the observed variable yi = ai Yi we have
E[E[yi |ai = 1, wi ]] = E[E[Yi |ai = 1, wi ]] = E[E[Yi |wi ]] = β0 .
Let x = (a, w) and γ0 (xi ) = E[yi |xi ]. Then for m(z, γ) = γ(1, w) we have β0 = E[m(zi , γ0)].
A third example is a weighted average derivative, where the object of interest is
Z
β0 = v(x) [∂γ0 (x)/∂x1 ] dx,
R
for some weight function v(x), with x1 continuously distributed and v(x)dx = 1. This object
is proportional to β10 in a conditional mean index model where E[yi |xi ] = τ (xTi β0 ) for some
unknown function τ (·), as in Stoker (1986). This object is included in the framework of this paper
R
for m(z, γ) = v(x) [∂γ(x)/∂x1 ] dx. Assuming that v(x) is zero at the boundary, integration by
parts gives
Z
m(z, γ) = m(γ) =

ω(x)γ(x)dx, ω(x) = −∂v(x)/∂x1 .

Throughout we will focus on the case where estimators of β0 have a finite semiparametric variance bound and so should be root-n consistently estimable under sufficient regularity
conditions. As discussed in Newey (1994), this corresponds to E[m (zi , γ)] being mean square
continuous as a function of γ, so that by the Riesz representation theorem the following condition
is satisfied:
Assumption 1: There is α0 (x) with E[α0 (xi )2 ] < ∞ and for all γ with E[γ(xi )2 ] < ∞,
E [m (zi , γ) − m(zi , 0)] = E [α0 (xi ) γ (xi )] .
12

(3.1)

The function α0 (x) has an important role in the asymptotic theory. The bias in a series
estimator of β0 will depend on the expected product of biases in approximating γ0 (x) and
α0 (x). Consequently there will be a trade-off in conditions that can be imposed on γ0 (x) and
α0 (x) so that the estimators of β0 have good properties.
To help explain this condition we give the form of α0 (x) in each of the examples. In the
expected conditional covariance example iterated expectations gives
E [m (zi , γ) − m(zi , 0)] = −E[ai γ(xi )] = −E[E[ai |xi ]γ(xi )] = E[α0 (xi )γ(xi )],

(3.2)

α0 (xi ) = −E[ai |xi ].

In the missing data example, for the propensity score Pr(ai = 1|wi ) = π0 (wi ), iterated expectations gives
π0 (wi )
ai
γ(1, wi )] = E[
γ(1, wi )]
π0 (wi )
π0 (wi )
ai
ai
γ(xi )] = E[α0 (xi )γ(xi )], α0 (xi ) =
.
= E[
π0 (wi )
π0 (wi )

E [m (zi , γ) − m(zi , 0)] = E[γ(1, wi )] = E[

In the average derivative example, multiplying and dividing by the pdf f0 (x) of xi gives
Z
Z
ω(xi )
ω(x)
γ(x)f0 (x)dx = E[
γ(xi )]
E [m (zi , γ) − m(zi , 0)] = ω(x)γ(x)dx =
f0 (x)
f0 (xi )
ω(xi )
= E[α0 (xi )γ(xi )], α0 (xi ) =
.
f0 (xi )

(3.3)

(3.4)

Our estimators of β0 will be based on a nonparametric estimator γ̂ of γ0 and possibly on a
nonparametric estimator α̃ of α0 . The CF plug-in estimator is given by
L

1 XX
m(zi , γ̂ℓ ),
β̂ =
n
i∈I
ℓ=1

ℓ

where Iℓ , (ℓ = 1, ..., L) is a partition of the observation index set {1, ..., n} into L distinct subsets
of about equal size and γ̂ℓ only uses observations not in Iℓ . We will consider a fixed number of
groups L in the asymptotics. It would be interesting to consider results where the number of
groups grows with the sample size, even ”leave one out” estimators where Iℓ only includes one
observation, but theory for those estimators is more challenging and we leave it to future work.
The DCDR estimator makes use of α̃ℓ that may be constructed from different observations
than γ̂ℓ . The doubly robust estimator is
L

1 XX
{m(zi , γ̂ℓ ) + α̃ℓ (xi )[yi − γ̂ℓ (xi )]} .
β̃ =
n ℓ=1 i∈I
ℓ

This estimator has the form of a plug-in estimator plus the sample average of α̃ℓ (xi )[yi − γ̂ℓ (xi )],
R
which is an estimator of the influence function of m(z, γ̂ℓ )F0 (dz). The addition of α̃ℓ (xi )[yi −
13

γ̂ℓ (xi )] will mean that the nonparametric estimators γ̂ℓ and α̃ℓ do not affect the asymptotic
distribution of β̃, i.e. the limit distribution would be the same if γ̂ℓ and α̃ℓ were replaced by
their true values and ∆n −→ 0. This estimator allows for full cross-fitting where α̃ and γ̂ may
be based on distinct subsamples.
The cross-fit estimator β̃ is doubly robust in the sense that β̃ will be consistent as long as
either γ̂ℓ or α̃ℓ is consistent, as shown by Chernozhukov et al.(2016) for this general class of
functionals. When γ̂(x) is a series estimator like that described above the CF plug-in estimator
β̂ is also doubly robust in a more limited sense. It will be consistent with fixed p(x) if either
γ0 (x) or α0 (x) is a linear combination of p(x), as shown for the mean with missing data in
Robins et al.(2007) and in Chernozhukov et al.(2016) for the general linear function case we are
considering.
Throughout the paper we assume that each data point zi is used for estimation for some
group ℓ and that the number of observations in group ℓ, the number used to form γ̂ℓ , and the
number used to form α̃ℓ grow at the same rate as the sample size. To make this condition precise
let n̄ℓ be the number of elements in Iℓ , n̂ℓ be the number used to form γ̂ℓ , and ñℓ be the number
of observations used to form α̃ℓ . We will assume throughout that all the observations are used
for each ℓ, i.e. that either n̄ℓ + n̂ℓ = n or n̄ℓ + n̂ℓ + ñℓ = n if different observations are used for
γ̂ℓ and α̃ℓ .
Assumption 2: There is a constant C > 0 such that either n̄ℓ + n̂ℓ = n and minℓ {n̄ℓ , n̂ℓ } ≥
Cn or n̄ℓ + n̂ℓ + ñℓ = n and minℓ {n̄ℓ , n̂ℓ , ñℓ } ≥ Cn. For the plug-in estimator groups are as close
as possible to being of equal size.
The assumption that the group sizes are as close to equal as possible for the plug-in estimator
is made for simplicity but could be relaxed.
We turn now to conditions for the regression spline estimators of γ0 (x) and α0 (x). We
continue to consider regression spline first steps where p(x) is a K × 1 vector of regression
splines. The nonparametric estimator of γ0 (x) will be a series regression estimator where
γ̂ℓ (x) = p(x)T δ̂ℓ , δ̂ℓ = Σ̂−
ℓ ĥℓ , Σ̂ℓ =

1 X
1 X
p(xi )p(xi )T , ĥℓ =
p(xi )yi ,
n̂ℓ ˆ
n̂ℓ ˆ
i∈Iℓ

i∈Iℓ

where a T superscript denotes the transpose, Iˆℓ is the index set for observations used to construct
γ̂ℓ (x), and A− denotes any generalized inverse of a positive semi-definite matrix A. Under
−1
conditions given below Σ̂ℓ will be nonsingular with probability approaching one so that Σ̂−
ℓ = Σ̂ℓ
for each ℓ.
The DCDR estimator β̃ uses an estimator of α0 (x). The function α0 (x) cannot generally
be interpreted as a conditional expectation and so cannot generally be estimated by a linear
regression. Instead we use Assumption 1 and equation (3.1) to construct an estimator. Let
14

v(z) = (m(z, p1 ) − m(z, 0), ..., m(z, pK ) − m(z, 0))T . Then by Assumption 1,
E[v(zi )] = E[p(xi )α0 (xi )],
so that h̃ℓα =
α0 (x) is then

P

i∈I˜ℓ

v(zi )/ñℓ is an unbiased estimator of E[p(xi )α0 (xi )]. A series estimator of

α̃ℓ (x) = p(x)T δ̃ℓα , δ̃ℓα = Σ̃−
ℓ h̃ℓα , Σ̃ℓ =

1 X
p(xi )p(xi )T .
ñℓ ˜
i∈Iℓ

Here δ̃ℓα is an estimator of the coefficients of the population regression of α0 (x) on p(x), but δ̃ℓα
is not obtained from a linear regression. This type of estimator of α0 (x) was used to construct
standard errors for functionals of series estimators in Newey (1994).
Now that we have specified the form of the estimators γ̂ℓ and α̃ℓ we can give a complete
description of the estimators in each of the examples. For the expected conditional covariance
recall that m(z, γ) = a[y − γ(x)]. Therefore the CF plug-in estimator will be
L

1 XX
ai [yi − γ̂ℓ (xi )].
β̂ =
n
i∈I
ℓ=1

(3.5)

ℓ

Also, as discussed above, for the expected conditional covariance α0 (x) = −E[ai |xi = x] and
P
v(zi ) = −ai p(xi ), so that α̃ℓ (x) = −γ̃aℓ (x) where γ̃aℓ (x) = p(x)T Σ̃−
ℓ
i∈I˜ℓ p(xi )ai /ñℓ is the
˜
regression of ai on p(xi ) for the observations indexed by Iℓ . Then the DCDR estimator is
L

β̃ =

L

1 XX
1 XX
[ai + α̃ℓ (xi )][yi − γ̂ℓ (xi )] =
{ai − Ẽ[ai |xi ]}[yi − γ̂ℓ (xi )],
n ℓ=1 i∈I
n ℓ=1 i∈I
ℓ

(3.6)

ℓ

where Ẽ[ai |xi ] = −α̃ℓ (xi ) is the predicted value from the regression of ai on p(xi ). This estimator
is the average of the product of two nonparametric regression residuals, where the average and
each of the nonparametric estimators can be constructed from different samples.
For the missing data example the estimators are based on series estimation of E[yi |ai = 1, wi ].
Let q(w) denote a K ×1 vector of splines, x = (a, w T )T , and p(x) = (aq(w)T , (1−a)q(w)T )T . The
predicted value γ̂(1, w) will be the same as from a linear regression of yi on q(wi ) for observations
with ai = 1. That is, γ̂(1, w) = q(w)T δ̂ℓ where
δ̂ℓ = Σ̂−
ℓ ĥℓ , Σ̂ℓ =

1 X
1 X
ai q(wi )q(wi )T , ĥℓ =
ai q(wi )yi .
n̂ℓ ˆ
n̂ℓ ˆ
i∈Iℓ

i∈Iℓ

The CF plug-in estimator is
L

1 XX
q(wi )T δ̂ℓ .
β̂ =
n ℓ=1 i∈I
ℓ

15

The DCDR estimator is based on an estimator of the inverse propensity score π0 (wi )−1 =
1/π0 (wi ) given by
1 X
1 X
−1
T α α
− α
^
π(w
ai q(wi )q(wi )T , h̃αℓ =
q(wi ),
i )ℓ = q(wi ) δ̃ℓ , δ̃ℓ = Σ̃ℓ h̃ℓ , Σ̃ℓ =
ñℓ ˜
ñℓ ˜
i∈Iℓ

i∈Iℓ

where ñℓ is the number of observation indices in I˜ℓ . This estimator of the inverse propensity
score is a version of one discussed in Robins et al.(2007). The DCDR estimator is


L
1 XX
T
−1
T
^
β̃ =
q(wi ) δ̂ℓ + ai π(wi )ℓ [yi − q(wi ) δ̂ℓ ] .
n ℓ=1 i∈I
ℓ

This has the usual form for a doubly robust estimator of the mean with data missing at random.
It differs from previous estimators in having the full CF form where the nonparametric estimators
are based on distinct subsamples of the data.
R
For the average derivative example m(z, γ) = ω(x)γ(x)dx does not depend on z so we can
use all the data in the construction of the plug-in estimator. That estimator is given by
β̂ =

Z

T

ω(x)γ̂(x)dx = v δ̂, v =

Z

n
n
X
X
T −
p(xi )p(xi ) ]
p(xi )yi .
ω(x)p(x)dx, δ̂ = [
i=1

(3.7)

i=1

As shown in equation (3.4), α0 (x) = f0 (x)−1 ω(x), where f0 (x) is the pdf of x. Also here v(z) = v
so the estimator of α0 (x) is p(x)T Σ̃−
ℓ v. The DCDR estimator is then
L

1 XX
{
β̃ =
n ℓ=1 i∈I
ℓ

Z

h
i
T −
ω(x)γ̂ℓ (x)dx + p(xi ) Σ̃ℓ v [yi − γ̂ℓ (xi )]}.

(3.8)

R
Both the plug-in and the DCDR estimators depend on the integral v = ω(x)p(x)dx. Generally
this vector of integrals will not exist in closed form so that construction of these estimators will
require numerical computation or estimation of v, such as by simulation.
We now impose some specific conditions on p(x).
Assumption 3: p(x) = aq(w) where i) the support of wi is [0, 1]r , wi is continuously
distributed with bounded pdf that is bounded away from zero; ii) q(w) are tensor product bsplines of order κ with knot spacing approximately proportional to the number of knots; iii)
√
q(w) is normalized so that λmin(E[q(wi )q(wi )T ]) ≥ C > 0 and supw∈[0,1]r kq(w)k ≤ C K; iv) ai
is bounded and E[a2i |wi ] is bounded away from zero.
Under condition i) it is known that there is a normalization such that condition iii) is
satisfied, e.g. as in Newey (1997). To control the bias of the estimator we require that the
true regression function γ0 (x) and the auxiliary function α0 (x) each be in a Holder class of
16

functions. We define a function g(x) to be Holder of order s if there is a constant C such that
g(x) is continuously differentiable of order s̄ = int[s] and each of its s̄ partial derivatives ∇s̄ g(x)
satisfies |∇s̄ g(x̃) − ∇s̄ g(x)| ≤ C kx̃ − xks−s̄ .
Assumption 4: γ0 (x) and α0 (x) are Holder of order sγ and sα respectively.
This condition implies that the population least squares approximations to γ0 (x) and α0 (x)
converge at certain rates. Let ζγ = min{1 + κ, sγ }/r, ζα = min{1 + κ, sa }/r, Σ = E[p(xi )p(xi )T ],
δ = Σ−1 E[p(xi )γ0 (xi )], γK (x) = p(x)T δ, δα = Σ−1 E[p(xi )α0 (xi )], αK (x) = p(x)T δa . Then standard approximation theory for splines gives
E[{γ0 (xi ) − γK (xi )}2 ] = O(K −2ζγ ), sup |γ0 (x) − γK (x)| = O(K −ζγ ),
x∈[0,1]r

E[{α0 (xi ) − αK (xi )}2 ] = O(K −2ζα ).
We will use these results to derive the rates at which certain remainders converge to zero.
We also impose the following condition:
Assumption 5: V ar(yi |xi ) ≤ C, K −→ ∞, and K ln(K)/n −→ 0.
These are standard conditions for series estimators of conditional expectations. A bounded
conditional variance for yi helps bound the variance of series estimators. The upper bound on
the rate at which K grows is slightly stronger than K/n −→ 0. This upper bound on K allows
us to apply the Rudelson (1999) law of large numbers for symmetric matrices to show that the
various second moment matrices of p(x) converge in probability. Another condition we impose
is:
Assumption 6: λmax (E[v(zi )v(zi )T ]) ≤ CdK and {E[{m(zi , γK ) − m(zi , γ0)}2 ]}

1/2

= O(K −ζm ).

The first condition will be satisfied with dK = 1 in the examples under specific regularity
conditions detailed below. The second condition gives a rate for the mean square error convergence of m(z, γK ) − m(z, γ0 ) as K grows. In all of the examples this rate will be ζm = ζγ . In
other examples, including those where m(z, γ) and v(z) depend on derivatives with respect to
x, we will have dK growing with K and ζm < ζγ .
For the statement of the results to follow it is convenient to work with the remainder term
r
√
¯ ∗ = nK −ζγ −ζα + K −ζγ + K −ζα + K .
∆
n
n

This remainder coincides with the fast remainder ∆∗n when the spline order is high enough with
κ ≥ max{sγ , sα } − 1. The only cases where it would not be possible to choose such a κ are for
the Haar basis where κ = 0.
17

4

The Plug-in Estimator

In this Section we derive bounds on the size of remainders for the plug-in estimator. Some
bounds are given for general plug-in estimators, some for plug-ins that are series regression with
Haar splines, and some for other splines. We begin with a result that applies to all plug-ins. We
drop the CF designation because all the estimators from this point on will use cross-fitting.
The cross-fit form of the plug-in estimator allows us to partly characterize its properties
under weak conditions on a general plug-in estimator that need not be a series regression. This
characterization relies on independence of γ̂ℓ from the observations in Iℓ to obtain relatively
simple stochastic equicontinuity remainders. Also, this result accounts for the overlap across
groups in observations used to form γ̂ℓ . Let An denote an event that occurs with probability
approaching one. For example, An could include the set of data points where Σ̂ℓ is nonsingular
for each ℓ.
Lemma 1: If Assumptions 1 and 2 are satisfied and there is ∆m
n such that
Z
1/2
2
1(An )
[m(z, γ̂ℓ ) − m(z, γ0 )] F0 (dz)
= Op (∆m
n ), (ℓ = 1, ..., L) ,
then for m̄(γ) =
√

R

m(z, γ)F0 (dz),

n
L
√ X
n̄ℓ
1 X
[m̄(γ̂ℓ ) − β0 ] + Op (∆m
[m(zi , γ0 ) − β0 ] + n
n(β̂ − β0 ) = √
n ).
n
n i=1
ℓ=1

If in addition there is ∆φn such that for each (ℓ = 1, ..., L) ,
p
1 X
α0 (xi )[yi − γ0 (xi )] + Op (∆φn ),
n̂ℓ [m̄(γ̂ℓ ) − β0 ] = √
n̂ℓ i∈I
/
ℓ

then for δ(z) = m(z, β0 ) − β0 + α0 (x)[y − γ0 (x)]
√

n

1 X
φ
−1
ψ(zi ) + Op (∆m
n(β̂ − β0 ) = √
n + ∆n + n ).
n i=1

This result gives a decomposition of remainder bounds into two kinds. The first ∆m
n is a
stochastic equicontinuity bound that has the simple mean-square form given here because of
the sample splitting. The second ∆φn is a bound that comes from the asymptotically linear
expansion of the linear functional estimator m̄(γ̂ℓ ). For general b-splines we can apply Ichimura
and Newey (2017) to obtain ∆φn . For zero order splines we give here sharper remainder bounds.
For series estimators the stochastic equicontinuity remainder bound ∆m
n will be
r
K
(dK + 1) + K −ζm ,
∆m
n =
n
18

where dK and ζm are as given in Assumption 6. As mentioned above, in the examples in this
¯∗
paper dK ≤ C and ζm = ζγ . Here we can take ∆m
n ≤ C ∆n , so the stochastic equicontinuity
¯ ∗.
remainder bound is the same size as ∆
n
Our next result gives remainder bounds for the Haar basis.
Theorem 2: If Assumptions 1-6 are satisfied, κ = 0, and K[ln(n)]2 /n −→ 0 then
√

n

1 X
¯ ∗ + ∆m + K −ζγ ln(n)).
n(β̂ − β0 ) = √
ψ(zi ) + Op (∆
n
n
n i=1

¯∗
If in addition dK is bounded as a function of K and ζm = ζγ then ∆m
n ≤ C ∆n .
Here we see that for a Haar basis the order of the remainder term for the plug-in estimator
−ζγ
¯∗
is a sum of the stochastic equicontinuity term ∆m
ln(n) being the size of
n and ∆n , with K
the fast remainder up to ln(n). In the examples and other settings where dK is bounded and
¯∗
ζm = ζγ the ∆m
n remainder will just be of order ∆n . The following result states conditions for
the examples.
Corollary 3: Suppose that Assumptions 1-3 and 5 are satisfied, κ = 0, K[ln(n)]2 /n −→ 0,
and γ0 (x) is Holder of order sγ . If either i) β̂ is the expected conditional covariance estimator,
E[ai |xi = x] is Holder of order sα , E[a2i |xi ] is bounded, or ii) β̂ is the missing data mean
estimator, Pr(ai = 1|xi ) is bounded away from zero and is Holder of order sα , or iii) β̂ is the
average derivative estimator, ω(x) and f0 (x) are Holder of order sα , and f0 (x) is bounded away
from zero on the set where ω(x) > 0, then
√

n

1 X
¯ ∗n + K −ζγ ln(n)).
ψ(zi ) + Op (∆
n(β̂ − β0 ) = √
n i=1

The remainder bound means that the plug-in estimator can attain root-n consistency under
¯ ∗n
minimal conditions, when the dimension r is small enough. There will exist K such that ∆
goes to zero if an only if
min{1, sγ } + min{1, sα}
.
(4.1)
r
This condition can be satisfied for r < 4 but not for r ≥ 4. For r = 1 this condition will be
satisfied if and only if
1
sγ + sα > ,
2
which is the minimal condition of Robins et al.(2009) for existence of a semiparametric efficient
estimator for the expected conditional covariance and missing data parameters when r = 1. For
r = 2 we note that
1/2 < ζγ + ζα =

min{1, sγ } + min{1, sα } ≥ 1 if and only if sγ + sα ≥ 1.
19

For r = 2 equation (4.1) is min{1, sγ } + min{1, sα } > 1, which requires both sα > 0 and sγ > 0
and so is slightly stronger than the Robins et al.(2009) condition sγ + sα > 1. For r = 3
the situation is more complicated. Equation (4.1) is stronger than the corresponding condition
sγ + sα > 3/2 of Robins et al.(2009), although it is the same for the set of (sγ , sα ) where sγ ≤ 1
and sα ≤ 1. Along the diagonal where sα = sγ the two conditions coincide as sγ > 3/4.
The limited nature of these results is associated with the Haar basis, which limits the degree to which smoothness of the underlying function results in a faster approximation rate. If
Theorem 2 and Corollary 3 could be extended to other, higher order b-splines, this limitation
could be avoided. For the present we are only able to do this for the doubly robust estimator of
a partially linear projection, as discussed in the next Section.
¯ ∗ in Theorem 2. Let
There is a key result that allows us to obtain the remainder bound ∆
n
Pn
Pn
ĥ2 = i=1 p(xi )[γ0 (xi ) − γK (xi )]/n, Σ̂ = i=1 p(xi )p(xi )T /n, and Σ = E[p(xi )p(xi )T . We show
in the Appendix that for the Haar basis

j
K −2ζγ CK
j
T
j
λmax (E[(Σ − Σ̂) ĥ2 ĥ2 (Σ − Σ̂) ]) ≤
.
(4.2)
n
n
If b-spline bases other than Haar also satisfied this condition then we could obtain results
analogous to Theorem 2 and Corollary 3 for these bases. We do not yet know if other bases
satisfy this condition. The Haar basis is convenient in p(x)T p(x) being piecewise constant.
Cattaneo and Farrell (2013) exploited other special properties of the Haar basis to obtain sharp
uniform nonparametric rates.
For b-splines of any order we can obtain remainder rates by combining Lemma 1 with
Theorem 8 of Ichimura and Newey (2017).
Theorem 4: If Assumptions 1-6 are satisfied then
√

n

1 X
¯ ∗n + ∆m
¯
¯
n(β̂ − β0 ) = √
ψ(zi ) + Op (∆
n + ∆n ), ∆n =
n i=1



K ln K
n

1/2

K (1/2)−ζγ .

¯∗
If in addition dK is bounded as a function of K and ζm = ζγ then ∆m
n ≤ C ∆n .
¯ n.
Here we see that the remainder bound for splines with κ > 0 has an additional term ∆
When ζγ is large enough, i.e. γ0 (x) is smooth enough and the order of the spline is big enough,
¯ n will be no larger than ∆
¯ ∗ . Also, when ζγ > 1/2 the condition
so that ζγ > 1/2, the additional ∆
n
of Robins et al.(2009) for semiparametric efficient estimation is met for the expected conditional
covariance and missing data examples for any ζα . Thus, when γ0 (x) is smooth enough to meet the
Robins et al.(2009) condition without imposing any smoothness on α0 (x) the plug-in estimator
¯ ∗.
will have the remainder bound ∆
n
¯n + ∆
¯ ∗n goes to zero if and only if
More generally there will exist a K such that ∆
2 min{κ + 1, sγ } + min{κ + 1, sα } > r.
20

(4.3)

This condition is slightly stronger than that of Robins et al.(2009) which is 2sγ + 2sα > r. Also,
the remainder may go to zero when when K is chosen to maximize the rate at which the mean
square error of γ̂0 (x) goes to zero. Setting K −2ζr proportional to K/n is such a choice of K. Here
the remainder term goes to zero for min{κ + 1, sγ } > r/ [2(1 + r)] and min{κ + 1, sα } > r/2, a
¯ ∗.
stronger condition for sγ and the same condition for sα as would hold if the remainder were ∆
n

5

Partially Linear Projection

In this Section we consider a series estimator of partially linear projection coefficients. We give
this example special attention because the DCDR estimator will have a remainder bound that
¯ ∗n . The remainder bounds we find for other doubly robust estimators may be larger.
is only ∆
What appears to make the partially linear projection special in this respect is that α0 (x) is a
conditional expectation of an observed variable. In other cases where α0 (x) is not a conditional
¯ ∗ for bases other than Haar.
expectation we do not know if the remainder bound will be ∆
n
The parameter vector of interest in this Section is
β0 = E[{ai − E[ai |wi ]}aTi ]

−1

E[{ai − E[ai |wi ]}yi ].

This vector β0 can be thought of as the coefficients of ai in a projection of yi on the set of
functions of the form aTi β + λ(xi ) that have finite mean square. Note that this definition of β0
places no substantive restrictions on the distribution of data, unlike the conditional expectation
partially linear model where E[yi |ai , wi ] = aTi β0 + ξ0 (xi ).
The object β0 is of interest in a treatment effects model where ai is a binary treatment, yi is
the observed response, xi are covariates, and outcomes with and without treatment are assumed
to be mean independent of ai conditional on wi . Under an ignorability condition that the
outcome is mean independent of treatment conditional on covariates, E[yi |ai = 1, xi ] − E[yi |ai =
0, xi ] is the average treatment effect conditional on xi . Also for πi = Pr(ai = 1|xi ),
β0 =

E[πi (1 − πi ){E[yi |ai = 1, xi ] − E[yi |ai = 0, xi ]}]
.
E[πi (1 − πi )]

Here we have the known interpretation of β0 as a weighted average of conditional average
treatment effects, with weights πi (1 − πi )/E[πi (1 − πi )].
It is straightforward to construct a DCDR estimator of β0 . Let γ0 (xi ) = E[yi |xi ] and α0 (xi ) =
−E[ai |xi ] as before, except that ai may now be a vector. Also let Iℓ denote the index set for the
ℓth group, and Iˆℓ and I˜ℓ the index sets for the observations used to obtain γ̂ℓ and α̃ℓ respectively.
For any function g(z) let
F̄ {g(z)} =

1 X
1 X
1 X
g(zi ), F̂ {g(z)} =
g(zi ), F̃ {g(z)} =
g(zi ).
n̄ℓ i∈I
n̂ℓ ˆ
ñℓ ˜
i∈Iℓ

ℓ

21

i∈Iℓ

These represent sample averages over each of the groups of observations. Let γ̂ℓ (x), α̂ℓ (x), and
α̃ℓ (x) be series estimators of γ0 (x) and α0 (x) given by
γ̂ℓ (x) = p(x)T δ̂ℓ , α̂ℓ (x) = p(x)T δ̂ℓα , α̃i (x) = p(x)T δ̃ℓα ,
δ̂ = Σ̂−1 ĥ, δ̂α = Σ̂− ĥα , δ̃α = Σ̃− h̃α , Σ̂ = F̂ {p(x)p(x)T }, Σ̃ = F̃ {p(x)p(x)T },

ĥ = F̂ {p(x)y}, ĥα = F̂ {p(x)a}, h̃α = F̃ {p(x)a}.
The estimator we consider is
β̃ =

L X
X
ℓ=1 i∈Iℓ

[ai − α̃ℓ (xi )][ai − α̂ℓ (xi )]

T

!−1

L X
X
ℓ=1 i∈Iℓ

[ai − α̃ℓ (xi )][yi − γ̂ℓ (xi )].

(5.1)

This estimator can be thought of as an instrumental variables estimator with left hand sides
variable yi − ĝi (xi ), right hand side variables ai − α̂i (xi ), and instruments ai − α̃i (xi ). Here
the instrumental variables form is used to implement the cross-fitting and not to correct for
endogeneity. This form means that every element of the matrix that is inverted and of the
vector it is multiplying is a DCDR estimator of an expected conditional covariance like that
described earlier.
Theorem 5: If Assumptions 1 - 3 and 5 are satisfied, λ0 (x) = E[yi − aTi β0 |xi = x] is Holder
of order sγ and each component of E[ai |xi = x] is Holder of order sα , H = E[V ar(ai |xi )] exists
and is nonsingular, and Ω = E[{ai − α0 (xi )}{ai − α0 (xi )}T ε2i ] exists then for εi = yi − aTi β0 −
λ0 (xi ) and ψ(zi ) = H −1 (ai − E[ai |xi ])εi ,
n
√
1 X
¯ ∗ ).
ψ(zi ) + Op (∆
n(β̃ − β0 ) = √
n
n i=1

The regularity conditions here are somewhat stronger than those of Donald and Newey
(1994), who do not require any restrictions on the marginal distribution of xi nor use any
sample splitting. This strengthening is useful to achieve the fast remainder for partially linear
projections rather than for the coefficients β0 in the conditional mean model E[yi |ai , xi ] =
aTi β0 + λ0 (xi ) of Donald in Newey (1994). The upper bound on the rate at which K can grow
is slightly stricter than in Donald and Newey (1994) due to the presence of the ln(K) term in
Assumption 5. Thus, under somewhat stronger conditions than those of Donald and Newey
(1994) the DCDR estimator of a partially linear projection has a fast remainder just as in
Donald and Newey (1994). Consequently, the estimator will be root-n consistent under minimal
conditions.
When the Robins et al. (2009) minimal condition (sγ + sα )/r > 1/2 holds, consider a spline
with κ > max{sγ , sα } − 1, so that ζγ + ζα = (sγ + sα )/r > 1/2. Then there will exist a K
22

¯ ∗ −→ 0 and hence β̃ will be semiparametric efficient. Thus we see that the DCDR
such that ∆
n
estimator β̃ of equation (5.1) will be semiparametric efficient under nearly minimal conditions
and has a fast remainder term.

6

The Doubly Robust Estimator

In this Section we show that the DCDR estimator has improved properties relative to the plug-in
estimator, in the sense that the remainder bounds are smaller for the DCDR robust estimator.
We have not yet been able to obtain the fast remainder for the doubly robust estimator for
general splines, for the same reasons as for plug-in estimators.
Before giving results for series estimators we give a result that applies to any doubly robust
estimator of a linear functional. Let An denote an event that occurs with probability approaching
one. For example, An could include the set of data points where Σ̂ℓ is nonsingular.
Lemma 6: If Assumptions 1 and 2 are satisfied, γ̂ℓ (x) and α̂ℓ (x) do not use observations in
γ
α
Iℓ , V ar(yi |xi ) is bounded, and there are ∆m
n , ∆m , and ∆m , such that for each (ℓ = 1, ..., L),
Z
1/2
2
1(An )
[m(z, γ̂ℓ ) − m(z, γ0 )] F0 (dz)
= Op (∆m
n ),
1(An )

Z

2

2

α0 (x) [γ̂ℓ (x) − γ0 (x)] F0 (dz)

1(An )

Z

2

[α̃ℓ (x) − α0 (x)] F0 (dz)

1/2

1/2

= Op (∆γn ),
= Op (∆αn ),

then
n
L
√
1 XX
1 X
γ
α
[α̃ℓ (xi ) − α0 (xi )][γ̂ℓ (xi ) − γ0(xi )] + Op(∆m
ψ(zi ) − √
n(β̃ − β0 ) = √
n + ∆n + ∆n ).
n i=1
n ℓ=1 i∈I
ℓ

This result does not require that γ̂ℓ (x) and α̂ℓ (x) be computed from different samples. It
only uses the sample splitting in averaging over different observations that are used to construct
γ̂ℓ and α̃ℓ . Also, it is known from Newey, Hsieh, and Robins (1998, 2004) and Chernozhukov et.
al. (2016) that adding the adjustment term to the plug-in estimator makes the remainder second
order. The conclusion of Lemma 6 gives an explicit form of that result. Under weak conditions
that only involve mean-square convergence the doubly robust estimator has a remainder that is
the sum of three stochastic equicontinuity remainders and the quadratic, split sample remainder
involving the product of the estimation remainders for the two nonparametric estimators γ̂ and
α̃.
23

¯ ∗ as its primary remainder for the
For series estimators the DCDR estimator will have ∆
n
Haar basis
Theorem 7: If Assumptions 1-6 are satisfied, κ = 0, and K[ln(n)]2 /n −→ 0 then
√

n

1 X
¯ ∗ + ∆m ).
n(β̃ − β0 ) = √
ψ(zi ) + Op (∆
n
n
n i=1

¯∗
If in addition dK is bounded as a function of K and ζm = ζγ then ∆m
n ≤ C ∆n .
One improvement of the DCDR estimator over the plug-in estimator is that the remainder
no longer contains the K −ζγ ln(n) term. The elimination of this term is the direct result of the
DCDR estimator having a smaller remainder than the plug-in estimator.
For splines of order κ > 0 we can obtain a result for the DCDR estimator that improves on
the plug-in remainder bound.
Theorem 8: If Assumptions 1-6 are satisfied then
√

n

1 X
¯ ∗ + ∆m + ∆
˜ n ), ∆
˜n =
n(β̃ − β0 ) = √
ψ(zi ) + Op (∆
n
n
n i=1

s

K 3 [ln(K)]2 (1 + dK ) (1/2)−ζγ
K
.
n3

¯∗
If in addition dK is bounded as a function of K and ζm = ζγ then ∆m
n ≤ C ∆n .
Here we see that the remainder bound for the DCDR estimator will generally be smaller
than the remainder bound for the plug-in estimator because the term K ln(K)/n is raised to
the 3/2 power rather than the 1/2 power. Here it turns out that there will exist a K such that
all of the remainder terms go to zero if
4ζγ + 3ζα ≥ 2.
For example, if sγ = sα and κ ≥ max{sγ , sα } − 1, this requires sγ > 2r/7, which is only slightly
stronger than the sγ > r/4 condition of Robins et al.(2009) that is required for existence of a
semiparametric efficient estimator. Also, existence of K such that the remainder will be of size
¯ ∗ requires
no larger than ∆
n
2ζγ + ζα ≥ 1.
For example, if ζγ = ζα this requires ζγ > 1/3, which is weaker than the condition ζγ > 1/2 for
the remainder for the plug-in estimator. In these ways the DCDR estimator improves on the
plug-in estimator.

24

7

Appendix

This Appendix gives the proofs of the results in the body of the paper. We begin with the proofs
of Lemma 1 and Lemma 6 because they are not restricted to series estimators.
ˆ iℓ = m(zi , γ̂) − m(zi , γ0 ) − m̄(γ̂ℓ ) + β0 for i ∈ Iℓ and let Z(Iℓ )c
Proof of Lemma 1: Define ∆
ˆ iℓ |Z(Iℓ )c ] = 0 by construction for
denote the set of observations zi for i ∈
/ Iℓ . Note that E[∆
ˆ iℓ ∆
ˆ jℓ |Z(Iℓ )c ] = 0 for i, j ∈ Iℓ . Furthermore,
i ∈ Iℓ . Also by independence of the observations, E[∆
R
ˆ 2 |Z(Iℓ )c ] ≤ [m(z, γ̂ℓ ) − m(z, γ0 )]2 F0 (dz) = Op ((∆m )2 ) for i ∈ Iℓ . Then we have
E[∆
n
iℓ
!2
!2
X
1
1 Xˆ
2
ˆ iℓ |Z(Iℓ )c ] = n̄ℓ E[∆
ˆ 2iℓ |Z(Iℓ )c ] = Op ((∆m
∆
∆iℓ |Z(Iℓ )c ] = E[
E[ √
n ) ).
n
n
n i∈I
i∈I
ℓ

ℓ

Therefore, by the Markov inequality we have
follows from
√

P

i∈Iℓ

ˆ iℓ /√n = Op (∆m ). The first conclusion then
∆
n

n
L
L
X
√ X
1 X
1 Xˆ
n̄ℓ
√
∆iℓ + √
[m(zi , γ0 ) − β0 ] + n
[m̄(γ̂ℓ ) − β0 ].
n(β̂ − β0 ) =
n i∈I
n i=1
n
ℓ=1
ℓ=1
ℓ

For the second conclusion note by the subsamples being as close to equal size as possible,

Then by

n̄ℓ
n̄ℓ /n
1/L
1
=
=
+ O(n−1) =
+ O(n−1 ).
n̂ℓ
n̂ℓ /n
(L − 1)/L
(L − 1)

L
L
√ X
n̄ℓ
1 X
n
[m̄(γ̂ℓ ) − β0 ] = √
n̄ℓ
n
n
ℓ=1
ℓ=1

=

1
1
√
L−1 n

r

L

X n̄ℓ 1 X
1p
√
n̂ℓ [m̄(γ̂ℓ ) − β0 ] =
φ(zi ) + Op (∆φn )
n̂ℓ
n̂ℓ n
ℓ=1

L X
X

i∈I
/ ℓ

φ(zi ) + Op (∆φn + n−1 )

ℓ=1 i∈I
/ ℓ

L
n
X
1 XX
1
√
=
φ(zi )) + Op (∆φn + n−1 )
(
φ(zi ) −
L − 1 n ℓ=1 i=1
i∈I
ℓ

n
1 X
φ(zi ) + Op (∆φn + n−1 ).
=√
n i=1

The conclusion then follows by the triangle inequality. Q.E.D.
Proof of Lemma 6: By adding and subtracting terms it follows that for εi = yi − γ0 (xi )
and φ(zi ) = α0 (xi )[yi − γ0 (xi )]
α̃ℓ (xi )[yi − γ̂ℓ (xi )] = φ(zi ) − α0 (xi )[γ̂(xi ) − γ0 (xi )] + [α̃ℓ (xi ) − α0 (xi )]εi
− [α̃ℓ (xi ) − α0 (xi )][γ̂(xi ) − γ0 (xi )].
25

The first conclusion of Lemma 1 with m(z, γ) = α0 (x)γ(x) gives
Z
L
L
√ X
1 XX
n̄ℓ
√
α(x)[γ̂ℓ (x) − γ0 (x)]F0 (dx) + Op (∆γn ).
α0 (xi )[γ̂ℓ (xi ) − γ0 (xi )] = n
n
n ℓ=1 i∈I
ℓ=1
ℓ

Assumption 1 and the first conclusion of Lemma 1 also give
Z
L
L
√ X
√ X
n̄ℓ
n̄ℓ
α(x)[γ̂ℓ (x) − γ0 (x)]F0 (dx) = n
[m̄(γ̂ℓ ) − β0 ]
n
n
n
ℓ=1
ℓ=1
L

1 XX
=√
[m(zi , γ̂ℓ ) − m(zi , γ0 )] + Op (∆m
n ).
n ℓ=1 i∈I
ℓ

R

In addition, if we take γ = α and m(z, α) = α(x)ε then m(z, α)F0 (dz) = 0, so that by Lemma
1,
L
1 XX
√
[α̃ℓ (xi ) − α0 (xi )]εi = Op (∆αn ).
n ℓ=1 i∈I
ℓ

Then collecting terms we have
√

n

1 X
n(β̃ − β0 ) = √
[m(zi , γ0 ) − β0 ]
n i=1
L

1 XX
{m(zi , γ̂) − m(zi , γ0) + α̃ℓ (xi )[yi − γ̂ℓ (xi )]}
+√
n ℓ=1 i∈I
ℓ

1
=√
n
1
√
n

n
X
i=1

L

1 XX
γ
α0 (xi )[γ̂ℓ (xi ) − γ0 (xi )] + Op (∆m
ψ(zi ) + √
n + ∆n )
n ℓ=1 i∈I
ℓ

L X
X
ℓ=1 i∈Iℓ

{−α0 (xi )[γ̂(xi ) − γ0 (xi )] + [α̃ℓ (xi ) − α0 (xi )]εi }

L

1 XX
−√
[α̃ℓ (xi ) − α0 (xi )][γ̂(xi ) − γ0 (xi )]
n ℓ=1 i∈I
ℓ

1
=√
n
+

n
X
i=1

Op (∆m
n

L

1 XX
[α̃ℓ (xi ) − α0 (xi )][γ̂(xi ) − γ0 (xi )]
ψ(zi ) − √
n ℓ=1 i∈I

+

ℓ

∆γn

+

∆αn ).Q.E.D.

We now turn to proofs of the results involving series estimators. Let Σ = E[p(xi )p(xi )T ]. It
follows from Assumption 3 that Σ is nonsingular, so we can replace p(x) by Σ−1/2 p(x) and so
normalize Σ = I without changing the assumptions. We impose this normalization throughout.
Also, throughout the Appendix C will denote a generic constant not depending on n or K.
26

We next prove the key result in eq. (4.2) for a zero order spline. Let r(x) = γ0 (x) − γK (x)
P
and ĥ2 = ni=1 p(xi )r(xi )/n as in the body of the paper. Also let kAkop denote the operator
norm of a symmetric matrix A, being the largest absolute value of eigenvalues.
Lemma A1: If Assumptions 1-6 are satisfied, κ = 0, K[ln(n)]2 /n −→ 0, then for Û =
PJ−1
j
−1
J
j=0 (I − Σ̂) ĥ2 , Ŵ = Σ̂ (I − Σ̂) ĥ2 , J = int[ln(n)] and any constant ∆ > 0,
E[Û Û T ]

op

≤C

K −2ζγ [ln(n)]2
, Ŵ T Ŵ = op (n−∆ ).
n

Proof: Let Qi = p(xi )p(xi )T , ∆i = I − Qi , and hi = p(xi )r(xi ). Note that E[∆i ] = 0 and
E[hi ] = 0. For each j let L = 2j + 2. Let Ûj = (I − Σ̂)j ĥ2 . Then we have
E[Ûj ÛjT ] =

1
n2j+2

n
X

i1 ,...,iL =1



E[ Πjℓ=1 ∆iℓ hij+1 hTij+2 ΠLℓ=j+3 ∆iℓ ].

Consider any (i1 , ..., iL ) such that ij+1 6= ij+2 . Let i∗ = ij+1 and let Zic∗ denote the vector of
observations other than zi∗ . Note that




E[ Πjℓ=1 ∆iℓ hij+1 hTij+2 ΠLℓ=j+3 ∆iℓ ] = E[E[ Πjℓ=1 ∆iℓ hi∗ hTij+2 ΠLℓ=j+3∆iℓ |Zic∗ ]].

We proceed to show that



E[ Πjℓ=1 ∆iℓ hi∗ hTij+2 ΠLℓ=j+3∆iℓ |Zic∗ ] = 0.

Note that conditional on Zic∗ we can treat all terms where iℓ 6= i∗ as constant. Also, because
ij+1 6= ij+2 all terms where iℓ = i∗ depend only on p(xi∗ ). Therefore for the scalar r(x) =
γ0 (x) − γK (x) we have


E[ Πjℓ=1 ∆iℓ hi∗ hTij+2 ΠLℓ=j+3 ∆iℓ |Zic∗ ] = E[A1 (p(xi∗ ))p(xi∗ )r(xi∗ )A2 (p(xi∗ ))] = E[A(p(xi∗ ))r(xi∗ )],

where A1 (p) and A2 (p) are K × K and 1 × K matrices of functions of p and A(p) = A1 (p)pA2 (p).
Let Xk denote the interval where pk (x) is nonzero. Note that pk (x) = 1(x ∈ Xk )ck for a constant
ck , and hence
A(p(xi∗ )) =

K
X
k=1

Ak 1(xi∗ ∈ Xk ), Ak = A((0, ..., 0, ck , 0, ..., 0)T ).

Therefore by orthogonality of each pk (xi ) with r(xi ) in the population,
K
K
X
X


Ak c−1
Ak E[1(xi∗ ∈ Xk )r(xi∗ )] =
E[ Πjℓ=1 ∆iℓ hi∗ hTij+2 ΠLℓ=j+3 ∆iℓ |Zic∗ ] =
k E[pk (xi∗ )r(xi∗ )] = 0.
k=1

k=1

27

Therefore by iterated expectations, if ij+1 6= ij+2 we have


E[ Πjℓ=1 ∆iℓ hij+1 hTij+2 ΠLℓ=j+3 ∆iℓ ] = 0.

˜ i = hi hTi − Ψ,
It then follows that for Ψ = E[hij+1 hTij+1 ] = E[r(xi )2 p(xi )p(xi )T ] and ∆
j+1
j+1
j+1
E[Ûj ÛjT ] =
T1j
T2j

=
=

1
n2j+2
1
n2j+1
1
n2j+2

n
X

i1 ,..,ij+1 ,ij+3 ....,iL=1
n
X

i1 ,..,ij ,ij+3 ....,iL=1
n
X



E[ Πjℓ=1 ∆iℓ hij+1 hTij+1 ΠLℓ=j+3∆iℓ ] = T1j + T2j ,



E[ Πjℓ=1 ∆iℓ Ψ ΠLℓ=j+3∆iℓ ],

i1 ,..,ij+1 ,ij+3 ....,iL=1



L
˜i
].
Π
∆
E[ Πjℓ=1 ∆iℓ ∆
i
j+1
ℓ=j+3
ℓ



˜ i are diagonal matrices, so that E[ Πj ∆i ∆
˜i
Consider first T2j . Note that ∆i and ∆
ΠLℓ=j+3 ∆iℓ ]
j+1
ℓ
ℓ=1


˜ k,i
ΠLℓ=j+3 ∆k,iℓ ]
is a diagonal matrix, with k th diagonal element given by E[ Πjℓ=1 ∆k,iℓ ∆
j+1
where
˜ k,i = r(xi )2 pk (xi )2 − E[r(xi )2 pk (xi )2 ].
∆k,i = pk (xi )2 − E[pk (xi )2 ], ∆
j+1
The largest absolute value of the eigenvalues of a diagonal matrix is the maximum of the absolute
values of the diagonal elements, so it suffices to show that the conclusion holds for these diagonal
elements. We will consider the k th diagonal element but for notational convenience drop the k
subscript in what follows.
Note that pk (xi )2 ≤ BK for some B that does not vary with k or j. Also, for any random
variable Yi and µ = E[Yi ], note that by Jensen’s inequality, |µ|s ≤ E[|Yi |s ] for s ≥ 1. Then for
any positive s,
E[|Yi − µ|s ] ≤ E[(|Yi | + |µ|)s ] ≤ E[2s−1 (|Yi |s + |µ|s)] ≤ 2s−1 (E[|Yi |s ] + |µ|s )] ≤ 2s E[|Yi |s ]
Then for any positive integer s, by the triangle inequality and the definitions of ∆i ,
|E[∆si ]| ≤ 2s E[pk (xi )2s ] ≤ 2s (BK)s−1 E[pk (xi )2 ] ≤ (4BK)s−1 ≤ (CK)s−1 .

(7.1)

Also, by r(xi )2 ≤ DK −2ζγ we have
˜ i ] ≤ E[|∆i |s (r(xi )2 pk (xi )2 + E[r(xi )2 pk (xi )2 ])]
E[(∆i )s ∆
≤ E[(pk (xi )2 + E[pk (xi )2 ])s+1 ]DK −2ζγ

≤ 2s+1E[pk (xi )2s+2 ]DK −2ζγ ≤ 2s+1 (BK)s DK −2ζγ
≤ (4(D + 1)BK)s K −2ζγ ≤ (CK)s K −2ζγ .

28

(7.2)

Next consider
T2j

=

n
X

1
n2j+2

i1 ,..,ij+1 ,ij+3 ....,i2j+2 =1



˜i
ΠLℓ=j+3∆iℓ ].
E[ Πjℓ=1 ∆iℓ ∆
j+1

The only terms in this sum that are nonzero are those where every index iℓ is equal to at
least one other index iℓ′ , i.e. where each index is ”matched” with at least one other. Let
ı̃ = (i1 , .., ij+1 , ij+3...., i2j+2 )T denote the 2j + 1 dimensional vector of indices where each iℓ
is an integer in [1, n]. Let Υd denote a set of all such ı̃ with specified indices that are equal
to each other, but those matched indices are not equal to any other indices. For example,
one Υd is the set of ı̃ with i1 = ij+1 = ij+3 = · · · = i2J+2 and another is the set of ı̃ with
i1 = i2 , i3 = · · · = i2J+2 , i2 6= i3 . For each d each group of index coordinates that are equal to
each other can be thought of as a group of matching indices that we index by gd . Let mgd denote
the number of indices in group gd and Gd denote the total number of groups. Note that the
P d
total number of indices is 2j + 1 = G
gd =1 mgd . Also, by eqs. (7.1) and (7.2) for each ı̃ ∈ Υd we
have
|E[


˜i
Πjℓ=1 ∆iℓ ∆
j+1

ΠLℓ=j+3∆iℓ



]| ≤ K

−2ζγ

Gd
Y

(CK)mgd −1 = K −2ζγ (CK)2j+1−Gd .

gd =1

Also, the number of indices in Υd is less than or equal to nGd since each match can be regarded
as a single index. Therefore,
1
n2j+2

n
X

E[

Πjℓ=1 ∆iℓ

ı̃∈Υd




˜i
∆
ΠLℓ=j+3∆iℓ ] ≤
j+1
≤

1
n2j+2


1

n
X



˜i
ΠLℓ=j+3 ∆iℓ ]
E[ Πjℓ=1 ∆iℓ ∆
j+1

ı̃∈Υd

n2j+2



1
= K −2ζγ
n

nGd K −2ζγ (CK)2j+1−Gd



CK
n

2j+1−Gd

.

By hypothesis K/n −→ 0 so that for large enough n we have CK/n < 1. For such n we have
(CK/n)2j+1−Gd decreasing in Gd . Also, the largest Gd is j, because each group must contain at
least two elements. Therefore, for large enough n we have
j+1

n


1 −2ζγ CK
1 X
j
L
˜
E[ Πℓ=1 ∆iℓ ∆ij+1 Πℓ=j+3 ∆iℓ ] ≤ K
.
n2j+2 ı̃∈Υ
n
n
d

Note that the bound on the right does not depend on d. Let D denote the total number of


˜i
/ ∪D
ΠLℓ=j+3 ∆iℓ ] = 0 if ı̃ ∈
possible Υd . Then since E[ Πjℓ=1 ∆iℓ ∆
d=1 Υd we have
j+1
T2j

≤

D
X
d=1

1

n2j+2

n
X

ı̃∈Υd

E[


˜i
Πjℓ=1 ∆iℓ ∆
j+1

29

ΠLℓ=j+3∆iℓ


D
] ≤ K −2ζγ
n



CK
n

j+1

.

Note that there are exactly j 2j+1 ways of forming 2j + 1 indices into j groups. Ignoring the fact
that we can exclude ways where any group has only one index we have the bound D ≤ j 2j+1 .
Plugging in this bound into the above inequality and maximizing over diagonal elements gives

j+1
j 2j+1 K −2ζγ CK
j
T2 op ≤
.
n
n
Arguing similarly for T1j gives
T1j op

j 2j K −2ζγ
≤
n



CK
n

j

,

where we take 00 = 1.
Next note that by K ln(n)2 /n −→ 0 we have CK/n ≤ 1/[2 ln(n)2 ] for large enough n. Also,
j/ ln(n) ≤ 1 for all j < J. Then for n large enough
J−1
X
j=0

j

2j



CK
n

j

≤

J−1
X

j

2j

j=0



1
2 ln(n)2

j

2j  j X
J−1  j
∞  j
J−1 
X
X
1
1
1
1
j
≤ 2.
≤
≤
=
≤
ln(n)
2
2
2
1 − εn
j=0
j=0
j=0

Similarly it follows that for large enough n,
J−1
X

j

j=0

2j+1



CK
n

j+1

2j+1  j
J−1 
j
1
1 X
1
.
≤
≤
2 ln(n) j=0 ln(n)
2
ln(n)

Then we have for large enough n,
J−1
X

E[Ûj ÛjT ]

j=0

op

≤

J−1
X

T1j + T2j

j=0

K −2ζγ
≤
n


2+



op

1
ln(n)
T

Also by the Cauchy Schwartz inequality, Û Û =
Therefore, for large enough n,
T

E[Û Û ]

op

≤J

2

J−1
X

≤


≤

+ T2j
op

op

≤



CK −2ζγ
.
n

j=0

op

T1j

j=0

P
J−1

E[Ûj ÛjT ]

j=0

J−1 
X

Ûj

 P

J−1
j=0 Ûj

T

≤ J2

C ln(n)2 K −2ζγ
,
n

giving the first conclusion.
For the second conclusion note that for any ∆ > 0,
ln{n∆ [ln(n)]−2 ln(n)+2 } = ln(n)[∆ − 2 ln(ln(n))] + 2 ln(ln(n)) −→ −∞.
30

PJ−1
j=0

Ûj ÛjT .

It follows that [ln(n)]−2 ln(n)+2 = o(n−∆ ) for any ∆. Also, by K/n = o([1/ ln(n)]2 ) we have
K ln (K) /n = o(1/ ln(n)), so that


K ln(K)
n

2J

= o([ln(n)]−2int(ln(n)) ) = o([ln(n)]−2(ln(n))+2 ) = o(n−∆ ),

for any ∆ > 0. Then we have
T

1̂Ŵ Ŵ ≤

4ĥT2 (I

2J

− Σ̂) ĥ2 ≤

4ĥT2 ĥ2

I − Σ̂

2J
op

K 1−2ζγ
= Op (
n



K ln(K)
n

2J

) = op (n−∆ ),

for any ∆ > 0 by Rudelson’s (1999) law of large numbers for random matrices, giving the second
conclusion. Q.E.D.
In the Appendix we focus on one subset I¯ = Iℓ of observations and let Iˆ and I˜ denote the
observations used to compute δ̂ and δ̃α respectively. Let n̄, n̂, ñ denote the number of elements
¯ I,
ˆ and I˜ respectively and
of I,
F̄ {g(z)} =

1X
1X
1X
g(zi ), F̂ {g(z)} =
g(zi ), F̃ {g(z)} =
g(zi ),
n̄ ¯
n̂ ˆ
ñ ˜
i∈I

i∈I

i∈I

denote averages over the respective subsets of observations.
Next we make a few definitions we will use throughout. Let ζγ , ζα , δ, γK , δα , and αK be as
defined following Assumption 4. Also, let
εi = yi − γ0 (xi ), ri = γ0 (xi ) − γK (xi ), ηi = v(zi ) − p(xi )α0 (xi ), riα = α0 (xi ) − αK (xi ),

ĥ1 = F̂ {p(x)ε}, ĥ2 = F̂ {p(x)r}, h̃α1 = F̃ {η}, h̃α2 = F̃ {p(x)r α }, Σ̂ = F̂ {p(x)p(x)T }, Σ̃ = F̃ {p(x)p(x)T },
ˆ 1 = Σ̂− ĥ1 , ∆
ˆ 2 = Σ̂− ĥ2 , ∆
˜ α = Σ̃− h̃α , ∆
˜ α = Σ̃− h̃α , Σ̄ = F̄ {p(x)p(x)T },
∆
1
1
2
2
One piece of algebra we will use throughout is that, when Σ̂ and Σ̃ are nonsingular, by adding
and subtracting Σ̂−1 F̂ {p(x)γ0 (x)} and Σ̃−1 F̃ {p(x)α0 (xi )} respectively we have
ˆ1 + ∆
ˆ 2 , δ̃α − δα = ∆
˜α + ∆
˜ α.
δ̂ − δ = ∆
1
2

(7.3)

Some properties of these objects will be useful in the proofs to follow. We collect these
properties in the following result. Let 1̂ and 1̃ denote the indicator function that the smallest
eigenvalue of Σ̂ or Σ̃ is larger than 1/2 respectively. As in Belloni et al.(2015) Pr(1̂ = 1) −→ 1
and Pr(1̄ = 1) −→ 1. Also, let Ẑ c , Z̃ c , Z̄ c denote all the other observations other than those
ˆ I,
˜ or I¯ respectively and X = (x1 , ..., xn ).
indexed by I,

31

Lemma A2: If Assumptions 1-6 are is satisfied then
 


K
−2ζγ K
ˆ
ˆ
i) 1̂ ∆1 = Op
; ii) 1̂ ∆2 = op K
;
n
n

 

K
(1 + dK )K
α
α
ˆ
ˆ
, iv) 1̂ ∆2 = Op
,
iii) 1̂ ∆1 = Op
n
n
 


2
2
K
dK K
v) 1̂ δ̂ − δ = Op
; vi) 1̃ δ̃α − δα = Op
,
n
n
Z
K
C
T
c
ˆ
ˆ
vii) 1̂E[∆1 ∆1 |X, Ẑ ] ≤ I, viii) 1̂ [γ̂(x) − γ0 (x)]2 F0 (dx) = Op ( + K −2ζγ ),
n
n


Z
(dK + 1)K
ix) 1̃ [α̃(x) − α0 (x)]2 F0 (dx) = Op
+ K −2ζα .
n
Proof: Note that for εi = yi − γ0 (xi ), E[ε2i |xi ] = V ar(yi |xi ) ≤ C. Note that 1̂Σ̂−2 ≤ 4I in
the positive semi-definite semi-order so that
X
2
4
4C
K
ˆ 1 ] ≤ 4E[ĥT ĥ1 ] = 4
E[p(xi )T p(xj )εi εj ] = E[kp(xi )k2 ε2i ] ≤
E[kp(xi )k2 ] = O( ).
E[1̂ ∆
1
2
n̂
n̂
n̂
n
ˆ
i,j∈I

The first conclusion then follows by the Markov inequality. Next, we have supx |γK (x) − γ0 (xi )| =
O(K −ζ ) and hence for


2
4 X
4
2
T
T
−2ζγ
−2ζγ K
ˆ
,
E[p(xi ) p(xj )ri rj ] = E[kp(xi )k ]O(K
)=O K
E[1̂ ∆2 ] ≤ 4E[ĥ2 ĥ2 ] = 2
n̂
n̂
n
ˆ
i,j∈I

so the second equality also follows by the Markov inequality. Next, note that
E[ηiT ηi ] ≤ 2E[v(zi )T v(zi )] + 2E[α0 (xi )2 kp(xi )k2 ] = O(K(dK + 1)).
Then we have
˜α
E[1̃ ∆
1

2

α
] ≤ 4E[h̃αT
1 h̃1 ] =

4 X
4
K(dK + 1)
E[ηiT ηj ] = E[ηiT ηi ] = O(
),
2
ñ
n̂
n
˜
i,j∈I

so the third conclusion follows from the Markov inequality. The fourth conclusion follows exactly
like the second conclusion. the fifth and sixth conclusions follow by eq. (7.3) and the triangle
inequality.
Next, note that by independence of the observations



1 X
T
ˆ 1∆
ˆ T |X, Ẑ c ] = 1̂Σ̂−1 E[ĥ1 ĥT |X]Σ̂−1 = 1̂Σ̂−1
p(x
)p(x
)
E[ε
ε
|X]
Σ̂−1
E[1̂∆
i
j
i
j
1
1

 n̂2
i,j∈Iˆ


1 X

2C
C
−1
T
2
= 1̂Σ̂
p(xi )p(xi ) E[εi |xi ] Σ̂−1 ≤ 1̂ Σ̂−1 ≤
I,
2
 n̂

n̂
n
ˆ
i∈I

32

giving the seventh conclusion.
R
Next, note that p(x)[γK (x) − γ0 (x)]F0 (dx) = 0, so that
Z
Z
2
1̂ [γ̂(x) − γ0 (x)] F0 (dx) = 1̂ [γ̂(x) − γK (x) + γK (x) − γ0 (x)]2 F0 (dx)


2
K
−2ζγ
−2ζγ
+K
,
= Op
= 1̂ δ̂ − δ + 1̂K
n
giving the eighth conclusion. The last conclusion follows similarly. Q.E.D.
Next, we give an important intermediate result:
Lemma A3: If Assumptions 1-6 are satisfied then


Z
dK K
2
−2ζm
1̂ [m(z, γ̂) − m(z, γ0 )] F0 (dz) = Op
+K
.
n
Proof: By linearity of m(z, γ) − m(z, 0), we have m(z, γ̂) − m(z, γK ) = v(z)T (δ̂ − δ). Then by
Lemma A2,
Z
Z
Z
2
2
1̂ [m(z, γ̂) − m(z, γ0 )] F0 (dz) ≤ 21̂ [m(z, γ̂) − m(z, γK )] F0 (dz) + 21̂ [m(z, γK ) − m(z, γ0 )]2 F0 (dz)
≤ 21̂(δ̂ − δ)T E[v(zi )v(zi )T ](δ̂ − δ) + O(K −2ζm )


2
dK K
−2ζm
−2ζm
. Q.E.D.
+K
≤ 2dK 1̂ δ̂ − δ + O(K
) = Op
n

The proof of the results for the doubly robust estimators will make use of a few Lemmas,
that we now state.
Lemma A4: If Assumptions 1-6 are satisfied then the hypotheses of Lemma 6 are satisfied
with
r
r
r
dK K
K
dK K
m
−ζm
−ζγ
γ
α
∆n =
+K
+ K , ∆n =
+ K −ζα .
, ∆n =
n
n
n
Proof: The first conclusion follows by Lemma A3 and the second and third by parts viii)
and ix) of Lemma A2. Q.E.D.
Lemma A5: If Assumptions 1-6 are satisfied and γ̂ℓ and α̃ℓ are computed from distinct
samples then for Σ̄ = F̄ {p(x)p(x)T }
√

nF̄ {[α̃ℓ (x) − α0 (x)][γ̂ℓ (x) − γ0 (x)]} =

33

√

ˆ T2 Σ̄∆
˜ α1 + Op (∆
¯ ∗n + ∆m
n∆
n ).

Proof: Let h̄2 = F̄ {p(x)[γK (x) − γ0 (x)]} and h̄α2 = F̄ {p(x)[αK (x) − α0 (x)]}. Note that
F̄ {[α̃ℓ (x) − α0 (x)][γ̂ℓ (x) − γ0 (x)]}

= F̄ {[p(x)T (δ̃α − δα ) + αK (x) − α0 (x)][p(x)T (δ̂ − δ) + γK (x) − γ0 (x)]}

= (δ̂ − δ)T Σ̄(δ̃α − δα ) + (δ̂ − δ)T h̄α2 + (δ̂α − δα )T h̄2 + F̄ {[αK (x) − α0 (x)][γK (x) − γ0 (x)]}.

By the Markov inequality
√

√
nF̄ {[αK (x) − α0 (x)][γK (x) − γ0 (x)]} = Op ( nK −ζγ −ζα ).

Note that
E[h̃α2 (h̃α2 )T ] =

(7.4)

1
1
E[p(xi )p(xi )T (riα )2 ] ≤ C I.
n̄
n

Therefore by Lemma A2 we have
E[{1̂(δ̂ − δ)T h̄α2 }2 |Z̄ c ] = 1̂(δ̂ − δ)T E[h̃α2 (h̃α2 )T ](δ̂ − δ) ≤ C 1̂

1
δ̂ − δ
n

2

= Op (

K
).
n2

Then by the Markov inequality it follows that
√

n(δ̂ − δ)T h̄α2 = Op (

r

K
).
n

(7.5)

dK K
).
n

(7.6)

It follows similarly that
√

T

n(δ̂α − δα ) h̄2 = Op (

Next, note that

r

ˆ T Σ̄(δ̃α − δα ) + ∆
ˆ T Σ̄∆
˜α + ∆
ˆ T Σ̄∆
˜ α.
(δ̂ − δ)T Σ̄(δ̃α − δα ) = ∆
1
2
2
2
1
Let 1̄ be the event that λmax (Σ̄) ≤ 2. Then by conclusion vii) of Lemma A2, and 1̄, 1̂, and 1̃ all
functions of X we have
ˆ T Σ̄(δ̃α − δα )}2 |X, Ẑ c ] = 1̄1̂1̃(δ̃α − δα )T Σ̄E[∆
ˆ 1∆
ˆ T |X, Ẑ c ]Σ̄(δ̃α − δα )
E[1̄1̂1̃{∆
1
1
1
1
≤ C 1̄1̃(δ̃α − δα )T Σ̄2 (δ̃α − δα ) ≤ 4C 1̃ δ̃α − δα
n
n
dK K
= Op ( 2 ).
n
Therefore we have

√

ˆ T Σ̄(δ̃α − δα ) = Op (
n∆
1

Finally, note that by the Cauchy-Schwartz inequality

r

dK K
).
n

ˆT∆
ˆ 2 ≤ 1̂2ĥT Σ̂−1 ĥ2 ≤ 2F̂ {[γK (x) − γ0 (x)]2 } = Op (K −2ζγ ).
1̂∆
2
2
34

2

(7.7)

ˆ α )T (∆
ˆ α ) = Op (K −2ζα ) so that
It follows similarly that 1̂(∆
2
2
q
q
√
√
√
T
α
T ˆ
ˆ
ˆ
˜
˜ α )T (∆
˜ α ) = Op ( nK −ζγ −ζα ).
n1̄1̂1̃∆2 Σ̄∆2 ≤ 2 n 1̂∆2 ∆2 1̃(∆
2
2

(7.8)

The conclusion then follows by eqs. (7.4), (7.5), (7.6), (7.7), (7.8), and the triangle inequality.
Q.E.D.
Proof of Theorem 2: It follows by Lemma A2 that the first hypothesis of Lemma 1 is
p
dK /n + K −ζm . Let
satisfied with ∆m
n =
Z
m̄(γ) = [m(z, γ) − m(z, γ)]F0 (dz) = E[α0 (xi )γ(xi )],
where the first equality is a definition and the second follows by Assumption 1. Then the first
conclusion of Lemma 1 holds.
Next let n = n̂ℓ and γ̂ = γ̂ℓ for some ℓ and φ(z) = α0 (x)[y − γ0 (x)]. Then it follows as in
Ichimura and Newey (2017), pp. 29 that
n


√
√
1X
1̂ n[m̄(γ̂) − β0 −
φ(zi )] = 1̂ R̂1 + R̂2 + R̂3 , R̂1 = nE[α0 (xi ){γK (xi ) − γ0 (xi )}],
n i=1

(7.9)

R̂2 =

√

n

1 X
nv Σ̂ ĥ2 , R̂3 = √
[αK (xi ) − α0 (xi )][yi − γ0 (xi )].
n i=1
T

−1

By γK (xi ) − γ0 (xi ) orthogonal to p(xi ) in the population and the Cauchy-Schwartz inequality,
√

n |E[{α0 (xi ) − αK (xi )}{γ0 (xi ) − γK (xi )}]| ≤
√
¯ ∗ ).
= O( nK −ζγ −ζα ) = O(∆
n

R̂1 =

√

n{E[{α0 (xi ) − αK (xi )}2 ]E[{γ0 (xi ) − γK (xi )}2 ]}1/2

Also,
E[R̂32 ] = E[{αK (xi ) − α(xi )}2 ε2i ] ≤ CE[{αK (xi ) − α(xi )}2 ] = O(K −2ζα ),
so by the Markov inequality,
¯ ∗ ).
R̂3 = Op (K −ζα ) = Op (∆
n
Next, note that R̂2 = R̂21 + R̂22 where R̂21 = v T ĥ2 and R̂22 =
following Assumption 4, supx |γK (x) − γ0 (x)| = O(K −ζγ ), so that

√

nv T (Σ̂−1 − I)ĥ2 . As noted

2
E[R̂21
] = v T E[p(xi )p(xi )T ri2 ]v ≤ O(K −2ζγ )v T v ≤ O(K −2ζγ )E[α0 (xi )2 ] = O(K −2ζγ ).

Then by the Markov inequality
¯ ∗ ).
R̂21 = Op (K −ζγ ) = Op (∆
n
35

Finally, note that (Σ̂−1 − I)ĥ2 = Û + Ŵ for Û and Ŵ defined in the statement of Lemma A1,
so that for any ∆ > 0 we have
2
1̂R̂22
= 1̂n · v T (Û + Ŵ )(Û + Ŵ )v ≤ 21̂n · v T (Û Û T + Ŵ Ŵ )v

≤ CK −2ζγ [ln(n)]2 + Op (n−∆+1 ),

for any C. It then follows by eq. (7.9) and the triangle inequality that
√

n

1 X
¯ ∗n + K −ζγ ln(n)).
n[m̄(γ̂) − β0 ] = √
φ(zi ) + Op (∆
n i=1

The first conclusion then follows from the second conclusion of Lemma 1. The second conclusion
p
¯ ∗ ) when dK is bounded and ζm = ζγ . Q.E.D.
follows by ∆m
=
C
K/n + K −ζγ = O(∆
n
n

Proof of Corollary 3: To prove this result it suffices to show that Assumptions 4 and 6
are satisfied in each of the examples with dK bounded and ζm = ζγ .
For the conditional covariance α0 (x) = −E[ai |xi = x]. This being Holder of order sα is a
hypothesis. Also, v(z) = a · p(x), so that
E[v(zi )v(zi )T ] = E[a2i p(xi )p(xi )T ] ≤ CE[p(xi )p(xi )T ] ≤ CI
by E[a2i |xi ] bounded. Also ζm = ζγ by
E[{m(zi , γK ) − m(zi , γ0 )}2 ] = E[a2i {γK (xi ) − γ0 (xi )}2 ] ≤ CK −2ζγ .
For the missing data mean α0 (x) = a/π0 (w) is Holder of order sα by π0 (wi ) being bounded
away from zero and Holder of order sα . Furthermore v(z) = q(w), so that by Assumption 3,
E[v(zi )v(zi )T ] = E[q(wi )q(wi )T ] ≤ CI,
and by ai bounded and π0 (wi ) bounded away from zero,
E[{m(zi , γK ) − m(zi , γ0 )}2 ] = E[{q(wi )T δ − E[yi |ai = 1, wi ]}2 ]
ai
{γK (xi ) − γ0 (xi )}2 ]
= E[
π0 (wi )
≤ CE[{γK (xi ) − γ0 (xi )}2 ] ≤ CK −2ζγ .
For the average derivative example α0 (x) = ω(x)/f0 (x) which is Holder of order sα by each
of ω(x) and f0 (x) being Holder of order sα and by f0 (x) bounded away from zero where ω(x) is
R
non zero. Furthermore v(z) = ω(x)p(x)dx, so that by Cauchy-Schwartz,
Z
Z
T
E[v(zi )v(zi ) ] = ω(x)p(x)dx ω(x)p(x)T dx
= E[α0 (xi )p(xi )]E[α0 (xi )p(xi )T ]

≤ E[α0 (xi )2 ]E[p(xi )p(xi )T ] ≤ CI.
36

Furthermore,
2

E[{m(zi , γK ) − m(zi , γ0 )} ] = {

Z

ω(x)[γK (x) − γ0 (x)]dx}2

= E[α0 (xi ){γK (xi ) − γ0 (xi )}]2

≤ E[α0 (xi )2 ]E[{γK (xi ) − γ0 (xi )}2 ] = O(K −2ζγ ). Q.E.D.
Proof of Theorem 4: The conclusion follows from Lemma 1 and Theorem 8 of Ichimura
and Newey (2017) similarly to the proof of Theorem 2 above, with the conclusion of Theorem
8 of Ichimura and Newey (2017) replacing the argument following eq. (7.9) in the proof of
Theorem 2. Q.E.D.
Proof of Theorem 5: Let λ̂ℓ (x) denote the series regression of ui = y −aTi β0 on p(xi ) in the
Iˆℓ sample. By a standard formula for instrumental variables estimation and series estimation,
√

L


1 XX
n(β̂ − β0 ) = Ĥ −1 √
[ai − α̂ℓ (xi )] yi − γ̂ℓ (xi ) − [ai − α̂ℓ (xi )]T β0
n ℓ=1 i∈I

(7.10)

ℓ

= Ĥ

−1

1
√
n

L X
X
ℓ=1 i∈Iℓ

h
i
[ai − α̂ℓ (xi )] ui − λ̂ℓ (xi )

h
i
Assume for the moment that ai is a scalar and let yi = ui . Then ℓ=1 i∈Iℓ [ai −α̂ℓ (xi )] ui − λ̂ℓ (xi ) /n
is the doubly robust estimator with m(z, γ) = a[y − γ(x)], i.e. for the expected conditional coγ
α
¯∗
variance. It then follows as in the proof of Corollary 3 that max{∆m
n , ∆n , ∆n } ≤ C ∆n . Then by
Lemmas 6 and A5, for ϕ(z) = [ai − α0 (xi )]εi ,
PL P

L

n

h
i
√ T
1 X
1 XX
¯ ∗ ) + n∆
ˆ Σ̄∆
˜ α.
√
[ai − α̂ℓ (xi )] ui − λ̂ℓ (xi ) = √
ϕ(zi ) + Op (∆
n
2
1
n ℓ=1 i∈I
n i=1
ℓ

Note that here α0 (xi ) = −E[ai |xi ] so that
h̃α1 = F̃ {v(z) − α0 (x)p(x)} = −F̃ {[a − α0 (x)]p(x)}.
Then we have
C
1 −1
T
−1
c
˜ α1 ∆
˜ αT
≤ I.
E[1̃∆
1 |X, Z̃ ] = 1̃ Σ̃ F̃ {p(x)p(x) V ar(ai |xi = x)}Σ̃
ñ
n
Therefore it follows by Lemma A2 that
C ˆT ˆ
2
c ˆ
ˆ T Σ̄∆
˜ α )2 |X, Z̃ c ] = 1̂∆
ˆ T E[1̃∆
˜ α∆
˜ αT
E[1̂1̃(∆
2
1
2
1 1 |X, Z̃ ]∆2 ≤ 1̂ ∆2 ∆2 = op (K/n ).
n
37

Then by the Markov inequality
√

ˆ T Σ̄∆
˜ α = op
n∆
2
1

r

K
n

!

¯ ∗ ).
= O p (∆
n

Consequently we have
L

n

h
i
1 XX
1 X
¯ ∗ ).
√
[ai − α̂ℓ (xi )] ui − λ̂ℓ (xi ) = √
ϕ(zi ) + Op (∆
n
n ℓ=1 i∈I
n i=1

(7.11)

ℓ

Next, note that
F̄ {[a − α̃ℓ (x)][a − α̂ℓ (x)]} = F̄ {[a − α0 (x) + α0 (x) − α̃ℓ (x)][a − α0 (x) + α0 (x) − α̂ℓ (x)]}
= F̄ {[a − α0 (x)]2 + [a − α0 (x)][α0 (x) − α̃ℓ (x)]

+ [a − α0 (x)][α0 (x) − α̂ℓ (x)] + [α0 (x) − α̃ℓ (x)][α0 (x) − α̂ℓ (x)]}
Note that by Lemma A3 and E[a2i |xi ] bounded,
Z
1̃
2
[a − α0 (x)]2 [α0 (x) − α̃(x)]2 F0 (dz)
1̃E[(F̄ {[a − α0 (x)][α0 (x) − α̃ℓ (x)]}) |Z̃] =
n̄
Z
1̃
E[a2i |xi = x][α0 (x) − α̃(x)]2 F0 (dz)
≤C
n
 

Z
1̃
1 K
2
−2ζγ
≤C
[α0 (x) − α̃(x)] F0 (dz) = Op
+K
.
n
n n
so that by the Markov inequality it follows that
¯ ∗n ).
F̄ {[a − α0 (x)][α0 (x) − α̃ℓ (x)]} = Op (∆

(7.12)

It follows similarly that
¯ ∗ ).
F̄ {[a − α0 (x)][α0 (x) − α̂ℓ (x)]} = Op (∆
n

(7.13)

Also, by the Cauchy-Schwartz inequality
1̂1̃ F̄ {[α0 (x) − α̃ℓ (x)][α0 (x) − α̂ℓ (x)]} ≤ (1̃F̄ {[α0 (x) − α̃ℓ (x)]2 })1/2 (1̂F̄ {[α0 (x) − α̂ℓ (x)]2 })1/2 .
Also,
2

E[1̃F̄ {[α0 (x) − α̃ℓ (x)] }|Z̃] = 1̃

Z

2

[α̃ℓ (x) − α0 (x)] F0 (dx) = Op



K
+ K −2ζγ
n



,

so that 1̃F̄ {[α0 (x)− α̃ℓ (x)]2 } = Op (K/n+K −2ζγ ). It follows similarly that 1̂F̄ {[α0 (x)− α̂ℓ (x)]2 } =
Op (K/n + K −2ζγ ), so that
¯ ∗)
= F̄ {[α0 (x) − α̃ℓ (x)][α0 (x) − α̂ℓ (x)]} = Op (∆
n
38

(7.14)

Also, note that by E[kai k4 ] < ∞,
2

2

F̄ {[a − α0 (x)] } = E[{a − α0 (x)} ] + Op



1
√
n



¯ ∗ ).
= E[{ai − α0 (xi )}2 ] + Op (∆
n

It then follows by eqs. (7.12), (7.13), (7.14) and the triangle inequality that
¯ ∗ ).
F̄ {[a − α̃ℓ (x)][a − α̂ℓ (x)]} = E[{ai − α0 (xi )}2 ] + Op (∆
n
P P
Applying this argument to each element of Ĥ = Lℓ=1 i∈Iℓ [ai − α̃ℓ (xi )][ai − α̂ℓ (xi )]T /n and each
¯ ∗n ). It then follows by a standard
group of observations Iℓ and summing up gives Ĥ = H + Op (∆
argument and nonsingularity of H that
¯ ∗ ).
Ĥ −1 = H −1 + Op (∆
n
Finally, it follows from eqs. (7.10), (7.11), (7.15) and from
√

n

n(β̂ − β0 ) = [H

−1

(7.15)
Pn

i=1

√
ϕ(zi )/ n = Op (1) that
n

X
X
∗
−1 1
¯
¯ ∗ ). Q.E.D.
¯ ∗ )][ √1
√
ϕ(z
)
+
O
(
∆
)]
=
H
ϕ(zi ) + Op (∆
+ O p (∆
i
p
n
n
n
n i=1
n i=1

√ ˆT ˜α
Proof of Theorem 7: By Lemmas 6 and A5 it suffices to show that 1̄1̂1̃ n∆
2 Σ̄∆1 =
m
Op (∆n ). Note that
ˆ 2 = 1̂ĥ2 + 1̂Û + 1̂Ŵ .
1̂∆
By E[ĥ2 ĥT2 ] ≤ Cn−1 K −2ζγ I and Lemma A2 iii) we have
2
 T
 T
 √
T
α
c
α
T
α
˜
˜
˜
˜ α Σ̄E[ĥ2 ĥT ]Σ̄∆
˜α
E[ 1̄1̂1̃ nĥ2 Σ̄∆1 |Ẑ ] = n1̄1̃ ∆1 Σ̄E[1̂ĥ2 ĥ2 ]Σ̄∆1 ≤ n1̄1̃ ∆
1
2
1


 T
1−2ζγ
2
˜ α Σ̄2 ∆
˜ α = Op (1 + dK )K
= Op ((∆m
≤ CK −2ζγ 1̄1̃ ∆
n ) ).
1
1
n
Also, by the first conclusion of Lemma A1 and by Lemma A2 iii),
 √
2
 T
 T
˜ α |Ẑ c ] = n1̄1̃ ∆
˜ α Σ̄E[1̂Û Û T ]Σ̄∆
˜ α ≤ n1̄1̃ ∆
˜ α Σ̄E[Û Û T ]Σ̄∆
˜α
E[ 1̄1̂1̃ nÛ T Σ̄∆
1
1
1
1
1


 T
1−2ζγ
2
(1
+
d
)K
[ln(n)]
K
2
−2ζγ
2
α
2
α
˜ = Op
˜
= Op ((∆m
≤ CK
ln(n) 1̄1̃ ∆
Σ̄ ∆
n ) ).
1
1
n
Also by the second conclusion of Lemma A1 and Lemma A2 iii), for ∆ > 0 large enough,
p
√ T
ˆ Σ̄∆
˜ α = Op (n(1/2)−∆ (1 + dK )/n)) = Op (∆m ).
1̄1̂1̃ n∆
2
1
n

The conclusion then follows by the Markov and triangle inequalities. Q.E.D.

39

√ ˆT ˜α
Proof of Theorem 8: By Lemmas 6 and A5 it suffices to show that 1̄1̂1̃ n∆
2 Σ̄∆1 =
m
˜
Op (∆n + ∆n ). Note that
√ T
√
ˆ Σ̄∆
˜ α = T1 + T2 + T3 , T1 = 1̄1̂1̃ nĥT Σ̄∆
˜ α,
1̄1̂1̃ n∆
2
1
2
1
√ T
α
ˆ (I − Σ̂)Σ̄h̃ ,
T2 = 1̄1̂1̃ n∆
2
1
√ T
ˆ (I − Σ̂)Σ̄(I − Σ̃)∆
˜ α.
T3 = 1̄1̂1̃ n∆
2
1
By Lemma A2 iii),
˜ α )T ∆
˜ α = Op (K −2ζγ (∆m )2 ),
˜ α )T Σ̄E[ĥ2 ĥT ]Σ̄∆
˜ α ≤ CK −2ζγ 1̃(∆
E[T12 |Ẑ c ] ≤ 1̄1̃n(∆
1
1
1
2
1
n
so by the Markov inequality, T1 = Op (∆m
n ). By Lemma A2 ii),
 T
√ T
ˆ (I − Σ̂)E[h̃α h̃α ](I − Σ̂)∆
ˆ 2 ≤ CdK ∆
ˆ T (I − Σ̂)2 ∆
ˆ2
E[T22 |Z̃ c ] ≤ 1̂ n∆
2
1
1
2
= Op ((1 + dK )

K 1−2ζγ K ln(K)
).
n
n

¯ ∗ + ∆m ).
Note that by the Markov inequality and K ln(K)/n −→ 0 it follows that T2 = Op (∆
n
n
Finally, by the Caucy-Schwartz inequality and Lemma A2,
r
K 3 ln(K)(1 + dK ) (1/2)−ζγ
˜ n ).
T3 = Op (
K
) = O p (∆
n3
The conclusion then follows by the triangle inequality. Q.E.D.

Acknowledgements
We appreciate the hospitality of the Cowles Foundation where much of the work for this paper
was accomplished. We also appreciate the comments of M. Cattaneo, X. Chen, M. Jansson and
seminar participants at UCL.
REFERENCES
Athey, S., G. Imbens, and S. Wager (2017): ”Efficient Inference of Average Treatment Effects in High Dimensions via Approximate Residual Balancing,” Journal of the Royal Statistical
Society, Series B, forthcoming.
Ayyagari, R. (2010): Applications of Influence Functions to Semiparametric Regression Models,
Ph.D. Thesis, Harvard School of Public Health, Harvard University.
Belloni, A., V. Chernozhukov, D. Chetverikov, K. Kato (2015): “Some New Asymptotic
Theory for Least Squares Series: Pointwise and Uniform Results,” Journal of Econometrics 186,
345–366.
40

Bickel, P.J. (1982): ”On Adaptive Estimation,” Annals of Statistics 10, 647-671.
Bickel, P. and Y. Ritov (1988): ”Estimating Integrated Squared Density Derivatives: Sharp
Best Order of Convergence Estimates,” Sankhya: The Indian Journal of Statistics, Series A 50,
381–393.
Blomquist, S. and M. Dahlberg (1999): ”Small Sample Properties of LIML and Jackknife IV
Estimators: Experiments with Weak Instruments,” Journal of Applied Econometrics 14, 69–88.
Cattaneo, M.D., and M. Farrell (2013): ”Optimal Convergence Rates, Bahadur Representation, and Asymptotic Normality of Partitioning Estimators,” Journal of Econometrics 174,
127-143.
Cattaneo, M.D., and M. Jansson (2017): ”Kernel-Based Semiparametric Estimators: Small
Bandwidth Asymptotics and Bootstrap Consistency,” Econometrica, forthcoming.
Cattaneo, M.D., M. Jansson, and X. Ma (2017): ”Two-step Estimation and Inference with
Possibly Many Included Covariates,” working paper, Michigan.
Chernozhukov, V., J.C. Escanciano, H. Ichimura, W.K. Newey, J.M. Robins (2016):
”Locally Robust Semiparametric Estimation,” arXiv 1608.00033.
Chernozhukov, V., D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W.K. Newey,
J.M. Robins (2017): ”Double/Debiased Machine Learning for Treatment and Structural Parameters,” Econometrics Journal, forthcoming.
Donald, S.G. and W.K. Newey (1994): “Series Estimation of Semilinear Models,” Journal of
Multivariate Analysis 50, 30-40.
Firpo, S. and C. Rothe (2016): ”Semiparametric Two-Step Estimation Using Doubly Robust
Moment Conditions,” working paper.
Gine, E. and R. Nickl (2008): ”A Simple Adaptive Estimator of the Integrated Square of a
Density,” Bernoulli 14, 47–61.
Hahn, J. (1998): ”On the Role of the Propensity Score in Efficient Semiparametric Estimation
of Average Treatment Effects,” Econometrica 66, 315-331.
Hirano, K., G. Imbens, and G. Ridder (2003): ”Efficient Estimation of Average Treatment
Effects Using the Estimated Propensity Score,” Econometrica 71: 1161–1189.
Hirschberg, D.A and S. Wager (2017): ”Balancing Out Regression Error: Efficient Treatment
Effect Estimation without Smooth Propensities,” arXiv:1712.00038.
Ichimura, H. and W.K. Newey (2017): ”The Influence Function of Semiparametric Estimators,” CEMMAP working paper CWP06/17.
Imbens G., J. Angrist, A. Krueger (1999): ”Jackknife Instrumental Variables Estimation,”
Journal of Applied Econometrics 14, 57-67.
Kandasamy, K., A. Krishnamurthy, B. Poczos, L. Wasserman, J. Robins (2015) ”Non41

parametric von Mises Estimators for Entropies, Divergences and Mutual Informations,” Advances in Neural Information Processing Systems 28 (NIPS 2015).
Laurent, B. (1996): ”Efficient Estimation of Integral Functionals of a Density,” Annals of Statistics 24, 659-681.
Mukherjee, R., W.K. Newey, J.M. Robins (2017): ”Semiparametric Efficient Empirical
Higher Order Influence Function Estimators,” arXiv:1705.07577.
Newey, W.K. (1994): ”The Asymptotic Variance of Semiparametric Estimators,” Econometrica
62, 1349-1382.
Newey, W.K. (1997): “Convergence Rates and Asymptotic Normality for Series Estimators,”
Journal of Econometrics 79, 147-168.
Newey, W.K., F. Hsieh, and J.M. Robins (1998): “Undersmoothing and Bias Corrected
Functional Estimation,” MIT Dept. of Economics working paper 72, 947-962.
Newey, W.K., F. Hsieh, and J.M. Robins (2004): “Twicing Kernels and a Small Bias Property
of Semiparametric Estimators,” Econometrica 72, 947-962.
Powell, J.L., J.H. Stock, and T.M. Stoker (1989): ”Semiparametric Estimation of Index
Coefficients,” Econometrica 57, 1403-1430.
Robins, J.M. and A. Rotnitzky (1995): ”Semiparametric Efficiency in Multivariate Regression Models with Missing Data,” Journal of the American Statististical Association 90, 122–129.
Robins, J.M., A. Rotnitzky, and M. van der Laan (2000): ”Comment on ’On Profile
Likelihood’ by S. A. Murphy and A. W. van der Vaart,” Journal of the American Statistical
Association 95, 431-435.
Robins, J., M. Sued, Q. Lei-Gomez, and A. Rotnitzky (2007): ”Comment: Performance of
Double-Robust Estimators When Inverse Probability’ Weights Are Highly Variable,” Statistical
Science 22, 544–559.
Robins, J.M., E.T. Tchetgen, L. Li, A. van der Vaart (2009): ”Semiparametric Minimax
Rates,” Electronic Journal of Statistics 3, 1305–1321.
Robins, J.M., L. Li, E. Tchetgen, and A. van der Vaart (2008) ”Higher Order Influence
Functions and Minimax Estimation of Nonlinear Functionals,” in IMS Collections Vol. 2, Probability and Statistics: Essays in Honor of David A. Freedman, D. Nolan and T. Speed (eds.),
Beachwood, Ohio: Institute of Mathematical Statistics, 335-421.
Robins, J.M, P. Zhang, R. Ayyagari, R. Logan, E. Tchetgen, L. Li, T. Lumley, A.
van der Vaart A, HEI Health Review Committee (2013): ”New Statistical Approaches
to Semiparametric Regression with Application to Air Pollution Research,” Research Report
Health Eff Instm 175:3-129.
Robins, J.M., L. Li, R. Mukherjee, E. Tchetgen, A. van der Vaart (2017): ”Minimax
42

Estimation of a Functional on a Structured High Dimensional Model,” Annals of Statistics,
forthcoming.
Rotnitzky, A. and J.M. Robins (1995): ”Semi-parametric Estimation of Models for Means and
Covariances in the Presence of Missing Data,” Scandinavian Journal of Statistics 22, 323–333.
Rudelson, M. (1999): ”Random Vectors in the Isotropic Position,” Journal of Functional Analysis 164, 60-72.
Scharfstein D.O., A. Rotnitzky, and J.M. Robins (1999): Rejoinder to “Adjusting For
Nonignorable Drop-out Using Semiparametric Non-response Models,” Journal of the American
Statistical Association 94, 1135-1146.
Stoker, T. (1986): ”Consistent Estimation of Scaled Coefficients,” Econometrica 54, 1461-1482.

43

