Title: Orthogonal Statistical Learning,
Abstract: We provide excess risk guarantees for statistical learning in the presence of
an unknown nuisance component. We analyze a two-stage sample splitting
meta-algorithm that takes as input two arbitrary estimation algorithms: one for
the target model and one for the nuisance model. We show that if the population
risk satisfies a condition called Neyman orthogonality, the impact of the first
stage error on the excess risk bound achieved by the meta-algorithm is of
second order. Our general theorem is agnostic to the particular algorithms used
for the target and nuisance and only makes an assumption on their individual
performance. This enables the use of a plethora of existing results from
statistical learning and machine learning literature to give new guarantees for
learning with a nuisance component. Moreover, by focusing on excess risk rather
than parameter estimation, we can give guarantees under weaker assumptions than
in previous works and accommodate the case where the target parameter belongs
to a complex nonparametric class. When the nuisance and target parameters
belong to arbitrary classes, we characterize conditions on the metric entropy
such that oracle rates---rates of the same order as if we knew the nuisance
model---are achieved. We also analyze the rates achieved by specific estimation
algorithms such as variance-penalized empirical risk minimization, neural
network estimation and sparse high-dimensional linear model estimation. We
highlight the applicability of our results via four applications of primary
importance: 1) heterogeneous treatment effect estimation, 2) offline policy
optimization, 3) domain adaptation, and 4) learning with missing data.