Title: Reducing variance in importance-weighted cross-validation under covariate shift,
Abstract: Covariate shift classification problems can in principle be tackled by
importance-weighting of training samples. However, the sampling variance of the
risk estimator is often scaled up dramatically by employing such weighting. One
of the consequences of this is that during cross-validation -- when the
importance-weighted risk is repeatedly estimated -- suboptimal hyperparameter
estimates are produced. We study the sampling variance of the
importance-weighted risk estimator as a function of the width of the source
distribution. We show that introducing a control variate can reduce its
sampling variance, which leads to improved regularization parameter estimates
when the training data is smaller in scale than the test data.