Title: On Asymptotic Properties of Hyperparameter Estimators for Kernel-based Regularization Methods,
Abstract: The kernel-based regularization method has two core issues: kernel design and
hyperparameter estimation. In this paper, we focus on the second issue and
study the properties of several hyperparameter estimators including the
empirical Bayes (EB) estimator, two Stein's unbiased risk estimators (SURE) and
their corresponding Oracle counterparts, with an emphasis on the asymptotic
properties of these hyperparameter estimators. To this goal, we first derive
and then rewrite the first order optimality conditions of these hyperparameter
estimators, leading to several insights on these hyperparameter estimators.
Then we show that as the number of data goes to infinity, the two SUREs
converge to the best hyperparameter minimizing the corresponding mean square
error, respectively, while the more widely used EB estimator converges to
another best hyperparameter minimizing the expectation of the EB estimation
criterion. This indicates that the two SUREs are asymptotically optimal but the
EB estimator is not. Surprisingly, the convergence rate of two SUREs is slower
than that of the EB estimator, and moreover, unlike the two SUREs, the EB
estimator is independent of the convergence rate of $\Phi^T\Phi/N$ to its
limit, where $\Phi$ is the regression matrix and $N$ is the number of data. A
Monte Carlo simulation is provided to demonstrate the theoretical results.