Title: Learning from a lot: Empirical Bayes in high-dimensional prediction settings,
Abstract: Empirical Bayes is a versatile approach to `learn from a lot' in two ways:
first, from a large number of variables and second, from a potentially large
amount of prior information, e.g. stored in public repositories. We review
applications of a variety of empirical Bayes methods to several well-known
model-based prediction methods including penalized regression, linear
discriminant analysis, and Bayesian models with sparse or dense priors. We
discuss `formal' empirical Bayes methods which maximize the marginal
likelihood, but also more informal approaches based on other data summaries. We
contrast empirical Bayes to cross-validation and full Bayes, and discuss hybrid
approaches. To study the relation between the quality of an empirical Bayes
estimator and $p$, the number of variables, we consider a simple empirical
Bayes estimator in a linear model setting.
We argue that empirical Bayes is particularly useful when the prior contains
multiple parameters which model a priori information on variables, termed
`co-data'. In particular, we present two novel examples that allow for co-data.
First, a Bayesian spike-and-slab setting that facilitates inclusion of multiple
co-data sources and types; second, a hybrid empirical Bayes-full Bayes ridge
regression approach for estimation of the posterior predictive interval.