Title: Sparsity information and regularization in the horseshoe and other shrinkage priors,
Abstract: The horseshoe prior has proven to be a noteworthy alternative for sparse
Bayesian estimation, but has previously suffered from two problems. First,
there has been no systematic way of specifying a prior for the global shrinkage
hyperparameter based on the prior information about the degree of sparsity in
the parameter vector. Second, the horseshoe prior has the undesired property
that there is no possibility of specifying separately information about
sparsity and the amount of regularization for the largest coefficients, which
can be problematic with weakly identified parameters, such as the logistic
regression coefficients in the case of data separation. This paper proposes
solutions to both of these problems. We introduce a concept of effective number
of nonzero parameters, show an intuitive way of formulating the prior for the
global hyperparameter based on the sparsity assumptions, and argue that the
previous default choices are dubious based on their tendency to favor solutions
with more unshrunk parameters than we typically expect a priori. Moreover, we
introduce a generalization to the horseshoe prior, called the regularized
horseshoe, that allows us to specify a minimum level of regularization to the
largest values. We show that the new prior can be considered as the continuous
counterpart of the spike-and-slab prior with a finite slab width, whereas the
original horseshoe resembles the spike-and-slab with an infinitely wide slab.
Numerical experiments on synthetic and real world data illustrate the benefit
of both of these theoretical advances.