arXiv:1803.10749v2 [stat.CO] 12 Apr 2018

On Model Selection with Summary Statistics
Erlis Ruli
ruli@stat.unipd.it
Department of Statistical Sciences, University of Padova, Italy

April 13, 2018

Abstract
Recently, many authors have cast doubts on the validity of ABC model choice. It has been shown
that the use of sufficient statistic in ABC model selection leads, apart from few exceptional cases in which
the sufficient statistic is also cross-model sufficient, to unreliable results. In a single model context and
given a sufficient summary statistic, we show that it is possible to fully recover the posterior normalising
constant, without using the likelihood function. The idea can be applied, in an approximate way, to more
realistic scenarios in which the sufficient statistic is not unavailable but a “good” summary statistic for
estimation is available.

1

Background

Approximate Bayesian Computation (ABC) is a useful tool for Bayesian (see, e.g., Marin et al., 2012) or
frequentist (see, e.g., Rubio and Johansen, 2013) inferences when the likelihood function is mathematically
or computationally unavailable. The successfulness of the ABC method relies on a careful choice of: the
summary statistics s(·), the distance metric ρ(·, ·) and tolerance level ǫ > 0; with the summary statistic
playing arguably the most crucial role.
1

To set the notation, let y = (y1 , . . . , yn ) be n realisations of the random variable Y ∼ Pθ , with θ ∈ Θ ⊆ Rd ,
d ≥ 1. Furthermore, let π(θ) be a prior distribution for θ, for simplicity assumed to be proper, let L(θ) be
the likelihood function based on model Pθ and data y and let π(θ|y) ∝ L(θ)π(θ) be the posterior distribution,
with normalising constant p(y) =

R

θ∈Θ

L(θ)π(θ) dθ. The Bayes factor (BF), the standard Bayesian solution

for model selection, involves the posterior normalising constants of the models under comparison. Thus, if
the likelihood for a single model is unavailable, the BF cannot be computed.
The ABC machinery comes equipped with an ABC model choice (ABC-MC) algorithm which works as
follows (Grelaud et al., 2009).
Result: A sample of model indices (j (1) , . . . , j (N ) )
for i = 1 → N do
repeat
1

draw j ∗ from π(M = j)

2

draw θj∗∗ from πj ∗ (θj ∗ )

3

draw y ∗ from pj ∗ (·; θj∗∗ )
until ρ(s(y ∗ ), s(y)) ≤ ǫ;

4

set j (i) = j ∗
end
Algorithm 1: ABC model choice (ABC-MC) sampler.

where ǫ is the threshold value and j = 1, . . . , m is the model index. The N -vector of indices j produced
form Algorithm 1 can be used, in principle, to compute posterior model probabilities and BFs.
Recently, many authors have cast doubts on the validity of the ABC model choice procedure (see, e.g.,
Marin et al., 2014; Robert et al., 2011). For instance, suppose y is a vector of counts and we wish to choose
between the Poisson and the Geometric model. In both cases, with ABC we can obtain (almost) the exact
posterior by using

P

i

yi as the summary statistic, since the latter is sufficient under both models. However,

the BF obtained with ABC-MC in this case converges asymptotically to a positive constant (Robert et al.,
2011). With the particular exception of Gibbs random fields (Grelaud et al., 2009), the BF obtained with

2

ABC-MC misses the exact BF by some unknown function of the data. Marin et al. (2014) give theoretical
conditions under which the summary statistic gives valid posterior model probabilities or BFs under the ABC
model choice framework.
Clearly, the issue is with the summary statistic s(·). Even though it can be sufficient for the parameters,
it is the cross-model sufficiency that plays the crucial role here, e.g., the summary statistic must be sufficient
for the models themselves (see also Marin et al., 2014). Finding cross-model sufficient statistics in practice
is impossible, and some efforts have been spent on constructing summary statistics for ABC model selection
(see, e.g., Barnes et al., 2012). However, at the best of our knowledge, the choice of summary statistics for
ABC model selection is still an open problem. Last but not the least, summary statistics for ABC model
selection are notoriously a bad choice for ABC posterior sampling (C.P. Robert, personal communication).
In Section 2 we show how the marginal likelihood can be approximated by using the sufficient summary
statistic and ABC. In Section 3 we conclude by pointing to future developments.

2

Marginal likelihood from sufficient statistics

Let us focus on a single model Pθ , and suppose that s(·) is sufficient for θ. By the sufficiency principle we
have that
π(θ|y) =

L(θ; y)π(θ)
= π(θ|s(y)) .
p(y)

From this we see that
p(y) =

L(θ; y)π(θ)
L(θ; y)π(θ)
=
.
π(θ|y)
π(θ|s(y))

To approximate p(y) we propose to approximate π(θ|s(y)) via ABC, and L(θ; y) by simulation as follows
(see Algorithm 2).

3

Result: approximation of p(s(y)).
1

use ABC to get a posterior sample and compute its mean θ̂;

2

compute π̂(θ̂|s(y)), the ordinate of the posterior at θ = θ̂;

3

draw a large sample of y ∗ from p(·; θ̂) and;

4

compute p̂(y; θ̂) = L̂(θ̂; y) an approximation of p(y; θ̂);

5

set p̂(y) =

p̂(y;θ̂)π(θ̂)
.
π̂(θ̂|y)

Algorithm 2: Marginal likelihood from summary statistic.
Steps 2 and 4 of Algorithm 2 can be performed by any density estimation method; in Step 5 we only need
to generate a (possibly) large sample of data from the model under θ̂, a fixed value of θ.

A toy example: the Poisson model
Suppose Y ∼ Po(λ), and a priori λ ∼ Exp(1). The marginal likelihood in case is
p(y) =

Z

0

∞

e

−λ(n+1)

Pn

λ

i=1

yi /

n
Y

i=1

yi !

!

P
Γ( i yi + 1)
P
dλ = Q
i y+1
i yi (n + 1)

(2.1)

As a numerical example, consider y = (2, 3, 1, 1, 2, 1, 3, 1, 3, 1), which are realisations of 10 random draws
from Po(2) distribution. Figure 1 shows on the left side the histogram of the ABC posterior against the exact
posterior (solid line). The ABC posterior is approximated by 1 × 104 final samples with ǫ = 0.001, where ρ(·)
is the Euclidean distance among the total number of counts. The right side of Figure 1 shows the scatter plot
of the logarithm of the marginal likelihood obtained from ABC against the logarithm of the exact marginal
likelihood (2.1), in 50 random samples of size 10 from Po(2). The approximate marginal likelihoods obtained
from ABC with the sufficient statistic and the exact marginal likelihoods are virtually indistinguishable.

3

Conclusion

Obviously, in realistic scenarios sufficient summary statistics are unavailable. However, if we have a set of
judiciously chosen summary statistics which give provably valid inference on the parameters of interest, then
the idea can still be usefully applied. The more close to sufficiency is the summary statistic the more close
4

−16

1.0

−18
−20

Exact marginal likelihood

0.8
0.4

0.6

Posterior density

−22

0.2
0.0
0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

−22

λ

−20

−18

−16

ABC marginal likelihood

Figure 1: Left: ABC (histogram) against the exact posterior (line). Right: scatter plot of the log-marginal
likelihoods obtained with ABC versus the exact log-marginal likelihoods in 50 random samples, each of size
10.

to the exact value is the proposed approximation.

Acknowledgement
This work was presented at the BayesComp 2018 conference (26–29 March, Barcelona) and is partially
supported by University of Padova (Progetti di Ricerca di Ateneo 2015, CPDA153257) and by PRIN 2015
(grant 2015EASZFS 003).

References
Barnes, C. P., Filippi, S., Stumpf, M. P. H., and Thorne, T. (2012). Considerate approaches to constructing
summary statistics for ABC model selection. Statistics and Computing, 22(6):1181–1197.
Grelaud, A., Robert, C. P., Marin, J.-M., Rodolphe, F., Taly, J.-F., et al. (2009). ABC likelihood-free
methods for model choice in Gibbs random fields. Bayesian Analysis, 4:317–335.
5

Marin, J.-M., Pillai, N. S., Robert, C. P., and Rousseau, J. (2014). Relevant statistics for bayesian model
choice. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 76:833–859.
Marin, J.-M., Pudlo, P., Robert, C. P., and Ryder, R. J. (2012). Approximate Bayesian computational
methods. Statistics and Computing, 22:1167–1180.
Robert, C. P., Cornuet, J.-M., Marin, J.-M., and Pillai, N. S. (2011). Lack of confidence in approximate
bayesian computation model choice. Proceedings of the National Academy of Sciences, 108:15112–15117.
Rubio, F. J. and Johansen, A. M. (2013). A simple approach to maximum intractable likelihood estimation.
Electronic Journal of Statistics, 7:1632–1654.

6

