Principled Bayesian Minimum Divergence Inference
Jack Jewson1,* , Jim Q Smith1 , and Chris Holmes2
1

Department of Statistics, University of Warwick, Coventry, CV4 7AL
2
University of Oxford, Oxford, OX1 3PA
*
Correspondence address J.E.Jewson@warwick.ac.uk

arXiv:1802.09411v1 [math.ST] 26 Feb 2018

February 2018
Abstract
When it is acknowledged that all candidate parameterised statistical models are misspecified
relative to the data generating process, the decision maker must concern themselves with the KLdivergence minimising parameter in order to maintain principled statistical practice Walker (2013).
However, it has long been known that the KL-divergence places a large weight on correctly capturing
the tails of the data generating process. As a result traditional inference can be very non-robust. In
this paper we advance recent methodological developments in general Bayesian updating (Bissiri,
Holmes, & Walker, 2016) to propose a statistically well principled Bayesian updating of beliefs targeting the minimisation of any statistical divergence. We improve both the motivation and the statistical foundations of existing Bayesian minimum divergence estimation (Hooker and Vidyashankar
(2014), Ghosh and Basu (2016)), for the first time allowing the well principled Bayesian to target
predictions from the model that are close to the data generating process in terms of some alternative
divergence measure to the KL-divergence. We argue that defining this divergence measure forms an
important, subjective part of any statistical analysis. We here illustrate our method a broad array
of divergence measures. We then compare the performance of the different divergence measures for
conducting simple inference tasks on both simulated and real data sets, and discuss then how our
methods might apply to more complicated, high dimensional models.

1

Introduction

In the M -closed world, where a fitted model class is implicitly assumed to contain a model exactly
corresponding to the data generating mechanism, Bayesian updating is highly compelling. However,
modern statisticians are increasingly acknowledging that their inference is taking place in the Mopen world (Bernardo & Smith, 2001). Within this framework we acknowledge that any class
of models we choose is unlikely to contain the data generating mechanism precisely. Any model
within the class is then at best an approximate description of our own beliefs and the underlying
real world process. In this situation it is no longer possible to learn about the parameter which
generated the data and a statistical divergence measure must be specified between the model and the
data generating process in order to define the parameter targeted by the inference Walker (2013).
Remarkably in the M -open world standard Bayesian updating can be seen as a method which
learns a model by minimising the predictive Kullback-Leibler (KL)-divergence to the data generating
process (Walker, 2013). See Bissiri et al. (2016) for a simple demonstration. Therefore traditional
Bayesian updating is still a well principled method for belief updating provided the decision maker
concerns themselves with the parameter of the model that is closest to the data generating process
in terms of KL-divergence (Walker, 2013). Recall the KL-divergence of functioning model f from

1

data generating density g is given by.
Z
dKL (g||f ) =

log(

g(x)
)dG(x).
f (x; θ)

(1)

Sadly it is also well known that when the KL-divergence is large, which is almost inevitable in
high dimensional problems, the KL-divergence minimising models can give a very poor approximation if our main interest is in getting the central part of the model well estimated. This is because
the KL-divergence gives disproportionate consideration to correctly specifying the tails of any data
generating process. Therefore the KL-divergence minimising predictive distribution is particularly
non-robust to misspecifications in the tails of the data generating process.
As a result any DM may actually want to explicitly target more robust divergences, a framework
commonly known as Minimum Divergence Estimation (MDE), see Basu, Shioya, and Park (2011).
Minimum divergence estimation is of course a well-developed field by frequentists, with Bayesian
contributions coming more recently. However when the realistic assumption of being in the M -open
world is considered the currently proposed Bayesian minimum divergence posteriors fail to fully
appreciate the principled justification and motivation required to produce a coherent updating of
beliefs. A Bayesian cannot therefore make principled inference using currently proposed methods
in the M -open setting, except in a way that Miller and Dunson (2015) describe as “tend(ing) to
be either limited in scope, computationally prohibitive, or lacking a clear justification”. In order to
make principled inference it appears as though the DM must currently concern themselves with the
KL-divergence. However in this paper we remove this reliance upon the KL-divergence by providing
a justification for Bayesian updating minimising alternative divergences, both theoretically and
ideologically. Our updating of beliefs does not produce an approximate or pseudo posterior, but
uses general Bayesian updating (Bissiri et al., 2016) to produce the coherent posterior beliefs of a
decision maker who wishes to produce predictions from a model that provide an explanation of the
data that is as good as possible in terms of some pre-specified divergence measure.
We are not the first to consider a principled alternative to Bayesian updating when it is difficult
to fully specify a model for the data generating process. Bayes linear methods Goldstein (1999)
only require the subjective specifications of expectations and covariances for various quantities the
DM is well informed about and interested in. Alternatively our method tackles the same problem
by being robust to routine assumptions making a full probability specification a much less strenuous
task.
In Section 2 of this article we review the currently available Bayesian minimum divergence
estimation techniques and introduce general Bayesian updating Bissiri et al. (2016). Then in Section
3 we identify the inadequacies in the justification provided by the currently available Bayesian
MDE methods and use general Bayesian updating to address these inadequacies. In Section 4
we examine the variety of divergences the Bayesian could wish to minimise. Lastly in Section 5
we demonstrate that under several model misspecifications superior robustness can be obtained
by minimising alternative divergences to the KL-divergence and that when the data generating
distribution is contained within the model class, these methods can be shown not to be too inferior
to standard approaches.

1.1

Solutions to the M -open problem

When the decision maker (DM) or practitioner acknowledges that they are in the M -open world,
they have several options currently available to them:
1. Proceed as though the model class will contain the data generating distribution and conduct
sensitivity analysis on the results.
2. Modify the model class in order to improve its robustness.

2

3. Abandon the given parametric class and conduct some form of routine machine learning exercise.
To this we add:
4. Acknowledge that the model class is only approximate, but is the best available, and seek to
fit the model parameters in a way that are most useful to the decision maker.
Method 1 is how Box (1980) recommends approaching parametric model estimation and is the
most popular approach amongst statisticians. Although it is acknowledged that the model is only an
approximation of the data generating process, Bayesian statistics is applied as though the statistician
believes the model to be correct. The results are then checked to examine how sensitive these are
to the approximations made. Authors Berger et al. (1994) provide a thorough review while Watson,
Holmes, et al. (2016) consider this in a decision focused manner.
Method 2 corresponds to the classical robustness approach. Within this approach, one model
within the parametric class is substituted for a model providing heavier tails (Berger et al., 1994).
Alternatively different estimators, for example M-estimators Huber and Ronchetti (1981) Hampel,
Ronchetti, Rousseeuw, and Stahel (2011), see Greco, Racugno, and Ventura (2008) for a Bayesian
analogue, are used instead of those justified by the model class. Bayes linear methods Goldstein
(1999), requiring many fewer belief statements than a full probability specification, form a special
subclass of these techniques.
The last possibility is to appear to abandon the model. In statistics this constitutes adopting empirical likelihood methods (Owen, 1991); considering the decision focused general Bayesian update
(Bissiri et al., 2016); or appealing to statistical learning methods such as Neural Networks, Gaussian
Processes, Support vector machines or Bayesian non-parametric. Except when the supporting data
sets are heterogeneous and massive, such methods simply substitute the assumptions and structure associated with the model class, for a much broader class of models with less transparent
assumptions.
Though alternatives 2 and 3 can be shown to be very powerful in certain scenarios, it is our
opinion that within the context described above they are far from optimal. In an applied statistical
problem the model(s) represents the DM’s best guess at the data generating process. The model
provides the only opportunity to input not only structural, but quantitative expert judgements
about the domain, something that is often critical to a successful analysis - see Lazer, Kennedy,
King, and Vespignani (2014) for example. The model provides an interpretable and transparent
explanation about how different factors might be related to each other, this type of evidence is
often essential when advising on decisions to policy makers. The model also represents a store of
information that can be logically updated in the presence of further data or transferred to different
data sets. Often, simple assumptions like linearity of predictors to response, or symmetric Gaussian
errors play important roles in providing interpretability to the model and in particular preventing
it from over fitting to any non-generic features contained in any one data set.
For the above reasons an unambiguous statement of a model, however simple, is in our opinion
an essential element for much of applied statistics. In light of this, statistical methodology should
also be sufficiently flexible to cope with the fact that the DM’s model is inevitably an approximation
both of their beliefs and the real world process. Currently, Bayesian statistics sometimes struggles
to do this well in the M -open world, because it implicitly minimises the KL-divergence to the
data generating process. By fitting model parameter in a way that can be non-robust, the DM is
having to combine their best guess belief model with something that will be robust to the parameter
fitting. The DM must seek the best representation of their beliefs about the data generating process
in order to make future predictions. However under traditional Bayesian updating they must also
give consideration to how robust these beliefs are. This seems an unfair task to ask of the DM. We
therefore propose to decouple what the DM believes about the data generating process, from how
the DM wishes the model to be fitted. This results in option 4.

3

This last method leads us to choose an updating of beliefs parallel to the usual Bayesian update but minimising an alternative divergence from the KL-divergence in order to fit the model
parameters in a subjective, problem specific manner. In the frequentist literature this refers to
minimum divergence estimation or minimum scoring rule inference. These methods aim to replace
the logarithmic score associated with the KL-divergence with a scoring function associated with a
more robust divergence function. In fact, de Finetti uses the minimisation of scoring rules in order
to justify producing (subjective) probability belief distributions and therefore considering a foundationally well principled Bayesian analogue to minimum scoring inference is more natural than it first
appears. By doing this the principled statistical practice of fitting model parameters to produce
predictions is adhered to, but the parameter fitting is done so acknowledging the M -open nature of
the problem.

1.2

Subjectivity

Authors Goldstein et al. (2006) identity that subjectivity is required to allow statistics to properly
answer the questions that are being asked of it. Very few problems seek answers that are connected
with a specific dataset or model, they seek answers about the real world process underpinning
these. Subjective judgements are therefore required in order to generalise conclusions from the
model and the data to the real world process. Authors Celeux, Jewson, Josse, Marin, and Robert
(2017) observed that while Gelman and Hennig (2015) advocate greater freedom for subjective
judgements to impact statistical methodology, they fail to consider the possibility of subjective
Bayesian parameter updating. In the M -closed world Bayes rule is objectively the correct thing to
do, but in the M -open world this is no longer so clear. In addition to the prior and model, the
choice of the divergence d(·, ·) is another way for the DM to impose their subjective judgements on
the analysis and control the way in which the analysis of the data and the model is used to inform
about the underlying real world process. If correctly matching the tails of the fitted belief model
with the tails of the data generating process is of importance to the DM’s problem, then they should
use the KL-divergence and document in their methodology why they have done this. Alternatively,
if how their fitted belief model performs for the centre of the distribution is of greater importance,
then they should use an alternative divergence and document the reasons for this choice also. We
discuss at the start of Section 3 that frequentists are naturally afforded subjectivity in the way they
fit their parameter estimates. We therefore view our contribution here as providing the tools to
afford the same types of subjectivity to Bayesian practitioners as well.

2

Related work

2.1

Bayesian theory and scoring functions

Authors Bernardo and Smith (2001) consider Bayesian inference through the lens of a Bayesian
decision problem, where quoting a probability belief distribution for future uncertainties is the
action to be taken. In this scenario, a scoring rule S(z, F ) is used to define the loss of quoting
a distribution F and observing a realisation z. They argue that the scoring rule associated with
scoring probabilistic predictions should be proper and local. A proper scoring rule results in the
DM’s expected loss being minimised when the DM quotes their true beliefs, and a local scoring rule
is one where the score only depends on the quoted probability of the actual observed outcome and
nothing else. The only proper, local scoring rule is the logarithmic scoring rule:
`(f (·; θ), z) =

n
X

S(zi , F ) =

i=1

n
X

− log (f (zi ; θ)) .

(2)

i=1

The expected extra loss when quoting distribution f (·; θ) for future outcomes when the data is

4

distributed according to distribution g is then:
EZ∼g(·) [− log(f (z; θ))] − EZ∼g(·) [− log(g(z))] = dKL (g||fθ ),

(3)

the Kullback-Leibler (KL) divergence between the two probability distributions in equation (1). Authors P. D. Grünwald and Dawid (2004) formalise the definition of a divergence. For a proper scoring
rule the associated divergence D(G, F ) = S(G, F ) − S(G, G) = EX∼G [S(z, F )] − EZ∼G [S(z, G)] is
the extra penalty for believing z was distributed according to F when it was actually distributed
according to G. As is well known but often forgotten, someone whose probabilities are elicited using
the logarithmic score, or the KL-divergence, will have to beware of approximating the probability
of an event by 0. Since if they are wrong, they will incur an infinite loss. As a result, accurate
probability specification of tail events will be important to the DM using the logarithmic-score
(Bernardo & Smith, 2001). However for entirely reasonable practical reasons these rare events are
precisely ones which the DM will find hard to accurately elicit O’Hagan et al. (2006), see Winkler and Murphy (1968) for a demosntration of this in the context of forecasting the probability of
precipitation. Therefore, basing inference on such severe penalties can be dangerous. The authors
Dawid, Musio, and Ventura (2016) consider minimising alternative proper scoring rules in order
to produce frequentist parameter estimates, demonstrating unbiasedness under the data generating
model and asymptotic normality.

2.2

General Bayesian updating

In Goldstein (1990) the author identifies that in order to correctly condition upon observed data in
a Bayesian updating of beliefs, a model for the whole world in which the observations occur must
be constructed. This requires many more probability specifications than any DM is ever going to
be able to make, and as a result these genuine beliefs must be approximated. This defines the
subjectivist opinion of the M -open world - the model used for the belief updating is only ever
feasibly an approximation of the DM’s true beliefs. In fact, in Goldstein et al. (2006) it is identified
that there exists no result suggesting that conditioning is the correct way to update beliefs. Using
Bayes rule describes now how a DM’s beliefs would change if they observed some data, but by
the time this data has been observed there is inevitably more information the DM would wish to
incorporate into their posterior belief, and Bayes rule provides no provision for this. Once free from
conditioning, Bayes linear methods Goldstein (1999) provide tools to move away from making a
full probability specification in a principled manner. Bayes linear methods take expectations as
primitive rather than probabilities, in order to simplify the probability specification required of the
DM. When expectation is primitive the DM need only concern themselves with the sub-collection of
probabilities and expectations they consider themselves to be able to specify Goldstein et al. (2006).
Following the above motivation, Bissiri et al. (2016) consider only conducting inference about the
factors of the world/problem that matter to the DM. They produce what they refer to as a general
Bayesian update - a coherent method to produce a posterior distribution over some quantity without
relying on a full model for the observations - when considering the non-inferential Bayesian decision
problem. Similarly to Bayes linear methods Goldstein (1999), they consider producing posterior
beliefs which do not require conditioning. They consider the Bayes act associated with beliefs
corresponding to the data generating distribution G(·) and quantity of interest θ as
θ∗ = arg min Ez∼G [`(θ, z)] = arg min
θ

θ

Z
`(θ, z)dG(z).

(4)

Rather than elicit a model representing the DM’s beliefs over z, Bissiri et al. (2016) argue that
given a prior, a loss function and some data x, an updating of beliefs about θ must be possible in
the absence of a model for the sampling distribution. They suggest that the posterior distribution
resulting from such an updating of beliefs, be as ‘close’ as possible to both the prior and the observed

5

data x, used for the updating. In this development the KL-divergence was the only way to measure
‘closeness’ to the prior which resulted in an additive Bayesian update - one where the order the data
arrives in does not affect that parameter updating and is consistent with the likelihood principle.
In the absence of a likelihood, the best choice to measure the ‘closeness’ of the posterior to the data
is to use the expected loss of the observed data under the proposed posterior, Bissiri et al. (2016)
show that the posterior minimising the sum of these criteria is:
π(θ|x) ∝ π(θ) exp(−w`(θ, x))

(5)

This posterior is not an approximation, or pseudo-posterior, but rather a valid, coherent representation of subjective uncertainty in the minimizer of the decision in equation (4). Often
Pn
`(β, x) =
i=1 l(β, xi ), the cumulative loss of the current data set. Updating using the cumulative loss amounts to replacing integrating over the data generating distribution in equation (4),
with empirical integration over data whose distribution is G(z). They also mention the need to
calibrate the loss-function used in the general Bayesian update against the prior, by setting the
value of w in equation (5). While any probability density is defined so that it integrates to 1, any
loss-function can be defined to be arbitrarily large or small. Therefore, it is important that the
weight the loss-function has in the updating process is calibrated against the prior such that the
posterior distributions resulting from the general Bayesian update retain probabilistic meaning in
the absence of a model. That is to say the general Bayesian must be prepared to post odds according
to their posterior similarly to any other Bayesian.
It is straightforward to see that if the logarithmic score is used in the general Bayesian update,
the traditional Bayesian update is recovered:
π(θ|x) ∝ exp(−

n
X

− log(fθ (xi ))) · π(θ) =

n
Y

fθ (xi )π(θ),

(6)

i=1

i=1

echoing the well-known result that the Bayesian predictive distribution finds the distribution that
is closest to the data generating distribution in terms of KL-divergence. General Bayesian updating
allows the goals of the statistical analysis to be coupled together with the parameter updating,
something not previously possible under traditional Bayesian statistics. While Bissiri et al. (2016)
provide the framework to update a probability belief distribution using a loss function, they only
consider the logarithmic score in an inferential scenario.

2.3

Minimum divergence estimation and Robustness

Minimum divergence estimation (MDE) is an approach to robust inference, largely overlapping with
minimum scoring rule inference Dawid et al. (2016). MDE considers making inference about the
parameters θ of a parametric model {f (x; θ) : θ ∈ Θ} by minimising the divergence between f (x; θ)
and the data generating function of the observed data. While Dawid et al. (2016) uses proper scores,
MDE traditionally minimises a member of the class of discrepancies or disparities. Disparity based
methods first build a non-parametric density estimate gn (x) of the data generating distribution
from the data (often a Kernel Density estimate), and then conduct inference by minimising the
divergence between the model and gn (x).
The Hellinger divergence between g and f is
d2H (g, f ) =

1
2

Z p
Z p
p
( g(z) − f (z))2 dz = 1 −
g(z)f (z)dz.

(7)

From the frequentist standpoint, Beran (1977) proposes using the minimum Hellinger distance
in order to discover robust parameter estimates. They observe that when the data is distributed
according to one member of the model class, g(x) = f (x; θ0 ), the Hellinger divergence between
the model and a data generating process is approximately equal to the KL-divergence between the

6

model a data generating process, when n is large. Therefore the minimum Hellinger divergence
estimate (MHDE) is asymptotically equivalent to the MLE, the estimate achieved by minimising
the KL-divergence. By the same argument Beran (1977) propose that the MHDE will also be
asymptotically efficient. Results regarding the robustness and efficiencies of general discrepancy
measures can be found in Lindsay (1994) for discrete distributions and Kuchibhotla and Basu
(2015) for the continuous analogue. Basu et al. (2011) provides a comprehensive review of minimum
divergence estimation. The frequentist literature in this area is vast, but in this article we choose to
focus on the Bayesian contributions. These have come first from Hooker and Vidyashankar (2014)
and more recently from Ghosh and Basu (2016) and Ghosh and Basu (2017a). Authors Hooker and
Vidyashankar (2014) use the approximate equality between the KL divergence and the Hellinger
divergence to produce a more robust posterior distribution over the model parameters, a direct
Bayesian analogue to the MHDE. Rather than considering a discrepancy, Ghosh and Basu (2016)
considers minimising the proper Tsallis score associated with the density power divergence in order
to produce a ‘pseudo-posterior’. The density power divergence between g and f is
dα
DP D (g, f )

1
=
α+1

Z
f

1+α

1
(z)dz −
α

Z

1
f (z)g(z)dz +
α(1 + α)
α

Z

g 1+α (z)dz.

(8)

This can be seen as a Bayesian analogue of the minimum proper score inference considered by
Dawid et al. (2016). Authors Ghosh and Basu (2016) identify the fact that the score associated with
the density power divergence does not require a density estimate, but can still be used to provide
robust inference. They term the distribution produced by their estimation method a ‘pseudoposterior’, but Ghosh and Basu (2017a) demonstrate exponential converge results illustrating that
this posterior is asymptotically optimal in exactly the same exponential rate as the traditional
Bayesian posterior when the model is correct Barron (1988).
The current Bayesian minimum divergence technology is thus only able to produce approximate
or pseudo posteriors. Therefore, Bayes rule is currently the only method explicitly available in
the literature that DMs can use to produce a well principled updating of their beliefs about the
parameters of a model. We address this issue in the next section.

3
3.1

Bayesian MDE
Why the current justification is not enough

Authors Hooker and Vidyashankar (2014) consider posteriors of the form
π(θ|x) ∝ π(θ) exp(−nd(g, f (·; θ))),

(9)

where g is the data generating density and d(·, ·) is either the Hellinger or negative exponential
divergence. Posteriors of this form are justified following the asymptotic approximation of the
divergence to the KL-divergence when the data comes from the model. This results from the
fact that the divergence between two probability distributions must be uniquely minimised to 0
when the probability distributions are equal P. D. Grünwald and Dawid (2004). Therefore, when
g(x) = f (x; θ0 ) and as n → ∞ the distribution minimising any divergences between it and the data
generating process will be f (x; θ0 ). The equivalence between the KL-divergence and the Hellingerdivergence of the model from the data generating process can be seen by taking Taylor expansions
about the KL-divergence minimising parameter θ̂KL = arg minθ∈Θ dKL (g(x), f (·; θ)). We have that

7



Z
gn (x) log

gn (x)
f (x; θ0 )



Z
dx =

gn (x) log

−

(θ − θ̂KL )2
2

0

gn (x)

0

dx − (θ − θ̂

KL

Z

∇θ f (x; θ̂KL )

gn (x)dx
f (x; θ̂KL )
!
∇2θ f (x; θ̂KL ) (∇θ f (x; θ̂KL ))2
−
gn (x)dx + · · · .
f (x; θ̂KL )
f (x; θ̂KL )2

f (x; θ̂KL )
Z

!
)

(10)
Z

!



q



Z
f (x; θ̂KL )
0
1
∇ f (x; θ̂KL )
KL

p
q θ
gn (x)dx − (θ − θ̂ )
gn (x)dx =
gn (x)dx
p
2
gn (x)
f (x; θ̂KL ) gn (x)


0
2
KL
KL 2 Z
KL 2
∇ f (x; θ̂ )
1
(θ − θ̂ )
1
(∇θ f (x; θ̂ ))
 gn (x)dx + · · · .
q θ
− p
−
p
KL ))3/2
2
2
2
g
n (x)(f (x; θ̂
f (x; θ̂KL ) gn (x)

p

f (x; θ0 )
1− p
gn (x)

Z

1 −

(11)
Now following by the same arguments used in Hooker and Vidyashankar (2014), if gn (x) is
consistent for g = f (·; θ̂KL ) = f (·; θ0 ) then equations (10) and (11) are equivalent in the limit as
n → ∞. However when θ̂KL 6= θ0 is not the data generating parameter because the model class
is misspecified, and if gn (x) is still converging to g(x) 6= f (x; θ0 ) as n → ∞ then equations (10)
and (11) will be different. In this setting the current literature gives no foundational reasoning why
updating using the Hellinger divergence constitutes a principled updating of beliefs.
Authors Ghosh and Basu (2016) refer to the distribution resulting from a parameter update
minimising the density power divergence as a ‘pseudo-posterior’. This update is justified as a valid
posterior by Ghosh and Basu (2017a) as the traditional posterior originating from an alternative
belief model which would have probability density equivalent to the exponential of the score associated with the density power divergence (see equation 16) standardised so it integrates to one.
However this transformed model will not even correspond to the DM’s approximate beliefs about
the data generating process and thus there is no justification for a DM to use Bayes rule to update
their beliefs using this.

3.2

Principled justification for KL in M -open world

Contrarily, Walker (2013) has provided a principled justification for Bayes’ rule in the M -open world
that requires the DM to think about the KL-divergence minimising parameter. When the model is
correctly specified, the Bayesian is trying to learn about the parameter θ0 that generated the data.
This, for any statistical divergence d(·, ·) is equivalent to the parameter minimising d(g(·), f (·; θ)).
When the model is considered to be incorrect, there is no longer any formal relationship between
the parameter θ and the data. That is to say the likelihood no longer represents the probability of
the observed data conditioned on the parameter. Therefore, the first definition loses its meaning
and the DM is required to appeal to the second definition. In order for M -open inference to be
meaningful a divergence measure must be chosen and the parameter of interest can then defined
as θ∗ = arg minθ∈Θ d(g(·), f (·; θ)). Authors Walker (2013) then state that once the parameter of
interest has been defined as the minimum of some divergence, it is then possible for a practitioner
to define their prior beliefs about where the minimiser of this divergence may lie. Then the final
job of the practitioner is to ensure that their Bayesian learning machine is learning about the same
parameter with which they defined their prior belief.
Recalling the well-known result that Bayesian updating learns about the parameters of the model
which minimises the KL divergence of the model from the data generating density, allows the DM
to continuing conducting belief updating in a principled fashion using Bayes rule provided they are
interested in, and specify prior beliefs about, the parameter θKL minimising the KL divergence of

8

the model from the data generating process.

3.2.1

Moving away from KL-divergence

Viewed conversely, Walker (2013) identified that using Bayes rule to update beliefs using an misspecified model results in the DM being implicitly concerned about producing prediction that are closest
to the data generating process in terms of KL-divergence. However, we have already identified that
this is undesirably non-robust
When inference is all the practitioner cares about the logarithmic score has optimal properties.
The logarithmic score is proper, local, guarantees coherence and places high weight on the tail
specification of the belief distribution, something which is generally speaking extremely important
in pure inference problems (Bernardo & Smith, 2001). However, in reality inference is being done in
order to produce estimates of expected utilities as part of a larger decision making process. While
the tail specification is important for inferential procedures, many loss functions connected to real
decisions may wish to place less impact on abnormal observations in the tails. For example a treatment regime is likely to put greater weight on correctly treating the majority of the population than
worrying about outlying extreme cases. Thus many loss functions for real problems are bounded.
Authors Watson et al. (2016) quite rightly point out that a bounded loss function is a reasonable
assumption for applied problems and any method that simulates parameters via MCMC makes this
assumption. As will be demonstrated in Section 4.2, inferential procedures assuming an unbounded
loss function such as the logarithmic score, provide no guarantees about performance in more general
decision making scenarios.
Once the consequences of using Bayes rule to solve decision problem in the M -open world is
understood, we believe that DMs may reasonably desire alternative options for parameter updating,
that are as well principled as Bayes rule, but places less importance on tail misspecification. In this
new era of “Big data” it becomes increasingly likely that the model used for inference is misspecified,
especially in the tails of the process. We believe many DMs would consider it undesirable for models
that approximate the distribution of the majority of the data to be disregarded because they poorly
fit a few outlying observations.

3.3

Principled minimum divergence estimation

Here we take a foundational approach to theoretically justify an updating of beliefs targeting the
parameter minimising any statistical divergence between the model and the data generating density.
Consider the general inference problem of wanting to find the parameters θ of the parametric model
{f (·; θ) : θ ∈ Θ}. The model here can be considered as the DM’s best guess at the data generating
process. We consider it important to continue to use a model even under the acknowledgement that
it is misspecified for the reasons outlined in Section 1.1. Following Bernardo and Smith (2001) and
Walker (2013) we consider fitting these parameters in a decision making scenario, minimising some
divergence function
d(·, ·) = Ex∼G [`d (x, f (·; θ))] − Ex∼G [`d (x, g)],

(12)

to the data generating process g(·). The goal is therefore to solve the decision problem
θ∗ = arg min d(g(·), f (·; θ)) = arg min
θ

Z
`d (x, f (·; θ))dG(x).

(13)

Here the entropy term in the definition of the divergence is removed from the minimisation
because it does not depend on θ. In this scenario Walker (2013) forces the Bayesian into concerns
associated with non-robustness by shackling them to minimising the KL-divergence in order to use
Bayes’ rule. However, equation (13) is identical to the setting consider by Bissiri et al. (2016) in

9

equation (4), except that now the loss function depends on the parameter through a model so is
a scoring rule. General Bayesian updating therefore gives us the tools to come up with a coherent
updating of beliefs that target the parameter minimising the general divergence d(·, ·) as
π (d) (θ|x) ∝ π (d) (θ) exp(−

n
X

`d (xi , f (·; θ))),

(14)

i=1

where we have set the calibration weight w = 1. This will be discussed further in Section 3.3.2.
In order to stay consistent with Walker (2013), we use the notation π (d) to indicate that the prior and
posterior belief distributions correspond to beliefs about the parameter minimising divergence d(·, ·).
Applying the general Bayesian update in an inferential scenario like this, provides a compromise
between purely loss based general Bayesian inference and traditional Bayesian updating, which is
conducted completely independently of the problem specific loss function. By continuing to fit
a model, the Bayesian minimum divergence posterior somewhat separates the inference from the
decision making. The inference is still concerned with estimating model parameters based on how
well the model’s predictions reflect the data generating process. However, the criteria for closeness
between predictions and reality takes decision making into account. As a result, the minimum
divergence posteriors can be seen as producing inferences that will be suitable for a broad range
of loss functions. The DM is no longer required to exactly define the loss function associated
with their decision problem at the inference stage, and need only consider broadly how robust
to tail misspecifications they want their inference to be in order to define the target divergence
(see Section 4.6). Considering the realistic M -open nature of the model class also justifies making
this compromise. The general Bayesian update, assumes absolutely no information about the data
generating process, while using Bayes rule traditionally1 assumes the data generating density is
known precisely. As we point out, it is actually more likely that the decision maker is able to
express informative but not exact beliefs about the data generating process and therefore a halfway-house between these two is appropriate in reality.
Plugging in the corresponding loss functions gives general Bayesian posteriors minimising the
Hellinger and density power divergence as
!
f
(x
;
θ)
i
p
π H (θ|x) ∝ π H (θ) exp
gn (xi )
i=1

!
Z
n
X 1
1
DP D
DP D
α
α+1
πα
(θ|x) ∝ πα
(θ) exp
f (xi ; θ) −
f (y; θ)
dy
.
α
1+α
i=1
n
X

p

(15)

(16)

Equation (15) introduces gn (·) to estimate the data generating density g (see Section 4.7 for
more on this). As a result, the general Bayesian updating is being conducted using an empirical
loss function, say `ˆH , which approximates the true loss function associated with minimising the
Hellinger divergence between the model and the data generating process `H .
Equation (16) is exactly the distribution resulting from the robust parameter update of Ghosh
and Basu (2016) while equation (15) is similar to the posterior produce by Hooker and Vidyashankar
(2014) except the divergence function has been decomposed into its score and entropy term here. The
posteriors above are not pseudo posteriors - as Ghosh and Basu (2016) suggests - or approximations
of posteriors - as Hooker and Vidyashankar (2014) suggests - they are how a DM should update their
prior beliefs about the parameter minimising an alternative divergence to the KL-divergence. This
provides the principled justification for a Bayesian to conduct inferences targeting an alternative
divergence to the KL-divergence in the M -open world that is currently lacking in the literature. In
fact the scope of this is not limited to the M -open world, the posterior of Ghosh and Basu (2016)
no longer needs to be considered a pseudo-posterior even when the model is correct. All divergences
1

Prior to the interpretation provided by Walker (2013) explained in section 3.2

10

are minimised when the distributions are equal, so if the data generating distribution is contained
within the model class then any minimum divergence posterior will be learning about the parameter
of the data generating distribution. In Section 5.2 we observe that although the KL-divergence is
optimal in this scenario the performance when minimising the alternative divergences is comparable.
Authors Basu et al. (2011) showed that θ̂n is a consistent estimate of θDP D , while Ghosh
and Basu (2016) then show that under certain regularity conditions παDP D follows a Bernstein-von
Mises (BVM) type result, but with convergence in probability, where the asymptotic variance of the
posterior is the expected second derivative of the loss function `DP D (xi , f (·, θ̂n )). Authors Hooker
and Vidyashankar (2014) prove the corresponding asymptotic normality result for the posterior
minimising the Hellinger divergence, but assume that the data generating process is contained
within the model class, as this was required to justify their posterior in the first place. Consistency
to the minimiser of any divergence provides consistency with the data generating process when it
is contained within the model class, as by definition all divergences are minimised to 0 when the
distributions are the same. The results of Ghosh and Basu (2017a) demonstrate that not only is
the posterior mean consistent, but the whole posterior is asymptotically optimal in the same sense
as the traditional Bayesian posterior is when the model class contains the data generating process.

3.3.1

The likelihood principle and Bayesian additivity

Authors Hooker and Vidyashankar (2014) identify that their posterior distribution minimising the
Hellinger divergence no longer satisfies Bayesian additivity. That is to say that the posteriors would
look different if the data were observed in one go or in two halves for example. This is caused by the
fact that the posterior now depends on the estimate of the data generating density, which depends
in a non-additive way on the data. This constitutes a departure from the likelihood principle
underpinning traditional Bayesian statistics. The likelihood principle says that the likelihood is
sufficient for the data. When the model is correct this is a sensible principle, the likelihood of
the observed data under the correct model represents all the information in the data. In the M open world the likelihood principle is no longer a reasonable requirement. When the model is only
considered as an approximation of the data generating process, it is not unreasonable to suspect that
the data contains more information than the likelihood of an incorrect model represents. However,
Bayesian additivity (defined as coherence by Bissiri et al. (2016)) is a central principle of the general
Bayesian update. Bissiri et al. (2016) chose the KL-divergence to measure the distance between the
prior and the posterior as it was the only divergence measure that left the belief update additive.
The general Bayesian posterior minimising the Hellinger divergence combines the loss function for
each observation in an additive way, which is consistent with the additivity demanded by Bissiri
et al. (2016), but a different density estimate is produced when the data is considered as a whole
or in parts. This causes the empirical loss function used for the updating to be different when the
data arrives in sections, as opposed to one go. If the data generating density were available then
the exact loss function associated with minimising the Hellinger divergence could be calculated and
the Bayesian update would be additive. However because an approximation of the loss function
associated with the Hellinger divergence is used, additivity is sacrificed.
While the posterior of Ghosh and Basu (2016) is still additive, the likelihood is also not considered
R
to be sufficient for the data, as the posterior also depends on the integral, f (y; θ)1+α dy. Using an
alternative divergence to the KL-divergence in order to conduct Bayesian updating, requires that
additional information to the likelihood is required for the updating. For the Hellinger-divergence
this information comes from the data in the way of a density estimate, while for the density powerR
divergence this comes from the model through f (y; θ)1+α dy.

11

3.3.2

A note on calibration

The two posteriors in equations (15) and (16) have set the general Bayesian calibration weight to 1.
Unlike probability distributions, loss functions can be of arbitrary size and therefore it is important
that they are calibrated with the prior. However, this is not a problem here. The posteriors above
both use well-defined models and the canonical form of well-defined divergence functions and as a
result there is no arbitrariness in the size of the loss. This is further demonstrated by the fact that
Bayes rule corresponds to using the canonical form of the KL-divergence and a probability model
with weight w = 1. We do note that when the model is correct the posterior variance of these
method will be comparatively bigger for finite data samples than that of the traditional Bayesian
posterior. The posterior is simply a subjective reflection of the DM’s uncertainty after seeing the
data, and if they believe their model is incorrect and therefore target a more robust divergence,
they are likely to have greater posterior uncertainty than if they naively believe their model to be
correct and proceed accordingly.

4

Possible divergences to consider

While currently only the Hellinger, Negative exponential and density power divergences have been
considered for producing a Bayesian updating of beliefs, the formulation of minimum divergences
estimation through general Bayesian updating opens the possibility to use any statistical divergence.
Below we list some divergences that a practitioner may wish to consider. We do not claim that the
list is exhaustive, but merely contains the divergences we have considered using. We qualitatively
describe the features of the inferences produced by targeting each specific divergence, which will be
demonstrated empirically in Section 5. Throughout we consider the Lebesgue measure to be the
dominating measure.

4.1

KL-divergence

In situations where the losses are unbounded, for example in gambling and odds setting scenarios,
correctly specifying the probability of unlikely events will be important and if this is the case then
minimising the KL-divergence is often (subjectively) the optimal thing to do. When the model is
correct, minimising the KL-divergence also has the best small sample efficiency (see Section 5.2),
this is because of the connection between Bayes rule and the KL-divergence. However, it can
be the case that when the model is incorrectly specified traditional Bayesian updating can learn
too fast about the parameter of the model minimising the KL-divergence to the data generating
process, which leads to proposed methods to slow this learning down (P. Grünwald and van Ommen
(2014),Holmes and Walker (2017)). In many scenarios using the KL-divergence is also the most
expedient thing to do, this probably explains why the KL-divergence is so prolific for statistical
applications. The logarithmic form of the KL-divergence makes it straightforward to calculate the
KL-divergence between many exponential families and the link with Bayes rule allows for conjugate
prior distributions to be considered.

4.2

Total Variation Divergence

The Total-Variation divergence between probability distributions g and f is given by
1
dT V (g, f ) = sup |g(A) − f (A)| =
2
A

Z
|g(z) − f (z)|dz,

(17)

and the general Bayesian update targeting the minimisation of the TV-divergence can be produced by using loss function

12

`T V (x, f (·; θ) =

1
f (·; θ)
1−
2
gn (x)

(18)

in equation (14). It is worth noting however that the loss function in equation (18) is not
monotonic in the predicted probability of each observation, this is discussed further in Section 5.2
. In fact when informed decision making is the goal of the statistical analysis, closeness in terms
of TV-divergence ought to be the canonical criteria the DM demands. If dT V (g, f ) ≤  then for
any utility function bounded by 1, the expected utility under g of making the optimal decision
believing the data was distributed according to f is at most 2 worse than the expected utility
gained from the optimal decision under g (see J. Q. Smith (2010) for example). Therefore if the
predictive distribution is close to the data generating process in terms of TV-divergence, then the
consequences of the model misspecification in terms of the expected utility of the decisions made is
small. However explicit expressions of the TV-divergence even between known families, are rarely
available. This somewhat hinders the algebraic analysis of the divergence.
It is straightforward to see that 0 ≤ dT V (g, f ) ≤ 1 and the KL-divergence form an upper bound
on the TV-divergence through Pinsker’s inequality. However, the KL-divergence does not bound
the TV-divergence below so there are situations where the TV-divergence is very small but the
KL-divergence is very large. In this scenario, a predictive distribution whose associated optimal
decisions achieve a close to optimal expected utility estimate (as the distribution is close in TVdivergence) will receive very little posterior mass. This is clearly undesirable in a decision making
context.
Authors Hooker and Vidyashankar (2014) identify some drawbacks of having a bounded score
function. The score function being upper bounded means that there is some limit to the score that
can be incurred in the tails of the posterior distribution. The score incurred for one value of θ, will
here be very similar to the score incurred by another value of θ past a given size. Therefore the tails
of the posterior will be equivalent to the tails of the prior. As a result the DM is required to think
more carefully about their prior distribution. Not only are improper priors prohibited, but more
data is required to move away from a poorly specified prior. This can result in poor finite sample
efficiency when the data generating process is within the chosen model class.

4.3

Hellinger Divergence

Authors Devroye and Gyorfi (1985) observed, that the Hellinger divergence can be used to bound
the TV-divergence both above and below:
d2H (P, Q) ≤ dT V (P, Q) ≤
where
d2H (g, f )

1
=
2

q
q
q
d2H (P, Q) 2 − d2H (P, Q) ≤ 2d2H (P, Q).

Z p
Z p
p
2
( g(z) − f (z)) dz = 1 −
g(z)f (z)dz

(19)

(20)

As a result, the Hellinger-divergence and the TV-divergence are geometrically equivalent. Thus, if
one of them is small, the other is small and similarly if one of them is large the other also is. So
if one distribution is close to another in terms of TV-divergence, then the two distributions will be
close in terms of Hellinger-divergence as well. Authors Beran (1977) first noted that minimising
the Hellinger-divergence gave a robust alternative to minimising the KL-divergence, while Hooker
and Vidyashankar (2014) proposed a Bayesian alternative (equation (15)) that has been discussed
at length above. While Hooker and Vidyashankar (2014) motivated their posterior through asymptotic approximations, identifying the geometric equivalence between the Hellinger-divergence and
TV-divergence proposes further justification for a robust Bayesian updating of beliefs similar to
that of Hooker and Vidyashankar (2014). Specifically, if being close in terms of TV-divergence is
the ultimate robust goal, then being close in Hellinger-divergence will guarantee closeness in TV-

13

divergence. Hellinger can therefore serve as a proxy for TV-divergence that retains some desirable
properties of the KL-divergence: it is possible to compute the Hellinger divergence between many
known families J. Smith (1995) and the score associated with the Hellinger divergence has a similar
strictly convex shape about it (this is discussed further in Section 5.2). A posterior targeting the
Hellinger-divergence does suffer from the same drawbacks associated with having a bounded scoring
function that are mentioned at the end of the previous section. Lastly along with TV-divergence,
the Hellinger-divergence is also a metric.

4.4

Negative Exponential disparity

Authors Hooker and Vidyashankar (2014) observe that while updating using the Hellinger-divergence
down-weights the influence that outliers can have on the inference, it also leads to an up-weighting
of inliers - areas of high mass that are unsupported by the data. Or equivalently, values of x for
which gn (x) is lower than f (x; θ). As a solution Hooker and Vidyashankar (2014) also propose
Bayesian updating using the negative exponential divergence
D

NE




Z 
g(z) − f (z)
(g, f ) =
exp −
− 1 f (z)dz.
f (z)

(21)

The general Bayesian posterior targeting the minimisation of the negative exponential divergence
is

!
n
X
f
(x
)
g(x
)
−
f
(x
)
i
i
i
π N E (θ|x) ∝ π N E (θ) exp −
exp −
.
g (x )
f (xi )
i=1 n i

(22)

Methods for inlier control are discussed in the frequentist literature with respect to discrete
models, see Mandal and Basu (2013) Ghosh and Basu (2017b), where inliers are a bigger issue.
Inliers and empty cells are considered to be the cause for the poor small sample efficiencies of
frequentist minimum divergence techniques targeting anything other than the KL-divergence.

4.5

αβ-divergences

A much wider class of divergences are provided by the two parameter αβ-divergence family proposed
in Cichocki, Cruces, and Amari (2011)

(α,β)

Z

DAB (g||f ) =

1 α
1
1
f α+β (x) −
g (x)f β (x) +
g α+β (x)dx,
α(α + β)
αβ
β(α + β)

α, β ≥ 0, α + β 6= 0.

(23)
Reparametrising this such that α = (1 + λ(1 − αS )) and β = (αS − λ(1 − αS )) gives the
S-divergence of Ghosh et al. (2017).
The general Bayesian posterior targeting the parameter minimising the αβ-divergence of the
model from the data generating density is
Z
n
X
1 α−1
1
π αβ (θ|x) ∝ π αβ (θ) exp(
gn (xi )f β (xi ; θ) −
f α+β (y; θ)dy)
αβ
α(α
+
β)
i=1

(24)

Considering g to be the data generating density, and fθ to be some predictions of g parametrised
by θ, Cichocki et al. (2011) provide intuition about how the parameters α and β impact the influence
each observation x has on the inference about θ. They consider influence to be how the observations
impact the estimating equation of θ, this is a frequentists setting but the intuition is equally valid
in the Bayesian setting.

14



α > 1, down weights x with smaller ratios g(x)/fθ (x) with respect to larger ones.




α < 1, down weights x with the larger ratios g(x)/f (x) with respect to smaller ones.
θ


β + α > 1, down weights x where fθ (x) is small.




β + α < 1, down weights x where fθ (x) is large.

(25)

The size of g(x)/fθ (x) for an observation x defines how outlying (large values) or inlying (small
values) the observation is. Choosing α < 1 ensures outliers relative to the model have little influence
on the inference, while adjusting the value of β such that β + α > 1 down-weights the influence of
unlikely observations from the model fθ .
While Ghosh et al. (2017) empirically demonstrate that superior estimation performance that
can be gained by considering a two-parameter divergence, we consider this no further. We believe
that considering the selection of the divergence that will be used for the analysis as a subjective
input, that to ask for values of α and λ to be specified by a DM is perhaps over ambitious, even
given the interpretation ascribed to these parameters above. However, using the αβ-divergence for
inference provides greater flexibility to the DM, which may in some cases be useful. Two important,
one parameter subfamilies within the αβ-divergence family are the alpha-divergence where β = 1−α
and the beta-divergence where α = 1.

4.5.1

alpha Divergence

The alpha-divergence, introduced by Csisz et al. (1967) and extensively studied by (Shun-ichi, 2012)
is


Z
1
α
1−α
dα (g, f ) =
1 − g(z) f (z)
dz ,
(26)
α(1 − α)
where α ∈ R\{0, 1}. There exists various reparametrisations of this: Amari notation uses αA with
A
α = 1−α
2 ; or Cressie-Read notation Cressie and Read (1984) introduces λ with α = λ + 1.
We generally restrict attention to values of α ∈ (0.5, 1). α = 1 corresponds to a KL-divergence
limiting case while α = 0.5 is 4 times the Hellinger divergence. We consider these to be two extremes
of efficiency and robustness within the alpha-divergence family with which a DM would want to
conduct inference between, α thus controls this trade-off.
The general Bayesian posterior targeting the minimisation of the alpha-divergence is
n

π α (θ|x) ∝ π α (θ) exp(

X
1
gn (xi )α−1 f (xi ; θ)1−α ).
α(1 − α) i=1

(27)

Note the power on g in equation (27) is reduced by 1 from equation (26) as we consider empirical
expectations in order to estimate the expected score associated with the alpha-divergence.
It was demonstrated in Sason and Verdú (2015) (Corollary 1) that for α ∈ (0, 1) the alphadivergence can be bounded above by TV-divergence :
α(1 − α)dα (P ||Q) ≤ dT V (P ||Q).

(28)

Therefore, if the TV-divergence is small then the alpha-divergence will be small (provided α 6=
{0, 1}). So a predictive distribution that is close to the data generating density in terms of TVdivergence will receive high posterior mass under an update targeting the alpha-divergence. Once
again, the general Bayesian posterior requires a density estimate to compute the empirical estimate
of the score associated with the divergence and the same issues associated with having a bounded
score function that were discussed in Section 4.2 apply here.

15

Considering the alpha divergence as a subfamily of the αβ-divergence takes β = 1 − α, and
therefore α + β = 1. Therefore the influence of observations x are weighted based on their ratio of
g(x)/fθ (x) only and not on the value of fθ (x) in isolation. α = 1 corresponded to the KL-divergence
and in order to obtained greater robustness α is chosen such that α < 1 in order to down-weight
the influence larger ratios of g(x)/fθ (x), corresponding to outlying values of x, have on the analysis.
However down-weighting some of these ratios relative to the KL-divergence sacrifices efficiency. This
demonstrates the efficiency and robustness trade-off associated with the alpha-divergence.

4.5.2

beta Divergence

The beta-divergence, also known as the density power-divergence Basu, Harris, Hjort, and Jones
(1998), is a member of the Bregman divergence family
Z
D(P, Q) =

ψ{p(y)} − ψ{q(y)} − ψ 0 {q(y)}(p(y) − q(y))dµ(u).

(29)

Taking ψ = tβ+1 , to be the Tsallis score returns the beta-divergence
dβ (g, f ) =

1
β+1

Z

f β+1 (x)dx −

1
β

Z

f β (x)g(x)dx +

1
β(β + 1)

Z

g β+1 (x)dx,

(30)

β ∈ R\{−1, 0}. This results in the density power divergence given in equation (8), parameterised
by β rather than α here. Both Basu et al. (1998) and Dawid et al. (2016) noticed that inference can
be made using the beta-divergence without requiring a density estimate. This was used in Ghosh
and Basu (2016) to produce a robust posterior distribution that did not require an estimate of the
data generating density, which has been extensively discussed in previous sections.
Considering the density power divergence as part of the αβ-divergence family takes α = 1, which
results in treating all values of g(x)/fθ (x) equally in isolation, but taking β > 0 results in α + β > 1
which means observations that have low predicted probability fθ (x) under the model are downweighted. Under the KL-divergence at β = 0, the influence of an observation x is inversely related
to its probability under the model, through the logarithmic score. Raising β above 0 will decrease
the influence of the smaller values of fθ (x), robustifying the inference to tail specification. However
this results in a decrease in efficiency relative to methods minimising the KL-divergence Ghosh
and Basu (2017a). As a result there may be some issues using the density power-divergence when
high dimensional observations and models are considered. As the dimension increases the predicted
probability of each (multivariate) observation shrinks towards 0. The density power divergence
down-weights the influence of observations with small predicted probabilities, and as a result β
(α when we consider the density power divergence) needs to be selected very carefully in order to
prevent the analysis from disregarding the majority of the data. This is a price that is paid in order
to not require a data generating density estimate.

4.5.3

The S-Hellinger divergence

One further one-parameter special case of the αβ-divergence (S-divergence) is the S-Hellinger divergence given by Ghosh et al. (2017)
2
dSH (g, f ) =
1 + αS

Z 

g(x)(1+αS )/2 − f (1+αS )/2

2

.

(31)

This is generated from the S-divergence by taking λ = − 21 . Taking αS = 0 recovers twice the
squared Hellinger divergence and α = 1 gives the L2 squared divergence. Ghosh et al. (2017) observe
that the S-Hellinger divergence is a proper distance metric. Translating these back into the notation
of αβ-divergences gives
1
1
α = (1 + αS ) , β = (αS + 1) .
(32)
2
2

16

Therefore αs ∈ [0, 1] gives α < 1 and α + β > 1. As a result, the squared Hellinger divergence
down-weights the influence of large ratios of g(x)/fθ (x) with respect to smaller ones and also downweights these ratios when fθ (x) is small. In fact, there is a trade-off here: αs moving closer to 1
increase α + β away from 1, thus increasing the amount that ratios of g(x)/fθ (x) are down-weighted
for small fθ (x). However as αs moves closer to 1, α also draws closer to 1, which reduced the amount
ratios of g(x)/fθ (x) are down weighted with respect to smaller ones. Once again we consider trading
these two off against each other to be too greater task for the DM to consider. We therefore do
not consider the S-Hellinger divergence again here and instead consider only the choice between
the one parameter alpha and density power divergences. In any case, bridging the gap between the
Hellinger and the KL-divergence, as the alpha-divergence does, appears to make more sense in an
inferential setting than bridging the gap between the Hellinger and L2 squared divergence.

4.6

Comparison

The bullet points below summaries the reasons, introduced above, for which a DM might prefer a
specific divergence. Once again we stress that we consider the choice of divergence to be a subjective
and context specific judgement to be made by the DM similarly to their prior and model.
• KL-divergence: suitable if the tail behaviour of the model is of paramount importance or
there is the belief that the tails of the model are exactly correctly specified. Minimising the
KL-divergence also makes best use out of the data so may be suitable if the sample size is very
small and the prior is not necessarily informative.
• TV-divergence/Hellinger-divergence: suitable if the data set has many observations and
robustness for a decision problem with a bounded loss function is of high importance. The
Hellinger-divergence appears to be computationally easier to work with (see Section 5.2). Both
require accurate density estimation
• alpha-divergence: suitable for when a trade-off between robustness and efficiency is considered. Requires access to a density estimation technique.
• The density power divergence: suitable again for when a trade-off between robustness
and efficiency is considered. Does not require a density estimate but does require α to be set
carefully such that the data is not all completely disregarded by the updating process.
Unfortunately there is no free lunch when it comes to applying any of these methods to complex,
high dimensions problems. Minimising the KL-divergence is the most computationally efficient, providing the possibility of implementing conjugate prior distributions and avoiding the computational
burden of producing a density estimate. However as the dimensions and complexity of the problem
increases, robustness to model misspecification will become more and more important. Minimising
the Hellinger-divergence, TV-divergence and alpha-divergence requires an estimate of the data generating density which is certainly not straightforward for high dimensional data (see the discussion
in Section 4.7), however once this has been estimated then α can just be selected based on how
important tail misspecifications are, with a guarantee on some reasonable efficiency. Minimising the
density power divergence has the computational advantage of not requiring an estimate of the data
generating density. However when the dimension of the problem is high, much more thought needs
to be put into specifying the parameter α to ensure that the updating does not completely disregard
the data. Without the data generating density, the density power divergence requires information
on the shape of the data from another form in order to fit efficiently and robustly.

4.7

Density estimation

As has been mentioned before, for the TV, Hellinger and αβ divergence, it is not possible to exactly
calculate the loss function associated with any value of θ and x because the data generating density
g(x) will not be available. In this case, a density estimate of g(x) is required to produce an empirical

17

loss function. This is of course a problem also faced in frequentist’s minimum density estimation
Basu et al. (2011), where the divergence is referred to as a disparity between the density estimate
and the model. The Bayesian can consider the density estimate as providing additional information
to the likelihood from the data (see Section 3.3.1’s discussion on the likelihood principle), and can
thus consider their general Bayesian posterior inferences to be made conditional upon the density
estimate as well as the data. The general Bayesian update is a valid update for any loss function,
and therefore conditioning on the density estimate as well as the data still provides a valid posterior.
However, how well this empirical loss function approximates the exact loss function associated with
each divergence ought to be of interest. The exact loss function is of course the loss function the
DM would prefer to use having made the subjective judgement to minimise that divergence. If the
density estimate is consistent to the data generating process, then provided the sample size is large
the density estimate will converge to the data generating density, and the empirical loss function
will then correctly approximate the loss function associated with that divergence. It is this fact that
ensures the consistency of the posterior estimates of the minimum Hellinger posterior Hooker and
Vidyashankar (2014).
Authors Hooker and Vidyashankar (2014) use a fixed width kernel density estimate (FKDE) to
estimate the underlying data generating density and in our examples in Section 5 we adopt this
practice using a RBF kernel for simplicity and convenience. However we note that Silverman (1986)
identifies practical drawbacks of FKDEs, including their inability to correctly capture the tails of
the data generating process, whilst not over smoothing the centre, as well as the number of data
points required to fit these accurately in high dimensions. In addition to this Tamura and Boos
(1986) observe that the variance of the FKDE when using a density kernel in high dimensions lead

to asymptotic bias in the estimate that is larger than O n−1/2 . Alternatives include using a kernel
with better mean-squared error properties (Epanechnikov (1969), Rosenblatt et al. (1976)), variable
width adaptive KDE Abramson (1982), which Hwang, Lay, and Lippman (1994) show to promising
in high dimensions, or a fully Bayesian Gaussian process as is recommended in Li and Dunson
(2016).

4.8

Selecting α

When the alpha and the density power divergence are considered for inference, the hyperparameter
α needs to be fixed in order to completely define the divergence. No guidance is currently provided
in the literature about how to specify α. For both the alpha and the density power divergence,
α represents a trade-off between robustness and efficiency. Moving the value of α away from the
values associated with the KL-divergence increase robustness and decreases efficiency. In Section 4.5
we discussed the different types of robustness induced by the alpha and density power divergences
which should help choose between the two divergences but provides little guidance in how to choose
the value of α. Here we argue that this simply makes up part of the subjective specification of the
divergence, with there being no general correct or incorrect procedure to take.

5

Illustrations

In this section we aim to illustrate some of the qualitative features associated with conducting
inference targeting the minimisation of the different divergences identified in Section 4. Throughout
these experiments stan Carpenter et al. (2016) is used to produce fast and efficient samples from
the general Bayesian posteriors of interest2 .
2

Code to reproduce examples is available in the Rmarkdown document in the supplementary material.

18

5.1

M -open robustness

The focus of this article is primarily on how a DM should go about making inferences in the M -open
world. Therefore we first consider exploring the performance of the Bayesian minimum divergence
estimation techniques on some real and synthetic examples where the model class is misspecified
relative to the data generating process.

5.1.1

Synthetic example

The experiments below demonstrate the robustness of the general Bayesian update targeting KLdivergence (red), Hellinger-divergence (blue), TV-divergence (pink), alpha-divergence (green) and
power-divergence (orange) (in future these may be referred to as KL-Bayes, Hell-Bayes, TV-Bayes,
alpha-Bayes and power-Bayes respectively). For simplicity we fix α = 0.75 for the alpha-divergence
and α = 0.5 for the power-divergence. Figure 1 plots the posterior predictive originating from fitting
a normal model f (·; θ) = N (µ, σ 2 ) to data of size n = 1000 coming from a normal -contamination
distribution
g = 0.99 × N (0, 1) + 0.01 × N (5, 52 ),

(33)

and a data of size n = 200 coming from Student’s t-distribution with degrees of freedom 4. Prior
distributions µ ∼ N (0, 102 ) and σ ∼ G(0.001, 0.001) were used for both examples.
t

0.3
density

0.2
0.0

0.1

density

0.0

0.1

0.2

0.3

0.4

g
KL
Hell
alpha
power
TV
N(0,1)

0.4

contaminated normal

−5

0

5

10

−6

−4

−2

0

2

4

x

x

Log density plot (contaminated normal)

Normal QQ−plot (contaminated normal)

6

●

−2

2

●
●

1
0

Predictive quantile

−1

−4
−6
−8

log(density)

●
●

●
●
●
●
●

−2

●
●

●
●

●
●
●
●
●
●

●

●

●
●
●

−10

●
●

●
● ●
●
● ●
●
● ●
●
●

●
●
●
●
●
●
●
●
● ●
● ●
●

●
●
●

●

●

−6

−4

−2

0

2

4

6

−2

x

−1

0

1

2

Theoretical quantile

Figure 1: Top: Posterior predictive distributions (smoothed from a sample) arising from Bayesian
minimum divergence estimation fitting a normal distribution to an -contaminated normal (left) and
a t-distribution (right) using the KL-Bayes (red), Hell-Bayes (blue), TV-Bayes (pink), alpha-Bayes
(green) and power-Bayes (orange). Bottom: Log-density plots (smoothed from a sample) (left) and
Normal QQ plots (right) for the -contaminated normal dataset
It is easy to see from the plots in figure 1 that minimising the KL-divergence, requires that the
density of the outlying contamination centred at x = 5 to be captured correctly, at the expense of
capturing the density of the remaining 99% of the data, this is especially clear in the log-density

19

plot. The TV-Bayes, alpha-Bayes and power-Bayes appear to correctly capture the distribution
for 99% of the data. The boundedness of the TV-divergence and the alpha-divergence means the
contamination is not allowed to unduly affect the analysis, while the power-divergence is able to
down-weight the influence of the contaminated points as they are ascribed low predicted probability
under the model for 99% of the data. The Hell-Bayes appears to fit too smaller variance even for
99% of the data, and this is due to the small sample efficiency problems which will be discussed in
Section 5.2.
The Student’s t-distribution has consistently heavier tails than the normal distribution and
thus the top right hand plot of figure 1 more clearly demonstrates the importance placed on tail
misspecification by each method. The KL-divergence fits the largest variance to correctly capture
these tails for the most extreme tail observations, the alpha-divergence is able to fit a slight smaller
variance because of its bounded nature, however the variance of the alpha-Bayes predictive is still
larger than the other methods produce because of the greater convexity of the scoring function
depicted in figure 3. Lastly Hell-Bayes, TV-Bayes and power-Bayes place the least weight on tail
misspecification and are therefore able to fit a smaller variance and produce a predictive more closely
resembling the data generating process for the majority of the data.

5.1.2

Real examples

Figure 2 plots smoothed posterior predictive distributions for Gaussian models fitted to three examples of real world data sets. Examples 1 and 2 originate from the covtype3 dataset. ‘covtype1’ refers
to the first variable ‘Elevation’ while ‘covtype9’ refers to the 9th variable ‘Hillshade 3pm’. Both
variables were chosen as KDE estimates from the data looked as though they were close to a normal
distribution. For computational convenience we only looked at a subset of size n = 10, 000 from the
dataset. The third example, tracks1, was taken as the first variable from the ‘Geographical Original
of Music Data Set’4 containing n = 1059 data points, where again a KDE of the data appeared to
be approximately normally distributed.
In figure 2 the tails of the model for the ‘covtype9’ example are only slightly misspecified compared with the KDE of the data and as a result all five methods produce very similar parameter
estimates. We can observe that the KL-divergence and alpha-divergence, the two methods giving the
highest importance to tail misspecification, fit marginally higher variances than the other methods
in response to the tails being slightly missepecified. The importance given to tail misspecification
is made even more evident by the ‘covtype1’ example. Here there is an ordering from TV, power,
Hellinger, alpha and KL divergence on both bias towards the left tail and on the size of the posterior
variance, in response to the slight negative skew of the KDE of the data. This effect is exaggerated
further by the large lump in the right tail of the ‘tracks1’ data. Here it is clear to see that the power,
TV and Hellinger divergence produce much better fits of the majority of the data than the other
methods do. Lastly, the bottom right plot demonstrates how several possible alternative models
to the Gaussian perform on the ‘tracks1’ data set5 . This shows that a Gaussian distribution was
actually the best fit for the bulk of the data, and the poor fit achieved is down to the importance
placed on tail misspecification by the estimating procedure rather than the model selected.

5.2

M-closed efficiency

The examples in Section 5.1, demonstrate how minimising an alternative divergence to the KLdivergence can lead to superior robustness to tail misspecifications. Next we aim to illustrate that
by placing less importance on tail misspecifications in order to gain improved robustness, the DM
3

downloaded from http://archive.ics.uci.edu/ml/datasets/Covertype
downloaded from https://archive.ics.uci.edu/ml/datasets/Geographical+Original+of+Music
5
the data set was transformed by adding min(tracks1)+0.001 to every value in order to make the data strictly positive
so the gamma and log-Normal distributions could be applied
4

20

covtype1

0.008
Density

0.000
2500

3000

3500

4000

0

50

100

150

x

x

tracks1

tracks1

200

0.4
0.0

0.0

0.1

0.1

0.2

0.3

density

0.4
0.3
0.2

250

Normal
t
log−Normal
gamma
data KDE

0.5

0.5

0.6

0.6

2000

density

0.004

0.0015
0.0010
0.0000

0.0005

Density

covtype9

data
KL
Hell
alpha
power
TV

−2

0

2

4

6

8

10

0

x

2

4

6

8

10

x

Figure 2: Posterior predictive distributions (smoothed from a sample) arising from Bayesian minimum
divergence estimation for the covtype1 (upper left), covtype9 (upper right), tracks1 (lower left) datasets
using the KL-Bayes (red), Hell-Bayes (blue), TV-Bayes (pink), α-Bayes (green) and power-Bayes
(orange). Lower right plots the posterior predictive distributions (smoothed from a sample) from
alternative models using the KL-Bayes, Normal (red), t (blue), logNormal (green) and gamma (orange).
is required to trade-off a decrease in efficiency when the model class does in fact contain the data
generating process. Minimising the KL-divergence uses Bayes rule and thus conditions on the
data coming from the model. As a result minimising the KL-divergence is bound to perform the
best when the model class contains the data generating distribution. Here we illustrate this but
also demonstrate the reassuring fact that the MDE methods perform somewhat similarly in this
situation.
PN
In order to examine the frequentist efficiency, the observed mean squared error (MSE) N1 j=1 (θ̂nj −
θ)2 , over N repeats of a simulated experiment are examined. The MSE’s are examined on two examples X1 ∼ N (0, 102 ) and X2 ∼ N (15, 102 ) when fitting the model Xi ∼ N (µi , σi2 ) with prior
µi ∼ N (0, 52 ) and σi ∼ G(0.01, 0.01) for i = 1, 2. The prior for µ1 is centred on the data while the
prior for µ2 is not. Table 1 plots the observed MSEs over N = 50 replications6 .
Table 1 demonstrates several interesting things about these different methods. Firstly it appears
as though the KL-Bayes, Hell-Bayes and alpha-Bayes perform very similarly when n = 500, while
the power-Bayes and TV-bayes performs marginally worse. As we should expect, for n < 500 the
KL-Bayes appears to be optimal for both priors with the α-Bayes appearing to perform the next
best. Table 1 also shows the differing impacts the prior has on the different Bayesian updates. The
KL-Bayes is least affected by the prior with the α-Bayes and the TV-Bayes the next least. Authors
Hooker and Vidyashankar (2014) recognised that the boundedness of the Hell-Bayes score function
compared with the KL-Bayes score function, may cause problems such as this. The α-Bayes score
function is larger than the Hell-Bayes score function so our findings would be consistent with this.
6

The stan output from a number of these experiments associated with the TV divergence suggested that the sample
form the posterior was not sufficient to use for posterior inference, for the purpose of this preliminary analysis these
experiments were disregarded here, but this suggests there exists computational issues associated with the TV-bayes.

21

Table 1: Table of posterior mean MSE values for µi , σi from data sets of size n = 50, 100, 200, 500
under the Bayesian minimum divergence technology.
MSE
µ

σ

n=50
n=100
n=200
n=500
n=50
n=100
n=200
n=500

KL
µ1 = 0
1.89
0.64
0.35
0.15
1.11
0.60
0.23
0.10

Hell

µ2 = 15
4.43
1.29
0.47
0.16
1.17
0.61
0.23
0.10

µ1 = 0
1.50
0.57
0.32
0.14
1.51
0.98
0.57
0.26

TV

µ2 = 15
16.88
5.19
1.50
0.33
1.28
0.87
0.52
0.26

µ1 = 0
2.68
1.20
0.48
0.28
3.47
1.30
0.42
0.29

alpha

µ2 = 15
6.77
1.83
0.50
0.31
6.09
1.61
0.44
0.27

µ1 = 0
2.03
0.67
0.35
0.15
1.13
0.65
0.30
0.12

power

µ2 = 15
4.29
1.25
0.47
0.16
1.13
0.64
0.29
0.13

µ1 = 0
0.76
0.44
0.31
0.20
8.23
1.99
0.46
0.18

µ2 = 15
93.80
33.80
7.83
1.42
67.94
10.64
1.09
0.21

The power-Bayes score function with α = 0.5 is smaller than the Hell-Bayes score function when
g < 0.25. This explains why the power-Bayes is impacted more by the prior, and provides a likely
explanation of why the power-Bayes appears to perform so well when the prior is specified around
the data. For small values of g, which occur when the variance of g is 100, the absolute size of
the TV-Bayes score function is generally larger than the Hell-Bayes score, again explaining why the
TV-Bayes performs better than the Hell-Bayes under a poorly specified prior.
While is appears as though the TV-Bayes performs comparably with the other methods here,
there were some experiments were the MCMC sampler ‘stan’ was unable to produce a sufficient
sample to use for posterior inference, these were disregarded for these preliminary investigations
but this could causes issues in practice.
Examining whether the estimates of σ were generally above or below the data generating parameter demonstrates how these different methods learn, see appendix table 2. For all sample sizes
the KL-Bayes appears to over or under estimate the true variance in equal proportions over the 50
repeats, while the Hell-Bayes and alpha-Bayes, to a lesser extent, generally under predict the true
variance.

0.2

0.4

0.6
f

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0
−2
−4

score

−8
−10
0.0

f

−6

−2
−4

score

−10

−8

−6

−2
−4

score

−6
−10

−8

8
6
score
4
2
0
0.0

g=0.75

0

g=0.5

0

10

g=0.25
KL
Hell
alpha=0.6
alpha=0.75
alpha=0.85

0.2

0.4

0.6
f

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

f

Figure 3: Score comparisons when quoting probability x for event with probability g. Red = KL, blue
= Hellinger (α = 0.5, then divide by 4), green = alpha (α = 0.6 = solid, α = 0.75 = dashed, α = 0.85
= dotted)
Figure 3 compares the scoring functions used in the general Bayesian update targeting the KL,
Hellinger and alpha divergences. As α increases, the upper bound on the score associated with
the alpha-divergence increases and the scoring functions become more convex, tending towards the
logarithmic score when α = 1. In contrast, the score associated with the Hellinger-divergence is
closer to being linear. We believe that this is responsible for causing the smaller variance estimates
when minimising the Hellinger-divergence and alpha-divergence when α = 0.75. The closer the
score is to being linear the more similar the penalty is for over and under predicting the probability
of an observation when compared with reality. The bounded nature of the scores prevents the
penalty for under predicting being too high, and therefore more posterior predictive mass is placed
near the MAP of the posterior predictive distribution. In contrast, a score function with greater

22

convexity will penalise under prediction compared with the data generating density to a greater
extent spreading the posterior mass out further. However, one must keep in mind that it is the
sever nature of the penalty for under prediction incurred by the KL-divergence that renders it nonrobust. On the other hand the TV-Bayes and the power-Bayes generally over predict the variance.
For the power-Bayes this appears to be because the second term of the power divergence score is
1
1+α

Z

f (y; θ)α+1 dy =

1
(1 +

α)1+d/2 (2π)dα/2 |Σ|α/2

(34)

for a multivariate Gaussian model, and this is decreased when the variances are made large. As
was established in Section 4.5, the power-Bayes with α = 0.5 decreases the influence observations
with small likelihood under the model have on the inference. In the example above the data has a
variance of σ 2 = 100 and therefore the likelihood of each observation is very small. As a result the
data is then excessively down-weighted by the power-Bayes when α = 0.5. When the sample size is
small, a small increase in the variance may only decrease the observed likelihood term of the powerdivergence score by a fraction and may increase the term in equation (34) by more, resulting in a
variance estimate that is too large. The score function for the TV-Bayes is the only score function
considered here that is not monotonic in the predicted probability of each observation. While
under the other scores, the score for each individual observation can be increased by predicting
that observation with greater probability, this is not the case for the TV-Bayes. We believe this
causes the TV-Bayes to produces variance estimates that are too large. We note that all of the
scores remain proper for large samples and that these are just observations about how the different
methods perform for small samples sizes when the model is correct.

6

Discussion

This article uses general Bayesian updating Bissiri et al. (2016) in order to theoretically justify a
Bayesian update that targets the parameters of a model that minimise a statistical divergence to
the data generating process that is not the KL-divergence. When the M -open world is considered,
moving away from targeting the minimisation of the KL-divergence can provide an important tool
in order to robustify a statistical analysis. The desire for robustness ought to only increase as bigger
and bigger models are built to approximate more complex real world processes. This paper provides
any statistical practitioner with the principled justification to select the divergence they use for their
analysis in a subjective manner allowing them the potential to make more useful predictions from
their best guess approximate belief model.
As it stands however more work is required to advise on the selection of the divergence used for
the updating. Ways to more clearly articulate the impact of choosing a certain divergence to a DM
needs further research along with ways to help guide the choice of any hyperparameters associated
with the divergence. Further experimentation with complex real world data sets is also required
to analyse how this robustness-efficiency trade-off associated with the selected divergence manifests
itself in practice.
Another area not addressed in this paper is the subject of tailored computational algorithms.
There exists a vast literature on optimising MCMC algorithms to sample from traditional Bayesian
posteriors and in order to fully take advantage of the subjectivity this paper allows a statistician,
a whole new class of computational algorithms tailored to different divergences may be required.
In addition several of the divergences mentioned in this paper require a density estimate of the
underlying process and further research into effectively doing this for complex high dimensional
datasets can only improve the performances of these methods for real world problems.

23

Appendix
Over or under estimation
Table 2: Table of sums of posterior mean over (positive) or under (negative) estimation for µi , σi from
data sets of size n = 50, 100, 200, 500 under the Bayesian minimum divergence technology
MSE

µ

σ

KL

Hell

TV

alpha

power

µ1 = 0

µ2 = 15

µ1 = 0

µ2 = 15

µ1 = 0

µ2 = 15

µ1 = 0

µ2 = 15

µ1 = 0

µ2 = 15

n=50

-12

-32

-10

-50

-5

-37

-12

-32

-8

-50

n=100

-16

-40

-14

-50

-15

-22

-18

-40

-12

-50

n=200

-2

-24

-8

-46

1

-10

-2

-26

-2

-50

n=500

-6

-16

-6

-42

1

3

-10

-18

-8

-48

n=50

4

6

-26

-10

25

27

-8

-6

46

50

n=100

0

0

-36

-32

19

12

-14

-14

38

48

n=200

-2

-2

-40

-40

23

22

-24

-24

28

40

n=500

-2

-2

-36

-40

27

21

-28

-28

14

20

References
Abramson, I. S. (1982). On bandwidth variation in kernel estimates-a square root law. The
annals of Statistics, 1217–1223.
Barron, A. R. (1988). The exponential convergence of posterior probabilities with implications for bayes estimators of density functions. Department of Statistics, University
of Illinois.
Basu, A., Harris, I. R., Hjort, N. L., & Jones, M. (1998). Robust and efficient estimation
by minimising a density power divergence. Biometrika, 85 (3), 549–559.
Basu, A., Shioya, H., & Park, C. (2011). Statistical inference: the minimum distance
approach. CRC Press.
Beran, R. (1977). Minimum hellinger distance estimates for parametric models. The Annals
of Statistics, 445–463.
Berger, J. O., Moreno, E., Pericchi, L. R., Bayarri, M. J., Bernardo, J. M., Cano, J. A., . . .
others (1994). An overview of robust bayesian analysis. Test, 3 (1), 5–124.
Bernardo, J. M., & Smith, A. F. (2001). Bayesian theory. IOP Publishing.
Bissiri, P., Holmes, C., & Walker, S. G. (2016). A general framework for updating belief
distributions. Journal of the Royal Statistical Society: Series B (Statistical Methodology).
Box, G. E. (1980). Sampling and bayes’ inference in scientific modelling and robustness.
Journal of the Royal Statistical Society. Series A (General), 383–430.
Carpenter, B., Gelman, A., Hoffman, M., Lee, D., Goodrich, B., Betancourt, M., . . . Riddell, A. (2016). Stan: A probabilistic programming language. Journal of Statistical
Software, 20 .
Celeux, G., Jewson, J., Josse, J., Marin, J.-M., & Robert, C. P. (2017). Some discussions
on the read paper” beyond subjective and objective in statistics” by a. gelman and

24

c. hennig. arXiv preprint arXiv:1705.03727 .
Cichocki, A., Cruces, S., & Amari, S.-i. (2011). Generalized alpha-beta divergences and
their application to robust nonnegative matrix factorization. Entropy, 13 (1), 134–
170.
Cressie, N., & Read, T. R. (1984). Multinomial goodness-of-fit tests. Journal of the Royal
Statistical Society. Series B (Methodological), 440–464.
Csisz, I., et al. (1967). Information-type measures of difference of probability distributions
and indirect observations. Studia Sci. Math. Hungar., 2 , 299–318.
Dawid, A. P., Musio, M., & Ventura, L. (2016). Minimum scoring rule inference. Scandinavian Journal of Statistics, 43 (1), 123–138.
Devroye, L., & Gyorfi, L. (1985). Nonparametric density estimation: the l1 view (Vol. 119).
John Wiley & Sons Incorporated.
Epanechnikov, V. A. (1969). Non-parametric estimation of a multivariate probability
density. Theory of Probability & Its Applications, 14 (1), 153–158.
Gelman, A., & Hennig, C. (2015). Beyond subjective and objective in statistics. Journal
of the Royal Statistical Society: Series A (Statistics in Society).
Ghosh, A., & Basu, A. (2016). Robust bayes estimation using the density power divergence.
Annals of the Institute of Statistical Mathematics, 68 (2), 413–437.
Ghosh, A., & Basu, A. (2017a). General robust bayes pseudo-posterior: Exponential
convergence results with applications. arXiv preprint arXiv:1708.09692 .
Ghosh, A., & Basu, A. (2017b). Improvements in the small sample efficiency of the minimum
s-divergence estimators under discrete models. arXiv preprint arXiv:1702.03557 .
Ghosh, A., Harris, I. R., Maji, A., Basu, A., Pardo, L., et al. (2017). A generalized
divergence for statistical inference. Bernoulli , 23 (4A), 2746–2783.
Goldstein, M. (1990). Influence and belief adjustment. Influence Diagrams, Belief Nets
and Decision Analysis, 143–174.
Goldstein, M. (1999). Bayes linear analysis. Wiley StatsRef: Statistics Reference Online.
Goldstein, M., et al. (2006). Subjective bayesian analysis: principles and practice. Bayesian
Analysis, 1 (3), 403–420.
Greco, L., Racugno, W., & Ventura, L. (2008). Robust likelihood functions in bayesian
inference. Journal of Statistical Planning and Inference, 138 (5), 1258–1270.
Grünwald, P., & van Ommen, T. (2014). Inconsistency of bayesian inference for misspecified
linear models, and a proposal for repairing it. arXiv preprint arXiv:1412.3730 .
Grünwald, P. D., & Dawid, A. P. (2004). Game theory, maximum entropy, minimum
discrepancy and robust bayesian decision theory. Annals of Statistics, 1367–1433.
Hampel, F. R., Ronchetti, E. M., Rousseeuw, P. J., & Stahel, W. A. (2011). Robust
statistics: the approach based on influence functions (Vol. 114). John Wiley & Sons.
Holmes, C., & Walker, S. (2017). Assigning a value to a power likelihood in a general
bayesian model. Biometrika, 104 (2), 497–503.
Hooker, G., & Vidyashankar, A. N. (2014). Bayesian model robustness via disparities.
Test, 23 (3), 556–584.
Huber, P. J., & Ronchetti, E. (1981). Robust statistics, series in probability and mathematical statistics. John Wiley, New York.
Hwang, J.-N., Lay, S.-R., & Lippman, A. (1994). Nonparametric multivariate density
25

estimation: a comparative study. IEEE Transactions on Signal Processing, 42 (10),
2795–2810.
Kuchibhotla, A. K., & Basu, A. (2015). A general set up for minimum disparity estimation.
Statistics & Probability Letters, 96 , 68–74.
Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014). The parable of google flu:
traps in big data analysis. Science, 343 (6176), 1203–1205.
Li, M., & Dunson, D. B. (2016). A framework for probabilistic inferences from imperfect
models. arXiv preprint arXiv:1611.01241 .
Lindsay, B. G. (1994). Efficiency versus robustness: the case for minimum hellinger distance
and related methods. The annals of statistics, 1081–1114.
Mandal, A., & Basu, A. (2013). Minimum disparity estimation: Improved efficiency through
inlier modification. Computational Statistics & Data Analysis, 64 , 71–86.
Miller, J. W., & Dunson, D. B. (2015). Robust bayesian inference via coarsening. arXiv
preprint arXiv:1506.06101 .
O’Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. R., Garthwaite, P. H., Jenkinson,
D. J., . . . Rakow, T. (2006). Uncertain judgements: eliciting experts’ probabilities.
John Wiley & Sons.
Owen, A. (1991). Empirical likelihood for linear models. The Annals of Statistics, 1725–
1747.
Rosenblatt, M., et al. (1976). On the maximal deviation of k-dimensional density estimates.
The Annals of Probability, 4 (6), 1009–1015.
Sason, I., & Verdú, S. (2015). Bounds among f-divergences. submitted to the IEEE Trans.
on Information Theory.
Shun-ichi, A. (2012). Differential-geometrical methods in statistics (Vol. 28). Springer
Science & Business Media.
Silverman, B. W. (1986). Density estimation for statistics and data analysis (Vol. 26).
CRC press.
Smith, J. (1995, October). Bayesian approximations and the hellinger metric. (Unpublished
manuscript)
Smith, J. Q. (2010). Bayesian decision analysis: principles and practice. Cambridge
University Press.
Tamura, R. N., & Boos, D. D. (1986). Minimum hellinger distance estimation for multivariate location and covariance. Journal of the American Statistical Association,
81 (393), 223–229.
Walker, S. G. (2013). Bayesian inference with misspecified models. Journal of Statistical
Planning and Inference, 143 (10), 1621–1633.
Watson, J., Holmes, C., et al. (2016). Approximate models and robust decisions. Statistical
Science, 31 (4), 465–489.
Winkler, R. L., & Murphy, A. H. (1968). Evaluation of subjective precipitation probability
forecasts. In Proceedings of the first national conference on statistical meteorology
(pp. 148–157).

26

