Qualitative Robustness in Bayesian Inference
Houman Owhadi and Clint Scovel
California Institute of Technology

arXiv:1411.3984v3 [math.ST] 20 Apr 2016

April 21, 2016

Abstract
The practical implementation of Bayesian inference requires numerical approximation when closed-form expressions are not available. What types of accuracy
(convergence) of the numerical approximations guarantee robustness and what types
do not? In particular, is the recursive application of Bayes’ rule robust when subsequent data or posteriors are approximated? When the prior is the push forward of a
distribution by the map induced by the solution of a PDE, in which norm should that
solution be approximated? Motivated by such questions, we investigate the sensitivity of the distribution of posterior distributions (i.e. of posterior distribution-valued
random variables, randomized through the data) with respect to perturbations of
the prior and data generating distributions in the limit when the number of data
points grows towards infinity.

1

Introduction and motivations

When we apply Bayesian inference with Gaussian priors and linear observations, we
do not actually compute Bayes’ rule but solve the linear system whose solution is the
conditional expectation, and robustness is guaranteed by that of our linear solver. This
paper is motivated by robustness questions that arise in the numerical implementation
of Bayes’ rule for continuous systems when closed-form expressions are not available for
the computation of posterior values/distributions. For example, what is the sensitivity
of posterior values or the sensitivity of the distribution of posterior values to perturbations introduced by the numerical approximation of the prior? When Bayes’ rule
is applied in a recursive manner and approximated posterior distributions are used as
prior distributions or approximated data is used in the conditioning process, do we have
robustness guarantees on subsequent posterior distributions/values? Is it possible to
numerically approximate the optimal prior/mixed strategy of a decision theory problem
(arising in the continuous setting under the complete class theorem [42]) when closed
form expressions are not available, and if it is, in which metric should we do so?
Figure 1.1 provides an illustration of partial answers to such questions, which, when
combined, can be used as a map to navigate robustness/non-robustness questions/issues
arising from numerical approximations. To begin, (a) [34, 33, 35] shows that the range of
posterior values of the quantity of interest under perturbations of the prior in Prokhorov
1

Figure 1.1: On the robustness of the Bayes’ rule. The top panel illustrates relationships
among probability metrics and, as in [23], a directed arrow from A to B means that
(for some function h) dA ≤ h(dB ). The central panel illustrates the generation of nonrobustness and the bottom panel illustrates the generation of robustness.

2

or TV metrics is the deterministic range of the quantity of interest (we will refer to
this maximal sensitivity property as brittleness). Moreover, (c.1) [41] shows that the
posterior distributions are controlled by the Hellinger metric under the application of
the Bayes rule with an exact prior under the perturbation of a finite number of finite
dimensional samples (see also [11]). Since the Hellinger and TV metrics are equivalent
[23], (c.1) and (a) suggest that, without specific restrictions, it is, in general, not possible to offer guarantees on the robustness of recursive Bayes under perturbations of
the data. Similarly (c.2) the convergence of Markov chains (on continuous state spaces)
used in MCMC algorithms (such as the Metropolis algorithm) is generallly analyzed
with respect to the TV topology [38, 22]. However it should be noted that, according
to Roberts and Rosenthal [37, Pg. 10], Gibbs [22], Gelman [20, Intro] and Madras and
Sezer [30, Intro], convergence in TV is in general not guaranteed, so that one might
say in general that convergence of MCMC is at best in TV. Therefore combining (c.2)
and (a) suggests that, without further restrictions, it is, in general, not possible to offer
guarantees on the robustness of recursive Bayes if posteriors are approximated using
MCMC. Moreover, in many cases, the prior may need to undergo a numerical approximation/discretization step prior to conditioning. For example, in popular applications
of Bayes’ rule to stochastic PDEs [44, 16] one pushes forward the prior from the space
of coefficients of the PDE to the solution space where it is conditioned. Consequently,
if the PDE is numerically approximated this implies an approximation of the pushedforward prior. Therefore (c.3) representing numerical discretization/approximation as
a continuous map between two Polish spaces, it is known that the push forward of a
measure under such maps is continuous in the weak topology, which is metrized by the
Prokhorov distance. Therefore, combining (c.3) and (a) suggests that, without further
restrictions, it is, in general, not possible to offer robustness guarantees to perturbations
caused by the numerical discretization of the prior.
For positive results on the other hand, observe that (b) [24] shows that posterior
values (given a finite number of data points) are robust to approximation errors of
the prior measured in Kullback-Leibler divergence. Therefore if (c.4) the numerical
approximation map is continuous in Kullback-Leibler divergence then posterior values
(given a finite number of data points) are robust to that numerical discretization step.
However observe that unless closed form expressions are available, one must keep track
of densities to achieve the continuity of the numerical approximation map in KullbackLeibler divergence, which is a task plagued by the curse of dimensionality.
Since the brittleness of posterior distributions and values with respect perturbations
of the prior defined in TV or Prokhorov metrics is an obstacle to obtaining robustness
guarantees to numerical approximation errors when closed form expressions are unavailable, it is natural to ask whether robustness could be guaranteed by considering the
distribution of posterior distributions or posterior values generated by the random generation of the data. To answer this question we develop a framework for quantifying the
sensitivity of the distribution of posterior distributions with respect to perturbations of
the prior and data generating distributions in the limit when the number of data points
grows towards infinity. In this generalization of Cuevas’ [13] extension of Hampel’s [27]

3

notion of qualitative robustness to Bayesian inference to include both perturbations of
the prior and the data generating distribution, posterior distributions are analyzed as
measure-valued random variables (measures randomized through the data) and their
robustness is quantified using the total variation, Prokhorov, and Ky Fan metrics. Our
results show that (1) the assumption that the prior has Kullback-Leibler support at the
parameter value generating the data, classically used to prove consistency, can also be
used to prove the non-robustness of posterior distributions with respect to infinitesimal
perturbations (in TV) of the class of priors satisfying that assumption, (2) for a prior
which has global Kullback-Leibler support on a space which is not totally bounded, we
can establish non-robustness and (3) consistency, and the unstable nature of the conditions which generate it, produces non-robustness, and a careful selection of the prior is
important if both properties (or their approximations) are to be achieved. The mechanisms supporting our results are different and complementary to those discovered by
Hampel and developed by Cuevas. To obtain them, we derive in Section 3 a corollary
to Schwartz’ consistency Theorem that leads to robustness or non-robustness results
depending on the topology defining the continuity of the numerical approximation map
(Kullback-Leibler, TV or Prokhorov). Moreover, this corrollary is further developed in
Proposition 7.1 to analyze the convergence of random measures in the qualitative robustness framework. More precisely, although in [24] it is shown that the Fréchet derivatives
of posterior values to Kullback-Leibler perturbations of the prior may diverge to infinity with the number of data points, a simple application of Proposition 7.1 implies the
robustness of the distribution of posterior distributions/values under Kullback-Leibler
perturbations of the prior in the limit where the number of data points goes to infinity.
On the other hand, application of Proposition 7.1 also suggest the lack of robustness of
the distribution of posterior distributions/values to TV or Prokhorov perturbations of
the prior, where we note that Prokhorov perturbations include classes of perturbations
defined by generalized moment constraints, as in [8, 6, 7, 5, 34, 33, 35].

2

Qualitative Robustness for Bayesian Inference

Hable and Christmann [25] have recently established qualitative robustness for support
vector machines. Consequently, it appears natural to inquire into the qualitative robustness of Bayesian inference. Hampel [27] introduced the notion of the qualitative
robustness of a sequence of estimators and Cuevas [13] has extended Hampel’s definition
and his basic structural results to Polish spaces. Since the space M(Θ) of priors and
posteriors equipped with the weak topology is Polish whenever Θ is, Cuevas’ extension
has direct applications to Bayesian inference. Boente et al. [10] have developed qualitative robustness for stochastic processes, Nasser et al. [32] for estimation, and Basu et
al.[3] for Bayesian inference with a single sample. The notion of qualitative robustness
introduce in this paper is a straightforward generalization of that introduced by Hampel
[27] and developed by Cuevas [13], see also Cuevas [15]. Indeed, this version requires
no introduction of loss and risk functions and concerns itself with not just expected
values but with the full distribution of the effects of the randomness of the observations.
4

It considers a fixed model so is not concerned with robustness with respect to model
specification, although such considerations can easily be included. Moreover, since it
is formulated with respect to classic notions in probability, it appears to us as simple,
natural, flexible, without calibration issues, and easy to interpret statistically.
Metrics on spaces of measures and random variables will be important in its formulation. Fortunately, this is a well studied field, see e.g. Rachev et al. [36] and Gibbs and
Su [23], but to keep the presentation simple, here we will restrict our attention to the
total variation, Prokhorov and Ky Fan metrics (we refer to [8, 6, 7, 5] for motivations
for considering classes of priors defined in the Prokhorov metric). For measurable spaces
Θ and X, we write M(Θ) and M(X) for the set of probability distributions on Θ and
X respectively. In this general setting, we can metrize the spaces of measures M(X)
and M(Θ) using total variation. This latter metrization makes M(Θ) into a topological space whose Borel structure can be used to define the space M2 (Θ) of probability
measures on M(Θ), which can also be metrized using the total variation. However, the
separability of these spaces will be extremely useful for us and, in general, these spaces
will not be separable under the total variation metric. Recall that for a metric space
(S, d), the Prokhorov metric dP r on the space M(S) of Borel probability measures is
defined by

dP r (µ1 , µ2 ) := inf  : µ1 (A) ≤ µ2 (A ) + , A ∈ B(S) , µ1 , µ2 ∈ M(S) ,
(2.1)
where
A := {x0 ∈ S : d(x, x0 ) <  for some x ∈ A} .
According to Dudley [18, Thm. 11.3.1], the Prokhorov metric is a metric on M(S). Consequently, when Θ and X are metric, , we can also metrize the spaces of measures M(Θ)
and M(X) with the Prokhorov metrics, and having done so we can define the space
M2 (Θ) := M(M(Θ)) of Borel probability measures on the metric space (M(Θ), dP rΘ )
of Borel probability measures on Θ and metrize it with the Prokhorov metric dP r2 Θ .
Furthermore, when (S, d) is a separable metric space, Dudley [18, Thm. 11.3.3] asserts
that the Prokhorov metric metrizes weak convergence and Aliprantis and Border [1,
Thm. 15.12] asserts that the metric space (M(S), dP r ) is separable.
Therefore, when X and Θ are separable metric spaces, the Prokhorov metrics dP rX
and dP rΘ metrize weak convergence in M(X) and M(Θ) respectively and both metric
spaces (M(X), dP rX ) and (M(Θ), dP rΘ ) are separable. Consequently, when Θ is a separable metric space, (M(Θ), dP rΘ ) is a separable metric space and therefore (M2 (Θ), dP r2 Θ )
is a separable metric space.
The separability of (M(Θ), dP rΘ ) is sufficient to define the Ky Fan metric on a space
of (M(Θ)-valued random variables. Indeed, for a separable metric space S, probability
space (Ω, Σ, P ), and two S-valued random variables Z : Ω → S and W : Ω → S, the Ky
Fan distance between Z and W , see e.g. Dudley [18, Pg. 289], is defined as

α(Z, W ) := inf  ≥ 0 : P (d(Z, W ) > ) ≤  .
(2.2)
By Dudley [18, Thm. 9.2.2], the Ky Fan metric is a metric on the space of S-valued
random variables from (Ω, Σ, P ) and metrizes convergence in probability for them. Con5

sequently, when Θ is a separable metric space and M(Θ) is metrized with the Prokhorov
metric dP rΘ , the Ky Fan metric α of (2.2) metrizes the space of M(Θ)-valued random
variables Z : (X ∞ , µ∞ ) → (M(Θ), dP rΘ ) for each µ ∈ M(X). Since this family of metrics depends on the measure µ we indicate this dependence by writing αµ . Moreover,
when Θ and X are separable metric spaces, their Borel σ-algebras are countably generated, which is required to apply Doob’s Theorem to assert that a domninated measurable
model has a jointly measurable family of densities, which is required in the consistency
theorem of Schwartz which we will need.
When Θ and X are Borel subsets of Polish metric spaces, they are separable metric spaces so the above applies. Let us now show that the assumption also facilitates
the measurability of Bayesian conditioning that will be needed to define its qualitative
robustness. To that end, from now on let us place as default the weak topologies on
M(X), M(Θ) and M2 (Θ) and metrize them with the Prokhorov metrics dP rX , dP rΘ
and dP r2 Θ . This is primarly to obtain well-defined Bayesesian conditioning while at
the same time applicability of Schwartz’s consistency theorem. We will also place other
metric structures on M(X), M(Θ) and M2 (Θ) to quantify the size of perturbations
and indicate them with the notation dM(X) , dM(Θ) and dM2 (Θ) . Consider a measurable
model P : Θ → M(X). Since Aliprantis and Border [1, Thm. 15.13] implies that the
map M(X) → R defined by µ 7→ µ(A) is Borel measurable for all A ∈ B(X), it follows
that P corresponds to a Markov kernel. Consider a prior π ∈ M(Θ). Then since Θ is
assumed to be a Borel subset of a Polish space, it follows from Schervish [39, Thm. B.46]
that there exists a family πx , x ∈ RX of conditional probability measures generated by the
model P such that the map x 7→ Θ f dπx , x ∈ X is B(X)-measurable for all bounded and
measurable functions f : Θ → R. Note that after the proof Schervish mentions that such
a family of conditional measures is not unique. Since both Θ and X are separable and
metrizable, it then follows from Aliprantis and Border [1, Thm. 19.7] that the resulting
map x 7→ πx from X to M(Θ) is measurable. For multiple samples, it is clear that X n
is a Borel subset of the n-th power of the ambient Polish space of X. By Billingsly’s
[9, Thm. 2.8] characterization of weak convergence on product spaces it follows that the
injection M(X) → M(X n ) defined by µ 7→ µn is continuous, so that it follows that
P n : Θ → M(X n ), defined by P n (θ) = P (θ)n , θ ∈ Θ, is measurable and therefore, by
the same arguments as above, we obtain a family of multisample conditional measures
πxn , xn ∈ X n such that the resulting map
π̄ : X n → M(Θ)
defined by the determination of the posteriors
π̄(xn ) := πxn ,

xn ∈ X n

(2.3)

is measurable. Therefore, its corresponding pushforward operator
π∗ : M(X n ) → M2 (Θ)
is well-defined, where we have removed the bar over π in the notation to emphasize
that this pushforward operator π∗ corresponds to the prior π. Then to consider how
6

the posteriors πxn vary as a function of the sample data xn when it is generated by
i.i.d. sampling from µ, since Mn (X) ⊂ M(X n ) it follows that µn ∈ M(X n ) so we can
utilize the pushforward operator π∗ to define
π∗ µn ∈ M2 (Θ)
the sampling distribution of the posterior distribution πxn when xn ∼ µn .
For a fixed prior π ∈ M(Θ), we say that the Bayesian inference is qualitatively
robust at a data generating distribution µ ∈ M(X) with respect to an admissible set P
containing µ, and metrics dM(X) and dM2 (Θ) on M(X) and M2 (Θ), if for any  > 0,
there exists a δ > 0 such that
µ́ ∈ P, dM(X) (µ, µ́) < δ =⇒ dM2 (Θ) (π∗ µn , π∗ µ́n ) < 
for large enough n. On the other hand, when the data generating distribution µ is fixed
and we vary the prior π, we consider the sequence of maps
πn : X ∞ → M(Θ)
defined by
πn (x∞ ) := πxn ,

x∞ ∈ X ∞ .

(2.4)

Since the projection Pn : X ∞ → X n is continuous and πn = π̄ ◦ Pn , it follows from the
measurability of π̄ : X n → M(Θ), that πn is measurable, and therefore the resulting
sequence
πn : (X ∞ , µ∞ ) → M(Θ)
is a sequence of M(Θ)-valued random variables. For each µ ∈ M(X), let αµ be a
metric on the space of M(Θ)-valued random variables whose domain is the probability
space (X ∞ , µ∞ ). Then, for a prior π ∈ M(Θ), we say that the Bayesian inference is
qualitatively robust at π with respect to an admissible set Π ⊂ M(Θ) containing π and
metrics αµ and dM(Θ) on M(Θ) if given  > 0, there exists a δ > 0 such that
π́ ∈ Π, dM(Θ) (π, π́) < δ =⇒ αµ (πn , π́n ) < 
for large enough n.
These two definitions can be combined in a straightforward manner to define robustness corresponding to a single prior/data generating pair. However, to consider a larger
2
class of distributions than a single pair, we let Z ⊂ M(Θ) × M(X) denote the admis
2
sible set of prior-data generating distribution pairs (π, µ), (π́, µ́) ∈ M(Θ) × M(X)
such that (π, µ) ∈ M(Θ) × M(X) is an admissible candidate for robustness and (π́, µ́) ∈
M(Θ) × M(X) is an admissible candidate for its perturbation. In particular, the projection Z1 ⊂ M(Θ) × M(X) denotes the set of admissible prior-data generating pairs.
Now combining in a straightforward manner we obtain:

7

Definition 2.1. Let X and Θ be Borel subsets of Polish metric spaces and let M(X) and
M(Θ) be equipped with the weak topology metrized by the Prokhorov metrics dP rX and
dP rΘ . Let M2 (Θ) := M(M(Θ)) be the space of Borel probability measures on the metric
space (M(Θ), dP rΘ ) of Borel probability measures on Θ equipped with its weak topology
metrized by its Prokhorov metric dP r2 Θ . Consider perturbation pseudometrics dM(X) ,
dM(Θ) and dM2 (Θ) on M(X), M(Θ) and M2 (Θ) respectively and for each µ ∈ M(X), let
αµ be a pseudometric on the space of M(Θ)-valued random variables on the probability
2
space (X ∞ , µ∞ ). Let Z ⊂ M(Θ) × M(X) denote the admissible set of prior-data
generating distribution pairs and suppose that P : Θ → M(X) is measurable. Then the
Bayesian inference is qualitatively robust with respect to Z, if given 1 , 2 > 0, there
exists δ1 , δ2 > 0 such that

(π, µ), (π́, µ́) ∈ Z, dM(Θ) (π, π́) < δ1 , dM(X) (µ, µ́) < δ2
=⇒ dM2 (Θ) (π́∗ µn , π́∗ µ́n ) < 1 ,

αµ (πn , π́n ) < 2

for large enough n.
Finite sample versions, as introduced in Hable and Christmann [26, Def. 2], are also
available. Note that unlike Hampel and Cuevas who require “for all n” in their definitions, we follow Huber [28] and Mizera [31] in only requiring closeness “for large enough
n”. The results of this paper are applicable to both versions. Of course the relevance
of the specific notion of qualitative robustness used depends on the perturbation metrics used. The results of this paper apply to the case when αµ is the Ky Fan metric,
metrizing convergence in probability on the space of M(Θ)-valued random variables
with domain the probability space (X ∞ , µ∞ ) and the dM(Θ) is any metric weaker than
the total variation.

3

Lorraine Schwartz’ Theorem

The fundamental mechanism generating non robustness for Bayesian inference will be
its consistency. The breakthrough in consistency for Bayesian inference is considered
to be Schwartz’ theorem [40, Thm. 6.1], so we use it as a model for consistency and
the conditions sufficient to generate it. Stated in Barron, Schervish and Wasserman
[2, Intro], Wasserman [43, Pg. 3] and Ghosal, Ghosh and Ramamoorthi [21, Cor. 1]
for the nonparametric case, for the parametric case we will operate in Borel subsets of
Polish metric spaces. By Schervish [39, Thm. B.32], regular conditional probabilities
exist for conditioning random variables with values in such a space. Moreover, when the
parametric model is a Markov kernel and is dominated by a σ-finite measure, then by
the Bayes’ Theorem for densities Schervish [39, Thm. 1.31] we have, in addition, that the
Bayes’ rule for densities determines a valid family of densities for the regular conditional
distributions. We say that a model P : Θ → M(X) is dominated if there exists a σ-finite
Borel measure ν on X such that Pθ  ν, θ ∈ Θ.

8

Recall the Kullback-Leibler divergence K between two measures µ1 and µ2 defined
by
Z
K(µ1 , µ2 ) :=

log

 dµ . dµ 
1
2
dµ1 ,
dν
dν

where ν is any measure such that both µ1 and µ2 are absolutely continuous with respect
to ν. It is well known thatRK is nonnegative, and that it is finite only if µ1  µ2 , and
1
in that case K(µ1 , µ2 ) = log dµ
dµ2 dµ1 . From this we can define the Kullback-Leibler
ball K (µ) of radius  about µ ∈ M(X) by K (µ) = {µ0 ∈ M(X) : K(µ, µ0 ) ≤ }.
For a model P : Θ → M(X), there is the pullback to a function K on Θ defined by
K(θ1 , θ2 ) := K(Pθ1 , Pθ2 ) and when the model is dominated by a σ-finite measure ν, if
θ
we let p(x|θ) := dP
dν (x), x ∈ X be a realization of the Radon-Nikodym derivative, then
the pullback has the form
Z
p(x|θ1 )
dPθ1 (x) .
K(θ1 , θ2 ) := log
p(x|θ2 )
From this we define a Kullback-Leibler neighborhood of a point θ ∈ Θ by

K (θ) := θ0 ∈ Θ : K(θ, θ0 ) ≤  .
Let us define the set of priors K(θ) ⊂ M(Θ) which have Kullback-Leibler support at θ
by
n
o

K(θ) := π ∈ M(Θ) : π K (θ) > 0,  > 0 ,
(3.1)
which implicitly requires that K (θ) be measurable1 for all  > 0. Also let K ⊂ M(Θ)
denote those measures with global Kullback-Leibler support, that is,
K := ∩θ∈Θ K(θ)
is the set of priors which have Kullback-Leibler support at all θ, and let Kae ⊃ K, defined
by
(
)
n
o

ae
K := π ∈ M(Θ) : π θ ∈ Θ : π K (θ) > 0,  > 0 = 1 ,
(3.2)
denote the set of priors with almost global Kullback-Leibler support.
Let us address the measurability of the Kullback-Leibler neighborhoods K (θ) ⊂
Θ,  > 0. For the nonparametric case, Barron, Schervish and Wasserman [2, Lem. 11]
demonstrate that the Kullback-Leibler neighborhoods K (Pθ∗ ) ⊂ M(X) are measurable
with respect to the strong topology restricted to the subspace of measures which are
absolutely continuous with respect to a common σ-finite reference measure. For the
parametric case, Dupuis and Ellis [19, Lem. 1.4.3] assert that on a Polish space that K
is lower semicontinuous in both arguments. Since the subset embedding : X → X 0 of
Note the change from the standard definition K (µ) = {µ0 : K(µ, µ0 ) < } to ours K (µ) = {µ0 :
K(µ, µ0 ) ≤ } does not affect which measures have Kullback-Leibler support, but is more convenient
since then K (µ) is closed, simplifying the proof that K (θ) is measurable.
1

9

a subset X of a metric space X 0 is isometric, when X is a Borel subset of a separable
metric space X 0 , it can be shown that the induced pushforward map i∗ : M(X) →
M(X 0 ) is isometric in the Prokhorov metrics, in particular it is continuous. Since the
composition of a continuous and a lower semicontinuous function is lower semicontinuous,
it follows from Dupuis and Ellis [19, Lem. 1.4.3] that on any realization of a standard
Borel space that the Kullback-Leibler divergence is lower semicontinuous in each of its
arguments separately, in particular, fixing the first, it is lower semicontinuous. Therefore
K (Pθ∗ ) ⊂ M(X) is closed, and therefore measurable for  > 0. Consequently, when P
is measurable, it follows that K (θ∗ ) ⊂ Θ is measurable for  > 0.
The following corollary to Schwartz’ Theorem, and its implications in Proposition
7.1, gives us the form of consistency that we will use in the robustness analysis. Since the
σ-algebra of a Borel subset of a Polish space is countably generated, Doob’s Theorem in
Dellacherie and Meyer [17, Thm. V.58] and the measurability of the dominated model P
implies that a family p(θ), θ ∈ Θ of densities can be chosen to be B(X)×B(Θ) measurable,
so that this assumption of Schwartz’ Theorem [40, Thm. 6.1] is satisfied. We note that
Dellacherie and Meyer emphasize that the countably generated condition is indispensable
for Doob’s Theorem to apply. Note the assumption that the map P : Θ → P (Θ) be
open.
Corollary 3.1 (Schwartz). Let X and Θ be Borel subsets of Polish metric spaces and
equip M(X) and M(Θ) with the Prokhorov metrics. Consider an injective measurable
dominated model P : Θ → M(X) such that P : Θ → P (Θ) is open. Then for every π ∈
M(Θ) with Kullback-Leibler support at θ∗ ∈ Θ and for every measurable neighborhood
U of θ∗ , we have
πxn (U ) → 1 n → ∞, a.e. Pθ∞
∗ .

4

Main Results

Now that we have defined qualitative robustness for Bayesian inference and presented
the consistency conditions of Schwartz’ Corollary 3.1, we are now prepared for our main
results. Indeed, the brittleness results of [34, 33, 35] and the non qualitative robustness
results of Cuevas [13, Thm. 7] suggest that we may obtain non qualitative robustness
according to Definition 2.1 by fixing the prior and varying the data generating distribution. However, according to Berk [4], in the misspecified case, although “there need be
no convergence (in any sense)”, in the limit the posterior becomes confined to a carrier
set consisting of those points which are closest in terms of the Kullback-Leibler divergence. Consequently, it appears possible that a generalization of the results of Hampel
[27, Lem. 3] and Cuevas [13, Thm. 1] which allows such a set-valued notion of consistency may be sufficient. Certainly it will require the more sophisticated notions of the
continuity, or semi-continuity, of the Kullback-Leibler set-valued information projection
and its dependence on the geometry of the model class P (Θ) ⊂ M(X). Although this
path will certainly be instructive and appears feasible, we instead find it simpler to
obtain non qualitative robustness by fixing the data generating distribution to be in

10

the model class and varying the prior. In particular, we show that the inference is not
robust according to Definition 2.1 when the metric αµ is the the Ky Fan metric and the
metric M(Θ) is any that is weaker than the total variation metric. It is important to
note that these results do not require any misspecification. Moreover, it appears that
Bayesian Inference’s dependence on both the data generating distribution and the prior
leads to two complementary mechanisms generating non qualitative robustness; whereas
Cuevas’ result [13, Thm. 7] utilizes consistency and the discontinuity of the infinite sample limit, this other component utilizes the non-robustness of consistency, namely that
the set of consistency priors, those with Kullback-Leibler support at the data generating
distribution, is not robust.
Now let us return to our main results. For θ ∈ Θ, let us recall from (3.1) the set of
priors K(θ) ⊂ M(Θ) with Kullback-Leibler support at θ and, for ρ > 0, define a total
variation uniformity Πρ (θ) ⊂ M(Θ) × M(Θ) by
Πρ (θ) := {(π, π́) ∈ M(Θ) × M(Θ) : π ∈ K(θ), dtv (π, π́) < ρ}
of prior pairs where the first component has Kullback-Leibler support at θ and the second
component is within ρ of the first in the total variation metric. For θ ∈ Θ, we define
2 an
admissible set of prior-data generating distribution pairs Zρ (θ) ⊂ M(Θ) × M(X) by
Zρ (θ) := Πρ (θ) × Pθ × Pθ ,

(4.1)

2
using the identification of M(Θ) × M(X) with M(Θ)2 × M(X)2 .
Our Main Theorem shows, under the conditions of Schwartz’ Corollary, that the
Bayesian inference is not robust under the assumption that the prior has KullbackLeibler support at the parameter value generating the data. This result, along with those
that follow, supports Cuevas’ [14] statement that “his results suggest the possibility of
proving the instability (i.e. the lack of qualitative robustness) for a wide class of usual
Bayesian models.”
Theorem 4.1. Consider Definition 2.1 with the total variation metric dM(Θ) := dtv on
M(Θ) and the Ky Fan metric αµ on the space of M(Θ)-valued random variables with
domain the probability space (X ∞ , µ∞ ). Given the conditions of Schwartz’ Corollary
3.1, for all θ ∈ Θ the Bayesian inference is not qualitatively robust with respect to Zρ (θ)
for all ρ > 0.
Remark 4.2. Actually the proof shows more; let D denote the diameter of Θ, then
for  < min ( D
2 , 1), there does not exist a δ > 0 such that robustness is satisfied. Since
D
min ( 2 , 1) is large, either half the diameter of the space or larger than 1, we say the
inference is brittle.
Theorem 4.1 does not assert that the Bayesian inference is not robust at any specified
prior, only that it is not robust under the assumption that the prior has Kullback-Leibler
support at the parameter value generating the data. To establish non-robustness at

11

specific priors we include variation in the data-generating distribution in the model class
as follows. Let ∆P ⊂ M(X) × M(X), defined by
∆P = {(Pθ , Pθ ), θ ∈ Θ} ,
denote the fact that we allow the data generating distribution to vary throughout the
model class but do not allow any perturbations to it. Then, for π ∈ M(Θ), define the
2
admissible set Zρ (π) ⊂ M(Θ) × M(X) by
Zρ (π) := π × Bρtv (π) × ∆P ,
where Bρtv (π) is the open ball in the total variation metric.
Since the following theorem is a corollary to the theorem after it, Theorem 4.4, we
do not include its proof. However, we state it here because it is the more fundamental
result.
Theorem 4.3. Consider the sitiuation of Theorem 4.1 with Θ not totally bounded. Then
if the prior π has Kullback-Leibler support for all θ ∈ Θ, the Bayesian inference is not
qualitatively robust with respect to Zρ (π) for all ρ > 0.
Since a metric space is totally bounded if and only if its completion is compact,
when Θ is totally bounded, we assume that it is a Borel subset of a compact metric
space. In this case, although Theorem 4.3 does not apply, utilizing the covering number
and packing number inequalities of Kolmogorov and Tikhomirov [29], we can provide
a natural quantification of qualitative robustness. To that end, we define covering and
packing numbers. For a finite subset Θ0 ⊂ Θ, the finite collection of open balls {B (θ), θ ∈
Θ0 } is said to constitute a covering of Θ if Θ ⊂ ∪θ∈Θ0 B (θ). For a finite set Θ0 we denote
its size by |Θ0 |. The covering numbers are defined by
n
o
N (Θ) = min |Θ0 | : Θ ⊂ ∪θ∈Θ0 B (θ) ,
that is, N (Θ) is the smallest number of open balls of radius  centered on points in
Θ which covers Θ. On the other hand, a set of points Θ0 ⊂ Θ is said to constitute an
-packing if d(θ1 , θ2 ) ≥ , θ1 6= θ2 ∈ Θ0 . The packing numbers are then defined by
n
o
M (Θ) := max |Θ0 | : Θ0 is an -packing of Θ .
Since the Kolmogorov and Tikhomirov [29, Thm. IV] inequalities
M2 (Θ) ≤ N (Θ) ≤ M (Θ)

(4.2)

are valid in the not totally bounded case, if we allow values of ∞, the following theorem
has Theorem 4.3 as its corollary.
Theorem 4.4. Given the conditions of Theorem 4.3 with Θ totally bounded. If the
Bayesian inference is qualitatively robust with respect to Zρ (π) for some ρ > 0, then
given 2 > 0, we must have


1
δ1 < min
,ρ .
N22 (Θ)
12

Figure 5.1: The data generating distribution is P (θ∗ ). π 0 has most of its mass around
θ. π is an arbitrarily small perturbation of π 0 so that π has Kullback-Leibler support at
θ∗ . Corollary 3.1 implies that πn converges towards δθ∗ while πn0 remains close to δθ .

5

Mechanisms generating non-robustness

For the clarity of the paper, in this subsection, we illustrate some of the mechanisms generating non qualitative robustness in Bayesian inference, which complement the mechanism discovered by Hampel [27, Lem. 3] and Cuevas [13, Thm. 1]. These mechanisms do
not utilize misspecification. Those which do are discussed in Subsection 5.1. The core
mechanism is derived from the nature of both the assumptions and assertions of results
supporting consistency. More precisely, Corollary 3.1 states that if the data generating
distribution is µ = P (θ∗ ) and if the prior π attributes positive mass to every KullbackLeibler neighborhood of θ∗ ∈ Θ, then the posterior distribution converges towards δθ∗
as n → ∞. The assumption that π attributes positive mass to every Kullback-Leibler
neighborhood of θ∗ ∈ Θ does not require π to place a significant amount of mass around
θ∗ , but instead can be satisfied with an arbitrarily small amount. Therefore, if, as in
Figure 5.1, π is a prior distribution with support centered around θ 6= θ∗ , but with a very
small amount of mass about θ∗ , so that it satisfies the assumptions of Corollary 3.1 at
θ∗ , then π can be slightly perturbed into a π 0 with support also centered around θ 6= θ∗ ,
but with no mass about θ∗ . In this situation, although π and π 0 can be made arbitrarily
close in total variation distance, the posterior distribution of π converges towards δθ∗ as
n → ∞, whereas that of π 0 remains close to δθ . Figure 5.2 gives an illustration of the
same phenomenon when the parameter space Θ is the interval [0, 1] and the probability
density functions of π and π 0 are p and p0 with respect to the uniform measure.
Note that the mechanism illustrated in Figures 5.1 and 5.2 does not generate non
qualitative robustness at all priors but instead for the full class of consistency priors, defined by the assumption of having positive mass on every Kullback-Leibler neighborhood
of θ∗ . One may wonder whether this non qualitative robustness can be avoided by selecting the prior π to satisfy Cromwell’s rule (that is, the assumption that π gives strictly
positive mass to every nontrivial open subset of the parameter space Θ). Theorem 4.3
shows that this is not the case if the parameter space Θ is not totally bounded. For
example, when Θ = R, for all δ > 0 one can find θ ∈ R such that the mass that π places

13

(a) Density of π

(b) Density of π

(c) Density of π 0

Figure 5.2: The data generating distribution is P (θ∗ ). The probability density functions
of π and π 0 are p and p0 with respect to the uniform distribution on [0, 1]. π is an
arbitrarily small perturbation of π 0 in total variation. πn converges towards δθ∗ while
the distance between the support of πn0 and θ∗ remains bounded from below by a > 0.

on the ball of center θ and radius one is smaller than δ, and by displacing this small
amount of mass one obtains a perturbed prior π 0 whose posterior distribution remains
asymptotically bounded away from that of π when the data-generating distribution is
P (θ). Similarly if Θ is totally bounded then Theorem 4.4 places an upper bound on the
size of the perturbation of the prior π that would be required as a function of the covering
complexity of Θ. Note that these observations suggest that a maximally qualitatively
robust prior should place as much mass as possible near all possible candidates θ for the
parameter θ∗ of the data generating distribution, thereby reinforcing the notion that a
maximally robust prior should have its mass spread as uniformly as possible over the
parameter space.

5.1

Robustness under misspecification

Although the main results of Section 4 do not utilize any model misspecification, the
brittleness results of [34] suggest that misspecification should also generate non qualitative robustness. Indeed, although, one may find a prior that is both consistent and
qualitatively robust when Θ is totally bounded and the model is well-specified, we now
show how extensions of the mechanism illustrated in Figures 5.1 and 5.2 suggest that
misspecification implies non qualitative robustness. Consider the example illustrated in
Figure 5.3. In this example the model P is the restriction of a well specified larger
model P̄ : Θ1 × Θ2 → M(X) to θ2 = 0. Assume that the data generating distribution
is P̄ (θ1∗ , θ2∗ ) where θ2∗ 6= 0, so that the restricted model P is misspecified. Let π be
any prior distribution on Θ1 × {θ2 = 0}. Although π may satisfy Cromwell’s rule the
mechanisms presented in this paper suggest that is not qualitatively robust with respect
to perturbed priors having support on Θ1 × Θ2 . Indeed, let π 0 be an arbitrarily small
perturbation of π obtained by removing some mass from the support of π and adding
that mass around θ∗ . Note that π 0 can be chosen arbitrarily close to π while satisfying
14

(b) π 0

(a) π

Figure 5.3: Non-robustness caused by misspecification. The parameter space of the
model is Θ1 . We assume that the model P : Θ1 → M(X) is the restriction of an injective
model P̄ : Θ1 × Θ2 → M(X) to Θ1 × {θ2 = 0}. The data generating distribution is
P̄ (θ∗ ) where θ∗ := (θ1∗ , θ2∗ ), with θ2∗ 6= 0, so that the model P is misspecified. π satisfies
Cromwell’s rule. π 0 is an arbitrarily small perturbation of π having Kullback-Leibler
support at θ∗ . Corollary 3.1 implies that πn0 converges towards δθ∗ while the distance
between the support of πn and θ∗ remains bounded from below by a > 0.

(a) P∗ π

(b) ν

Figure 5.4: Non-robustness caused by misspecification. The parameter space of the
model is Θ. The data generating distribution is µ 6∈ P (Θ), so that the model is misspecified. ν is an arbitrarily small perturbation of P∗ π in total variation distance having
non-zero mass on µ. Note that if a small mass on µ is sufficient to ensure the consistency
of ν then such a mechanism also implies the non-robustness of the model with respect to
misspecification since in such a case, νn , the posterior distribution of ν would converge
towards δµ whereas the distance between the support of P∗ πn and µ remains bounded
from below by the distance from the model to the data generating distribution.

15

the local consistency assumption of Corollary 3.1, which implies that the posterior distributions of π 0 concentrate on θ∗ while the posterior distributions of π remain supported
on Θ1 × {θ2 = 0}. Note that if P̄ is interpreted as an extension of the model P , then this
mechanism suggests that we can establish conditions under which Bayesian inference is
not qualitatively robust under model extension.
Figure 5.4 represents a non-parametric generalization of the mechanism of Figure
5.3. Assume that the data generating distribution is µ 6∈ P (Θ), so that the model is
misspecified. Let π ∈ M(Θ) be an arbitrary prior distribution and P∗ π ∈ M2 (X) its
corresponding non-parametric prior. By removing an arbitrarily small amount of mass
from P∗ π and placing it on µ one obtains an arbitrarily close prior distribution ν that
is consistent with respect to the data generating distribution µ. Therefore although
P∗ π and ν may be made arbitrarily close, their posterior distributions would remain
asymptotically separated by a distance corresponding to the degree of misspecification
of the model (the distance from µ to P (Θ)).

6
6.1

Proofs
Proof of Corollary 3.1

We seek to apply Schwartz’ theorem [40, Thm. 6.1]. Since Θ and X are separable metric
spaces, their Borel σ-algebras are countably generated, so Doob’s Theorem [17, Thm.
V.58] and the measurability of the dominated model P implies that a family of densities
can be chosen to be B(X) × B(Θ) measurable, thus satisfying this requirement of [40,
Thm. 6.1]. Since U is a neighborhood it follows that it contains an open neighborhood
O of θ. Since O is open and P : Θ → P (Θ) is open, it follows that P (O) is open in
P (Θ), and therefore there is an open set V∗ ⊂ M(X) such that V∗ ∩ P (Θ) = P (O).
Moreover, V∗ is an open neighborhood of Pθ∗ . Since X is a separable metric space, it
follows that dP rX metrizes the weak topology, and since V∗ is open, it is well known
(see e.g. [2, 43, 21]) that there exists a uniformly consistent test of Pθ∗ against V∗c , see
Schwartz [40] for the definition of uniformly consistent test. It follows trivially that
there exists a uniformly consistent test of Pθ∗ against V∗c ∩ P (Θ). Moreover, since P is
injective it follows that Oc = P −1 (V∗c ). Therefore, there exists a uniformly consistent
test of Pθ∗ against V∗c ∩ P (Θ) = {Pθ : θ ∈ Oc }.
Since V∗ is open, it also follows that there is a Prokhorov metric ball Bs (Pθ∗ ) of
radius s > 0 about Pθ∗ such that Bs (Pθ∗ ) ⊂ V∗ . Now consider the Kullback-Leibler
2
ball Kτ (Pθ∗ ) for τ < s2 . It follows from Csiszar, Kemperman and Kullback’s [12]
improvement K ≥ 12 d2tv of Pinsker’s inequality and the inequality dtv ≥ dP rX , that
Kτ (Pθ∗ ) ⊂ Bs (Pθ∗ ). Since then Kτ (Pθ∗ ) ⊂ Bs (Pθ∗ ) ⊂ V∗ it follows that

P −1 Kτ (Pθ∗ ) ⊂ P −1 (V∗ ) = O .
Consider now the Kullback-Leibler neighborhood Wτ (θ∗ ) ⊂ Θ of θ∗ defined by pulling
Kτ (Pθ∗ ) back to Θ by the model P :

Wτ (θ∗ ) := P −1 Kτ (Pθ∗ ) .
16

Then the previous inequality states that
Wτ (θ∗ ) ⊂ O .
Since the Kullback-Leibler neighborhoods are measurable in the weak topology and P
is assumed measurable, it follows that Wτ (θ∗ ) is measurable.
Therefore, O and Wτ (θ∗ ) satisfy the assumptions of the sets V and and W in [40,
Thm. 6.1]. Consequently, since by assumption, the prior π has Kullback-Leibler support,
it follows that we can apply Schwartz’ theorem [40, Thm. 6.1] to obtain the assertion
for O and since U ⊃ O is measurable the assertion follows.

6.2

Proof of Theorem 4.1

Let us prove the assertion for a weaker pseudometric άµ ≤ αµ derived from the Prokhorov
metric on dP r2 Θ on M2 (Θ). Since it is weaker the assertion follows. To that end, consider
µ ∈ M(X). Then for two random variables Z, W : (X ∞ , µ∞ ) → M(Θ) it follows that
Z∗ µ∞ , W∗ µ∞ ∈ M2 (Θ), so we can define a pseudometric άµ by
άµ (Z, W ) := dP r2 Θ (Z∗ µ∞ , W∗ µ∞ ) .

(6.1)

Since Dudley [18, Thm. 11.3.5] asserts that
dP r2 Θ (Z∗ µ∞ , W∗ µ∞ ) ≤ αµ (Z, W ) ,

(6.2)

άµ ≤ αµ .

(6.3)

we conclude that
For fixed π and µ and n, the M(Θ)-valued random variable
πn : (X ∞ , µ∞ ) → M(Θ)
defined by πn (x∞ ) := πxn in (2.4) satisfies
(πn )∗ µ∞ = π∗ µn

(6.4)

where π∗ : M(X n ) → M2 (Θ) is the pushforward operator corresponding to the map
π̄ : X n → M(Θ) defined by π̄(xn ) := πxn in (2.3). Consequently, we obtain
άµ (πn , π́n ) = dP r2 Θ (π∗ µn , π́∗ µn )

(6.5)

for π, π́ ∈ M(Θ) and n fixed. From the triangle inequality we then obtain
dP r2 Θ (π∗ µn , π́∗ µ́n ) ≤ dP r2 Θ (π∗ µn , π́∗ µn ) + dP r2 Θ (π́∗ µn , π́∗ µ́n )
≤ αµ (πn , π́n ) + dP r2 Θ (π́∗ µn , π́∗ µ́n ) ,
bounding the simple single term dP r2 Θ (π∗ µn , π́∗ µ́n ) in terms of the sum of two terms
αµ (πn , π́n ) and dP r2 Θ (π́∗ µn , π́∗ µ́n ) of qualitative robustness in Definition 2.1. Conseuquently, the assumption of the pseudometric άµ amounts to Definition 2.1 with one
epsilon instead of two corresponding to the metric
dP r2 Θ (π∗ µn , π́∗ µ́n ),

(π, π́, µ, µ́) ∈ M(Θ)2 × M(X)2 .
17

(6.6)

Moreover, non qualitative robustness with respect to this definition implies non qualitative robustness with respect to the original Definition 2.1 with the Ky Fan metric.
Now lets turn to the proof that the inference is non qualitatively robust with respect
to the objective metric (6.6). Fix θ∗ ∈ Θ and consider another point θ ∈ Θ and the
Dirac mass δθ ∈ M(Θ) situated at θ. For π ∈ K(θ∗ ), the convex combination
π α := απ + (1 − α)δθ
is a probability measure with Kullback-Leibler support, that is, π α ∈ K(θ∗ ), α > 0 and
dtv (π α , δθ ) ≤ α.

(6.7)

Therefore, it follows that
(π α , δθ ) ∈ Πρ (θ∗ ) ,

α < ρ,

and therefore

π α , δθ , Pθ∗ , Pθ∗ ∈ Zρ (θ∗ ) ,

α < ρ,

where Zρ (θ∗ ) is the admissible set defined in (4.1).
α ∞ ) := π α , x∞ ∈ X ∞ ,
For the prior π α , let πnα : (X ∞ , Pθ∞
∗ ) → M(Θ), defined by πn (x
xn
denote the corresponding sequence of posterior random variables, and let (πnα )∗ Pθ∞
∗ ∈
2
M (Θ) denote its induced sequence of laws. On the other hand, for the prior δθ , it is
easy to see that (δθ )xn = δθ , xn ∈ X n , so that if we denote the corresponding sequence
n
of posterior random variables by δθn , then (δθn )∗ Pθ∞
∗ = (δθ )∗ Pθ ∗ = δδθ .
Since the assumptions of Schwartz’ Corollary 3.1 are satisfied and π α has KullbackLeibler support at θ∗ , we can apply the assertion (7.1) of Proposition 7.1
n
o
dP rΘ (πnα , δθ∗ ) >  → 0 n → ∞ ,
Pθ∞
∗
for  > 0. To complete the proof we simply use the fact that convergence in law to a
Dirac mass is equivalent to convergence in probability to a constant random variable,
that is use the equivalent assertion (7.2) of Proposition 7.1


dP r2 Θ (πnα )∗ Pθ∞
→ 0 n → ∞.
(6.8)
∗ , δδ ∗
θ
Now the proof is very simple. Indeed, from the triangle inequality we have






α
∞
dP r2 Θ (πnα )∗ Pθ∞
,
δ
≥
d
δ
,
δ
−
d
(π
)
P
,
δ
2
2
∗
δθ ∗
δθ
P r Θ δθ ∗ δθ
Pr Θ
n ∗ θ∗
and, by two applications of Proposition 7.4, we have




 
dP r2 Θ δδθ∗ , δδθ
= min dP rΘ δθ∗ , δθ , 1

 
= min min d(θ∗ , θ), 1 , 1

= min d(θ∗ , θ), 1 .
18

Therefore, since (δθn )∗ Pθ∞
∗ = δδθ , the convergence (6.8) implies that



n
∞
dP r2 Θ (πnα )∗ Pθ∞
,
(δ
)
P
→ min d(θ∗ , θ), 1 , n → ∞ .
∗
θ ∗ θ∗
Finally, since dP rΘ ≤ dtv , it follows from (6.7) that
dP rΘ (π α , δθ ) ≤ α.
Then, for any δ > 0, if we restrict α so that α < min (δ, ρ), it follows that dtv (π α , δθ ) < ρ
and dP rΘ (π α , δθ ) < δ, so that

π α , δθ , Pθ∗ , Pθ∗ ∈ Zρ (θ∗ ) ,
(6.9)
dP rΘ (π α , δθ ) < δ .

(6.10)

Let D := sup {d(θ1 , θ2 ) : θ1 , θ2 ∈ Θ} denote the diameter of Θ. Then it follows from the
triangle inequality that, for any  > 0, there exists a θ ∈ Θ such that d(θ∗ , θ) ≥ D
2 − .
Consequently, for any ¯ < min ( D
,
1),
no
matter
how
small
δ
is,
there
is
an
α
>
0
such
2
that, in addition to (6.9) and (6.10), we have


∞
n
> ¯ ,
dP r2 Θ (πnα )∗ Pθ∞
∗ , (δθ )∗ Pθ ∗
for large enough n. Consequently, the assertion is proved.

6.3

Proof of Theorem 4.4

As in the proof of Theorem 4.1, we establish the assertion with respect to the modified
form of qualitative robustness defined by (6.6), and since this form is weaker it implies
the assertion. It follows from the definition of the packing numbers that, for  > 0, there
is a packing {θi , i = 1, .., M2 (Θ)} and therefore the collection of open balls B (θi ), i =
1, .., M2 (Θ) is a disjoint union. Denoting N2 := N2 (Θ) and M2 := M2 (Θ), we
therefore obtain
1 = π(Θ)
2
≥ π ∪M
i=1 B (θi )

=

M
2
X

π B (θi )





i=1


≥ M2 min π B (θi ) .
i=1,M2

Consequently, since (4.2) implies M2 ≥ N2 , there exists a point θ∗ ∈ Θ such that

1
π B (θ∗ ) ≤
.
N2

19

(6.11)

Let B := B (θ∗ ) denote the open ball about θ∗ and let Bc denote its complement. Let
π  ∈ M(Θ), defined by
π  (B) :=

π(Bc ∩ B)
, B ∈ B(Θ) ,
π(Bc )

denote the normalization of the restriction of π to Bc which, by the inequality (6.11), is
well defined. Since π = π(Bc )π  + π|B it follows that π − π  = π|B − π(B )π  so that
we obtain
1
dtv (π  , π) ≤ π(B ) ≤
N2
from which we obtain
dP rΘ (π  , π) ≤
In particular, when

1
N2

1
.
N2

(6.12)

< ρ, we obtain
π  ∈ Bρtv (π)

and therefore

π, π  , Pθ∗ , Pθ∗ ∈ Zρ (π) .

That is, when N12 < ρ, the point π, π  , Pθ∗ , Pθ∗ ∈ Zρ (π).
∞ ) := π  , x∞ ∈ X ∞ ,

For the prior π  , let πn : (X ∞ , Pθ∞
∗ ) → M(Θ), defined by πn (x
xn
denote the corresponding sequence of posterior random variables, and let (πn )∗ Pθ∞
∗ ∈
M2 (Θ) denote its induced sequence of laws. Since the assumptions of Schwartz’ Corollary 3.1 are satisfied and π has Kullback-Leibler support at θ∗ , we can apply the assertion
(7.2) of Proposition 7.1 to the sequence of posterior laws (πn )∗ Pθ∞
∗ corresponding to π:


→ 0 n → ∞.
(6.13)
dP r2 Θ (πn )∗ Pθ∞
∗ , δδ ∗
θ
From the triangle inequality we have






∞

∞
,
δ
(π
)
P
−
d
≥ dP r2 Θ (πn )∗ Pθ∞
dP r2 Θ (πn )∗ Pθ∞
2
∗
∗ , δδ ∗
∗ , (πn )∗ Pθ ∗
n ∗ θ
δθ∗ , (6.14)
Pr Θ
θ
so to lower bound the lefthand side it is sufficient in the limit to lower bound the first
term on the right. To that end, we use a quantitative version of the partial converse [18,
Thm. 11.3.5] of convergence in probability implies convergence in law, valid when the
convergence in law is to a Dirac mass. Indeed, if we denote the Ky Fan metric determined
from the measure Pθ∞
∗ by αθ ∗ , Lemma 7.3 asserts that



(6.15)
dP r2 Θ (πn )∗ Pθ∞
,
δ
∗
δθ∗ = αθ∗ (πn , δθ∗ ) .
To evaluate the Ky Fan distance on the righthand side, first observe that since π  has
support contained in the closed set Bc , it follows from Schervish [39, Thm. 1.31] that

20

πx n also has support contained in Bc a.e Pθn∗ . Therefore, if we define B0 := {θ∗ } and
Br := Br (θ∗ ), it follows that B0r = Br , so that
πx n (B0r ) = 0 ,

a.e. Pθn∗ ,

r<

and
(δθ∗ )xn (B0 ) = 1 ,

a.e. Pθn∗ .

It follows from Lemma 7.2 that
dP rΘ πx n , (δθ∗ )xn



≥ min (, 1)

a.e. Pθ∞
∗ ,

and, since  ≤ 1, we obtain



Pθ∞
dP rΘ πx n , (δθ∗ )xn ≥  = 1 .
∗
Therefore, by the definition (2.2) of the Ky Fan metric, we obtain αθ∗ (πn , δθ∗ ) ≥  and,
by the identity (6.15), we conclude that


≥ .
dP r2 Θ (πn )∗ Pθ∞
∗ , δδ ∗
θ
Consequently, from the triangle inequality (6.14) and the convergence (6.13), we conclude, for any ´ > 0, that for large enough n we have



∞
dP r2 Θ (πn )∗ Pθ∞
,
(π
)
P
≥  − ´ .
(6.16)
∗
n ∗ θ∗
Consequently, if this Bayesian inference is qualitatively robust, then for  > 0, it follows
from (6.16) and (6.12) that δ < N12 . The requirement that perturbations be admissible,
that is determine members in Zρ (π), implies that δ < ρ.

7
7.1

Appendix
Schwartz’ Theorem and the convergence of random measures

It will be useful to express the assertion of Corollary 3.1 and some of its consequences
in terms of the convergence of measures and random measures. To that end, recall the
notation M2 (Θ) := M(M(Θ)), and consider the corresponding sequence of random
∞ ) := π n , x∞ ∈ X ∞ , and its
variables πn : (X ∞ , Pθ∞
∗ ) → M(Θ), defined by πn (x
x
2
induced sequence of laws (πn )∗ Pθ∞
∗ ∈ M (Θ). Note especially that δδθ ∗ is the Dirac
mass in M2 (Θ) situated at the Dirac mass δθ∗ in M(Θ) situated at θ∗ .
Proposition 7.1. The assertion of Corollary 3.1 is equivalent to
πxn 7→ δθ∗

a.e. Pθ∞
∗ ,

where 7→ is weak convergence. This in turn implies that
n
o
Pθ∞
dP rΘ (πn , δθ∗ ) >  → 0 n → ∞ ,
∗
21

(7.1)

for  > 0, which is equivalent to


dP r2 Θ (πn )∗ Pθ∞
→0
∗ , δδ ∗
θ

n → ∞,

(7.2)

where dP r2 Θ is the Prokhorov metric on M2 (Θ) defined with respect to the Prokhorov
metric dP rΘ on M(Θ).
Proof. Let O denote the open sets in Θ and Oθ∗ ⊂ O denote the open neighborhoods of
θ∗ . Then, under the conditions of Corollary 3.1, for O ∈ Oθ∗ , it follows that
πxn (O) → 1

n → ∞,

a.e. Pθ∞
∗ .

Since δθ∗ (O) = 1, O ∈ Oθ∗ and δθ∗ (O) = 0, O ∈ O \ Oθ∗ it easily follows that
a.e. Pθ∞
∗ .

lim inf πxn (O) ≥ δθ∗ (O), ∀O ∈ O,
n

which, by the Portmanteau theorem [18, Thm. 11.1.1], is equivalent to
a.e. Pθ∞
∗ .

πxn 7→ δθ∗

where 7→ denotes weak convergence.
Now consider the corresponding sequence of random variables πn : (X ∞ , Pθ∞
∗ ) →
∞
∞
∞
n
M(Θ), defined by πn (x ) := πx , x ∈ X , and its induced sequence of laws (πn )∗ Pθ∞
∗ ∈
M2 (Θ). Then πxn 7→ δθ∗ a.e. Pθ∞
∗ is equivalent to
πn 7→ δθ∗

a.s. Pθ∞
∗ .

Since Θ is a separable metric space it follows that M(Θ) equipped with the Prokhorov
metric is a separable metric space. Since a.s. convergence implies convergence in probability for random variables with values in a separable metric space, it follows that
πn 7→ δθ∗ in Pθ∞
∗ − probability ,
that is,
n
o
∗
d
(π
,
δ
)
>

→0
Pθ∞
∗
P rΘ n θ

n → ∞.

Since M(Θ) is a separable metric space it follows that M2 (Θ) equipped with the
Prokhorov metric is also a separable metric space. Therefore, since on separable metric
spaces convergence in probability to a constant valued random variable is equivalent to
the weak convergence of the corresponding set of laws to the Dirac mass situated at that
value, see e.g. Dudley [18, Prop. 11.1.3], it follows that the convergence in probability,
πn → δθ∗ in Pθ∞
∗ − probability, is equivalent to the corresponding convergence of laws
(πn )∗ Pθ∞
∗ 7→ δδ ∗
θ

n → ∞.

Finally, since the Proprokhorov metric dP r2 Θ on M2 (Θ) metrizes the weak topology on
M2 (Θ) = M(M(Θ)), it follows that the latter is equivalent to


dP r2 Θ (πn )∗ Pθ∞
,
δ
∗
δθ ∗ → 0 n → ∞ .

22

7.2

Some Prokhorov Geometry

We establish a basic mechanism to bound from below the Prokhorov distance between
two measures based on the values of the measures on the neighborhood of a single set.
Lemma 7.2. Let Z be a metric space and consider the space M(Z) of Borel probability
measures equipped with the Prokhorov metric. Consider µ ∈ M(Z) and suppose that
there exists a set B ∈ B(Z) and α, δ ≥ 0 such that
µ(B  ) ≤ δ,

 < α.

Then, for any µ0 ∈ M(Z), we have

dP r (µ, µ0 ) ≥ min α, µ0 (B) − δ .
Proof. If dP r (µ1 , µ2 ) ≥ α the assertion is proved, so let us assume that dP r (µ1 , µ2 ) < α.
∗
Then, denoting d∗ := dP r (µ1 , µ2 ), it follows from the assumption that µ(Ad ) ≤ δ, so
that
∗

µ0 (A) ≤ µ(Ad ) + d∗
≤ δ + d∗
from which we conclude that µ0 (A) − δ ≤ d∗ . Therefore, either dP r (µ1 , µ2 ) ≥ α or
dP r (µ1 , µ2 ) ≥ µ0 (A) − δ, proving the assertion.
Lemma 7.3. Let S be a separable metric space. Then, for an S-valued random variable
X we have
α(X, s) = dP r (L(X), δs )
where α is the Ky Fan metric and s denotes the random variable with constant value s.
Proof. Let us denote α := α(X, s) and ρ := dP r (L(X), δs ). Define the set B0 := {s} and
Br := Br (s), r > 0 and observe that B0r = Br , r > 0. Therefore, by the definition of ρ
we have
L(s)(B0 ) ≤ L(X)(B0ρ ) + ρ
and since L(s)(B0 ) = 1 we obtain
L(X)(B0ρ ) ≥ 1 − ρ
from which we obtain P (d(X, s) ≥ ρ) ≤ ρ . Since this implies that
P (d(X, s) > ρ) ≤ P (d(X, s) ≥ ρ) ≤ ρ
we conclude that ρ ≤ α. Since Dudley [18, Thm. 11.3.5] asserts that α ≤ ρ, the assertion
follows.
Proposition 7.4.
dP r (δx1 , δx2 ) = min 1, d(x1 , x2 )
23



Proof. Consider the set B := {x1 }. Then since B  = B (x1 ), it follows that for  <
d(x1 , x2 ) that x2 ∈
/ B  . Consequently, since δx1 (B) = 1, the inequality
δx1 (B) ≤ δx2 (B  ) + 
requires either  ≥ 1 or x2 ∈ B  which implies that  ≥ d(x1 , x2 ). Consequently,
dP r (δx1 , δx2 ) ≥ min 1, d(x1 , x2 ) . To obtain equality, suppose that dP r (δx1 , δx2 ) >
d(x1 , x2 ). Then, for any d0 which satisfies dP r (δx1 , δx2 ) > d0 > d(x1 , x2 ) there exists
a measurable set B such that
0

δx1 (B) > δx2 (B d ) + d0
0

Consequently, x1 ∈ B, but d0 > d(x1 , x2 ) implies that x2 ∈ B d , which implies the
contradiction 1 > 1 + d0 .

Acknowledgments
The authors gratefully acknowledges this work supported by the Air Force Office of
Scientific Research and the DARPA EQUiPS Program under awards number FA9550-121-0389 (Scientific Computation of Optimal Statistical Estimators) and number FA955016-1-0054 (Computational Information Games).

References
[1] C. D. Aliprantis and K. C. Border. Infinite Dimensional Analysis: A Hitchhiker’s
Guide. Springer, Berlin, third edition, 2006.
[2] A. Barron, M. J. Schervish, and L. Wasserman. The consistency of posterior distributions in nonparametric problems. Ann. Statist., 27(2):536–561, 1999.
[3] S. Basu, S. R. Jammalamadaka, and W. Liu. Stability and infinitesimal robustness
of posterior distributions and posterior quantities. Journal of statistical planning
and inference, 71(1):151–162, 1998.
[4] R. H. Berk. Limiting behavior of posterior distributions when the model is incorrect.
Ann. Math. Statist. 37 (1966), 51–58; correction, ibid, 37:745–746, 1966.
[5] B. Betrò. Numerical treatment of Bayesian robustness problems. Internat. J. Approx. Reason., 50(2):279–288, 2009.
[6] B. Betrò and A. Guglielmi. Numerical robust Bayesian analysis under generalized
moment conditions. In Bayesian robustness (Rimini, 1995), volume 29 of IMS
Lecture Notes Monogr. Ser., pages 3–20. Inst. Math. Statist., Hayward, CA, 1996.
With a discussion by Elı́as Moreno and a rejoinder by the authors.

24

[7] B. Betrò and A. Guglielmi. Methods for global prior robustness under generalized
moment conditions. In Robust Bayesian analysis, volume 152 of Lecture Notes in
Statist., pages 273–293. Springer, New York, 2000.
[8] B. Betrò, F. Ruggeri, and M. Meczarski. Robust Bayesian analysis under generalized
‘
moments conditions. J. Statist. Plann. Inference, 41(3):257–266, 1994.
[9] P. Billingsley. Convergence of Probability Measures. Wiley, New York, second
edition, 1999.
[10] G. Boente, R. Fraiman, and V. J. Yohai. Qualitative robustness for stochastic
processes. The Annals of Statistics, pages 1293–1312, 1987.
[11] T. Bui-Thanh and O. Ghattas. An analysis of infinite dimensional Bayesian inverse shape acoustic scattering and its numerical approximation. SIAM/ASA J.
Uncertain. Quantif., 2(1):203–222, 2014.
[12] I. Csiszár. I-divergence geometry of probability distributions and minimization
problems. The Annals of Probability, pages 146–158, 1975.
[13] A. Cuevas. Qualitative robustness in abstract inference. Journal of Statistical
Planning and Inference, 18(3):277–289, 1988.
[14] A. Cuevas. Comment on ‘Bounds on posterior expectations for density bounded
class with constant bandwidth’ by Sivaganesan. Journal of Statistical Planning and
Inference, 40(2):340–343, 1994.
[15] A. Cuevas González. Una definición de robustez cualitativa en inferencia Bayesiana.
Trabajos de Estadı́stica y de Investigación Operativa, 35(2):170–186, 1984.
[16] M. Dashti and A. M. Stuart. Uncertainty quantification and weak approximation
of an elliptic inverse problem. SIAM J. Numer. Anal., 49(6):2524–2542, 2011.
[17] C. Dellacherie and P.-A. Meyer. Probabilities and Potential. B, Vol. 72 of NorthHolland Mathematics Studies. North-Holland Publishing Co., Amsterdam, 1982.
[18] R. M. Dudley. Real Analysis and Probability, volume 74 of Cambridge Studies in
Advanced Mathematics. Cambridge University Press, Cambridge, 2002. Revised
reprint of the 1989 original.
[19] P. Dupuis and R. S. Ellis. A Weak Convergence Approach to the Theory of Large
Deviations, volume 902. John Wiley & Sons, 2011.
[20] A. Gelman. Inference and monitoring convergence. In Markov chain Monte Carlo
in practice, pages 131–143. Springer, 1996.
[21] S. Ghosal, J. K. Ghosh, and R. V. Ramamoorthi. Consistency issues in Bayesian
nonparametrics. Statistics Textbooks and Monographs, 158:639–668, 1999.
25

[22] A. L. Gibbs. Convergence in the Wasserstein metric for Markov chain Monte Carlo
algorithms with applications to image restoration. Stoch. Models, 20(4):473–492,
2004.
[23] A. L. Gibbs and F. E. Su. On choosing and bounding probability metrics. International Statistical Review, 70(3):419–435, 2002.
[24] P. Gustafson and L. Wasserman. Local sensitivity diagnostics for Bayesian inference.
The Annals of Statistics, 23(6):2153–2167, 1995.
[25] R. Hable and A. Christmann. On qualitative robustness of support vector machines.
Journal of Multivariate Analysis, 102(6):993–1007, 2011.
[26] R. Hable and A. Christmann. Robustness versus consistency in ill-posed classification and regression problems. In Classification and Data Mining, pages 27–35.
Springer, 2013.
[27] F. R. Hampel. A general qualitative definition of robustness. The Annals of Mathematical Statistics, pages 1887–1896, 1971.
[28] P. J. Huber and E. M. Ronchetti. Robust Statistics. Wiley Series in Probability and
Statistics. John Wiley & Sons Inc., Hoboken, NJ, second edition, 2009.
[29] A. N. Kolmogorov and V. M. Tikhomirov. ε-entropy and ε-capacity of sets in
function spaces. Trans. Amer. Math. Soc, 17:277–364, 1961.
[30] N. Madras and D. Sezer. Quantitative bounds for Markov chain convergence:
Wasserstein and total variation distances. Bernoulli, 16(3):882–908, 2010.
[31] I. Mizera. Qualitative robustness and weak continuity: the extreme unction. Nonparametrics and Robustness in Modern Statistical Inference and Time Series Analysis: A Festschrift in honor of Professor Jana Jurecková, 1:169, 2010.
[32] M. Nasser, N. A. Hamzah, and Md. A. Alam. Qualitative robustness in estimation.
Pakistan Journal of Statistics and Operation Research, 8(3):619–634, 2012.
[33] H. Owhadi and C. Scovel. Brittleness of Bayesian inference and new Selberg formulas. Commun. Math. Sci., 13(75), 2013.
[34] H. Owhadi, C. Scovel, and T. J. Sullivan. Brittleness of Bayesian inference under
finite information in a continuous world. Electron. J. Statist., 9:1–79, 2015.
[35] H. Owhadi, C. Scovel, and T. J. Sullivan. On the brittleness of Bayesian inference.
SIAM Review, 57(4):566–582, 2015.
[36] S. T. Rachev, L. B. Klebakov, S. V. Stoyanov, and F. J. Fabozzi. The Methods of
Distances in the Theory of Probability and Statistics. Springer, New York, 2013.

26

[37] G. O. Roberts and J. S. Rosenthal. Markov-chain Monte Carlo: Some practical
implications of theoretical results. Canadian Journal of Statistics, 26(1):5–20, 1998.
[38] G. O. Roberts and J. S. Rosenthal. General state space Markov chains and MCMC
algorithms. Probability Surveys, 1:20–71, 2004.
[39] M. J. Schervish. Theory of Statistics. Springer, 1995.
[40] L. Schwartz. On Bayes procedures. Z. Wahrscheinlichkeitstheorie und Verw. Gebiete, 4:10–26, 1965.
[41] A. M. Stuart. Inverse problems: a Bayesian perspective. Acta Numer., 19:451–559,
2010.
[42] A. Wald. Statistical Decision Functions. John Wiley & Sons Inc., New York, NY,
1950.
[43] L. Wasserman. Asymptotic properties of nonparametric Bayesian procedures. In
Practical nonparametric and semiparametric Bayesian statistics, pages 293–304.
Springer, 1998.
[44] A. D. Woodbury and T. J. Ulrych. A full-bayesian approach to the groundwater
inverse problem for steady state flow. Water Resources Research, 36(8):2081–2093,
2000.

27

