1

How biased is your model?
Concentration Inequalities, Information and Model Bias
Konstantinos Gourgoulias, Markos A. Katsoulakis, Luc Rey-Bellet and Jie Wang

arXiv:1706.10260v1 [cs.IT] 30 Jun 2017

Abstract
We derive tight and computable bounds on the bias of statistical estimators, or more generally of quantities
of interest, when evaluated on a baseline model P rather than on the typically unknown true model Q. Our
proposed method combines the scalable information inequality derived by P. Dupuis, K.Chowdhary, the authors
and their collaborators together with classical concentration inequalities (such as Bennett’s and Hoeffding-Azuma
inequalities). Our bounds are expressed in terms of the Kullback-Leibler divergence R(QkP ) of model Q with
respect to P and the moment generating function for the statistical estimator under P . Furthermore, concentration
inequalities, i.e. bounds on moment generating functions, provide tight and computationally inexpensive model bias
bounds for quantities of interest. Finally, they allow us to derive rigorous confidence bands for statistical estimators
that account for model bias and are valid for an arbitrary amount of data.
Index Terms
Uncertainty quantification, information theory, information bounds, model bias, model uncertainty, goal-oriented
divergence, concentration inequalities, Kullback-Leibler divergence, statistical estimators

I. I NTRODUCTION
An essential ingredient of predictive modeling is the reliable calculation of specific statistics/quantities
of interest of the predictive distribution. Such statistics are typically tied to the application domain,
for instance moments, covariance, failure probabilities, extreme events, arrival times, average velocity,
energy and so on. Predictive models can involve (a) statistical aspects or data collection, and (b) physical/mathematical mechanisms with choices in complexity/resolution, some of them potentially computationally intractable. Therefore, to improve the predictive capabilities of models we face fundamental
trade-offs between model complexity, amount of available data, computational efficiency, and model bias.
The main focus of the paper is the understanding and control of model bias which often inevitably occurs
in model building and which is itself a measure of reliable predictions. Our primary tool are informationtheoretic Uncertainty Quantification methods. Uncertainty quantification (UQ) methods address questions
related to model selection, model sensitivity, model reduction and misspecification, [1], [2], [3]. Sources
of uncertainty are broadly classified in two categories: aleatoric, due to the inherent stochasticity of
probabilistic models and the limited availability of data, and epistemic, stemming from the inability
to accurately model all aspects of a complex system, [4], [2], [5]. Model bias is closely related to
epistemic uncertainty, and probability metrics (Wasserstein, total variation) and divergences (KullbackLeibler, Renyi, χ2 ) [6] are important tools to quantify uncertainty by comparing models. Among the
divergences, the Kullback-Leibler (KL) divergence (also known as relative entropy) is widely used because
of its computational tractability. Specifically, KL-based methods have been used successfully in variational
inference and expectation propagation [7], model selection [8], model reduction (coarse-graining) [9], [10],
[11], [12], optimal experiment design, [13], and UQ [14], [15], [16], [17].
K. Gourgoulias is with Babylon Health, London, UK. He contributed to this work while he was at the Department of Mathematics and
Statistics, University of Massachusetts Amherst, Amherst, MA, 01003, USA (e-mail: kostis.gourgoulias@babylonhealth.com)
M. Katsoulakis is with the Department of Mathematics and Statistics, University of Massachusetts Amherst, Amherst, MA, 01003, USA
(e-mail:markos@math.umass.edu)
L. Rey-Bellet is with the Department of Mathematics and Statistics, University of Massachusetts Amherst, Amherst, MA, 01003, USA
(e-mail:luc@math.umass.edu)
J. Wang is with the Department of Mathematics and Statistics, University of Massachusetts Amherst, Amherst, MA, 01003, USA (email:wang@math.umass.edu)

2

Information-theoretic methods for model building will typically induce bias for the various statistics
and the QoIs of the predictive distribution compared to the “true” model–if known–or the available data.
Managing the corresponding trade-offs between a range of less biased but more computationally expensive
models naturally leads to the following main question for the paper :
Can we provide performance guarantees for model bias in models built via KL-based approximate
inference, model misspecification, or model selection methods?
In this paper we ultimately seek to understand how a decrease in KL-divergence–associated with an
increase in modeling and/or computational effort–can guarantee a model bias tolerance; and in addition,
we seek the tightest possible control of model bias. Note that bounds on the model bias of a QoI
between two distributions P and Q can be obtained, for example, in terms of their KL or χ2 divergences
using the classical Pinsker or Chapman-Robbins inequalities respectively, [18], [6]. Clearly a decrease
in divergence will improve bounds on the model bias. However, these classical inequalities are typically
non-tight and non-discriminating, in the sense that they scale poorly with the size of data sets, with the
number of variables in high-dimensional models (e.g. molecular systems), or with time in the context of
stochastic processes; we refer to Sections 2.2–2.3 in [19] for a complete discussion, see also the example
in Remark 19.
To tackle these challenges a class of new information inequalities have been introduced by Paul Dupuis
in [4] and further developed in [20], [19] by the authors and their collaborators. The resulting bounds on
model bias bounds involve (a) the KL divergence R(QkP ) between a baseline model P and an alternative
models Q, and (b) the moment generating function (MGF) for the QoI under the baseline model P . This
inequality inherits the asymmetry of R(QkP ), which in turn allows us to exchange the roles of P and
Q, depending on the context and/or availability of data from either P or Q. Considering a neighborhood
of models around the baseline P , defined by the KL divergence R(QkP ), can be associated with a
specified error tolerance and is non-parametric in nature. The crucial mathematical ingredient behind the
inequality is the Donsker-Varadhan variational principle [21, Appendix C.] for the KL divergence, also
known as the Gibbs variational formula [22]. This variational representation actually implies that the
new inequalities are tight, i.e. they become an equality for a suitable model Q within a KL divergence
neighborhood of the baseline model P . Furthermore, the dependence on the MGF renders the bounds
scalable and discriminating for high-dimensional data sets and models, e.g. Markov Random Fields, longtime dynamics of stochastic processes and molecular models, [19] as demonstrated recently in [19]. Finally,
broadly related methods in model misspecification and sensitivity analysis in financial risk measurement
and queuing theory, using a robust optimization perspective, were proposed recently in [23] and [24], we
also refer to references therein for other related work in operations research, finance and macroeconomics.
The primary goal of this paper is to use these new theoretical advances to develop practical tools to
estimate and control model bias, and this raises new theoretical questions and implementation challenges.
In particular evaluating or estimating MGFs can be very costly due to high variance of the estimators
thus requiring either a large amount of data, see also Table 1, or multi-level/sequential Monte Carlo
methods [25], [7], [26], [27]. In this paper we rather pursue the use of a variety of QoI-dependent
concentration inequalities [28], [29], [30] to bypass the evaluation or estimation of the MGF and this leads
to computable, tight bounds for model bias. Concentration inequalities are a fundamental mathematical
tool in the study of rare events [31], model selection methods [32], statistical mechanics [30, Section
8.4], random matrix theory [33]. Usually concentration inequalities are used to bound tail events, i.e. to
provide bounds on the probability that a random variable deviates from typical behavior. In this paper we
use concentration inequalities for the purpose of uncertainty quantification, specifically to control model
bias, by implementing efficiently the new information inequalities developed in [4], [20], [19], while at
the same time maintaining and expanding their theoretical advantages.
The new inequalities proved in this paper— which we call concentration/information inequalities—
combine concentration inequalities with the variational principles underlying the bounds and lead to
model bias bounds with the following key features:

3

(a) Easily computable bounds in terms of simple properties of the QoIs such as their mean, upper and
lower bounds, suitable bounds on their variance, and so on; that is, without requiring the costly
computation of MGFs.
(b) Scalability for QoIs that depend on large numbers of data such as statistical estimators, or for high
dimensional probabilistic models.
(c) Derivation of rigorous confidence bands for statistical estimators that account for model bias and are
valid for an arbitrary amount of data.
(d) Applicability to families of QoIs satisfying a concentration inequality, and not to just a single QoI.
(e) Tightness of the model bias bounds in the sense that the bounds are always attained within a prescribed
KL-divergence and the class of QoIs in (d).
The structure of the paper is as follows. In Section II we set-up the mathematical framework for
the paper and discuss the information inequalities for QoIs of [4], [20], [19]. In Section III we use
concentration inequalities to derive new concentration/information inequalities on model bias that are
typically straightforward to implement. In Section IV, we discuss the tightness properties of the new
concentration/information bounds. Finally in Section V we study the bias of statistical estimators, noting
that such QoIs will require results that scale properly with the amount of available data. We also illustrate
the bounds in a variety of examples. In Section VI we consider two elementary examples with bounded
or unbounded QoIs. Two examples of systems with epistemic uncertainty are discussed in Section VII;
the first one deals with failure probabilities for batteries and the second with Markov Random Fields such
as Ising systems.
II. T IGHT M ODEL B IAS B OUNDS USING KL D IVERGENCE
In coarse-graining, model reduction, model selection, or variational inference, as well as in other
uncertainty quantification and approximate inference problems, a baseline model P is compared to a
”true” or simply a different model Q. In this case the notion of risk or mean square error plays a key
role in assessing the quality of the corresponding estimators. Namely, if fˆ is an unbiased estimator of
the quantity of interest f for the baseline model P (but not of the ”true” model Q) then the risk of the
estimator is the mean squared error
(1)
RISK := EP [(fˆ − EQ [f ])2 ] = VarP [fˆ] + |EP [f ] − EQ [f ]|2 .
| {z } |
{z
}
Variance

ModelBias

If available computational resources can be used to control the variance of the baseline model P, then
the model bias becomes the dominant source of risk thus must be carefully controlled. The main goal of
this work is to understand how to transfer quantitative results on information metrics, specifically the KL
divergence R(QkP ) (also known as relative entropy), to bounds on the bias for quantities of interest f .
We formulate the corresponding mathematical problem next.
Mathematical Formulation. Let us consider a baseline model given by the probability measure P on
the state space X which we assume to be a Polish (i.e. complete separable metric) space and we consider
a QoI f , that is a measurable function f : X → R. We specify next a family of alternative probability
distributions in terms of the Kullback-Leibler (KL) divergence (or relative entropy) R(QkP ), which is
defined as
Z
dQ
R(QkP ) = log
dQ.
(2)
dP
if Q is absolutely continuous with respect to P (and +∞ otherwise). Note that R(QkP ) the properties
of a divergence that is R(QkP ) ≥ 0 for all Q and R(QkP ) = 0 if and only if Q = P , see e.g. [18].
We fix a positive number η which we interpret as a level of model misspecification, quantified in terms
KL divergence or, alternatively, as an error tolerance level between the baseline model P and alternative
models described Q. We then define the set of alternative probability as
Qη = {Q : R(QkP ) ≤ η 2 } .

(3)

4

and any Q ∈ Qη is referred to as an η-admissible model. We remark that our approach is non-parametric,
i.e. it does not rely on any parametric form of the probability distributions considered. The relative entropy
R(QkP ) is convex and lower-semicontinuous in (Q, P ). In general the set Qη is infinitely dimensional,
although it is compact with respect to the weak topology, [21]. The fact that the KL divergence is not
symmetric in its arguments can be advantageous in some situations. For example, in variational inference,
it naturally imposes a constraint on the support of the possible approximations Q of a target model P [7].
Our primary mathematical challenge in this work lies in quantifying the model bias in (1) if we use an
η-admissible model in Qη rather than the baseline model P . That is we need to
Compute (or estimate) sup {EQ [f ] − EP [f ]}

and

Q∈Qη

inf {EQ [f ] − EP [f ]} .

Q∈Qη

Note that this approach is intrinsically goal-oriented since it includes not only a family of alternative
models Q but also a specific choice of QoI f .
Goal-oriented divergence. We now define a divergence which incorporates the QoI f and hence is called
goal-oriented; it was first introduced in the current form in [20] based on earlier work in [4]. Consider a
QoI f and the moment-generating function (MGF)
˜
MP (c; f˜) := EP [ecf ]

(4)

f˜(x) := f (x) − EP [f ].

(5)

of the centered QoI f˜,

In general (see [31] for details) the MGF MP (c; f˜) is finite for c in some interval I and equal to +∞
otherwise. Throughout this paper we will make the standing assumption that MP (c; f˜) is finite in the
interval I = (d− , d+ ) with d− < 0 < d+ , then under this assumption, MP (c; f˜) is C ∞ in I and f has
finite moments of any order. We next define the goal-oriented (GO) divergence as


1
1
˜
Ξ(QkP ; f ) = inf
log MP (c; f ) + R(QkP ) .
(6)
c>0
c
c
for P, Q with R(Qkp) < ∞. Note that if d+ is finite then the infimum can be taken on (0, d+ ) and note
also that if R(QkP ) = ∞ then the goal oriented divergence can naturally be then set equal to +∞.
In [4], [20] the following bound on the model bias was proved, along with certain mathematical
properties:
Theorem 1. Let P be a probability measure and let f be such that its MGF MP (c; f˜) is finite in a
neighborhood of the origin. Then for any Q with R(QkP ) < ∞ we have
− Ξ(QkP ; −f ) ≤ EQ [f ] − EP [f ] ≤ Ξ(QkP ; f ).

(7)

The GO divergence has the following properties
1) Divergence: Ξ(Q || P ; f ) ≥ 0 and Ξ(Q || P ; f ) = 0 if and only if either Q = P or f is constant
P -a.s.
2) Linearization:
p
p
Ξ(Q || P ; ±f ) = varP [f ] 2R(Q || P ) + O(R(QkP ))
and thus
| EQ (f ) − EP (f ) | ≤

p
p
varP [f ] 2R(Q || P ) + O(R(Q || P )).

Tightness of goal-oriented divergence. Our next result complements Theorem 1 and demonstrates the
tightness of the GO divergence bounds (7) for the bias of a QoI f ; for the complete proof, we refer to
Appendix (A). An equivalent tightness result for the upper bound in (13) was first shown in [4], while here

5

we present a new formulation for a complete tightness result in Theorem 2, based on the goal-oriented
divergence formulation in [20].
To state our result we introduce the exponential family P c given by
dP c
ecf
,
= ecf −log MP (c;f ) = R cf
dP
e dP

(8)

which is well-defined for c in the interval I = (d− , d+ ) where MP (c; f ) is finite.
Theorem 2. Let P be a probability measure and let f be such the MGF MP (c; f˜) is finite in a
neighborhood of the origin. Let Qη = {Q : R(QkP ) ≤ η 2 } be the set all approximate probability
Q within a KL tolerance η 2 .
1) There exists 0 < η± ≤ ∞ such that for any η ≤ η± there are probability measures Q± such that
Ξ(Q+ kP ; f ) = EQ+ [f ] − EP [f ] = max EQ [f ] − EP [f ],

(9)

−Ξ(Q− kP ; −f ) = EQ− [f ] − EP [f ] = min EQ [f ] − EP [f ]

(10)

Q∈Qη

Q∈Qη

The measures Q± are given by the elements P c± of the exponential family (8) where c± are the
unique solution of R(P c± kP ) = η 2 .
2) If η± is finite then f is necessarily bounded above/bounded below P -almost surely with upper/lower
bound f± . In that case, for η > η± and any Q with R(QkP ) = η 2 we have
Ξ(QkP ; f ) = f+ − EP [f ] = sup EQ [f ] − EP [f ],
Q∈Qη

−Ξ(QkP ; −f ) = f− − EP [f ] = inf EQ [f ] − EP [f ]
Q∈Qη

(11)
(12)

The main result of the theorem provides performance guarantees in the sense that EQ [f ] belongs to the
interval
−Ξ(Q− kP ; −f )+EP [f ] ≤ EQ [f ] ≤ EP [f ] + Ξ(Q+ kP ; f ) for all Q ∈ Qη
(13)
and the bounds are tight in Qη , in the sense that inequalities become equalities for Q = Q∓ respectively.
This tightness property is crucial for our discussion because it implies that the GO divergence bounds
in (7) are the best possible in the sense that they have attainable worst-case model scenarios Q± among
all probability distributions Q within a KL tolerance η 2 > 0, see the schematic in Figure 1.

ecf1 dP

P c+

ecf2 dP

Q+

P

Q−

P c−

Qη = {Q : R(Q || P) ≤ η 2 }

Fig. 1. The schematic depiction of Theorem 2 for the QoIs f1 , f2 with tolerance η 2 . The solid lines depict the one-parameter tilted probability
distributions P c corresponding to the QoIs. The theorem implies that the upper and lower bounds in the family Qη = {Q : R(QkP ) ≤ η 2 }
are attained at the probability measures Q± = P c± for the QoI f1 .

Remark 3. The tightness property (13) is a non-parametric result: the family Qη of all alternative models
Q cannot be parametrized in general and is only characterized by the property R(QkP ) ≤ η 2 . In spite

6

TABLE I
F OR THE ESTIMATION OF varP [Y ], WE ASSUME THAT EP [Y ] IS UNKNOWN AND THAT THE BIAS - ADJUSTED ESTIMATOR IS USED . F OR
cY
THE VARIANCE OF EP [e ], A FIRST- ORDER APPROXIMATION IS USED ( SEE [34]), ASSUMING THAT EP [Y ] IS SMALL .
Quantity
EP [Y ]
varP [Y ]
MP (c; Y )

Variance of estimator
varP [Y ]/n
2(varP [Y ])2 /(n − 1)
c2 e2cEP [Y ] varP [Y ]/n

of this non-parametric framework, we showed in Theorem 2 that the extremal models Q± that yield the
tight bounds (13) belong to the parametrized family (8), see also Figure 1.
The attractive properties of the GO bounds demonstrated in Theorem 1 and Theorem 2, come at a
potentially significant cost since they require the knowledge or calculation of the MGF MP (c; f˜) with
respect to model P . If no simple formula for MP (c; f˜) is known, this can be a data-intensive operation—
compare the estimator variance of the MGF with that of other QoIs in Table I. Controlling the variance
of an MGF estimator will require a large amount of data and/or the use of a multi-level Monte Carlo
method, see also the discussion in Section I and Section VII.
In the next section we introduce a new class of inequalities that share the aforementioned features
of the GO divergence and (7), but they can bypass the estimation of an MGF by using the concept of
concentration inequalities.
III. C ONCENTRATION /I NFORMATION I NEQUALITIES FOR M ODEL B IAS
To bypass the estimation or computation of the MGF in (6) we will use a QoI-dependent concentration
bound for the MGF, i.e., a function Φ(c) taking values in (0, ∞] such that
MP (c; f˜) ≤ Φ(c)

(14)

for all c ∈ R. Since the moment generating function MP (c; f˜) can take the value +∞ it is natural to
allow the same for Φ(c).
Bounds of the form (14), for explicitly computable functions Φ(c), are called concentration inequalities
and we discuss several such examples in Section III-A and Section III-B, as well as in Section V.
Although we use only the simplest concentration inequalities here, the results are indicative to what
can be accomplished using such information on f and P . In upcoming work, we will consider further
applications for stochastic processes and interacting particle systems arising in Kinetic Monte Carlo and
molecular dynamics models. Concentration inequalities is an important mathematical tool since they allow,
via a Chernov bound, to control tail events, i.e. they provide explicit bounds on the probability that a
random variable deviates from typical behavior. More specifically, such methods can address, among
others, questions on rare events [31], model selection methods [32], statistical mechanics [30, Section
8.4] and random matrices [33]. Here we propose the use of concentration inequalities in tandem with
the information inequalities (7) for uncertainty quantification and especially for providing model bias
guarantees. In Theorem 4 we show how to construct new bounds for the model bias using a function Φ
satisfying (14).
Theorem 4. Let P be a probability measure and let f be a QoI such that its MGF MP (c; f˜) is finite in
a neighborhood of the origin. Let Φ : R → (0, ∞] be a function with Φ(0) = 1, Φ0 (0) = 0 and such that
MP (c; f˜) ≤ Φ(c)

(15)

for all c ∈ R. We define the set of admissible QoIs by
FP = {g : MP (c; g̃) ≤ Φ(c)}.

(16)

7

Then, f ∈ FP , and for every Q ∈ Qη = {Q : R(QkP ) ≤ η 2 } we have
−U− (η; FP ) ≤ EQ [g] − EP [g] ≤ U+ (η; FP )
where


U± (η; FP ) := inf

c>0

for every g ∈ FP ,


1 2
1
log Φ(±c) + η .
c
c

(17)
(18)

Proof. The proof follows immediately from (6) and (7), combined with the concentration inequality (15)
and the definition of the admissible QoIs, FP . We discuss specific examples of inequalities of the type
(15) and their corresponding admissible sets FP , in Section III-A and Section III-B below.
Remark 5 (Admissible set of QoIs). We note that the function Φ depends both on the QoI f and on
P through (15) and therefore the set of admissible functions FP also depends on the QoI f and on P .
However, to keep notation simple, we suppress this dependence for both Φ and FP .
Remark 6 (Computing U± (η; FP )). Some concentration bounds (15) such as the sub-Gaussian and
Hoeffding bounds discussed below provide explicit formulas for U± (η; FP ), see for instance (26) and
(33). However, in general—see the sharper Bennett bounds in (28 and (30)—we have an explicit formula
for Φ but no explicit closed form solution of the optimization over c. The elementary one-dimensional
optimization in (18) can be carried out with standard solvers, e.g., Newton’s method.
Divergence structure of U± (η; FP ): The following properties of the bounds U± in (18) are analogous
to the properties of the GO divergence (6) outlined in Theorem 1. One notable difference is that here the
divergence structure defined by U± (η; FP ) contains information about the entire family FP in (16) and
not just a single QoI f as was the case in the GO divergence (6).
Theorem 7. Under the assumptions of Theorem 4 and, in addition, if
Φ(c) = MP̄ (c; h̃) ,

(19)

for some probability P̄ and QoI h and all c ∈ R then U± (η; FP ) satisfy:
1) Divergence Properties:
a. U± (η; FP ) ≥ 0, and
b. U± (η; FP ) = 0 if and only if η = 0 or FP is trivial, i.e. consists only of functions which are
constant P -a.s.
2) Linearization: If Φ = Φ(c)
in a neighborhood of c = 0, then we have the
p is twice differentiable
2
00
asymptotics U± (η; FP ) = 2Φ (0)η + O(η ) and thus,
p
(20)
|EQ [g]−EP [g]| ≤ 2Φ00 (0)η + O(η 2 ) for all g ∈ FP and all Q ∈ Qη .
Proof. The proof follows from Theorem 1. Indeed since, by assumption, Φ(c) = MP̄ (c; h̃) we have
U± (η; FP ) = Ξ(QkP̄ ; ±h)

(21)

for any probability Q such that R(QkP̄ ) = η 2 . Therefore, by Theorem 1, U± (η; FP ) ≥ 0 and U± (η; FP ) =
0 if and only if η = 0 or h is constant P̄ a.s. But if h is constant P̄ a.s then Φ(c) = MP̄ (c; h̃) = 1 for
all c and thus the set of admissible QoIs (16) becomes:
FP = {g : MP (c; g̃) ≤ Φ(c) = 1} .

(22)

However for any g ∈ FP , by Jensen’s inequality, MP (c; g̃) ≥ 1 since EP [g̃] = 0. Therefore the admissible
set FP consists only of constant functions thus g is constant P -a.s. Finally, the linearization in Theorem 7
is proved similarly to the linearization result of the GO divergence in Theorem 1, (see the proof in Section
3 of [20]).

8

Theorem 4 and Theorem 7 motivate the following definition, in analogy to the goal oriented (GO)
divergence (6) defined for a single QoI f :
Definition 8 (Concentration/Information Divergence). Given the notation and assumptions of Theorem 4
and Theorem 7, we define the concentration/information divergence between a baseline model P and the
family of models Qη , satisfying (17) for all QoIs in FP :


1 2
1
,
(23)
log Φ(±c) + η
U± (η; FP ) := inf
c>0
c
c
where Qη and FP , are defined in (3) and (16) respectively.
Remark 9 (Features of Concentration/Information Inequalities). While the GO divergence bounds (7)
are defined for a specific QoI f , key features of the new bounds in Theorem 4 include: (a) allow to
consider whole families of admissible QoIs FP defined in (16), and (b) they bypass the costly MGF
calculations needed in the GO divergence (6). Finally, we next show that the new bounds (17) still share
the advantages of the GO divergence bounds, namely: in Section IV we prove that (17) is, (c) tight in
the family of models Qη , (3), and the family of QoIs FP , (16). in Section V we show that (17) is, (d)
scalable to QoIs that depend on large numbers of data such as statistical estimators and to high dimensional
probabilistic models.
We will next discuss specific examples of the bound Φ(c) in the concentration bounds (15) and
Theorem 4; furthermore, we also demonstrate how we can select such concentration bounds depending on
the information we have regarding the distribution P . We divide our presentation into two cases, namely
bounded and unbounded QoIs f .
A. Sub-Gaussian Bounds
For an unbounded QoI f and a probability distribution P , we can characterize the type of concentration
by bounding either the tail probabilities P (f (X) − EP [f ] > a) for all a or MP (c; f˜) for all c for which
the MGF is finite. In this section, we discuss the (classical) sub-Gaussian concentration bounds which
are characterized by Gaussian decay of the tails. Sub-exponential bounds (see Section VI-A) and subPoissonian bounds could also be useful in various situations but we will not discuss them further here
(see e.g. [30]).
Sub-Gaussian concentration bounds [28] : We say that f = f (X) is a sub-Gaussian random variable
if there exists a σB > 0 such that
MP (c; f˜) ≤ Φ(c) := exp(c2 σB2 /2) for all c ∈ R.

(24)

Now given a fixed σB , we can consider the family of QoIs defined in (16),
FP := {g : MP (c; g̃) ≤ Φ(c) = exp(c2 σB2 /2)} ,

(25)

i.e. we consider all random variables with MGF bounded by the MGF of a normal random variable with
2
variance σB2 . Furthermore, using (18) we can write an explicit formula for U± (η; FP ) = inf c>0 { cσ2B + ηc }
as
√
U± (η; FP ) = σB 2η.
(26)
By expanding MP (c; f˜) around c = 0, we can readily show that σB2 is an upper bound of varP [f (X)].
Relation (26) also implies that there is no η-admissible model Q ∈ Qη for which the QoIs under
consideration lie beyond the uncertainty region given by Theorem 4:
√
√
− σB 2η ≤ EQ [g] − EP [g] ≤ σB 2η
(27)

9

for all models Q ∈ Qη and QoIs g ∈ FP . In Corollary 10, we consider the special case where P is a
normal distribution which is compared against any models Q–possibly not normal–from Qη .
Corollary 10. Consider the QoI f (x) = x where P = N (µ, σ 2 ). Also, let Q be any distribution such
that R(QkP ) ≤ η 2 . Then, if the coefficient of variation (also known as relative standard deviation) is
cv := σ/|µ|, the relative model bias satisfies:
√
√
EQ [f ] − EP [f ]
≤ cv 2η .
−cv 2η ≤
|EP [f ]|
In general, sub-Gaussianity is a strong assumption for an unbounded random variable. For example, if
P = Laplace(1), i.e., a two-sided exponential distribution centered at zero, then MP (c; X) = 1/(1 − c2 ),
|c| < 1, which cannot be bounded by any exp(c2 σB2 /2) for all c. Finally, we note that results like the
McDiarmid’s inequality, see Section V below, or the logarithmic Sobolev inequalities [29], [35], can
provide values for the constant σB2 for QoIs that satisfy specific properties, e.g., (43).
B. Bennett and Hoeffding Bounds
Many quantities of interest are bounded such as failure probabilities or functions of random variables
with bounded support. Bounded random variables are necessarily sub-Gaussian [28], but much sharper
bounds for their MGFs, (15), can be derived and used to bound the worst-case bias through Theorem 4.
In this direction, we next discuss some additional concentration bounds for bounded QoIs, that we will
also showcase in examples in this work. This list is not complete by any means and other concentration
inequalities can be used here, see for instance [29] for other bounds. For each case below, the family of
QoIs FP is defined in terms of the concentration bound on the MGF, (16), as in Theorem 4.
Bennett concentration bound [31, Lemma 2.4.1]: Consider the random variable X where X ∼ P and
the QoI f = f (X) such that f (X) ≤ b, for some 0 ≤ b < ∞. Setting µ := EP [f (X)], b̃ := b − µ, we
have
σB2
b̃2
2
(28)
exp(−cσ
/
b̃)+
exp(cb̃),
MP (c; f˜) ≤ Φ(c) :=
B
b̃2 + σB2
b̃2 + σB2
for all c ≥ 0 and where σB2 is any upper bound of varP [f ]. Therefore, keeping in mind Remark 5, we
define
FP ={g : MP (c; g̃) ≤ Φ(c) } , where Φ is defined in (28).
(29)
Bennett-(a, b) concentration bound [31, Corollary 2.4.5]: If the QoI f is such that a ≤ f (X) ≤ b,
X ∼ P , then we can set σB2 = (µ − a)(b − µ) in the Bennett bound to obtain
MP (c; f˜) ≤ Φ(c) :=

b̃
ã
exp(cã) −
exp(cb̃) for all c ∈ R.
b−a
b−a

(30)

The right-hand side of (30) is the MGF of a Bernoulli-distributed random variable with values {a, b}.
Note that the Bernoulli is the distribution with the most “spread” around the mean value between all
bounded random variables in [a, b]. Similarly to (29) we have,
FP ={g : MP (c; g̃) ≤ Φ(c) } , where Φ is defined in (30).

(31)

Hoeffding concentration bound [36], [31]: When the QoI f is bounded as in the Bennett-(a, b) case, we
can further bound the Bennett-(a, b) bound by a Gaussian MGF, giving rise to the (less tight) Hoeffding
MGF bound,
MP (c; f˜) ≤ Φ(c) := exp(c2 (b − a)2 /8) for all c ∈ R.

(32)

10

TABLE II
T HE DIFFERENT MGF BOUNDS ALONG WITH THE CONDITIONS THEY IMPOSE ON P AND f AND THE QUANTITIES THEY DEPEND ON FOR
THEIR IMPLEMENTATION IF WE ARE INTERESTED IN QUANTIFYING THE WORST- CASE BIAS . H OWEVER , BOUNDING THE WORST- CASE
EQ [f ] does not REQUIRE EP [f ]. G AUSSIAN DECAY OF THE TAILS OF THE DISTRIBUTION OF f (X) IMPLIES THE SUB -G AUSSIAN MGF
BOUND ( SIMILAR ASSUMPTIONS ABOUT THE TAILS EXIST FOR THE REST OF THE BOUNDS ). I N TERMS OF INFORMATION
REQUIREMENTS , THE H OEFFDING BOUND REQUIRES THE LEAST AMOUNT, BUT IT IS ALSO THE LEAST TIGHT. A S AVAILABLE
INFORMATION / DATA FOR THE BOUNDS GROW, THE BOUNDS GET TIGHTER .
Name
Hoeffding (32)
Bennett-(a, b) (30)
Bennett (28)
sub-Gaussian (24)
GO bound (6)

Conditions on f, P
a ≤ f (X) ≤ b
a ≤ f (X) ≤ b
2
f (X) ≤ b, varP [f ] ≤ σB
2
2
MP (c; f˜) ≤ exp(σB c /2)
MP (c; f˜) < ∞

Φ = Φ(c) input
a, b
EP [f ], a, b
EP [f ], b, σB
σB
k

EP [(f ) ] for all k

As in the sub-Gaussian case of Section III-A, we can calculate U± (η; FP ) explicitly:
√
U± (η; FP ) = (b − a) 2η.

(33)

Finally the set of QoIs is
FP ={g : MP (c; g̃) ≤ Φ(c) }, where Φ is defined in (32).

(34)

Remark 11 (Hierarchy of bounds). It is straightforward to demonstrate that we can order the bounds
in terms of accuracy, noting that if the QoI f is bounded in [a, b], then we always have the bound
σB2 ≤ (EP [f ]−a)(b−EP [f ]) in the Bennett bound (28). Therefore, we have the hierarchy of concentration
bounds:
MP (c; f˜) ≤ Bennett ≤ Bennett-(a,b) ≤ Hoeffding.

(35)

Unlike the two Bennett bounds, the Hoeffding bound is independent of the location of the mean µ within
the interval [a, b] and only depends on the length of the interval b−a. As such, it requires the least amount
of information about f and P and is the least sharp of the bounds, see Table II and the requirements
for the QoI families FP , (29), (31) and (34). On the other end, the GO divergence bound—involving
MP (c; f˜)—is the tightest, as we see in (35), but also the most expensive to implement, see Table I.
We also refer to a demonstration of this hierarchy in the example in Section VI-B. Overall, as available
information/data on the QoI f and and the baseline model P grows, concentration bounds and therefore
model bias bounds become tighter. Finally, we refer to Figure 2, where we demonstrate the tightness of
the model bias bounds (17), (18), in terms of both η 2 = R(QkP ) and σB , for the Bennett bounds (28).
Remark 12. [How large is the class FP ?] A plausible question is how rich is the set of admissible QoIs,
FP derived by the various concentration bounds on the MGF MP (c; g̃) in (25), (29), (31) and (34). Here
we address this question in the context of the Bennett bound, however the same argument also applies to
the Bennett-(a, b) and Hoeffding bounds, as well as to the sub-Gaussian case in Section III-A. We can get
a simple first insight in this direction based on (28). Indeed, based on the conditions for this inequality
to hold, we readily have that
FP ⊃ {g : g(X) ≤ b , varP [g] ≤ σB2 , EP g = µ } .
We also note that enforcing the condition on the mean, EP g = µ, is trivial and involves only a translation
of the QoI g.

11

2
. Knowing
Fig. 2. Level curves of the upper model bias bound (17) with the Bennett bound (28) and assuming b = 1, varP [f ] ≤ σB
η 2 = R(QkP ) for some model Q and an upper bound on the variance provides model-bias guarantees (through Theorem 4). Further reduction
of the model bias bound requires a corresponding—and potentially expensive—decrease in KL and/or a tighter upper-bound for varP [f ],
2
= varP [f ]
for example, by incorporating additional data. The tightest possible guarantee afforded by the Bennett bound is gained when σB
2
and η = minQ R(QkP ).

IV. T IGHTNESS OF THE C ONCENTRATION /I NFORMATION I NEQUALITIES
In this section we show that, under suitable assumptions, for the concentration/information bounds
derived in Section III the divergence U± (η; FP ) retains some of the tightness properties of the GO
divergence Ξ(QkP ; f ) established in Section II.
Theorem 13. Let P be a probability and Qη = {Q : R(QkP ) ≤ η 2 }. Assume Φ(c) = MP (c; f˜) is a MGF
for some QoI f with respect to P and let
FP = {g : MP (c; g̃) ≤ Φ(c) for all c ∈ R} .

(36)

Then, there exist probabilities P c± ∈ Qη (see (8)) that satisfy R(P c± kP ) = η 2 and
U+ (η; FP ) = EP c+ [f ] − EP [f ] =

max

Q∈Qη ,g∈FP

−U− (η; FP ) = EP c− [f ] − EP [f ] =

min

EQ [g] − EP [g],

Q∈Qη ,g∈FP

EQ [g] − EP [g] ,

(37)
(38)

i.e., the maximum and minimum for model bias is attained within the family of models Qη and the family
of QoIs FP , see the schematic in Figure 3. We also have the “confidence band” around the baseline
model P ,
−U− (η; FP ) + EP [g] ≤EQ [g] ≤ EP [g] + U+ (η; FP ) for all Q ∈ Qη , g ∈ FP ,

(39)

with the two equalities holding if Q = P c∓ respectively and for g = f ∈ FP .
Proof. Since f ∈ FP , Theorem 2 implies that the probabilities P c± in (8), with c± chosen such that
R(P c± kP ) = η 2 satisfy
Ξ(P c± kP ; ±f ) = U± (η; FP ) .

(40)

12

ecf dP

ecg1 dP

P c+

ecg2 dP

P
P c−

Qη = {Q : R(Q || P) ≤ η 2 }

Fig. 3. The schematic depiction of Theorem 13 for a family of QoIs FP and tolerance η 2 . The solid lines depict the one-parameter tilted
probability distributions P c corresponding to the QoI g1 , g2 , f ∈ FP . The theorem implies that the upper and lower bounds in the family
Qη = {Q : R(QkP ) ≤ η 2 } are attained at the probability measures Q± = P c± .

Therefore, by Theorem 4, (17), for all Q ∈ Qη , g ∈ FP
− Ξ(P c− ||P ; −f ) ≤ EQ [g] − EP [g] ≤ Ξ(P c+ ||P ; f ) .

(41)

Finally, we apply (9) and (10) of Theorem 2 and use (40) to conclude the proof.
Remark 14 (Connections to Mass Transport). The proof of Theorem 13 is quite straightforward and we
discuss here one approach to verify the crucial assumption of the Theorem, namely that
Φ(c) = MP (c; f˜)

for some QoI f .

(42)

One natural way to ensure this is intimately related to mass transport methods, [37]. Instead of (42) we
may assume the more easily checkable hypothesis that Φ(c) = MP̄ (c; h̃) for some h and some model P̄ ;
e.g. h(x) = x and P̄ a Gaussian distribution for the Hoeffding’s bound, see also Example 16 below. To
prove (42) one shows then that there exists a transport map between P and P̄ , namely a map T such
that P̄ (A) = P (T −1 (A)) for any measurable set A [37]. If a transport map exists we have
Z
Z
EP [h ◦ T ] = h(T x)P (dx) = h(y)P̄ (dy) = EP̄ [h] .
and hence with f = h ◦ T

f˜ = f − EP [f ] = h ◦ T − EP̄ [h] = h̃ ◦ T .

This implies that
Z
Φ(c) = MP̄ (c; h̃) =

e

ch̃(y)

Z
P̄ (dy) =

ech̃(T x) P (dx) = MP (c; f˜) ,

and thus the assumption (42) holds.
To ensure the existence of such a transport map T one needs some assumptions on P (and P̄ ). For
example, if P and P̄ are non-atomic measures then a transport map always exists. If the measure P has
a density then T can be constructed using the Knothe-Rosenblatt rearrangement or Brenier’s L2 optimal
transport map; we refer to Chapter 1 [37], [38] for more details on these maps, and several other such
transport maps and relevant conditions for their existence.
Next we demonstrate how to use Theorem 13 by interpreting Φ(c) as the MGF of a suitable QoI f with
respect to the distribution P . In Example 15, we illustrate the tightness of the concentration bounds for the
case of bounded random variables supported in [−1, 1], while the arguments can be trivially generalized
to any other bounded interval.

13

Example 15 (Bennett-(a,b) QoIs). Consider a distribution P such that there is an event A ⊂ R such that
P (A) = 1/2; we consider the family of QoIs, FP , for which (30) is true with a = −1, b = 1, EP [g] = 0
for all g ∈ FP . The corresponding Bennett-(a,b) bound is
1
1
Φ(c) = ec + e−c .
2
2
Then, if we choose f (x) := 2 · 1A (x) − 1, where 1A is the characteristic function of the set A, we have
Φ(c) = MP (c; f˜). Therefore Theorem 13 is immediately applicable.
The next example covers the case of sub-Gaussian QoIs which contains both bounded and unbounded
random variables.
Example 16 (sub-Gaussian QoIs). Consider the probability measure P on R which has a density. For
sub-gaussian QoIs (24) we have the bound Φ(c) = exp(c2 σB2 /2), however, we can rewrite the bound as
Φ(c) = MP̄ (c; h̃)
where h(x) = x and P̄ = N (0, σB2 ) is a normal distribution. Since P has a density, we can use the
measurable isomorphism, or any other applicable map discussed in Remark 14, to construct a transport
map T between P and P̄ . Thus, we can show the existence of a a QoI f that satisfies the condition (42)
and we can readily apply Theorem 13 to show the tightness of the bounds given by (26).
V. M ODEL B IAS FOR S TATISTICAL E STIMATORS
As discussed in Section II a key challenge is to control the risk involved in evaluating statistical estimator
using the baseline model P rather than the true model Q. In addition it is important to control the bias of
QoI which are not necessarily expected values, for example the bias in the variance, i.e, varP X − varQ X,
or other statistics such as correlation of skewness, or mean and quantiles, see [34]. Generally, given data
X1 , . . . , Xn , we aim to control the bias of statistical estimator ψ = ψ(X1 , . . . , Xn ), for example the sample
variance (49).
To obtain useful bounds on the bias of statistical estimators ψ, we need to exhibit and control the
dependence of the inequalities in Sections III-B and III-A on the amount of data available, i.e. the
dependence on n. We will exhibit a large and natural class of statistical estimators for which inequalities
asymptotically independent on n. As demonstrated in [19] the Concentration/Information inequalities of
Sections II and III are the only known information equalities which scale properly with n.
The main tool we shall use is the key result used in the proof of the McDiarmid’s inequality, see also
the Hoeffding-Azuma bound, [31]. We refer to Chapter 2 of [29] or [39] for the proof.
Proposition 17. Let X1 , . . . , Xn be independent random variables with joint distribution P n = P1 × · · · ×
Pn . Let ψ(x1 , . . . , xn ) satisfy the Lipschitz condition
sup
x1 ,...,xn ,x0k

|ψ(x1 , . . . , xk , . . . , xn ) − ψ(x1 , . . . , x0k , . . . , xn )| ≤ dk .

(43)

for some constants dk , k = 1, . . . , n. Then ψ(X1 , . . . , Xn ) is a sub-Gaussian random variable and for all
c ∈ R we have
!
n
c2 X 2
d
.
MP n (c; ψ̃) = EP n [exp (c(ψ − EP n [ψ]))] ≤ exp
(44)
8 k=1 k
By combining the bound in (44) with the definition of U± (η; FP ) in Theorem 4 for the sub-Gaussian
case (24) we obtain immediately
Theorem 18. For X1 , . . . , Xn and ψ(x1 , . . . , xn ) as in Proposition 17 we have
!1/2 v
u n
n
X
u X
2
t2
|EP n [ψ(X1 , . . . , Xn )] − EQn [ψ(X1 , . . . , Xn )]| =
d
R(Qi kPi ) .
k

k=1

k=1

(45)

14

If X1 , . . . , Xn are identically distributed with common distribution P and if there exists a constant C
such that
C
dk ≤ , k = 1, . . . , n
n
then we have for any n
p
(46)
|EP n [ψ(X1 , . . . , Xn )] − EQn [ψ(X1 , . . . , Xn )]| ≤ C 2R(QkP ).
Next, we apply these results towards obtaining model bias bounds for statistical estimators.
CDF estimator: If X is a real-valued with cumulative distribution function (CDF) FP (x) = P {X ≤ x}
then given i.i.d. data X1 , . . . , Xn
n

1X
F̂n (x) =
I{Xk ≤x} ,
n k=1

(47)

where IA is the indicator function of the set A, is an estimator for the CDF FP = FP (x). It is easily
verified that the conditions of Theorem 18 are satisfied with C = 1. Since the bound is uniform in x, and
F̂n (x) is an unbiased estimator of FP (x), we obtain
h
i
h
i p
sup |FQ (x) − FP (x)| = sup EQn F̂n (x) − EP n F̂n (x) ≤ 2R(QkP ) ,
(48)
x

x

for any alternative model Q to the baseline P . As we also note in the sample variance example below,
the estimator does not need to be unbiased.
Sample variance and general statistical estimators: The McDiarmid’s inequality and the condition
(43) can be used to control bias of QoIs which are not simply expected values, for example the sample
variance
!2
n
n
n
X
1 X
1X
1
(49)
Vn (X1 , . . . , Xn ) =
Xi −
Xj
=
(Xi − Xj )2 .
n − 1 i=1
n j=1
2n(n − 1) i,j=1
If we assume that |Xi | ≤ M for some M > 0 then we have
sup |Vn (x1 , . . . , xk , . . . , xn ) − Vn (x1 , . . . , x0k , . . . , xn )| ≤

|xi |≤M,
|x0k |≤M

8M 2
.
n−1

Then the sample variance satisfies (43) with dk = 8M 2 /(n − 1) for all k. Thus, we can bound the
corresponding model bias by
n p
|varP [X] − varQ [X]| = |EP n [Vn ] − EQn [Vn ]| ≤ 8M 2
2R(QkP ).
(50)
n−1
valid for all n > 1. Note that if take n → ∞ we obtain the variance bound
p
|varP [X] − varQ [X]| ≤ 8M 2 2R(QkP )
which shows how KL-divergence R(QkP ) control the misspecification for QoIs beyond expected values.
The same analysis also applies to the (biased) plug-in estimator for the variance, namely
!2
n
n
1X
1X
Ṽn (X1 , . . . , Xn ) := varF̂n [X] =
Xi −
Xj
.
n i=1
n j=1

15

Finally, we can easily generalize the sample variance calculation to more general QoIs and statistical
Pn
1
n−1
)
only
on
the
two
sample
averages
estimators.
The
sample
variance
depends
(up
to
a
factor
i=1 Xi
n
n
Pn
1
2
and n i=1 Xi . It is not difficult to see that if |Xi | ≤ M and the QoI has the form
!
n
n
1X
1X
ψn (X1 , . . . , Xn ) = g
f1 (Xi ), . . . ,
fk (Xi )
(51)
n i=1
n i=1
for some f1 , · · · fk (say the the first k moments), and for some Lipschitz continuous function g, then one
can apply Theorem 18 for a constant C which depends on M , the Lipschitz constant for g and f1 , · · · fk .
One important example of the type (51) is the sample correlation, we refer to Example 2.16 in [40].
Confidence Bands and Model Bias To further illustrate our results we construct a non-parametric
confidence band for the CDF FQ (x). We combine the bound (48) with the Dvoretzky-Kiefer-Wolfowitz
(DKW) inequality [34], [40], i.e. the bound


2
P sup |F̂n (x) − FP (x)| ≥  ≤ 2e−2n ,
(52)
x

which itself is obtained though concentration inequalities. For any n and α > 0, we set n =
and
√
Ln (x; η) = max{F̂n (x) − 2η − n , 0}
√
Un (x; η) = min{F̂n (x) + 2η + n , 1} .

p
log(2/α)/2n
(53)

Since F̂n (x) is an unbiased estimator for the baseline model P rather than for the (unknown) “true” model
Q we obtain the α–confidence band for FQ (x):
P {Ln (x; η) ≤ FQ (x) ≤ Un (x; η) for all x} ≥ 1 − α ,

for all Q ∈ Qη .

(54)

Due to the fact that both our bound (48) and the DKW inequality (52) are valid for any data size n, the
confidence band (54) does not require any asymptotic normality assumptions or a large data set n  1.
Connections to the Vapnik-Chervonenkis inequality The DKW inequality is an effective tool for
controlling deviations from the average for one dimensional distributions and their corresponding CDFs.
However, the Vapnik-Chervonenkis (VC) theory [40] allows us to address the same issues in a more general
setting that is applicable to higher-dimensional distributions, by considering the empirical probability
distribution instead of the CDF. In particular, corresponding inequalities to (52), but for the empirical
probability distribution, can be derived based on the VC theory, see for instance Theorem 2.41 and
Theorem 2.43 in [40]. In turn the VC inequalities, along with our concentration information bounds (46)
can allow us to obtain confidence intervals for higher dimensional distributions, similarly to (54).
Remark 19. [Poor scalability of certain information inequalities] A notable feature of the concentration/information inequalities is that they scale independently of the number of data/random variables n, at
least for classes of QoIs that satisfy (43), as demonstrated in Theorem 18 and the subsequent examples.
Furthermore, the bias bound (46) remains discriminating even if n → ∞. The same scaling features
are also shared with the GO divergence bounds (7), see [19]. On the other hand, classical information
inequalities scale poorly with n. For example, in the case of the Pinsker inequality [18], [6], let us consider
the QoI (estimator) (51) for the i.i.d. random variables X1 , . . . , Xn ,
n

1X
ψn (X1 , . . . , Xn ) =
f (Xi ) .
n i=1
Then, the Pinsker inequality becomes
|EP n [ψ(X1 , . . . , Xn )] − EQn [ψ(X1 , . . . , Xn )]| ≤ kf k∞

p
√
2R(Qn || P n ) = O( n) ,

(55)

16

where we used that kψk∞ = kf k∞ , and R(Qn || P n ) = nR(Q || P ). Therefore the Pinsker bound
(55) blows up as n >> 1, in contrast to the concentration/information inequality (46) that remains
discriminating and informative for any n. Other model bias bounds based on the Renyi or χ2 divergences
(the latter known as the Chapman-Robbins inequality) or the Hellinger metric, also scale poorly with the
size of data set and/or with the number of variables n; we refer to Sections 2.2–2.3 in [19] for a complete
discussion.
VI. E LEMENTARY EXAMPLES
Prior to discussing applications involving more complex models in Section VII, here we demonstrate
the concentration/information inequalities we developed earlier to two elementary examples that allow
easy analytic and computational implementations.
A. Exponential distribution
We first consider the model bias bounds using the GO divergence in Theorem 1, contrasted to the
concentration/information divergence in Theorem 4. In our first example, the baseline model P is an
exponential distribution. The models Q can be any distributions which are absolutely continuous with
respect to P , hence R(QkP ) < ∞. Let P be an exponential distribution with parameter λP = 1. The QoI
is f (X) = X. The MGF of P is MP (c; X) = 1/(1 − c) and thus it is finite in (0, 1), while otherwise
it is infinite. Next, we let η be a model uncertainty threshold and Q any distribution, not necessarily
exponential or in any parametric family, such that R(QkP ) ≤ η 2 . We note that the distribution P exhibits
sub-exponential behavior, namely
MP (c; f ) = 1 + c +

c2
≤ 1 + c + 2c2 ≤ exp(c + c2 /(2σB2 )) := Φ(c) ,
1−c

c ∈ (−0.5, 0.5) ,

(56)

where σB = 1/2 and the interval (−0.5, 0.5) is selected so that the bounds remain finite. In general, if
we different information on the location of λP , e.g., from data, then we can adjust the interval that c
lies in accordingly. Here, the concentration/information bound (18) is then adjusted according to (56),
using Theorem 4 and the general concentration bound (15). Although the MGF is known in this particular
example, the use of the concentration bound (56) allows us to quantify the worst-case model bias for all
QoIs g ∈ FP , where
FP = {g : MP (c; g̃) ≤ Φ(c), c ∈ (−0.5, 0.5), σB2 ≤ 1/4}.

(57)

Figure 4 is a comparison of the GO-divergence and the concentration/information bound based on (56),
along with the exact model bias for the case that Q is also an exponential distribution with R(QkP ) ≤ η 2
and η ∈ [0, 1.6].
Finally, we can consider other types of tail decay, and thus corresponding concentration inequalities,
besides the sub-Gaussian and the sub-exponential cases discussed thus far. For example, we can also
consider Poisson-type tail decay, see for instance [29, Section 3.3.5] and [41].
Remark 20. The bias is an unbounded function of the KL divergence in this example—a consequence
of the QoI f (X) = X being unbounded under P . Therefore, any decrease in KL divergence translates to
an improvement in worst-case model bias, see Figure 4; this fact is in sharp contrast with the truncated
Normal example in Section VI-B, where even large improvements to larger values of the KL divergence
may not help much in reducing model bias, see Figure VI-B.

17

Fig. 4. Comparison of model bias bounds based on the GO divergence and the concentration/information (56), with the exact model bias
1 − λQ , λQ ∈ (1.01, 10). The sub-exponential bound (56) is less sharp as the KL divergence increases, since it captures the worst-case
performance over the family of QoIs FP . Although R(QkP ) is computed with Q being an exponential distribution, the bounds to the model
bias are valid for any Q that is absolutely continuous with respect to P and has R(QkP ) in the range of the figure.

B. Truncated Normal
In this example the distributions we consider are bounded, allowing us to deploy the hierarchy of
concentration/information bounds (35) developed in Section III-B. We assume the random variable X
follows the truncated Normal distribution, P = T N (0, 1, −1, 1), where [−1, 1] is the interval of support.
Here we will bound the model bias, EQ [f ]−EP [f ], for any Q such that R(QkP ) = η 2 , where η ∈ [0.01, 1]
and for any f in a suitable family of QoIs, FP . Apart from these, the bounds make no other assumptions
on Q, f . Figure 5 contains a comparison of the different concentration/information bounds (35) from
Section III-B.
p
As a general observation, we notice that for large values η = R(QkP ), small perturbations of η will
not change the Bennett/GO (see Relations (6) and (28)) bounds significantly. Therefore, for some QoIs,
e.g., f (X) = X, small improvements to large values of the KL will barely improve the worst-case bias
(as captured by the bounds, see Figure 5). The existence of such QoIs is guaranteed by the sharpness of
the bounds demonstrated in Section IV. Finally, we not that even for the tighter concentration/information
bounds, i.e., the ones associated with the two Bennett bounds (28) and (30), there is some discrepancy
with the GO divergence bound. This discrepancy is due to the fact that the GO bound is applied only
for a specific QoI, while the concentration/information bounds are tight over the broad classes of QoIs
defined in Section III-B, see also Remark 12.
VII. E PISTEMIC U NCERTAINTY Q UANTIFICATION VIA C ONCENTRATION /I NFORMATION
I NEQUALITIES
In this Section, we apply the concentration/information inequalities to control model bias between
baseline and alternative models in two more complex examples. The type of model bias considered here
arises in epistemic uncertainty quantification, where modelers are unsure if their baseline model included
all necessary complexity or lacks sufficient data, [2], [5]. The KL divergence and in particular the GO
divergence bounds provide a non-parametric framework to mathematically describe this type of epistemic
uncertainties, as first shown in [4]. Here, we consider two such examples that illustrate different aspects

18

Fig. 5. Comparison of the different bounds for the bias in the truncated Normal example (see Section VI-B), assuming that the observable
of interest is f (X) = X. This plot makes no assumptions on the form of Q except that R(QkP ) = η 2 ∈ (0.0, 4.0). As in Figure 4, here
the concentration/information bounds capture the worst-case performance over the family of QoIs FP , hence perform worse than the GO
divergence bounds which are suitable only for a single QoI, see also (35). Notice that Bennett and Bennett-(a, b) track better the bound of
the GO divergence for large values of the KL whereas the Hoeffding is sufficient only for small values of the KL, i.e., at the linearized
regime of the GO bounds. Only the upper bounds for the bias are shown here.

of epistemic uncertainty, namely a data-driven model for the lifetime of lithium batteries, as well as a
high-dimensional Markov Random Field model subject to various localized uncertainties such as local
defects. A key aspect of our discussion in both examples is the necessity and the (ease of) implementation
of concentration/information model bias bounds, see for instance Remark 21.
A. Epistemic Uncertainty for Failure Probabilities
Here, we apply the bounds of Theorem 4, and in particular the inequalities in Section III-B, to the lifetime analysis of lithium secondary batteries. Firstly, we introduce the Weibull distribution which is widely
used in for analyzing life-time data, see [42] and references therein. The probability density function of
a Weibull random variable is
 β−1
t β
β t
f (t) =
e−( ξ ) , t > 0,
(58)
ξ ξ
where β > 0 is called a shape parameter and ξ > 0 is a scale parameter of the distribution [43]. The
shape parameter explains the types of failure and the scale parameter explains the characteristic life cycle
of devices. The cumulative distribution function F can be expressed as:
T β

F (T ) = 1 − e−( ξ ) ,
where T denotes the time of failure (or the lifetime) of the battery.
In Table III, experimental data based on life cycle tests are obtained from [42]. By fitting the data in
Table III to the parameters of the Weibull distribution, we obtain the corresponding maximum likelihood
estimator (MLE) for ξ and β are ξˆ = 1138 and β̂ = 3.55, respectively. Now, we consider this MLE Weibull
distribution as the baseline model P , which is a data-driven approximation to the unknown true model.
Next we consider the family of alternative models within a fixed tolerance η 2 , namely the non-parametric

19

TABLE III
FAILURE TIMES OF TEST SAMPLES [42]
Specimen number
Failure time

01
1373

02
1470

03
1520

04
1427

05
892

06
814

07
777

08
637

09
927

10
688

11
857

12
866

1
0.9

Failure Probabilities

0.8
0.7
0.6
0.5
0.4
0.3

Failure Probability
Failure Probability(Logistic)
GO Bounds (22=0.1)
Bennett-(a,b)(Logistic, 22=0.1)
2
Bennett-(a,b)(Logistic, 2 =0.01)

0.2
0.1
0
600

800

1000

1200

1400

1600

Time to Failure T

Fig. 6. The blue line is the failure probability based on the logistic function f2 ; The red line is the failure probability based on the indicator
function f1 ; The black lines are the GO bounds based on f1 with η 2 = 0.1; The green lines are the Bennett-(a,b) bounds based on f2 with
model uncertainty η 2 = 0.1. The magenta lines are the Bennett-(a,b) bounds based on f2 with η 2 = 0.01.

family of models Qη , see (3). This family accounts for unknown features not necessarily captured in the
baseline model which was arbitrarily assumed to be Weibull. Furthermore, the family Qη can account
for perturbations in the baseline model—constructed based on the specific dataset in Table III—due to
additional data that may become available or for any errors in the data used in the MLE step.
Next, we assess the impact of model uncertainty within the family of models Qη on two QoIs associated
with lifetime probabilities of the batteries:
f1 (t) = 1{0≤t≤T } (t), t > 0,
1
f2 (t; w) =
, t > 0.
1 + ew(t−T )

(59)
(60)

The function f2 (t; w) is a commonly used smooth approximation to the indicator function f1 (t) and is
usually referred as the logistic function, see Section 39.1 of [44]). The parameter w, w ≥ 1, controls the
smoothness of the approximation. The QoI for the life-time probability is defined exactly as FP [T ] :=
EP [f1 (t)] = P (0 ≤ t ≤ T ) or through the smooth approximation EP [f2 ].
Since the QoI f1 (t) is bounded in [0, 1], we can apply the Bennett (28), Bennett-(a,b) (30) and Hoeffding
bounds (32)) to obtain the uncertainty region, where a = 0, b = 1, a˜1 = −FP (T ), b˜2 = 1 − FP (T ) and
σB2 = V arP [f1 (t)], the latter needed just in the Bennet bound. For f2 , we estimate EP [f2 ] by sampling
from P , thus computing µ2 = EP [f2 ], needed in both Bennett bounds. Then, ã2 = −µ2 and b̃2 = 1 − µ2 .
In Figure 6 we compare the lifetime probabilities given by f1 and f2 , where for the latter we set w = 5.
In this Figure, we also observe that the logistic function f2 gives a good approximation of the indicator
function f1 since lifetime probabilities based on them are almost the same. Moreover, we set η 2 = 0.1
and also plot the GO divergence bounds of Theorem 1 based on f1 and Bennett-(a,b) bounds based on f2 .
We notice that the bounds almost coincide. We also consider the Bennett-(a,b) bounds based on a smaller
tolerance η 2 = 0.01. As we see in the figure, we obtain a significantly narrower model bias region.

20

Remark 21 (Why concentration/information inequalities?). As shown in Lemma 2.11, Equation (2.28)
of [20], the c∗ that solves the optimization problem of the GO divergence bound in Equation (6) behaves
like
c∗ = c1 η + O(η 2 ),

(61)

for some explicit constant c1 and η 2 = R(QkP ). Due to (61) and since estimator variance for the MGF
increases exponentially with c, a larger uncertainty threshold η will quickly make the accurate estimation
of MP (c∗ ; f ) more demanding, as is readily clear from Table I and (61). This drawback becomes especially
problematic when sampling from P is computationally expensive, e.g., requires MCMC sampling, P is
multi-modal, etc., see also the Markov Random Field example in Section VII-B, where sampling challenges
can become more pronounced in higher dimensions. Even when P is simple to sample, as is the case
with the baseline models in [45] and here, avoiding the estimation of MP (c∗ ; f ) in the GO divergence
can still save significant computational time, as Table I strongly suggests. For instance, the Bennett-(a,b)
bound in (30) only requires (a) the bounds of the QoI, a, b, and (b) the expected value of the QoI with
respect to P .
B. Uncertainty Quantification for Markov Random Fields
Here we consider the impact on QoIs of localized perturbations to statistical probability distributions of
Markov Random Fields [44] such as Gibbs measures. Such distributions are inherently high-dimensional,
allowing us to focus on this aspect of model bias bounds. In particular, we consider Gibbs measures for
particle systems defined on a fixed finite subset ΛN of the infinite dimensional lattice Zd . Specifically
we consider ΛN = {x ∈ Zd , |xi | ≤ n} the square lattice with N = (2n + 1)d lattice sites, where
typically n  1. Before we describe the model, we will specify some necessary notation: we let S be
the configuration space of a single particle at a lattice site x ∈ Zd . For example in a lattice gas model
S = {0, 1}, i.e. the lattice site can be empty or occupied, and in a Potts model S = {0, 1, ..., q}, i.e. the
site is empty or occupied by particles of q different species. In Ising magnetization models studied below,
we have that S = {−1, 1}, corresponding to down or up spins respectively. Then S X is the configuration
space for the particles in any subset X ⊂ Zd ; we denote by σX = {σx }x∈X an element of S X . Next, in
order to define a Gibbs measure on ΛN , we first specify the Hamiltonian HN (σΛN ) of a set of particles in
the region ΛN . An interaction Φ = {ΦX : X ⊂ Zd , X finite} associates to any finite subset X a function
ΦX (σX ) which depends only on the particle configuration in X and accounts for all particle interactions
within X, see [22] for details. Given an interaction Φ we then define the Hamiltonian HNΦ (with free
boundary conditions) by
X
HNΦ (σΛN ) =
ΦX (σX ),
(62)
X⊂ΛN

and Gibbs measure µΦ
N by

1 −HN (σΛ )
N dP (σ
e
(63)
N
ΛN ),
ZNΦ
P
and ZNΦ = σΛ e−HN (σΛN ) is the normalization constant, also

dµΦ
N (σΛN ) =

where PN is the counting measure on S ΛN
N
known as the partition function, [22].
Here we consider classes of perturbed models with corresponding interaction Ψ that includes only local
perturbations to the interaction Φ, e.g. local defects encoded in the interaction potential J, or localized
perturbations to the external field h in the example of the Ising-type Hamiltonian (65). We also note
that defects of finite temperature multi-scale probability distributions are a continuous source of interest
in the computational materials science community, see, for instance, [46]; in fact, lattice probability
distributions such as (63), constitute an important class of simplified prototype problems. In the case of
localized perturbations to the interaction Φ in (62), the Hamiltonians scale as follows:
HNΨ (σΛN ) = HNΦ (σΛN ) + O(1) .

21

Thus the corresponding relative entropy satisfies
Φ
(e∆H ) + EµΨN (−∆H)
R(µΨ
N || µN ) = log EµΨ
N

(64)

= O(1) ,

uniformly in the system size N , where we define ∆H = HNΨ − HNΦ . However, in most cases, we do not
know the exact local perturbation as well as the perturbed Gibbs measure µΨ . Instead, based on(64) we
can consider a family of perturbed models:
Qη = {µΨ : R(µΨ kµΦ ) ≤ η 2 } .
This family will include any perturbation within that tolerance η 2 , for example: defects located at different
lattice sites, and of different magnitudes, as the scaling (64) demonstrates rigorously.
As a concrete example of a Hamiltonian(62), we consider µΦ to be a one-dimensional Ising model
probability distributions on the one-dimensional lattice ΛN , labeled successively by x = 1, 2, ..., N . To
each site corresponds a spin σ(x), with two possible values: +1 or −1. The Hamiltonian is given by
HNΦ (σΛN ) = −β

N
−1
X

J(x)σ(x)σ(x + 1) − βh

x=1

N
X

σ(x).

(65)

x=1

Using the concentration/information inequalities developed in Section III, we can obtain model bias bounds
for QoIs, such as the localized average around any lattice site x,
X
1
σ(y) ,
(66)
f (σΛN ) =
2m + 1
{y:|y−x|≤m}

for a fixed radius m. In the demonstration below we we pick m = 1 for concreteness. Since the QoI f
(66) is bounded, −1 ≤ f ≤ 1, we can use the Bennett-(a,b) bound (30). Alternatively, we can use the
Bennett bound (28)), which however requires estimating in addition to EµΦN [f ], the variance V arµΦN [f ] by
sampling from µΦ , see also Section III-B. The latter is not unreasonable given that variance computations
are necessary in many applications because they ensure suitable confidence intervals for the averaged
QoIs. In Figure 7, we implement both Bennett and Bennett-(a,b) bounds by considering two different KL

1.5

1

0.5

1.5

1

Mean(Baseline model)
Bennett Bounds(22=0.5)
Bennett Bounds(22=0.05)

0.5

0

0

-0.5

-0.5

-1

-1

-1.5
-3

-2

-1

Mean(Baseline model)
Bennett-(a,b) Bounds(22=0.5)
Bennett-(a,b) Bounds(22=0.05)

0

1

2

3

-1.5
-3

-2

-1

0

h

(a)

1

2

3

h

(b)

Fig. 7. (a) The red line is the mean of the QoI (66) with m = 1 for the baseline model (65) with J = 1 and β = 1. The green and blue
lines are the Bennett bounds for η 2 = 0.05 and η 2 = 0.5, respectively. (b) The red line is the mean of the same QoI for the baseline model
(65) with J = 1 and β = 1 ; The green and blue lines are the Bennett-(a,b) bounds for η 2 = 0.05 and η 2 = 0.5, respectively. In both
figures the lattice size is N=100.

22

divergence tolerances, η 2 = 0.5, 0.05. A comparisons between Figure 7a and Figure 7b indicates that
Bennett and Bennett-(a,b) bounds are fairly close for this example.
Notable computational advantages of these concentration/information inequalities over direct numerical
simulation of alternative models Q = µΨ , as well as over the GO divergence bounds in Theorem 1 are
the following: (1) when using Theorem 4 along with Bennett-type bounds (28) or (30), we can deploy
computational resources to estimate EµΦN [f ] or possibly VarµΦN [f ]—see also Table II—just for the baseline
model P = µΦ , instead of simulating all alternative models Q = µΨ models; (2) we do not need to use
the full GO divergence bounds in Theorem 1, which require potentially expensive full MGF calculations,
also recalling Remark 21.
VIII. C ONCLUSION
In this paper we combined the uncertainty quantification information inequality of [4], [20], [19] together
with classical concentration inequalities [28] to obtain easily implementable bounds for the model bias of
quantities of interest (QoIs). The bounds control the model bias in terms of the relative entropy between
different models and intrinsic statistical quantities associated to the QoIs in a baseline model, e.g. mean,
variance, L∞ bound. Our results improve substantially on classical information bounds such as the Pinsker
inequality. First, our bound scales correctly with the size of the data sets/number of degrees of freedom
while classical inequalities do not, see Remark 19. This scaling property is illustrated in Section V where
we discuss bias bounds for general statistical estimators. In addition, we demonstrate the tightness of our
bounds in Sections II and IV: given suitable families of QoIs and a family of models whose KullbackLeibler divergence with respect to a given baseline model is less than a tolerance η 2 , there always exists
a QoI and models which saturate the upper and lower bounds. This demonstrates rigorously the precise
sense our model bias bound is optimal. In forthcoming work we will apply and generalize our results
to quantify model bias between different stochastic dynamics, e.g. Markov processes, in their long-time
regime, bias in phase diagrams of Gibbs-Markov random fields, as well as model bias of coarse-grained
models for equilibrium and non-equilibrium molecular dynamics built via variational inference methods,
[9], [47].
A PPENDIX A
P ROOF OF T HEOREM 2
While all the ingredients in the proof of Theorem 2 are already present in [4], [20], (see in particular
[20][Theorem 2.9]), its formulation is new and we provide here a proof for completeness. Theorem 2
follows immediately from following lemma
Lemma 22. Let P be a probability measure and f to be a non-constant function such that its moment
generating function MP (c; f˜) is finite in a neighborhood of 0. Let Q be such that R(QkP ) = M .
1) For any M ≥ 0 the optimization problems
log MP (±c; f˜) + M
(67)
c>0
c
have unique minimizers c± ∈ [0, +∞]. Moreover there exists 0 < M± ≤ ∞ such that the minimizers
c± = c± (M ) are finite for M ≤ M± and c± (M ) = +∞ if M > M± .
2) If c± = c± (M ) is finite
Ξ(QkP ; ±f ) = inf

log MP (±c± ; f˜) + M
d log MP
=
(±c± ; f˜) = ±(EP ±c± [f ] − EP [f ]) ,
±
c
dc
where c± (M ) is strictly increasing in M and is determined by the equation
Ξ(QkP ; ±f ) =

±

R(P ±c kP ) = M .

(68)

(69)

23

3) If M± < ∞ then f is necessarily P almost surely bounded above/bounded below respectively with
upper/lower bound f± . For M > M± we have that c± (M ) = +∞ and
Ξ(QkP ; ±f ) = ±(f± − EP [f ]) .

(70)

Proof. For notational ease, in the proof, let us set H(c) = log MP (c; f˜) and note that since f˜ is centered
we have H(0) = H 0 (0) = 0. We have H 0 (c) = EP c [f ] − Ep [f ] and H 00 (c) = VarP c (f ) > 0 since f is not
constant P almost surely
If d+ < ∞ then we have limc→d+ H(c) = ∞ and limc→d+ H 0 (c) = ∞. If d+ = ∞ then

f+ − EP [f ] if f is bounded
0
lim H (c) =
.
(71)
+∞
otherwise
c→∞
Rc
Since c−1 H(c) = c−1 0 H 0 (t)dt and H 0 (c) is strictly increasing c−1 H(c) is a strictly increasing function
and we have limc→∞ c−1 H(c) = limc→∞ H 0 (c) which is finite if only if f is bounded. Let us set
B(c; M ) =

H(c) + M
log MP (c; f ) + M
=
c
c

and then distinguish two cases:
(a) If d+ ≤ ∞ or if d+ = ∞ and f is unbounded then we have limc→0 B(c; M ) = limc→d+ B(c; M ) = +∞
and thus B(c; M ) has at least one minimum for some 0 < c < d+ . By calculus the minimum must be a
solution of
cH 0 (c) − H(c) − M
∂
0 = B(c; M ) =
∂c
c2
∂
that is me must have cH 0 (c) − H(c) = M . Since ∂c
(cH 0 (c) − H(c)) = cH 00 (c) > 0 the function cH 0 (c) −
H(c) is strictly increasing and thus there is a unique minimizer c+ for B(c; M ).
(b) If d+ = ∞ but f is bounded, since cH 0 (c)−H(c) is strictly increasing we have limc→∞ cH 0 (c)−H(c) =
M+ which may or may not be finite depending on P . If M ≤ M+ we can proceed as in (a) to find a
unique minimizer for a finite c+ , while if M > M+ , B(c; M ) is strictly decreasing and thus the minimizer
is attained at c+ = ∞.
To conclude the proof we note that if c+ < ∞ then c+ H 0 (c+ ) − H(c+ ) = M and thus

B(c+ , M ) = H 0 (c+ )
which proves (68). On the other hand a simple computation shows that for any c
R(P c kP ) = cH 0 (c) − H(c)
and this establishes (69). Finally if c+ = ∞ the infimum is equal to limc→∞

H(c)
c

and this establishes (70).

ACKNOWLEDGMENT
The research of KG and MAK was partially supported by the Office of Advanced Scientific Computing
Research, U.S. Department of Energy, under Contract No. de-sc0010723. The research of MAK and LRB
was partially supported by the National Science Foundation (NSF) under the grant DMS-1515712. The
research of JW was partially supported by the Defense Advanced Research Projects Agency (DARPA)
EQUiPS program under the grant W911NF1520122.

24

R EFERENCES
[1] M. S. Eldred, B. M. Adams, D. M. Gay, L. P. Swiler, K. Haskell, W. J. Bohnhoff, J. P. Eddy, W. E. Hart, J. paul Watson, P. D. Hough,
and T. G. Kolda, “Dakota, a multilevel parallel object-oriented framework for design optimization, parameter estimation, uncertainty
quantification, and sensitivity analysis: version 5.0 users manual,” Tech. Rep. SAND2010-2183, Sandia National Laboratories, Tech.
Rep., Dec. 2009.
[2] R. C. Smith, Uncertainty quantification: theory, implementation, and applications. Society for Industrial and Applied Mathematics,
2013.
[3] A. Saltelli, M. Ratto, T. Andres, F. Campolongo, J. Cariboni, D. Gatelli, M. Saisana, and S. Tarantola, Global sensitivity analysis: the
primer. John Wiley & Sons, 2008.
[4] K. Chowdhary and P. Dupuis, “Distinguishing and integrating aleatoric and epistemic variation in uncertainty quantification,” ESAIM:
Mathematical Modelling and Numerical Analysis, vol. 47, no. 03, pp. 635–662, 2013.
[5] T. J. Sullivan, Introduction to uncertainty quantification. Springer, 2015.
[6] A. B. Tsybakov, Introduction to nonparametric estimation. Springer Science & Business Media, 2008.
[7] C. M. Bishop, Pattern recognition and machine learning. Springer-Verlag New York, Inc., 2006.
[8] K. P. Burnham and D. R. Anderson, Model selection and multimodel inference: a practical information-theoretic approach. Springer
Science & Business Media, 2003.
[9] A. Chaimovich and M. S. Shell, “Relative entropy as a universal metric for multiscale errors,” Physical Review E, vol. 81, no. 6, p.
060104, 2010.
[10] M. A. Katsoulakis and P. Plecháč, “Information-theoretic tools for parametrized coarse-graining of non-equilibrium extended systems,”
The Journal of chemical physics, vol. 139, no. 7, p. 074115, 2013.
[11] J. F. Rudzinski and W. G. Noid, “Coarse-graining, entropy, forces and structures,” Journal of Chemical Physics, vol. 135, no. 21, 2011.
[12] A. J. Majda, R. V. Abramov, and M. J. Grote, Information theory and stochastics for multiscale nonlinear systems, ser. CRM monograph
series. Providence, R.I. American Mathematical Society, 2005.
[13] A. Atkinson, A. Doney, and R. Tobias, Optimum experimental designs, with SAS. Oxford University Press, May 2007, vol. (2007).
Oxford science publications. Oxford.
[14] A. J. Majda and B. Gershgorin, “Quantifying uncertainty in climate change science through empirical information theory,” Proceedings
of the National Academy of Sciences, vol. 107, no. 34, pp. 14 958–14 963, 2010.
[15] ——, “Improving model fidelity and sensitivity for complex systems through empirical information theory,” Proceedings of the National
Academy of Sciences, vol. 108, no. 25, pp. 10 044–10 049, 2011.
[16] M. Komorowski, M. J. Costa, D. A. Rand, and M. P. Stumpf, “Sensitivity, robustness, and identifiability in stochastic chemical kinetics
models,” Proceedings of the National Academy of Sciences, vol. 108, no. 21, pp. 8645–8650, 2011.
[17] Y. Pantazis and M. A. Katsoulakis, “A relative entropy rate method for path space sensitivity analysis of stationary complex stochastic
dynamics,” The Journal of chemical physics, vol. 138, no. 5, p. 054115, 2013.
[18] T. M. Cover and J. A. Thomas, Elements of information theory. Wiley-Interscience, July 2006.
[19] M. A. Katsoulakis, L. Rey-Bellet, and J. Wang, “Scalable information inequalities for uncertainty quantification,” Journal of
Computational Physics, vol. 336, pp. 513–545, 2017.
[20] P. Dupuis, M. A. Katsoulakis, Y. Pantazis, and P. Plechac, “Path-space information bounds for uncertainty quantification and sensitivity
analysis of stochastic dynamics,” SIAM/ASA Journal on Uncertainty Quantification, vol. 4, no. 1, pp. 80–111, 2016.
[21] P. G. Dupuis and R. S. Ellis, A weak convergence approach to the theory of large deviations, ser. Wiley series in probability and
statistics. New York: Wiley, 1997.
[22] B. Simon, The statistical mechanics of lattice gases. Princeton University Press, 2014.
[23] P. Glasserman and X. Xu, “Robust risk measurement and model risk,” Quantitative Finance, vol. 14, no. 1, pp. 29–58, 2014.
[24] H. Lam, “Robust sensitivity analysis for stochastic systems,” Mathematics of Operations Research, vol. 41, no. 4, pp. 1248–1275, Nov.
2016.
[25] J. S. Liu, Monte carlo strategies in scientific computing, ser. Springer Series in Statistics. New York: Springer-Verlag, 2001.
[26] T. Lelièvre, G. Stoltz, and M. Rousset, Free energy computations: A mathematical perspective. World Scientific, 2010.
[27] P. Del Moral, A. Doucet, and A. Jasra, “On adaptive resampling strategies for sequential monte carlo methods,” Bernoulli, vol. 18,
no. 1, pp. 252–278, Feb. 2012.
[28] S. Boucheron, G. Lugosi, P. Massart, and M. Ledoux, Concentration inequalities : a nonasymptotic theory of independence. Oxford:
Oxford University Press, 2013.
[29] M. Raginsky, I. Sason, et al., “Concentration of measure inequalities in information theory, communications, and coding,” Foundations
and Trends R in Communications and Information Theory, vol. 10, no. 1-2, pp. 1–246, 2013.
[30] M. Ledoux, The concentration of measure phenomenon. American Mathematical Society, 2005.
[31] A. Dembo and O. Zeitouni, “Large deviations techniques and applications,” 2010.
[32] P. Massart, Concentration inequalities and model selection. Springer, 2007.
[33] J. A. Tropp, “An introduction to matrix concentration inequalities,” Foundations and Trends R in Machine Learning, vol. 8, no. 1-2,
pp. 1–230, 2015.
[34] L. Wasserman, All of statistics: a concise course in statistical inference. Springer-Verlag, New York, 2004.
[35] S. G. Bobkov and F. Götze, “Exponential integrability and transportation cost related to logarithmic sobolev inequalities,” Journal of
Functional Analysis, vol. 163, no. 1, pp. 1–28, 1999.
[36] W. Hoeffding, “Probability inequalities for sums of bounded random variables,” Journal of the American statistical association, vol. 58,
no. 301, pp. 13–30, 1963.
[37] C. Villani, Optimal transport: old and new. Springer Science & Business Media, 2008.
[38] G. Carlier, A. Galichon, and F. Santambrogio, “From knothe’s transport to brenier’s map and a continuation method for optimal
transport,” SIAM Journal on Mathematical Analysis, vol. 41, no. 6, pp. 2554–2576, 2010.

25

[39] C. McDiarmid, “On the method of bounded differences,” Surveys in combinatorics, vol. 141, no. 1, pp. 148–188, 1989.
[40] L. Wasserman, All of nonparametric statistics. Springer-Verlag New York, 2006.
[41] S. Boucheron, G. Lugosi, P. Massart, et al., “On concentration of self-bounding functions,” Electronic Journal of Probability, vol. 14,
no. 64, pp. 1884–1899, 2009.
[42] S.-W. Eom, M.-K. Kim, I.-J. Kim, S.-I. Moon, Y.-K. Sun, and H.-S. Kim, “Life prediction and reliability assessment of lithium
secondary batteries,” Journal of Power Sources, vol. 174, no. 2, pp. 954–958, 2007.
[43] A. Papoulis and S. U. Pillai, Probability, random variables, and stochastic processes. Tata McGraw-Hill Education, 2002.
[44] D. J. C. MacKay, Information theory, inference and learning algorithms. Cambridge University Press, 2003.
[45] J. Li and D. Xiu, “Computation of failure probability subject to epistemic uncertainty,” SIAM Journal on Scientific Computing, vol. 34,
no. 6, pp. A2946–A2964, 2012.
[46] J. Marian, G. Venturini, B. L. Hansen, J. Knap, M. Ortiz, and G. H. Campbell, “Finite-temperature extension of the quasicontinuum
method using langevin dynamics: entropy losses and analysis of errors,” Modelling and Simulation in Materials Science and Engineering,
vol. 18, no. 1, p. 015003, 2010.
[47] V. Harmandaris, E. Kalligiannaki, M. Katsoulakis, and P. Plecháč, “Path-space variational inference for non-equilibrium coarse-grained
systems,” Journal of Computational Physics, vol. 314, no. C, pp. 355–383, June 2016.

