A mathematical characterization of confidence
as valid belief

arXiv:1707.00486v1 [math.ST] 3 Jul 2017

Ryan Martin
Department of Statistics
North Carolina State University
rgmarti3@ncsu.edu
July 4, 2017
Abstract
Confidence is a fundamental concept in statistics, but there is a tendency to misinterpret it as probability. In this paper, I argue that an intuitively and mathematically more appropriate interpretation of confidence is through belief/plausibility
functions, in particular, those that satisfy a certain validity property. Given their
close connection with confidence, it is natural to ask how a valid belief/plausibility
function can be constructed directly. The inferential model (IM) framework provides such a construction, and here I prove a complete-class theorem stating that,
for every nominal confidence region, there exists a valid IM whose plausibility
regions are contained by the given confidence region. This characterization has
implications for statistics understanding and communication, and highlights the
importance of belief functions and the IM framework.
Keywords and phrases: confidence distribution; inferential model; plausibility
function; probability; random set.

1

Introduction

Confidence is a fundamental concept in statistics, “arguably the most substantive ingredient in modern model-based theory” (Fraser 2011b), dating back to Neyman (1941)
and also, indirectly, to Fisher (1973), through its close ties to fiducial inference (Efron
1998; Seidenfeld 1992; Zabell 1992). However, like the controversial p-value (e.g., Ionides et al. 2017; Trafimowa and Marks 2015; Wasserstein and Lazar 2016), interpretation
of this fundamental concept is somewhat elusive. For example, as instructors teaching
confidence intervals to students in an introductory statistics course, we are careful to distinguish confidence from probability: “95% confidence” does not mean that the unknown
parameter resides in the stated interval with probability 0.95. Unfortunately, there apparently is no fully satisfactory explanation of what the “95% confidence” feature of the
stated interval actually does mean. In practice, a stated confidence interval is informally
interpreted as a set of parameter values that, together, is “sufficiently and justifiably
believable” or, equivalently, as a collection of parameter values that, individually, are
1

“sufficiently and justifiably plausible.” Statisticians are reluctant to adopt the use of
words like “believable” and “plausible” because of their seemingly non-scientific connotations, but remaining silent about the interpretation of confidence and, in particular,
leaving the door open for incorrect interpretations, is not any more scientific. Fortunately,
belief, like probability, is a well-defined mathematical object, so if a rigorous connection
can be made between it and confidence, then we have for our students a clear and honest
explanation of what confidence means, consistent with how confidence intervals are used
in practice. Beyond the classroom, there is obvious value—both within the statistics
community and in our efforts to communicate with others—in having an agreeable understanding of confidence that is both intuitive and mathematically precise. A goal of
this paper is to formally develop this connection between confidence and belief.
To set the scene, suppose observable data X ∈ X is modeled by a distribution PX|θ
indexed by a parameter θ ∈ Θ; here either X, θ, or both can be scalars, vectors, or
something else. Let φ = φ(θ) be an interest parameter, taking values in Φ = φ(Θ), and
assume existence of a family of set-valued maps Cα : X → 2Φ where, for any α ∈ (0, 1),
Cα (X) is a 100(1 − α)% confidence region for φ. That is,
inf PX|θ {Cα (X) 3 φ(θ)} ≥ 1 − α,

θ∈Θ

∀ α ∈ (0, 1).

(1)

Despite the warnings given in textbooks, many might be tempted to convert this “X|θ”probability statement into a “θ|(X = x)”-probability statement by constructing a probability density function for φ = φ(θ), on Φ, that has Cα (x) as its level-(1 − α) contour,
α ∈ (0, 1). This defines a so-called confidence distribution (e.g., Schweder and Hjort
2016) for φ, but, for reasons described in Section 2.1, it is risky to treat this as a genuine
probability distribution. Basically, certain operations afforded to probabilities are inconsistent with the features of confidence, so the issues with probability go beyond semantics.
Therefore, I conclude that it is inappropriate to convert confidence to probability. Fisher
(1973, p. 74) seems to agree with this conclusion when he writes:
[Confidence regions] were I think developed and advocated under the impression that in a wider class of cases they could provide information similar to
that of the probability statements derived by the fiducial argument. It is clear,
however, that no exact probability statements can be based on them...
As an alternative to ordinary probability, here I will focus specifically on uncertainties
described by distributions of random sets (Molchanov 2005; Nguyen 2006; Shafer 1979),
a special case of the Dempster–Shafer theory of belief and plausibility functions (e.g.,
Dempster 2008, 2014; Shafer 1976). In particular, I will argue in Section 2.2 that a belief
function is more appropriate than a probability distribution for describing the uncertainty
about the unknown parameter encoded in a confidence statement, both intuitively and
mathematically (cf., Balch 2012). In particular, confidence can be propagated through
the operations afforded to belief functions, but not through the usual integration with
respect to probability measures; see Section 2.2.
In addition to re-expressing confidence regions as belief functions, it is of interest to
see what insights this connection provides. Certainly not all belief functions will admit
plausibility regions, defined in (4) and (8), that meet the coverage probability condition
(1), so important questions include: what additional properties are needed? and how can
2

a belief function satisfying these properties be constructed? Answers to both questions
can be found in the recent work on inferential models (IMs, Martin and Liu 2016), and I
review the relevant details in Section 3. Given that the IM approach provides a framework
for constructing valid belief functions with plausibility regions satisfying (1), a relevant
question is if there are confidence regions that cannot be obtained via the IM approach.
Analogous to the classical decision-theoretical results, I prove here, in Section 4, the
following complete-class theorem for IMs: given a confidence region Cα for φ = φ(θ), be
it Bayes, fiducial, or whatever, as long as it satisfies (1) and a mild compatibility condition,
there exists a valid IM on the full parameter space Θ such that the corresponding (naive)
marginal plausibility region for φ is contained by Cα . That is, at least for the purpose of
uncertainty quantification via confidence regions, there is no loss of generality or efficiency
in adopting an IM approach. Therefore, based on this theorem and the fact that IMs
provide more than just plausibility regions, I would argue, in the same spirit as Fraser
(2011a), that confidence regions are quick and dirty IMs.
While the focus here is on confidence and its ties to belief functions and IMs, there is
a larger context in which the points here are relevant. A primary source of disagreement
between the different schools of thought—Bayes, frequentist, fiducial, etc—is the source
of the probability used to describe uncertainty for the purpose of making inference, and
how should those probabilities be interpreted. But having different theories, and methods
built from each that can give different results in applications, hurts our field’s reputation
in other areas of science (Fraser 2014). According to the results presented below, if
confidence regions are an inferential objective—and they are in virtually every application
of statistical inference—then probability is not the appropriate description of that type
of uncertainty, so old questions like “where should probability come from?” and “how
should these probabilities be interpreted?” are actually irrelevant. This realization can
potentially put an end to the debates over whose probability is “right,” and provide
opportunities for better understanding and communication.

2
2.1

What is confidence?
Not probability

Introductory statistics textbooks make a distinction between confidence and probability,
but what, then, are confidence distributions? As the name suggests, a confidence distribution is a probability measure on the parameter space, derived from an α-indexed family
of confidence regions. In the simplest context, given a set of upper confidence limits, a
confidence distribution can be defined by taking the 100(1 − α)th percentile of the distribution to be the corresponding 100(1 − α)% upper confidence limit, α ∈ (0, 1). More
precise details about the construction of confidence distributions and their properties can
be found in Xie and Singh (2013), Schweder and Hjort (2002, 2016), and Nadarajah et al.
(2015). Ultimately, this approach boils down to interpreting confidence regions in the
way that students in introductory statistics courses are specifically warned not to, i.e.,
“the parameter of interest falls in the 95% confidence region with probability 0.95.” The
aforementioned warning is given to students because the stated inference problem does
not come equipped with a probability measure on the parameter space, so probability
statements about the parameter have no connection to the “real world.” The confi3

dence distribution approach side-steps this issue by treating confidence as a user-defined
subjective probability. While there is nothing principally wrong with a subjective interpretation, there are various reasons to be concerned about calling this a “probability,”
which I describe below.
First, there are cases where the only confidence regions that have the desired coverage
probabilities are unbounded, so stacking these up does not lead to a genuine probability
distribution—instead, one gets something with positive mass at “infinity.” One example
of this phenomenon is the Fieller–Creasy problem (Creasy 1954; Fieller 1954) which, in
its simplest form, concerns inference about the ratio φ = θ1 /θ2 based on two independent
normal observations, X1 ∼ N(θ1 , 1) and X2 ∼ N(θ2 , 1). In particular, for certain θ =
(θ1 , θ2 ), any 100(1 − α)% confidence interval for φ that has coverage probability 1 − α
must be unbounded with positive probability under PX|θ . This implies that certain nonextreme quantiles of the confidence distribution would be ∞, i.e., the corresponding
confidence density for φ integrates to something less than 1, hence, it is not a genuine
“distribution.” This phenomenon arises in the entire class of problems investigated in
Gleser and Hwang (1987), which includes regression models with measurement errors and
other problems of interest in econometrics (Dufour 1997).
Second, it is well known that confidence distributions for a full parameter cannot generally be marginalized like probabilities, via integration, to obtain a confidence distribution for an interest parameter φ = φ(θ). A standard example is given in Stein (1959), but
here I will reconsider the Fieller–Creasy problem described above. It is straightforward to
derive a joint confidence distribution for θ = (θ1 , θ2 ) based on the two independent observations, X = (X1 , X2 ). In particular, this confidence distribution would be N2 (X, I2 ), an
independent bivariate normal with mean X and unit variances. If one derives a marginal
distribution for φ = θ1 /θ2 from this joint distribution for θ, then one gets the following
distribution function for φ, depending on x = (x1 , x2 ),
Z
Gx (ϕ) = F (ϕ z − x1 )f (z − x2 ) dz, ϕ ∈ R,
where, F and f = F 0 denote the standard normal distribution and density functions,
respectively; see, also, Hinkley (1969). However,
−1
α
α
Cα (x) = [G−1
x ( 2 ), Gx (1 − 2 )]

is not a 100(1 − α)% confidence interval for φ since (1) is not satisfied. The problem
is that there are certain (extreme) values of (θ1 , θ2 ) such that the coverage probability
is arbitrarily small; for example, if θ1 = 1 and θ2 = 20, then the interval above, with
α = 0.05, has coverage probability approximately 0.12, so clearly (1) fails.
This known marginalization failure, along with the reality that users will inevitably be
tempted to do this marginalization, has led some researchers to reject the notion of a joint
confidence distribution, focusing instead on scalar confidence distributions. But these
marginalization issues can also arise in scalar problems. For example, there is a reasonable
confidence distribution for θ based on X ∼ N(θ, 1), but integration does not lead to a
genuine confidence distribution for φ = |θ|. Indeed, Figure 1 shows the distribution of
the confidence cumulative distribution function, denoted by CD(|θ|), evaluated at the
true |θ|, under sampling from N(0.5, 1). Clearly this distribution is not uniform, hence
4

1.0
0.8
0.6
0.4
0.0

0.2

Uniform quantiles

0.0

0.2

0.4

0.6

0.8

1.0

CD(|θ|)

Figure 1: According to Definition 1 in Xie and Singh (2013), CD(|θ|) is a confidence distribution if the gray dots—its empirical distribution based on 5000 Monte Carlo samples
under the scenario described in the text—lie on the diagonal line.
the confidence distribution features are not preserved under marginalization, even in the
scalar case.
To summarize, while there are some simple cases where it might seem safe to think of
a confidence distribution as a probability, the above points highlight that this probabilistic interpretation might be dubious. If there are cases where a confidence distribution is,
mathematically, not a distribution and, furthermore, even in cases where it is a distribution, it cannot be manipulated like one, then calling it a distribution is both inaccurate
and potentially misleading. And the same issues with marginalization hold for other inferential methods that summarize uncertainty via probability, including Bayes, fiducial,
generalized fiducial (Hannig et al. 2016), and maybe others; see Balch et al. (2017). Fortunately, there is an alternative to probability that seems particularly suited for describing
this type of inferential uncertainty.

2.2

If not probability, then what?

For the present discussion, consider the case where interest is in the full parameter θ.
Recall the standard connection between confidence regions and significance tests. In
particular, for a test that rejects the null hypothesis H0 : θ = ϑ in favor of the alternative
H1 : θ 6= ϑ, at level α ∈ (0, 1), if Cα (x) 63 ϑ, the p-value is given by
px (ϑ) = sup{α ∈ (0, 1) : Cα (x) 3 ϑ}.

(2)

Allowing ϑ to vary determines a “p-value function” (Martin 2016) that has many other
names, including confidence curve (Birnbaum 1961; Schweder and Hjort 2002, 2013,
2016), possibility function (Zadeh 1978), preference function (Blaker and Spjøtvoll 2000;
Spjøtvoll 1983), and significance function (Fraser 1990, 1991), among others. Here, I opt
for the plausibility contour terminology because there is a formal mathematical theory
5

available for describing these objects that will provide some statistical insights. Specifically, assume that {Cα : α ∈ (0, 1)} is nested in the sense that (i) Cα (x) ⊆ Cα0 (x) for
α ≥ α0 and for all x, and (ii) there exists θ̂(x) in Cα (x) for all α ∈ (0, 1). For example,
Bayesian highest posterior density credible regions have this property, with θ̂(x) being
the maximum a posteriori, or MAP, estimator. Note that θ̂(x) need not be unique and,
in the case where Cα (x) is “one-sided,” like a confidence lower/upper bound, then θ̂(x)
can be infinite. In general, nested Cα implies that px in (2) satisfies
sup px (ϑ) = 1.
ϑ

According to Shafer (1976, 1987), a plausibility contour satisfying this optimization property defines a plausibility function via
plx (A) = sup px (ϑ),

A ⊆ Θ,

(3)

ϑ∈A

which, in turn, corresponds to a consonant belief function given by
belx (A) = 1 − plx (Ac ) = 1 − sup px (ϑ),

A ⊆ Θ.

ϑ∈Ac

Note that, intuitively, belief in an assertion need not correspond to belief against its
complement, and it is easy to see that belx and plx meet this intuition, i.e., belx (A) ≤
plx (A) for all A ⊆ Θ. Given a consonant belief with plausibility function plx , it is possible
to back out the plausibility contour, i.e., px (ϑ) = plx ({ϑ}); henceforth, I will use the latter
expression for the contour function instead of carrying a separate notation.
Two immediate insights about the interpretation of Cα (x), for fixed x, emerge based
on the derived belief and plausibility functions. First, if a plausibility region Πα (x) is
defined as

Πα (x) = ϑ ∈ Θ : plx ({ϑ}) > α , α ∈ (0, 1), x ∈ X,
(4)
then it follows immediately from the definition of plausibility above that Cα (x) and Πα (x)
are the same. In other words, each ϑ in Cα (x), for fixed x, is individually sufficiently
plausible according to plx . Second, again from the definitions above,
belx {Cα (x)} = 1 − α,
so Cα (x), for fixed x, can be viewed as a set of sufficiently believable parameter values.
Note that these two assessments of Cα (x) based on belx and plx agree with those practical “sufficiently believable/plausible” interpretations in Section 1. There, an additional
adjective—“justifiably”—was used, and this is based on the coverage probability property
(1) of the confidence region. That is, my 1 − α belief in Cα (x) is justified on the basis
that the method used to construct it is reliable in the sense of (1). This is in line with
the reliabilist perspective in epistemology (e.g., Goldman 1979).
The belief function described above is determined by the given family of confidence
regions {Cα : α ∈ (0, 1)}. Since the confidence region is closely tied to the coverage
probability condition (1), it is reasonable to ask how that condition might look in terms
of its corresponding belief/plausibility function. Certainly, there are belief functions that
fail to meet this condition, so an answer to this question will provide insight as to which
6

belief functions are consistent with the classical notion of confidence. In terms of the
contour function in (2), it is easy to check that the coverage probability condition (1) is
equivalent to

sup PX|θ plX ({θ}) ≤ α ≤ α, ∀ α ∈ (0, 1).
(5)
θ∈Θ

That is, plX ({θ}) should be stochastically no smaller than Unif(0, 1) under PX|θ . This is
reminiscent of the familiar (right-of-)uniform null distribution of p-values in the hypothesis testing context. This is as far as the classical analysis goes, drawing a connection
between coverage of confidence regions and Type I error of the corresponding significance
test. Here I will take this analysis further to separate from the classical theory and
demonstrate the benefit of this extended point of view.
Consonance of the derived belief/plausibility function implies something stronger than
(5). Indeed, by the definition of plx in (3) and the property (5), it follows that
sup PX|θ {plX (A) ≤ α} ≤ α,

∀ A ⊆ Θ,

∀ α ∈ (0, 1).

(6)

θ∈A

This means that plX (A) will tend to be not small, as a function of X ∼ PX|θ , when
the assertion A is true in the sense that θ ∈ A. Since the above property holds for all
assertions A ⊆ Θ and belx (A) = 1 − plx (Ac ), a similar statement can be made in terms
of the belief function, i.e.,
sup PX|θ {belX (A) ≥ 1 − α} ≤ α.
θ6∈A

I will say that a belief/plausibility function is valid if it satisfies (6).
That consonance allows for an extension of the basic property (5) for singletons to the
general validity property has some important consequences in the context of marginalization. Indeed, if φ = φ(θ) is an interest parameter, taking values in Φ = φ(Θ), then a
marginal plausibility function for φ can be defined as
mplx (B) = plx ({ϑ : φ(ϑ) ∈ B}),

B ⊆ Φ,

(7)

and the corresponding marginal belief function is mbelx (B) = 1 − mplx (B c ). That is,
the marginal belief assigned to an assertion B about the interest parameter φ is just the
belx -belief assigned to the assertion {ϑ : φ(ϑ) ∈ B} about the full parameter θ. Note that
consonance implies marginalization is achieved by optimization,
mplx (B) =

sup plx ({ϑ}),
ϑ:φ(ϑ)∈B

and, in particular, a marginal plausibility contour obtains by taking B a singleton, i.e.,
mplx ({ϕ}) =

sup plx ({ϑ}),

ϕ ∈ Φ.

ϑ:φ(ϑ)=ϕ

Since the validity property for belx covers all assertions about θ, it immediately follows
for mbelx . That is, (6) implies
sup PX|θ {mplX (B) ≤ α} ≤ α,
θ:φ(θ)∈B

7

∀ B ⊆ Φ,

∀ α ∈ (0, 1).

In particular, this implies that the marginal plausibility region
Πφα (x) = {ϕ ∈ Φ : mplx ({ϕ}) > α}

(8)

is a nominal 100(1 − α)% confidence region for φ. This analysis and conclusions hold
for general belief/plausibility functions on Θ satisfying the validity property (6); see
Section 3. However, in the present case where belx is derived from a given family of
confidence regions Cα , it can be verified that the region Πφα (x) is exactly the “naive”
marginal confidence region obtained by projecting Cα (x) to the φ margin, i.e.,
{ϕ : φ(ϑ) = ϕ for some ϑ ∈ Cα (x)}.
In this case, that the coverage property is preserved is no surprise, but it reveals that
the belief/plausibility function representation and operations are natural for expressing
and manipulating confidence (Balch 2012). For comparison, recall from Section 2.1 that
confidence properties may not be preserved under probability-style marginalization via
integration. Therefore, as mentioned in Section 1, belief and plausibility is a more appropriate framework in which to express confidence than ordinary probability.

3

Constructing valid belief functions

The previous section demonstrated that confidence is most appropriately expressed in
terms of valid belief functions, and manipulated accordingly. There, the belief function
was derived from a given confidence region and its validity was a consequence of the
assumed coverage probability feature (1). Here, to build on those insights, we consider
the question of how to directly construct a valid belief function without a confidence
region to start with. To my knowledge, the only available construction is via the socalled inferential model (IM) framework. Martin and Liu (2013) present the following
three-step IM construction, whose key feature is the introduction of a suitably calibrated
random set on a specified auxiliary variable space.
A-step. Associate data X and parameter θ with an auxiliary variable U , taking values in
U, with known distribution PU . In particular, let
X = a(θ, U ),

U ∼ PU .

(9)

This is just a mathematical description of an algorithm for simulating from PX|θ , and it
is not necessary to assume that data is actually generated according to this process. The
inferential role played by the association is to shift primary focus from θ to U . To see
this, define the sets
Θx (u) = {θ : x = a(θ, u)},

u ∈ U,

x ∈ X.

(10)

Given data X = x, if an oracle tells me a value u? that satisfies x = a(θ, u? ), then my
inference is θ ∈ Θx (u? ), the “strongest possible” conclusion.
P-step. Predict the unobserved value of U in (9) with a random set S. This step is
motivated by the shift of focus from θ to U , and is the feature that distinguishes the IM
framework from fiducial. The distribution PS of S is to be chosen by the data analyst,
subject to certain conditions; see Theorem 1 below.
8

C-step. Combine the association map in (10) and the observed data X = x with the
random set S to obtain a new random set
[
Θx (S) =
Θx (u).
(11)
u∈S

Note that Θx (S) contains the true θ if (and often only if) S likewise contains the unobserved value u? of U . The IM output is the distribution of Θx (S), which, if Θx (S) is
non-empty with PS -probability 1 for each x, can be summarized by a belief and plausibility
function pair, given by
belx (A) = PS {Θx (S) ⊆ A}
plx (A) = PS {Θx (S) ∩ A 6= ∅},

A ⊆ Θ.

The pair (belx , plx ) measures the user’s degree of belief about θ, given data x and
the specified sampling model. That is, large belx (A) and small plx (A) indicate strong
and weak support in x for the truthfulness of “θ ∈ A,” respectively; intermediate cases
correspond to certain degrees of “don’t know” (Dempster 2008, 2014). Ideally, this output
should represent something more than just the user’s degrees of belief. Indeed, Reid
and Cox (2015) write that “it is unacceptable if a procedure. . . of representing uncertain
knowledge would, if used repeatedly, give systematically misleading conclusions.” To
protect against this, the IM approach insists that its belief and plausibility functions
are valid in the sense of (6), which effectively calibrates the plausibility function values,
leading to a connection with classical notions of confidence as described in Section 2.
How can one check that the validity condition (6) holds for the IM constructed above?
The following result, given in Martin and Liu (2013), shows that validity holds for a wide
class of predictive random sets S introduced in the P-step. Let (U, U ) be the measurable
space on which PU is defined, and assume that U contains all closed subsets of U.
Theorem 1. Suppose that the predictive random set S, supported on S, with distribution
PS , satisfies the following:
P1. The support S ⊂ 2U contains ∅ and U, and:
(a) is closed, i.e., each S ∈ S is closed and, hence, in U , and
(b) is nested, i.e., for any S, S 0 ∈ S, either S ⊆ S 0 or S 0 ⊆ S.
P2. The distribution PS satisfies PS {S ⊆ K} = supS∈S:S⊆K PU (S), for each K ⊆ U.
In addition, if Θx (S) is non-empty with PS -probability 1 for all x, then the IM is valid
in the sense that its plausibility function satisfies (6).
Constructing a random set S that satisfies P1–P2 is relatively easy; see Corollary 1
in Martin and Liu (2013). Note that the nested support property P1(b) implies that the
corresponding belief function is consonant, which was important to the development and
understanding in Section 2.2. The non-emptiness condition, namely, Θx (S) 6= ∅ with
PS -probability 1, holds trivially in many examples but not universally; often some refinements are needed, e.g., auxiliary variable dimension reduction techniques (Martin and
Liu 2015a,b) and/or random set stretching (Ermini Leaf and Liu 2012). The necessary
details for the present setting will be given in Section 4.
9

4
4.1

An IM characterization of confidence
IMs with a family of random sets

The developments in the IM literature have focused primarily on the case of a single
predictive random set S ∼ PS , but this is not the only possibility. Here it will be
advantageous to consider a family of predictive random sets indexed by the parameter
space Θ. That is, consider a fixed auxiliary variable space U and a family of supports
{Sϑ : ϑ ∈ Θ}, each consisting of subsets of U satisfying Condition P1 of Theorem 1 for
each ϑ. Then, on each support Sϑ , define a distribution PS|ϑ to satisfy Condition P2.
Given an association X = a(θ, U ), for U ∼ PU , where PU is supported on U, each
of the predictive random sets S ∼ PS|ϑ described above, for ϑ ∈ Θ, defines an IM, with
corresponding plausibility function
plx (A | ϑ) = PS|ϑ {Θx (S) ∩ A 6= ∅},

A ⊆ Θ,

ϑ ∈ Θ.

Here, I propose to fuse these ϑ-specific plausibility functions together into a single plausibility function via the following formula:
plx (A) = sup plx ({ϑ} | ϑ),

A ⊆ Θ.

(12)

ϑ∈A

This kind of fused plausibility function has been used informally in applications of the
so-called “local conditional IMs” (e.g., Cheng et al. 2014; Martin and Lin 2016; Martin
and Liu 2015a), designed to achieve a level of dimension reduction beyond that which
is available via sufficiency and/or conditioning arguments, leading to exact and efficient
solutions in some challenging non-regular problems.
According to Shafer (1987), the quantity defined in (12) is a genuine plausibility
function, corresponding to a consonant belief function, if
sup plx ({ϑ} | ϑ) = 1.

(13)

ϑ

Moreover, validity of the IM corresponding to this fused plausibility function follows
immediately from that for the individual ϑ-specific IMs implied by Theorem 1.
Theorem 2. Suppose that (13) holds. If the ϑ-specific IM is valid for each ϑ ∈ Θ, then
the fused IM with plausibility function (12) is also valid.
Proof. The essential observation is that, for the plausibility function plx (·) defined in (12),
plx (A) ≤ α implies plx ({θ} | θ) ≤ α for all θ ∈ A. Therefore,
sup PX|θ {plX (A) ≤ α} ≤ sup PX|θ {plX ({θ} | θ) ≤ α}.
θ∈A

θ∈A

Validity of the IM with plausibility function plx (· | θ) for each θ ∈ Θ implies that the
right-hand side above is no more than α. Since this holds for all A ⊆ Θ and all α ∈ (0, 1),
the claimed validity of the fused IM follows.

10

4.2

A complete-class theorem

The goal of this section is to establish the advertised complete-class result, i.e., given a
suitable confidence region, there exists a valid (fused) IM such that the corresponding
plausibility region matches the given confidence region. Recall the sampling model X ∼
PX|θ with unknown parameter θ ∈ Θ. Suppose that the parameter of interest is φ = φ(θ),
possibly vector-valued, and let Φ = φ(Θ) be its range. Let Cα : X → 2Φ be the rule that
determines, for any specified level α ∈ (0, 1), based on data X, a 100(1 − α)% confidence
region Cα (X) for φ = φ(θ). Its defining property is that Cα (X), a random set as a
function of X ∼ PX|θ , satisfies the coverage probability condition (1). I will also assume
that the collection {Cα : α ∈ (0, 1)} is nested in the sense that Cα (x) ⊆ Cα0 (x) for α ≥ α0
and for all x ∈ X, and the following limiting properties hold:
[
\
Cα (x) = Φ and
Cα (x) 6= ∅.
α

α

The latter non-emptiness condition amounts to there existing a point, say, φ̂(x) that
belongs to every confidence region Cα (x), which is a standard relationship between point
estimators and confidence regions. For example, the Bayesian MAP estimator belongs to
all of the highest posterior density credible regions.
Next, take an association, X = a(θ, U ) where U ∼ PU , like in (9), consistent with the
posited model PX|θ . Technically, PU is defined on a measurable space (U, U ), and I will
assume that the σ-algebra U is sufficiently rich that it contains all the closed subsets of
U relative to the topology T on U. Given this association, define the collection of subsets
Θx (u) = {ϑ : x = a(ϑ, u)} as in (10), and the new collection
Sα (ϑ) = clos({u : Cα (a(ϑ, u)) 3 φ(ϑ)}),

(α, ϑ) ∈ (0, 1) × Θ,

(14)

where clos(B) denotes the closure of B ⊆ U with respect to the topology T . In addition
to the confidence-regions-are-nested assumption, the complete-class result requires the
following compatibility condition on the confidence region and the association:
[
Θx (u) 6= ∅ ∀ (x, ϑ, α) ∈ X × Θ × (0, 1).
(15)
u∈Sα (ϑ)

In most “regular” problems, it is possible to arrange the association such that Θx (u)
is non-empty for all u, hence compatibility (15) is trivial, but I will postpone detailed
discussion of this condition to Section 5.1.
Theorem 3 characterizes confidence regions as IM-based plausibility regions, forging
a rigorous connection between the two; a similar result connecting p-values to IMs is
given in Martin and Liu (2014). Some remarks on the conditions and implications of the
theorem are given in Section 5.
Theorem 3. Let Cα be a family of nested confidence regions for φ = φ(θ) that satisfies
(1). Suppose that the sampling model PX|θ admits an association (9) that is compatible
with the family of confidence regions in the sense that (15) holds. Then there exists a
valid (fused) IM for θ, with marginal plausibility regions Πφα as in (8), such that
Πφα (x) ⊆ Cα (x)

∀ (x, α) ∈ X × (0, 1).
11

(16)

Equality holds in (16) for a particular α if the coverage probability function
φ 7→

inf
θ:φ(θ)=φ

PX|θ {Cα (X) 3 φ}

(17)

is constant equal to 1 − α.
Proof. For the A-step of the IM construction, take any association X = a(θ, U ), U ∼ PU ,
consistent with the above sampling distribution PX|θ and compatible with the given
confidence region in the sense that (15) holds. For the P-step, take a family of predictive
random sets S ∼ PS|ϑ , indexed by ϑ ∈ Θ, with support
Sϑ = {Sα (ϑ) : α ∈ (0, 1)},
closed and nested by the definition of Sα (ϑ) in (14), and distribution PS|ϑ satisfying
PS|ϑ (S ⊆ K) =

sup

PU {Sα (ϑ)}.

(18)

α:Sα (ϑ)⊆K

That Sα (ϑ) is PU -measurable follows from the fact that Sα (ϑ) is closed and the σ-algebra
U contains all closed subsets of U. Note, also, that Θx (S) in (11) is non-empty with
PS|ϑ -probability 1 according to (15). For the special singleton assertion {ϑ} about θ, the
C-step returns the plausibility function
plx ({ϑ} | ϑ) = PS|ϑ {Θx (S) 3 ϑ} = PS|ϑ {S ∩ Ux (ϑ) 6= ∅},
where Ux (ϑ) is defined as
Ux (ϑ) = {u : x = a(ϑ, u)}.
This determines the fused plausibility function (12) and the corresponding fused IM is
valid based on Theorem 2. To establish the desired connection between the plausibility
region of this valid IM and the given confidence region, define the index
α(x, ϑ) = sup{α ∈ (0, 1) : Sα (ϑ) ∩ Ux (ϑ) 6= ∅},

(19)

so that
plx ({ϑ} | ϑ) = 1 − PS|ϑ {S ⊆ Sα(x,ϑ) (ϑ)} ≤ α(x, ϑ),
where the inequality is due to (18) and (1). The marginal plausibility function mplx for
φ = φ(θ) is given in (7), and the corresponding marginal plausibility region Πφα (x) is in
(8). This region satisfies:
ϕ ∈ Πφα (x) ⇐⇒

sup plx ({ϑ} | ϑ) > α
ϑ:φ(ϑ)=ϕ

⇐⇒ plx ({ϑ} | ϑ) > α
=⇒ α(x, ϑ) > α
⇐⇒ Sα (ϑ) ∩ Ux (ϑ) 6= ∅
⇐⇒ ϕ ∈ Cα (x).

for some ϑ with φ(ϑ) = ϕ
for some ϑ with φ(ϑ) = ϕ
for some ϑ with φ(ϑ) = ϕ

Therefore, the valid IM constructed above has plausibility regions for φ = φ(θ) that are
contained in the given confidence regions, completing the proof of the first claim.
For the claim about when equality holds in (16), note that the one-sided implication
“=⇒” in the above display becomes two-sided if plx ({ϑ} | ϑ) = α(x, ϑ), which is the case
if the coverage probability function (17) is constant equal to 1 − α as in the statement of
the theorem.
12

5
5.1

Remarks
On the conditions of Theorem 3

The confidence regions considered under the formulation of Theorem 3 are those that
are nested, i.e., Cα (x) ⊆ Cα0 (x) for α ≥ α0 . This is an intuitively appealing property,
but certainly one could entertain confidence regions which are not nested. In the proof
of Theorem 3, this property was used only to ensure that the support sets {Sα (ϑ) : α ∈
(0, 1)} were also nested, which was one of the conditions imposed in Theorem 1. But even
this latter theorem does not strictly require nested support sets. Indeed, validity holds
for more general supports, but Theorem 3 in Martin and Liu (2013) says that any IM
based on a predictive random set with a non-nested support can be made more efficient
with the choice of a nested random set. In other words, IMs based on non-nested random
sets are inadmissible in the usual decision-theoretic sense, hence my choice here to focus
on nested confidence regions and the corresponding nested predictive random sets.
The compatibility condition (15) in Theorem 3, on the other hand, is less transparent. To provide some further insight into this condition, below I describe three different
scenarios, some general and some more specific.
First, consider the special case where both the statistical model and the confidence
region have a certain transformation structure. Let G be a group of transformations
g : X → X, with associated group G , such that X ∼ Pθ implies gX ∼ Pḡθ ; as is customary
in this context, I write gx instead of g(x), and assume that G acts transitively on Θ. Next,
define another associated group Ge such that φ(gθ) = g̃φ(θ). Then a confidence region Cα
for φ is invariant if Cα (gx) = C̃α (x). Arnold (1984) gives further details on this setup,
along with some examples. For the present context, the key point is that the model can
be described by first picking any “baseline” value of θ, say, 0, identifying the mapping
gθ that converts U ∼ P0 to X ∼ Pθ . Therefore, the association (9) looks like X = gθ U ,
with U ∼ P0 , and the support sets Sα (θ) are free of θ, i.e.,
Sα (θ) = {u : Cα (gθ u) 3 φ(θ)} = {u : Cα (u) 3 φ(0)}.
Therefore, no gluing is required in the IM construction in the proof of Theorem 3. Moreover, if the invariance structure is considered after reducing to a minimal sufficient statistic, as in Arnold, then, e.g., G acting transitively on X implies Θx (u) 6= ∅ for all (x, u),
hence (15).
Second, the compatibility condition is trivial whenever Θx (u) is non-empty for every
pair (x, u). This would hold for most “regular” problems when X represents a minimal
sufficient statistic of the same dimension as θ, there are no non-trivial constraints on the
parameter space, and the confidence region Cα is a function of only the minimal sufficient
statistic. Many problems are of this type; see Section 6.
Third, regularity is not necessary for compatibility to hold. Suppose that Y1 , . . . , Yn
are iid Unif(θ, θ + 1), and define the minimal sufficient statistic X = (X1 , X2 ), the sample
minimum and maximum, respectively. Since X is two-dimensional but θ is a scalar, the
model is considered to be “non-regular.” Here I will consider a Bayesian approach with
a flat prior for θ; then the posterior is Unif(X2 − 1, X1 ), and the equi-tailed 100(1 − α)%
credible interval is


Cα (x) = x1 − (1 − dx )(1 − α2 ), x1 − (1 − dx ) α2 , dx = x2 − x1 ,
13

and it is nested and satisfies the coverage condition (1) with equality. The natural
association is X = θ + U , where U = (U1 , U2 ) denotes the minimum and maximum of an
iid sample of size n from Unif(0, 1). With this choice,
Sα (ϑ) = {(u1 , u2 ) : [u1 + ϑ − (1 − du )(1 − α2 ), u1 + ϑ − (1 − du ) α2 ] 3 ϑ}
α
= {(u1 , u2 ) : 2−α
(1 − u2 ) ≤ u1 ≤ 2−α
(1 − u2 )}.
α
Note that any (u1 , u2 ) with u1 ≤ u2 and u1 = 1 − u2 belongs to Sα (ϑ) for all α and for
all ϑ. In particular, if θ̂(x) = 21 {x1 + (x2 − 1)} and u(x) = x − θ̂(x)12 , then
u(x) ∈ Sα (ϑ) and Θx (u(x)) = {θ̂(x)} =
6 ∅,
which implies (15).
To summarize, the compatibility condition seems to be rather weak and, in fact, I
have not been able to find a scenario where it fails. On the other hand, however, there
are many degrees of freedom in the choice of Cα , so I have not been able to prove that
compatibility always holds either.
Finally, Theorem 3 suggests that the IM’s marginal plausibility regions can in some
cases be proper subsets of the given confidence regions. The binomial example in Section 6.1 illustrates this. In cases with a vector θ with, say, a scalar interest parameter
φ = φ(θ), it is common for confidence regions Cα to be conservative at particular θ values.
However, strict inequality in (16) requires that Cα be uniformly conservative for θ having
a given value of φ, which is less common. Even confidence regions that are conservative
for some parameter values are typically motivated by their efficient performance for other
parameter values. For example, in the mixed-effects model setting considered by E et al.
(2008), their simulations show that certain confidence intervals for the random-effect
variance are conservative under some parameter settings but efficient under others. So,
a given Cα being conservative at a particular θ does not imply that the corresponding
IM-based plausibility region will be smaller; a type of uniform conservatism is required
for automatic improvement. The Behrens–Fisher illustration in Section 6.2 provides some
more details.

5.2

On connections to the Bayesian literature

A fundamental result in Bayesian analysis is the probability matching property, namely,
in certain cases, the Bayesian posterior credible regions will be first- or higher-order accurate in the sense that their frequentist coverage probability will be within a suitably
narrow range around the target nominal value; see Datta and Mukerjee (2004) and Ghosh
(2011) for a review, and see Fraser (2011a) and Fraser et al. (2016) for some limitations.
One interpretation of these results is that there exists a prior such that, asymptotically,
the corresponding Bayesian credible regions will roughly agree with a standard frequentist confidence region based on, say, likelihood theory. This is a type of complete-class
theorem, similar to others that can be found in the literature (e.g., Berger 1985, Chap. 8),
generally providing some frequentist justification for certain Bayesian methods. But having to choose a particular prior in order to get valid credible regions strips away almost
all that is genuinely Bayesian about this approach, so a characterization of confidence
through a Bayesian argument seems out of reach. Compare this to Theorems 1–2 that
14

give sufficient conditions for IMs to yield valid plausibility regions, and Theorem 3 that
says all valid confidence regions correspond to an IM.

5.3

On implications for the IM framework

Confidence regions are used in all areas of statistical applications as they provide a means
for uncertainty quantification. So being able to identify all reasonable confidence regions
as corresponding to output from a valid IM has some important implications for the
IM framework itself. That is, there is no systematic bias incurred by adopting the IM
approach in the sense that there is no “good answer” that cannot be reached through
the use of IMs. Some potential users of IMs may complain that the A- and P-steps of
the IM construction require certain subjective choices, but Theorem 3 indicates that this
is not a restriction: if the user would be happy with a valid Bayesian credible region,
then there is an IM that would achieve that, as well as an “algorithm” for finding it. It
is important to emphasize, however, that the IM approach is constructive and does not
rely on there being an already-known “good answer” to start with or compare to. That
is, one can start from a given sampling model and, through the three-steps described in
Section 3, directly construct a valid IM that may produce results different than existing
Bayesian or frequentist solutions. In other words, not only are the user’s choices in the
A- and P-steps not a restriction, they actually provide flexibility, though not so much
flexibility that the user could inadvertently violate the essential validity property. To
conclude this remark, I want to emphasize that the IM approach is a general framework,
a fundamental extension to Fisher’s work on fiducial, and should not be viewed just as a
method for constructing confidence regions or other frequentist procedures.

6
6.1

Examples
Binomial problem

Let X ∼ Bin(n, θ), where the size n is known but the success probability θ ∈ [0, 1]
is unknown. A standard confidence interval for θ, satisfying the coverage probability
condition (1), is the Clopper–Pearson interval
Cα (x) = {θ : Fθ (x) ≥ α2 , 1 − Fθ (x − 1) ≥ α2 },

α ∈ (0, 1),

where Fθ is the binomial distribution function. It is well-known (e.g., Brown et al. 2001,
Figure 11) that the coverage probability of Cα varies wildly as a function of θ so it
does not have the nice “uniform exactness” properties that pivot-based intervals enjoy.
Therefore, the IM plausibility region derived using the strategy in the proof of Theorem 3
may provide some improvement to the Clopper–Pearson interval.
To start, I will take the association in (9) as
Fθ (X − 1) < U ≤ Fθ (X),

U ∼ Unif(0, 1).

This is a standard formula for simulating a binomial variate, and I write the solution X
as Fθ−1 (U ), which can be computed via the qbinom function in R. Then the support set
Sα (θ) is given by
Sα (θ) = {u : Fθ (Fθ−1 (u)) ≥ α2 , 1 − Fθ (Fθ−1 (u) − 1) ≥ α2 }.
15

1.0
0.8
0.6
0.4
0.0

0.2

Plausibility

C−P
IM

0.4

0.6

0.8

1.0

θ

Figure 2: Plot of the fust IM and the Clopper–Pearson (C–P) plausibility contours for
the binomial problem, with n = 25 and X = 17.
The set Ux (θ) = {u : Fθ (x − 1) < u ≤ Fθ (x)} is an interval, and I need the supremum of
all α such that Sα (θ) ∩ Ux (θ) 6= ∅. This value is the index α(x, θ) in (19) which, in this
case, is given by

1

2{1 − Fθ (x − 1)}, if Fθ (x − 1) ≥ 2
α(x, θ) = 2Fθ (x),
if Fθ (x) ≤ 12


1,
if Fθ (x − 1) < 21 < Fθ (x).
After adjusting for the non-zero width of the smallest Sα(x,θ) (θ), the fused IM plausibility
function is
(
1,
if Fθ (x − 1) < 21 < Fθ (x)
plx ({θ} | θ) =
1 − g(θ, x), otherwise.
where
g(θ, x) = PU {Fθ (Fθ−1 (U )) >

α(x,θ)
,
2

Fθ (Fθ−1 (U ) − 1) < 1 −

α(x,θ)
},
2

which can be readily computed numerically, with or without Monte Carlo.
As an illustration, consider an experiment with n = 25 trials and X = 17 observed
successes. Figure 2 shows a plot of the plausibility contour extracted directly from the
Clopper–Pearson confidence interval, which is actually just α(x, θ), and that for the
fused IM based on the construction in the proof of Theorem 3. Note that the IM-based
plausibility contour is no wider than that from Clopper–Pearson, which demonstrates the
potential improvement from the particular IM construction. Finally, I mention that this
is not the only IM-based plausibility interval available for this binomial problem; e.g.,
Martin (2015) derives one from a likelihood-based IM.

16

6.2

Behrens–Fisher problem

Consider two independent data sets Y11 , . . . , Y1n1 and Y21 , . . . , Y2n2 from N(µ1 , σ12 ) and
N(µ2 , σ22 ), respectively, where θ = (µ1 , µ2 , σ12 , σ22 ) is unknown, but the goal is inference
on φ(θ) = µ1 − µ2 . Let X = (M1 , M2 , V1 , V2 ) denote the minimal sufficient statistic,
consisting of the two sample means and two sample variances, respectively. Then the
famous Hsu–Scheffé interval (Hsu 1938; Scheffé 1970) is given by
Cα (x) = (m1 − m2 ) ± t?α,n f (v1 , v2 ),
where f (v1 , v2 ) = (v1 /n1 + v2 /n2 )1/2 and t?α,n is the 1 − α2 quantile of a Student-t distribution with min{n1 , n2 } − 1 degrees of freedom. It follows from results in, e.g., Mickey
and Brown (1966) that the above interval achieves the coverage probability condition (1),
but could be somewhat conservative for small n and/or certain θ configurations. Here I
derive the plausibility counterpart for Cα as in the proof of Theorem 3.
For the association (9), I will take
D = φ + f (σ12 , σ22 ) U1 ,

Vk = σk2 U2k ,

k = 1, 2,

where D = M1 − M2 and the auxiliary variable U = (U1 , U21 , U22 ) consists of independent
components with U1 ∼ N(0, 1) and U2k ∼ ChiSq(nk − 1)/(nk − 1), k = 1, 2. Since
f (σ12 , σ22 )
D−φ
=
U1 ,
f (V1 , V2 )
f (σ12 U21 , σ22 U22 )
the support sets Sα (θ) can be written as
n
Sα (θ) = u = (u1 , u21 , u22 ) :

o
|u1 |
?
≤
t
α,n ,
{λθ u21 + (1 − λθ )u22 }1/2

where λθ = {1 + (n1 σ12 )/(n2 σ22 )}−1 , which takes values in [0, 1] as a function of θ. Note
that the set Sα (θ) depends on θ only through λθ , not on φ or on the specific values of σ12
and σ22 . Then the index α(x, θ) in (19) is given by

α(x, θ) = 2 Fn t(x, θ) − 1 ,
where t(x, θ) = (d − φ)/f (v1 , v2 ) and Fn is the Student-t distribution function with
min{n1 , n2 } − 1 degrees of freedom, which actually only depends on φ. Then the fused
IM plausibility function satisfies
plx ({θ} | θ) = PU {Sα(x,θ) (θ)},
which can be readily evaluated via Monte Carlo; then the corresponding marginal plausibility function for φ in (7) can be obtained by optimization over λ.
For illustration, consider the example in Lehmann (1975, p. 83) on travel times for
two different routes; summary statistics are:
n1 = 5
n2 = 11

m1 = 7.580
m2 = 6.136
17

v1 = 2.237
v2 = 0.073.

1.0
0.6
0.4
0.0

0.2

Plausibility

0.8

IM
H−S

−4

−2

0

2

φ

Figure 3: Plausibility functions for the Hsu–Scheffé interval (H–S) and for the derived
IM in the travel times illustration of the Behrens–Fisher problem. Differences between
the solid and dashed lines are the result of optimizing over only a finite range of λ.
The goal is to compare the mean travel times for two different routes. Figure 3 shows
a plot of the plausibility contour α(x, θ) for φ extracted directly from the Hsu–Scheffé
confidence interval, the plausibility contour above for a range of λ, and the corresponding
marginal plausibility based on optimizing over these λ. As expected, the λ-dependent
plausibilities (gray lines) are individually more efficient than that of Hsu–Scheffé. However, marginalizing over λ via optimization widens the plausibility contours to agree
exactly with Hsu–Scheffé, as predicted by Theorem 3.

6.3

A nonparametric problem

Although the developments in the previous sections focus primarily on finite-dimensional
or parametric models, there is no reason that they cannot be applied to infinite-dimensional
or nonparametric problems just the same. As a simple illustration, consider real-valued
data X = (X1 , . . . , Xn ) iid with distribution function F , where the goal is inference on the
“parameter” F . A standard nonparametric confidence region is based on the Dvoretsky–
Kiefer–Wolfowitz inequality (DKW, Dvoretzky et al. 1956) and corresponds to a ball with
respect to sup-norm k · k∞ , i.e.,
Cα (x) = {F : kF̂x − F k∞ ≤ δn,α },
where F̂x denotes the empirical distribution function based on a sample x = (x1 , . . . , xn ),
and δn,α = {log(2/α)/(2n)}1/2 . Equivalently, Cα (x) can be viewed as a confidence band,
giving pointwise lower and upper confidence limits; see Figure 4 below. Since the DKW
inequality upon which the coverage probability results for Cα are based holds for all
F , there will be some F at which the coverage is conservative, so there is reason to
expect that the IM-based confidence region derived from this will be smaller and less
conservative, similar to the binomial case presented above.
18

To keep things simple here, I will write F −1 for the inverse of a distribution function
F , and ignore the possible non-uniqueness due to F flattening out on some interval. Then
a natural association is
Xi = F −1 (Ui ), i = 1, . . . , n,
where U = (U1 , . . . , Un ) are iid Unif(0, 1). Writing this in vectorized form, i.e., X =
F −1 (U ), it follows that
Sα (F ) = {u : Cα (F −1 (u)) 3 F } = {u : kF̂F −1 (u) − F k∞ ≤ δn,α },
and it is easy to confirm that Sα (F ) ≡ Sα does not depend on F , hence there is no need
for a fused IM. Furthermore, the index α(x, F ) defined in the proof of Theorem 3 is

2
α(x, F ) = min 1, 2e−2nkF̂x −F k∞ .
Therefore, the plausibility contour for a singleton assertion {F } is
plx ({F }) = 1 − PU {Sα(x,F ) }
which can readily be evaluated via Monte Carlo for any fixed F .
For an illustration, consider the nerve data set from Cox and Lewis (1966), analyzed
in Wasserman (2006, Example 2.1) on n = 799 waiting times between pulses along a
nerve fiber. A plot of the empirical distribution function along with the lower and upper
95% confidence bands based on the DKW inequality is shown in Figure 4. While it is
difficult to draw a new pair of confidence bands corresponding to IM-based plausibility
analysis, it is simple to verify the claim that the bands would indeed be narrower. Take
the lower bound in Figure 4 as a candidate F ; then α(x, F ) ≈ 0.05 but plx ({F }) ≈ 0 and,
therefore, the 95% lower confidence bound based on the DKW inequality is not included
in the 95% plausibility region.

7

Discussion

Confidence is one of the most fundamental and widely used concepts in all of statistics, and while the majority of statisticians would agree that confidence is different from
probability, there seems to be no agreement about what specifically confidence is. The
goal of this paper was to provide a rigorous characterization of confidence via suitable
belief/plausibility functions. This goal was achieved by showing that every suitable confidence region corresponds to the plausibility region of a valid IM. This characterization
provides at least two important contributions.
First, statisticians have tried (e.g., ASA statement on p-values, etc) to drive home
the standard/textbook interpretation of confidence, but without much success. It is easy
for statisticians to complain that practitioners “just don’t get it,” but it would be more
productive to consider the possibility that this textbook explanation is not clear or otherwise not fully satisfactory. The fact is, despite our efforts to fit confidence into a familiar
framework, probability simply does not give an adequate characterization. “Plausibility intervals” is a natural and intuitively clear way to describe confidence intervals to
non-statisticians, and there is now no reason to shy away from this explanation since the
present paper provides a rigorous justification for it.
19

1.0
0.8
0.6
0.0

0.2

0.4

F(x)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

x

Figure 4: Empirical distribution function (gray) and lower and upper 95% confidence
bands (black) for the nerve data example.
Second, it establishes that the IM framework is more than just an alternative to Bayes,
frequentist, fiducial, etc. At the recent BFF4 meeting at Harvard University—see Meng
(2017) for details, including explanation of the name—a part of Nancy Reid’s invited
talk summarized each of the different approaches discussed at the meeting, with a oneline quote from the respective advocates of those approaches. For the IM framework,
Professor Reid summarized my position by “IMs are the only answer,” and the results in
the present paper provide a justification for this point of view: if there is a good answer,
then there is an IM solution that is the same or better.

References
Arnold, S. F. (1984). Pivotal quantities and invariant confidence regions. Statist. Decisions, 2(3-4):257–280.
Balch, M. S. (2012). Mathematical foundations for a theory of confidence structures.
Internat. J. Approx. Reason., 53(7):1003–1019.
Balch, M. S., Martin, R., and Ferson, S. (2017). Coverage probability fails to ensure
reliable inference. Unpublished manuscript, arXiv:1706.08565.
Berger, J. O. (1985). Statistical Decision Theory and Bayesian Analysis. Springer-Verlag,
New York, second edition.
Birnbaum, A. (1961). Confidence curves: an omnibus technique for estimation and testing
statistical hypotheses. J. Amer. Statist. Assoc., 56:246–249.
Blaker, H. and Spjøtvoll, E. (2000). Paradoxes and improvements in interval estimation.
Amer. Statist., 54(4):242–247.
20

Brown, L. D., Cai, T. T., and DasGupta, A. (2001). Interval estimation for a binomial
proportion. Statist. Sci., 16(2):101–133. With comments and a rejoinder by the authors.
Cheng, Q., Gao, X., and Martin, R. (2014). Exact prior-free probabilistic inference on
the heritability coefficient in a linear mixed model. Electron. J. Stat., 8(2):3062–3076.
Cox, D. R. and Lewis, P. A. W. (1966). The Statistical Analysis of Series of Events.
Methuen & Co., Ltd., London; John Wiley & Sons, Inc., New York.
Creasy, M. A. (1954). Symposium on interval estimation: Limits for the ratio of means.
J. Roy. Statist. Soc. Ser. B., 16:186–194.
Datta, G. S. and Mukerjee, R. (2004). Probability Matching Priors: Higher Order Asymptotics, volume 178 of Lecture Notes in Statistics. Springer-Verlag, New York.
Dempster, A. P. (2008). The Dempster–Shafer calculus for statisticians. Internat. J.
Approx. Reason., 48(2):365–377.
Dempster, A. P. (2014). Statistical inference from a Dempster–Shafer perspective. In Lin,
X., Genest, C., Banks, D. L., Molenberghs, G., Scott, D. W., and Wang, J.-L., editors,
Past, Present, and Future of Statistical Science, chapter 24. Chapman & Hall/CRC
Press.
Dufour, J.-M. (1997). Some impossibility theorems in econometrics with applications to
structural and dynamic models. Econometrica, 65(6):1365–1387.
Dvoretzky, A., Kiefer, J., and Wolfowitz, J. (1956). Asymptotic minimax character of the
sample distribution function and of the classical multinomial estimator. Ann. Math.
Statist., 27:642–669.
E, L., Hannig, J., and Iyer, H. (2008). Fiducial intervals for variance components in
an unbalanced two-component normal mixed linear model. J. Amer. Statist. Assoc.,
103(482):854–865.
Efron, B. (1998). R. A. Fisher in the 21st century. Statist. Sci., 13(2):95–122.
Ermini Leaf, D. and Liu, C. (2012). Inference about constrained parameters using the
elastic belief method. Internat. J. Approx. Reason., 53(5):709–727.
Fieller, E. C. (1954). Symposium on interval estimation: Some problems in interval
estimation. J. Roy. Statist. Soc. Ser. B., 16:175–185.
Fisher, R. A. (1973). Statistical Methods and Scientific Inference. Hafner Press, New
York, 3rd edition.
Fraser, D. A. S. (1990). Tail probabilities from observed likelihoods. Biometrika, 77(1):65–
76.
Fraser, D. A. S. (1991). Statistical inference: likelihood to significance. J. Amer. Statist.
Assoc., 86(414):258–265.

21

Fraser, D. A. S. (2011a). Is Bayes posterior just quick and dirty confidence? Statist. Sci.,
26(3):299–316.
Fraser, D. A. S. (2011b). Rejoinder: “Is Bayes posterior just quick and dirty confidence?”.
Statist. Sci., 26(3):329–331.
Fraser, D. A. S. (2014). Why does statistics have two theories? In Lin, X., Genest, C.,
Banks, D. L., Molenberghs, G., Scott, D. W., and Wang, J.-L., editors, Past, Present,
and Future of Statistical Science, chapter 22. Chapman & Hall/CRC Press.
Fraser, D. A. S., Bédard, M., Wong, A., Lin, W., and Fraser, A. M. (2016). Bayes,
reproducibility and the quest for truth. Statist. Sci., 31(4):578–590.
Ghosh, M. (2011). Objective priors: An introduction for frequentists. Statist. Sci.,
26(2):187–202.
Gleser, L. J. and Hwang, J. T. (1987). The nonexistence of 100(1 − α)% confidence sets
of finite expected diameter in errors-in-variables and related models. Ann. Statist.,
15(4):1351–1362.
Goldman, A. I. (1979). Reliabilism: What is justified belief? In Pappas, G. S., editor,
Justification and Knowledge, page 11. Reidel, Dordrecht, Holland.
Hannig, J., Iyer, H., Lai, R. C. S., and Lee, T. C. M. (2016). Generalized fiducial inference:
a review and new results. J. Amer. Statist. Assoc., 111(515):1346–1361.
Hinkley, D. V. (1969). On the ratio of two correlated normal random variables.
Biometrika, 56:635–639.
Hsu, P. L. (1938). Contributions to the theory of “Student’s” t-test as applied to the problem of two samples. In Statistical Research Memoirs, pages 1–24. University College,
London.
Ionides, E. L., Giessing, A., Ritov, Y., and Page, S. E. (2017). Response to the ASA’s
Statement on p-Values: Context, Process, and Purpose. Amer. Statist., 71(1):88–89.
Lehmann, E. L. (1975). Nonparametrics: Statistical Methods Based on Ranks. HoldenDay Inc., San Francisco, Calif.
Martin, R. (2015). Plausibility functions and exact frequentist inference. J. Amer. Statist.
Assoc., 110(512):1552–1561.
Martin, R. (2016). A statistical inference course based on p-values. Amer. Statist., to
appear; arXiv:1606.02352.
Martin, R. and Lin, Y. (2016). Exact prior-free probabilistic inference in a class of nonregular models. Stat, 5:312–321.
Martin, R. and Liu, C. (2013). Inferential models: a framework for prior-free posterior
probabilistic inference. J. Amer. Statist. Assoc., 108(501):301–313.

22

Martin, R. and Liu, C. (2014). A note on p-values interpreted as plausibilities. Statist.
Sinica, 24(4):1703–1716.
Martin, R. and Liu, C. (2015a). Conditional inferential models: combining information
for prior-free probabilistic inference. J. R. Stat. Soc. Ser. B. Stat. Methodol., 77(1):195–
217.
Martin, R. and Liu, C. (2015b). Marginal inferential models: prior-free probabilistic
inference on interest parameters. J. Amer. Statist. Assoc., 110(512):1621–1631.
Martin, R. and Liu, C. (2016). Inferential models, volume 147 of Monographs on Statistics
and Applied Probability. CRC Press, Boca Raton, FL. Reasoning with uncertainty.
Meng, X.-L. (2017). X-L Files: Bayesian, Fiducial and Frequentist: BFF4EVER. IMS
Bulletin, 46(4):17.
Mickey, M. R. and Brown, M. B. (1966). Bounds on the distribution functions of the
Behrens-Fisher statistic. Ann. Math. Statist., 37:639–642.
Molchanov, I. (2005). Theory of Random Sets. Probability and Its Applications (New
York). Springer-Verlag London Ltd., London.
Nadarajah, S., Bityukov, S., and Krasnikov, N. (2015). Confidence distributions: a
review. Stat. Methodol., 22:23–46.
Neyman, J. (1941). Fiducial argument and the theory of confidence intervals. Biometrika,
32:128–150.
Nguyen, H. T. (2006). An Introduction to Random Sets. Chapman & Hall/CRC, Boca
Raton, FL.
Reid, N. and Cox, D. R. (2015). On some principles of statistical inference. Int. Stat.
Rev., 83(2):293–308.
Scheffé, H. (1970). Practical solutions of the Behrens–Fisher problem. J. Amer. Statist.
Assoc., 65:1501–1508.
Schweder, T. and Hjort, N. L. (2002). Confidence and likelihood. Scand. J. Statist.,
29(2):309–332.
Schweder, T. and Hjort, N. L. (2013). Discussion: “Confidence distribution, the frequentist distribution estimator of a parameter: a review” [mr3047496]. Int. Stat. Rev.,
81(1):56–68.
Schweder, T. and Hjort, N. L. (2016). Confidence, Likelihood, Probability, volume 41 of
Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University
Press, New York.
Seidenfeld, T. (1992). R. A. Fisher’s fiducial argument and Bayes’ theorem. Statist. Sci.,
7(3):358–368.

23

Shafer, G. (1976). A Mathematical Theory of Evidence. Princeton University Press,
Princeton, N.J.
Shafer, G. (1979). Allocations of probability. Ann. Probab., 7(5):827–839.
Shafer, G. (1987). Belief functions and possibility measures. In Bezdek, J. C., editor, The
Analysis of Fuzzy Information, Vol. 1: Mathematics and Logic, pages 51–84. CRC.
Spjøtvoll, E. (1983). Preference functions. In A Festschrift for Erich L. Lehmann,
Wadsworth Statist./Probab. Ser., pages 409–432. Wadsworth, Belmont, Calif.
Stein, C. (1959). An example of wide discrepancy between fiducial and confidence intervals. Ann. Math. Statist., 30:877–880.
Trafimowa, D. and Marks, M. (2015). Editorial. Basic Appl. Soc. Psych., 37(1):1–2.
Wasserman, L. (2006). All of Nonparametric Statistics. Springer Texts in Statistics.
Springer, New York.
Wasserstein, R. L. and Lazar, N. A. (2016). The ASA’s statement on p-values: context,
process, and purpose [Editorial]. Amer. Statist., 70(2):129–133.
Xie, M.-g. and Singh, K. (2013). Confidence distribution, the frequentist distribution
estimator of a parameter: a review. Int. Stat. Rev., 81(1):3–39.
Zabell, S. L. (1992). R. A. Fisher and the fiducial argument. Statist. Sci., 7(3):369–387.
Zadeh, L. A. (1978). Fuzzy sets as a basis for a theory of possibility. Fuzzy Sets and
Systems, 1(1):3–28.

24

