ON EXTENDED ADMISSIBLE PROCEDURES
AND THEIR NONSTANDARD BAYES RISK
B Y H AOSUI D UANMU

AND

DANIEL M. ROY

arXiv:1612.09305v2 [math.ST] 13 Feb 2017

University of Toronto
For finite parameter spaces under finite loss, every Bayes procedure derived from a prior with full support is admissible, and every
admissible procedure is Bayes. This relationship already breaks down
once we move to finite-dimensional Euclidean parameter spaces. Compactness and strong regularity conditions suffice to repair the relationship, but without these conditions, admissible procedures need not be
Bayes. Under strong regularity conditions, admissible procedures can
be shown to be the limits of Bayes procedures. Under even stricter
conditions, they are generalized Bayes, i.e., they minimize the Bayes
risk with respect to an improper prior. In both these cases, one must
venture beyond the strict confines of Bayesian analysis. Using methods from mathematical logic and nonstandard analysis, we introduce
the class of nonstandard Bayes decision procedures—namely, those
whose Bayes risk with respect to some prior is within an infinitesimal of the optimal Bayes risk. Among procedures with finite risk
functions, we show that a decision procedure is extended admissible
if and only if its nonstandard extension is nonstandard Bayes. For
problems with continuous risk functions defined on metric parameter
spaces, we derive a nonstandard analogue of Blyth’s method that can
be used to establish the admissibility of a procedure. We also apply
the nonstandard theory to derive a purely standard theorem: when risk
functions are continuous on a compact Hausdorff parameter space, a
procedure is extended admissible if and only if it is Bayes.

CONTENTS
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . .
1.1 Overview of the paper . . . . . . . . . . . . . . . . . .
2 Standard preliminaries . . . . . . . . . . . . . . . . . . . .
2.1 Admissibility . . . . . . . . . . . . . . . . . . . . . .
2.2 Bayes optimality . . . . . . . . . . . . . . . . . . . .
2.3 Convexity . . . . . . . . . . . . . . . . . . . . . . . .
3 Prior work . . . . . . . . . . . . . . . . . . . . . . . . . . .
4 Nonstandard admissibility . . . . . . . . . . . . . . . . . .
4.1 Nonstandard extension of a statistical decision problem
4.2 Nonstandard admissibility . . . . . . . . . . . . . . .
5 Nonstandard Bayes . . . . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

MSC 2010 subject classifications: Primary 62C07, 62C10, 62A01; secondary 28E05
Keywords and phrases: decision theory, complete class theorems, nonstandard analysis

1

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

2
3
4
5
7
8
9
13
14
15
18

2

DUANMU AND ROY

5.1 Hyperdiscretized risk set . . . . . . . . . . . . . . . . . . .
5.2 Nonstandard complete class theorems . . . . . . . . . . . .
6 Applications to compact statistical decision problems . . . . . . .
7 Admissibility of nonstandard Bayes procedures . . . . . . . . . .
8 Some Examples . . . . . . . . . . . . . . . . . . . . . . . . . . .
9 Miscellaneous remarks . . . . . . . . . . . . . . . . . . . . . . .
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . .
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A Nonstandard analysis — Basic Notions and Key Results . . . . . .
A.1 The hyperreals . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Nonstandard extensions of more general spaces . . . . . . .
A.2.1 Vector spaces . . . . . . . . . . . . . . . . . . . . .
B Internal probability theory . . . . . . . . . . . . . . . . . . . . .
B.1 Hyperfinitely additive probability measures and Loeb theory
Author’s addresses . . . . . . . . . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

20
22
24
28
30
32
34
34
36
38
40
41
42
43
45

1. Introduction. There is a long line of research, originating in the seminal
work of Wald [29, 30], connecting admissibility and Bayes optimality. For finite
parameters spaces, one can use intuitive geometric arguments to establish that every admissible decision procedure is Bayes (see, e.g., [10, §2.10 Thm. 1]). In the
other direction, elementary arguments show that every procedure that is Bayes with
respect to a prior with full support is admissible [10, §2.3 Thms. 2 and 3]. This
close relationship between admissibility and Bayes optimality already breaks down
for finite-dimensional parameter spaces: Here, admissible decision procedures are,
in general, not Bayes procedures (see, e.g., Stein [27, §4]). Under some regularity conditions, however, every admissible procedure is a limit of Bayes procedures
[6, 16, 31], although, limits of Bayes procedures are in general neither admissible
nor Bayes, as famously demonstrated by Stein [27] in the multivariate normal location model (see also [13]). Under more stringent conditions, admissible procedures
are generalized Bayes [3, 5, 23, 28], i.e., procedures derived from the mechanical—
also known as formal—application of Bayes rule with respect to improper priors. In both cases, we must leave the strict confines of the Bayesian formalism to
reëstablish the link between admissibility and Bayes optimality. The price paid for
abandoning the standard Bayesian framework—namely, nonconglomerability and
its side effects, including marginalization paradoxes—is the subject of an extensive
literature (see, e.g., [9, 24, 25]).
Here we take a different approach, working within the standard Bayesian theory
but carrying out that work in an unusual setting. In particular, we rely on results in
mathematical logic that establish the existence of nonstandard models of the reals
satisfying three principles: extension, which associates every ordinary mathematical
object with a nonstandard counterpart called its extension; transfer, which permits
us to use first order logic to relate standard and nonstandard structures; and satura-

NONSTANDARD BAYES

3

tion, which gives us a powerful mechanism for proving the existence of nonstandard
structures defined in terms of finitely satisfiable collections of first order formula.
Informally speaking, the utility of nonstandard models for statistical decision theory stems from two sources: First, every nonstandard model possesses nonstandard
reals numbers, including infinitesimal / infinite positive real numbers that are, respectively, smaller than / larger than any standard positive real number. Using such
numbers, we can, e.g., construct uniform probability measures over infinite intervals that contain the entire standard real line, or construct probability measures on
the positive real line concentrating all their mass on a positive infinitesimal. As priors, these structures can be used as extreme statements of uncertainty that do not
correspond to any standard prior. Second, standard real numbers look discrete in
a nonstandard model. Indeed, in a suitably saturated model of the reals, the standard reals are contained within a hyperfinite set, i.e., an infinite set that nonetheless
possesses all the first order properties of a standard finite set.
Using nonstandard analysis and probability theory, we are able to reëstablish the
link between admissibility and Bayesian optimality without regularity conditions.
In particular, using a separating hyperplane argument in concert with the three principles outline above, we show that a standard decision procedure δ is extended
admissible if and only if, for some nonstandard prior, the Bayes risk of its extension ∗δ is within an infinitesimal of the minimum Bayes risk among all extensions.
Such a decision procedure is said to be nonstandard Bayes. Assuming Θ is a metric
space and risk functions are continuous, we are able to show that a procedure is
admissible if its extension is nonstandard Bayes with respect to a prior that assigns
sufficient mass to every standard open ball. The result is a nonstandard variant of
Blyth’s method, but a single nonstandard prior witnesses the admissibility, rather
than a sequence. We also apply our nonstandard theory to give a standard result: on
compact Hausdorff spaces when risk functions are continuous, a decision procedure
is extended admissible if and only if is Bayes.
1.1. Overview of the paper. In Section 2, we introduce basic notions and key
results in standard statistical decision theory: domination, admissibility, and its variants; Bayes optimality; and basic complete class and essentially complete class results. (Classic treatments can be found in [10] and [4], the latter emphasizing the
connection with game theory, but restricting itself to finite discrete spaces. A modern treatment can be found in [17].) In Section 3, we follow this introduction of basic
principles with a summary of the extensive literature on complete class theorems.
In Section 4, we define nonstandard counterparts of admissibility, extended admissibility, and essential completeness, which we obtain by ignoring infinitesimal
violations of the standard notions, and then give key theorems relating standard and
nonstandard notions for standard decision procedures and their nonstandard extensions, respectively. For readers unfamiliar with nonstandard analysis and probability, we summarize basic notions and key results in Appendices A and B.
In Section 5, we define the nonstandard counterpart to Bayes optimality, which

4

DUANMU AND ROY

we also obtain by ignoring infinitesimal violations of the standard notion. Using
saturation and a hyperfinite version of the classical separating hyperplane argument
on a hyperfinite discretization of the risk set, we show that a decision procedure is
extended admissible if and only if it its extension is nonstandard Bayes.
In Section 6, we apply the nonstandard theory to obtain a standard result: assuming the parameter space is compact and risk functions are continuous, a decision
procedure is extended admissible if and only if it is Bayes.
In Section 7, we employ the results of the previous section to connect admissibility and nonstandard Bayes optimality under various regularity conditions on the
space and the nonstandard prior. In the process, we give a nonstandard variant of
Blyth’s method.
In Section 8, we study several simple statistical decision problems to highlight
the nonstandard theory and its connections to the standard theory.
In Section 9, we list some open problems.
2. Standard preliminaries. A (nonsequential) statistical decision problem is
defined in terms of a parameter space Θ, each element of which represents a possible state of nature; a set A of actions available to the statistician; a function
ℓ : Θ × A → R≥0 characterizing the loss associated with taking action a ∈ A in
state θ ∈ Θ; and finally, a family P = (Pθ )θ ∈Θ of probability measures on a measurable sample space X . On the basis of an observation from Pθ for some unknown
element θ ∈ Θ, the statistician decides to take a (potentially randomized) action a,
and then suffers the loss ℓ(θ , a).
Formally, having fixed a σ -algebra on the space A of actions, every possible
response by the statistician is captured by a (randomized) decision procedure, i.e.,
a map δ from X to the space M1 (A) of probability measures A. As is customary,
we will write δ (x, A) for (δ (x))(A). The expected loss, or risk, to the statistician in
state θ associated with following a decision procedure δ is
Z hZ
i
(2.1)
ℓ(θ , a)δ (x, da) Pθ (dx).
rδ (θ ) = r(θ , δ ) =
X

A

R

For the risk function to be well-defined, the maps x 7→ A ℓ(θ , a)δ (x, da), for θ ∈ Θ,
must be measurable, and so we will restrict our attention to those decision procedures satisfying this weak measurability criterion. A decision procedure δ is said
to have finite risk if rδ (θ ) ∈ R for all θ ∈ Θ. Let D denote the set of randomized
decision procedures with finite risk.
The set D may be viewed as a convex subset of a vector space. In particular, for
all δ1 , . . . , δn ∈ D and p1 , . . ., pn ∈ R≥0 with ∑i pi = 1, define ∑i pi δi : X → M1 (A)
by (∑i pi δi )(x) = ∑i pi δi (x) for x ∈ X . Then r(θ , ∑i pi δi ) = ∑i pi r(θ , δi ) < ∞, and
so we see that ∑i pi δi ∈ D and r(θ , ·) is a linear function on D for every θ ∈ Θ.
For a subset D ⊆ D, let conv(D) denote the set of all finite convex combinations of
decision procedures δ ∈ D.

NONSTANDARD BAYES

5

A decision procedure δ ∈ D is called nonrandomized if, for all x ∈ X , there
exists d(x) ∈ A such that δ (x, A) = 1 if and only if d(x) ∈ A, for all measurable sets
A ⊆ A. Let D0 ⊆ D denote the subset of all nonrandomized decision procedures.
Under mild measurability assumptions, every δ ∈ D0 can be associated with a map
x 7→ d(x) from X to A for which the risk satisfies
(2.2)

r(θ , δ ) =

Z

X

ℓ(θ , d(x))Pθ (dx).

Finally, writing S[<∞] for the set of all finite subsets of a set S, let
(2.3)

D0,FC =

[

conv(D)

D∈D0 [<∞]

be the set of randomized decision procedures that are finite convex combinations
of nonrandomized decision procedures. Note that D0 ⊂ D0,FC ⊂ D and D0,FC is
convex.
2.1. Admissibility. In general, the risk functions of two decision procedures are
incomparable, as one procedure may present greater risk in one state, yet less risk
in another. Some cases, however, are clear cut: the notion of domination induces a
partial order on the space of decision procedures.
D EFINITION 2.1.

Let ε ≥ 0 and δ , δ ′ ∈ D. Then δ is ε -dominated by δ ′ if

1. ∀θ ∈ Θ r(θ , δ ′ ) ≤ r(θ , δ ) − ε , and
2. ∃θ ∈ Θ r(θ , δ ′ ) 6= r(θ , δ ).
Note that δ is dominated by δ ′ if δ is 0-dominated by δ ′ . If a decision procedure δ is ε -dominated by another decision procedure δ ′ , then, computational issues
notwithstanding, δ should be eliminated from consideration. This gives rise to the
following definition:
D EFINITION 2.2.

Let ε ≥ 0, C ⊆ D, and δ ∈ D.

1. δ is ε -admissible among C unless δ is ε -dominated by some δ ′ ∈ C .
2. δ is extended admissible among C if δ is ε -admissible among C for all ε > 0.
Again, note that δ is admissible among C if δ is 0-admissible among C . Clearly
admissibility implies extended admissibility. In other words, the class of all extended admissible decision procedures contains the class of all admissible decision
procedures.
Admissibility leads to the notion of a complete class.
D EFINITION 2.3. Let A , C ⊆ D. Then A is a complete subclass of C if, for
all δ ∈ C \ A , there exists δ0 ∈ A such that δ0 dominates δ . Similarly, A is an

6

DUANMU AND ROY

essentially complete subclass of C if, for all δ ∈ C \ A , there exists δ0 ∈ A such
that r(θ , δ0 ) ≤ r(θ , δ ) for all θ ∈ Θ. An essentially complete class is an essentially
complete subclass of D.
If a decision procedure δ is admissible among C , then every complete subclass
of C must contain δ . Note that the term complete class is usually used to refer to a
complete subclass of some essentially complete class (such as D itself or D0 under
the conditions described in Section 2.3.)
The next lemma captures a key consequence of essential completeness:
L EMMA 2.4. Suppose A is an essentially complete subclass of C , then extended admissible among A implies extended admissible among C .
The class of extended admissible estimators plays a central role in this paper. It
is not hard, however, to construct statistical decision problems for which the class
is empty, and thus not a complete class.
E XAMPLE 2.5. Consider a statistical decision problem with sample space X =
{0}, parameter space Θ = {0}, action space A = (0, 1], and loss function ℓ(0, d) = d.
Then every decision procedure is a constant function, taking some value in A. For all
c ∈ (0, 1], the procedure δ ≡ c is c/2-dominated by the decision procedure δ ′ ≡ c/2.
Hence, there is no extended admissible estimator, hence the extended admissible
procedures do not form a complete class.
The following result gives conditions under which the class of extended admissible estimators are a complete class. (See [4, §5.4–5.6 and Thm. 5.6.3] and [10, §2.6
Cor. 1] for related results for finite spaces.)
T HEOREM 2.6. Let C ⊆ D. Suppose that, for all sequences δ , δ1 , δ2 , . . . ∈ C
and nondecreasing sequences ε1 , ε2 , · · · ∈ R>0 such that ε0 = limi εi exists and δ is
εi -dominated by δi for all i ∈ N, there is a decision procedure δ0 ∈ C such that δ is
ε0 -dominated by δ0 . Then the set of procedures that are extended admissible among
C form a complete subclass of C .
P ROOF. Let S = {x ∈ RΘ : (∃δ ∈ C ) (∀θ ∈ Θ) x(θ ) = r(θ , δ )} denote the risk
set of C . Pick δ ∈ C and suppose δ is not extended admissible among C . Let
(2.4)

Qε (δ ) = {x ∈ RΘ : (∀θ ∈ Θ)(x(θ ) ≤ r(θ , δ ) − ε )}.

Let M be the set {ε ∈ R>0 : Qε (δ ) ∩ S 6= 0},
/ which is nonempty because δ is
not extended admissible among C . As the risk is nonnegative and finite, M is
also bounded above. Hence there exists a least upper bound ε0 of M. Pick a nondecreasing sequence ε1 , ε2 , . . . ∈ M that converges to ε0 . We now construct a (potentially infinite) sequence of decision procedures inductively:

NONSTANDARD BAYES

7

1. Choose δ1 ∈ C such that δ is ε1 -dominated by δ1 . Because M is nonempty,
there must exist such a procedure.
2. Suppose we have chosen δ1 , . . ., δi ∈ C , and suppose there is an index j ∈ N
such that δ is ε j -dominated by δi but δ is not ε j+1 -dominated by δi . Then we
choose δi+1 ∈ C such that δ is ε j+1 -dominated by δi+1 . Because M contains
ε j+1 , there must exist such a procedure. If no such index j exist, the process
halts at stage i.
Suppose the process halts at some finite stage i0 . Then for, all j ∈ N, δ is not
ε j -dominated by δi0 or δ is ε j+1 -dominated by δi0 . But δ is ε1 -dominated by δi0 and
so, by induction, δ is ε j -dominated by δi0 for all j ∈ N. As the sequence ε1 , ε2 , . . .
is non-decreasing and has a limit ε0 , it follows easily via a contrapositive argument
that δ is even ε0 -dominated by δi0 . If δi0 were not extended admissible among C ,
then this would contradict the fact that ε0 is a least upper bound on M.
Now suppose the process continues indefinitely. Then the claim is that δ is εi dominated by δi for all i ∈ N. Clearly this holds for i = 1. Supposing it holds for
i ≤ k. Then δ is εi -dominated by δk for all i ≤ k and there exists j ∈ N such that δ is
ε j -dominated by δk but δ is not ε j+1 -dominated by δk . It follows that j ≥ k, hence
δ is εk+1 -dominated by δk+1 , as was to be shown.
Thus, by hypothesis, there is a decision procedure δ ′ ∈ C such that δ is ε0 dominated by δ ′ . As ε0 is the least upper bound of M, δ ′ is also extended admissible
among C , completing the proof.
2.2. Bayes optimality. Consider now the Bayesian framework, in which one
adopts a prior, i.e., a probability measure π defined on some σ -algebra on Θ. Irrespective of the interpretation of π , we may define the Bayes risk of a procedure as
the expected risk under a parameter chosen at random from π .1
D EFINITION 2.7.

Let δ ∈ D, ε ≥ 0, and C ⊆ D, and let π0 be a prior.
R

1. The Bayes risk under π0 of δ is r(π0 , δ ) = Θ r(θ , δ )π0 (dθ ).
2. δ is ε -Bayes under π0 among C if r(π0 , δ ) < ∞ and, for all δ ′ ∈ C , we have
r(π0 , δ ) ≤ r(π0 , δ ′ ) + ε .
3. δ is Bayes under π0 among C if δ is 0-Bayes under π0 among C .
4. δ is extended Bayes among C if, for all ε > 0, there exists a prior π such that
δ is ε -Bayes under π among C .
1

We must now also assume that r(·, δ ) is a measurable function for every δ ∈ D. Normally, there
is a natural choice of σ -algebra on Θ that satisfies this constraint. Even if there is no natural choice,
there is always a sufficiently rich σ -algebra that renders every risk function measurable. In particular,
the power set of Θ suffices. Note that the σ -algebra determines the set of possible prior distributions.
In the extreme case where the σ -algebra on Θ is taken to be the entire power set, the set of prior
distributions contain the purely atomic distributions and these are the only distributions if and only
if there is no real-valued measurable cardinal less than or equal to the continuum [12, Thm. 1D]. As
we will see, the purely atomic distributions suffice to give our complete class theorems.

8

DUANMU AND ROY

5. δ is ε -Bayes among C (resp., Bayes among C ) if there exists a prior π such
that δ is ε -Bayes under π among C (resp., Bayes under π among C ).
We will sometimes write Bayes among C with respect to π0 to mean Bayes under
π0 among C , and similarly for ε -Bayes among C .
The following well-known result establishes a basic connection between Bayes
optimality and admissibility (see, e.g., [4, Thm. 5.5.1]). We give a proof for completeness.
T HEOREM 2.8. If δ is Bayes among C , then δ is extended Bayes among C , and
then δ is extended admissible among C .
P ROOF. That Bayes implies extended Bayes follows trivially from definitions.
Now assume δ is not extended admissible among C . Then there exists ε > 0 and
δ ′ R∈ C such that r(θR, δ ′ ) ≤ r(θ , δ ) − ε forR all θ ∈ Θ. But then,
for every prior
R
π , r(θ , δ ′ )π (dθ ) ≤ r(θ , δ )π (dθ ) − ε or r(θ , δ ′ )π (dθ ) = r(θ , δ )π (dθ ) = ∞,
hence δ is not ε /2-Bayes among C , hence not extended Bayes among C .
Note that neither extended admissibility nor admissibility imply Bayes optimality, in general. E.g., the maximum likelihood estimator in a univariate normallocation problem is admissible, but not Bayes.
Essential completeness allows us to strengthen a Bayes optimality claim:
T HEOREM 2.9. Suppose A is an essentially complete subclass of C , then ε Bayes among A implies ε -Bayes among C for every ε ≥ 0.
P ROOF. Let δ0 be Bayes under π among A for some prior π . Let δ ∈ C . Then
there exists δ ′ ∈ A such that, for all r(
θ , δ ′ ) ≤ r(θ , δ ) for
all θ ∈ Θ. By hypotheR
R
′
′
′
sis, r(π , δ0 ) ≤ r(π , δ ), but r(π , δ ) = r(θ , δ )π (dθ ) ≤ r(θ , δ )π (dθ ) = r(π , δ ).
Hence r(π , δ0 ) ≤ r(π , δ ) for all δ ∈ C .
2.3. Convexity. An important class of statistical decision problems are those in
which the action
space A is itself a vector space over the field R. In that case, the
R
mean estimate A a δ (x, da) is well defined for every δ ∈ D0,FC and x ∈ X , which
motivates the following definition.
D EFINITION 2.10. For δ ∈ D0,FC , define E(δ ) : X → M1 (A) by E(δ )(x, A) = 1
if A a δ (x, da) ∈ A and 0 otherwise, for every x ∈ X and measurable subset A ⊆ A.
R

When the loss function is assumed to be convex, it is well known that the mean
action will be no worse on average than the original randomized one. We formalize
this condition below and prove several well-known results for completeness.
C ONDITION LC (loss convexity). A is a vector space over the field R and the
loss function ℓ is convex with respect to the second argument.

NONSTANDARD BAYES

9

L EMMA 2.11. Let δ and E(δ ) be as in Definition 2.10, and suppose (LC) holds.
Then r(·, δ ) ≥ r(·, E(δ )), hence E(δ ) ∈ D0 .
P ROOF. Let θ ∈ Θ. By convexity of ℓ in its second parameter and a finitedimensional version of Jensen’s inequality [10, §2.8 Lem. 1], we have
Z hZ
i
(2.5)
ℓ(θ , a)δ (x, da) Pθ (dx)
r(θ , δ ) =
(2.6)

≥

ZX

X

A

ℓ(θ ,

R

A a δ (x, da))Pθ (dx) = r(θ , E(δ )).

R EMARK 2.12. Irrespective of the dimensionality of the action space A, we
may use a finite-dimensional version of Jensen’s inequality because the procedure
δ ∈ D0,FC is a finite mixture of nonrandomized procedures. The proof for a general
randomized procedure δ ∈ D and a general action space A, would require additional
hypotheses to account for the possible failure of Jensen’s inequality (see [20]) and
the possible lack of measurability of E(δ ) (see [10, S2.8]).
L EMMA 2.13.
of D0,FC .

Suppose (LC) holds. Then D0 is an essentially complete subclass

P ROOF. Let δ ∈ D0,FC . Then E(δ ) ∈ D0 . By Lemma 2.11, E(δ ) is well defined
and r(θ , δ0 ) ≥ r(θ , E(δ )), completing the proof.
R EMARK 2.14. See the remark following [10, §2.8 Thm. 1] for a discussion
of additional hypotheses needed for establishing that D0 is an essentially complete
subclass of D.
3. Prior work. The first key results on admissibility and Bayes optimality are
due to Abraham Wald, who laid the foundation of sequential decision theory. In
[30], working in the setting of sequential statistical decision problems with compact parameter spaces, Wald showed that the Bayes decision procedures form an
essentially complete class. Sequential decision problems differ from the decision
problems we will be discussing in this paper in the sense that it gives the statistician
the freedom to look at a sequence of observations one at a time and to decide, after
each observation, whether to stop and take an action or to continue, potentially at
some cost. The decision problems we will be discussing in this paper can be seen
as special cases of sequential decision problems with only one observation.
In order to prove his results, Wald required a strong form of continuity for his
risk and loss functions.
D EFINITION 3.1. A sequence of parameters {θi }i∈N converges in risk to a parameter θ when supδ ∈D |r(θi , δ ) − r(θ , δ )| → 0 as i → ∞, and converges in loss

10

DUANMU AND ROY

when supa∈A |ℓ(θi, a) −ℓ(θ , a)| → 0 as i → ∞. Similarly, a sequence of decision procedures {δi }i∈N in D converges in risk to a decision procedure δ when supθ ∈Θ |r(θ , δi )−
r(θ , δ )| → 0 as i → ∞. A sequence of actions {ai }i∈N converges in loss to an action
a ∈ A when supθ ∈Θ |ℓ(θ , ai) − ℓ(θ , a)| → 0 as i → ∞.
Topologies on Θ, A, and D are generated by these notions of convergence. In
the following result and elsewhere, a model P is said to admit (a measurable family
of) densities
( fθ )θ ∈Θ (with respect to a dominating (σ -finite) measure ν ) when
R
Pθ (A) = A fθ (x) ν (dx) for every θ ∈ Θ and measurable A ⊆ X . In terms of these
densities, there is a unique Bayes solution with respect to a prior π on Θ when, for
every x ∈ X , except perhaps for a set of ν -measure 0, there exists one and only one
action a∗ ∈ A for which the expression
(3.1)

Z

Θ

ℓ(θ , a) fθ (x) π (dθ )

takes its minimum value with respect to a ∈ A. (Another notion of uniqueness used
in the literature is to simply demand that the risk functions of two Bayes solutions
agree.) The main result can be stated in the special case of a nonsequential decision
problem as follows:
T HEOREM 3.2 ([30, Thms. 4.11 and 4.14]). Assume Θ and D are compact in
risk, and that Θ and A are compact in loss. Assume further that P admits densities
( fθ )θ ∈Θ with respect to Lebesgue measure, that these densities are strictly positive outside a Lebesgue measure zero set. Then every extended admissible decision
procedure is Bayes. If the Bayes solution for every prior π is unique, the class of
nonrandomized Bayes procedures form a complete class.
Wald’s regularity conditions are quite strong; he essentially requires equicontinuity in each variable for both the loss and risk functions. For example, the standard
normal-location problem under squared error does not satisfy these criteria.
A similar result is established in the nonsequential setting in [29]:
T HEOREM 3.3 ([29, Thm. 3.1]). Suppose that P admits densities ( fθ )θ ∈Θ , that
Θ is a compact subset of a Euclidean space, that the map (x, θ ) 7→ fθ (x) is jointly
continuous, that the loss ℓ(θ , a) is a continuous function of θ for every action a,
that the space A is compact in loss, and that there is a unique Bayes solution for
every prior π on Θ. Then every Bayes procedure is admissible and the collection of
Bayes procedures form an essentially complete class.
In many classical statistical decision problems, one does not lose anything by assuming that all risk functions are continuous. The following theorem, taken from
[17], formalizes this intuition: We will say that a model P has a continuous likelihood function ( fθ )θ ∈Θ when P admits densities ( fθ )θ ∈Θ such that θ 7→ fθ (x) is
continuous for every x ∈ X .

NONSTANDARD BAYES

11

T HEOREM 3.4 ([17, §5 Thm. 7.11]). Suppose P has a continuous likelihood
function ( fθ )θ ∈Θ and a monotone likelihood ratio. If the loss function ℓ(θ , δ ) satisfies
1. ℓ(θ , a) is continuous in θ for each action a;
2. ℓ(θ , a) is decreasing in a for a < θ and increasing in a for a > θ ; and
3. there exist functions f and g, which are bounded on all bounded subsets of
Θ × Θ, such that for all a
(3.2)

ℓ(θ , a) ≤ f (θ , θ ′ )ℓ(θ ′ , a) + g(θ , θ ′ ),

then the estimators with finite-valued, continuous risk functions form a complete
class.
If we assume the loss function is bounded, then all decision procedures have finite
risk. The following theorem gives a characterization of continuous risk assuming
boundedness of the loss.
T HEOREM 3.5 ([10, §3.7 Thm. 1]). Suppose P admits densities ( fθ )θ ∈Θ with
respect to a dominating measure ν . Assume
1. ℓ is bounded;
2. ℓ(θ , a) is continuous in θ , uniformly
in a;
R
3. for every bounded measurable φ , φ (x) fθ (x)ν (dx) is continuous in θ .

Then the risk r(θ , δ ) is continuous in θ for every δ .

If we assume continuity of the risk function with respect to the parameter and
restrict ourselves to Euclidean parameter spaces, we have the following theorem
from [2, Sec. 8.8, Thm. 12].
T HEOREM 3.6. Assume that A and Θ are compact subsets of Euclidean spaces
and that the model P admits densities ( fθ )θ ∈Θ with respect to either Lebesgue or
counting measure such that the map (x, θ ) 7→ fθ (x) is jointly continuous. Assume
further that the loss ℓ(θ , a) is a continuous function of a ∈ A for each θ , and that
all decision procedures have continuous risk functions. Then the collection of Bayes
procedures form a complete class.
In the noncompact setting, Bayes procedures generally do not form a complete
class. With a view to generalizing the notion of a Bayes procedure and recovering
a complete class, Wald [31] introduced the notion of “Bayes in the wide sense”,
which we now call extended Bayes (see Definition 2.7). The formal statement of
the following theorem is adapted from [10]:
T HEOREM 3.7. Suppose that there exists a topology on D such that D is compact and r(θ , δ ) is lower semicontinuous in δ ∈ D for all θ ∈ Θ. Then the set of
extended Bayes procedures form an essentially complete class.

12

DUANMU AND ROY

Wald also studied taking the “closure” (in a suitable sense) of the collection of
all Bayes procedures, and showed that every admissible procedure was contained in
this new class. The first result of this form appears in [31] and is extended later in
[16]. Brown [6, App. 4A] extended these results and gave a modern treatment. The
following statement of Brown’s version is adapted from [17, §5 Thm. 7.15].
T HEOREM 3.8. Assume P admits strictly positive densities ( fθ )θ ∈Θ with respect to a σ -finite measure ν . Assume the action space A is a closed convex subset
of Euclidean space. Assume the loss ℓ(θ , a) is lower semicontinuous and strictly
convex in a for every θ , and satisfies
(3.3)

lim ℓ(θ , a) = ∞ for all θ ∈ Θ.
|a|→∞

Then every admissible decision procedure δ is an a.e. limit of Bayes procedures,
i.e., there exists a sequence πn of priors with support on a finite set, such that
(3.4)

δ πn (x) → δ (x) as n → ∞ for ν -almost all x,

where δ πn is a Bayes procedure with respect to πn .
In the normal-location model under squared error loss, the sample mean, while
not a Bayes estimator in the strict sense, can be seen as a limit of Bayes estimators,
e.g., with respect to normal priors of variance K as K → ∞ or uniform priors on
[−K, K] as K → ∞. (We revisit this problem in Example 8.2.) In his seminal paper,
Sacks [23] observes that the sample mean is also the Bayes solution if the notion of
prior distribution is relaxed to include Lebesgue measure on the real line. Sacks [23]
raised the natural question: if δ is a limit of Bayes estimators, is there a measure
m on the real line such that δ is “Bayes” with respect to this measure? A solution
in this latter form was termed a generalized Bayes solution by Sacks [23]. The
following definition is adapted from [28]:
D EFINITION 3.9. A decision procedure δ0 is a normal-form generalized Bayes
procedure
with respect to a σ -finite measure π on Θ when δm minimizes r(π , δ ) =
R
r(θ , δ )π (dθ ), subject to the restriction that r(π , δm ) < ∞. If P admits densities
to a σ -finite measure ν and δ0 minimizes the unnormalized
( fθ )θ ∈Θ with respect
R
posterior risk ℓ(θ , δ0 (x)) fθ (x) π (dθ ) for ν -a.e. x, then δ0 is a (extensive-form)
generalized Bayes procedure with respect to π .
When a model admits densities, Stone [28] showed that every normal-form generalized Bayes procedure is also extensive-form.
(Sacks defined generalized Bayes
R
in extensive form, but demanded also that fθ (·) π (dθ ) be finite ν -a.e. The notion of normal- and extensive-form definitions of Bayes optimality were introduced
by Raiffa and Schlaifer [21].) For exponential families, under suitable conditions,
one can show that every admissible estimator is generalized Bayes. The first such

NONSTANDARD BAYES

13

result was developed by Sacks [23] in his original paper: he proved that, for statistical decision
problems where the model admits a density of the form exθ /Zθ with
R xθ
Zθ = e ν (dθ ), every admissible estimator is generalized Bayes. Stone [28] extended this result to estimation of the mean in one-dimensional exponential families
under squared error loss. These results were further generalized in similar ways by
Brown [5, Sec. 3.1] and Berger and Srinivasan [3]. The following theorem is given
in [3]. We adapt the statement of this theorem from [17].
T HEOREM 3.10 ([17, §5 Thm. 7.17]). Assume the model is a finite-dimensional
exponential family, and that the loss ℓ(θ , a) is jointly continuous, strictly convex in
a for every θ , and satisfies
(3.5)

lim ℓ(θ , a) = ∞ for all θ ∈ Θ.
|a|→∞

Then every admissible estimator is generalized Bayes.
Other generalized notions of Bayes procedures have been proposed. Heath and
Sudderth [11] study statistical decision problems in the setting of finitely additive
probability spaces. The following theorem is their main result:
T HEOREM 3.11 ([11, Thm. 2]). Fix a class D of decision procedures. Every
finitely additive Bayes decision procedure is extended admissible. If the loss function is bounded and the class D is convex, then every extended admissible decision
procedure in D is finitely additive Bayes in D.
The simplicity of this statement is remarkable. However, the assumption of boundedness is very strong, and rule out many standard estimation problems on unbounded spaces. We will succeed in removing the boundedness assumption by moving to a sufficiently saturated nonstandard model.
4. Nonstandard admissibility. As we have seen in the previous section, strong
regularity appears to be necessary to align Bayes optimality and admissibility. In
noncompact parameter spaces, the statistician must apparently abandon the strict
use of probability measures in order to represent certain extreme states of uncertainty that correspond with admissible procedures. Even then, strong regularity conditions are required (such as domination of the model and strict positiveness of densities, ruling out estimation in infinite-dimensional contexts). In the remainder of
the paper, we describe a new approach using nonstandard analysis, in which the
statistician retains the use of probability measures, but has access to a much richer
collection of real numbers to express their beliefs.
Let (Θ, A, ℓ, X , P) be a standard statistical decision problem.
We will assume the reader is familiar with basic concepts and key results in nonstandard analysis. (See Appendices A and B for a review tailored to this paper.) For

14

DUANMU AND ROY

a set S, let P(S) denote its power set. We assume that we are working within a nonstandard model containing V ⊇ R ∪ Θ ∪ A ∪ X , P(V ), P(V ∪ P(V )), . . ., and we
assume the model is as saturated as necessary. We use ∗ to denote the nonstandard
extension map taking elements, sets, functions, relations, etc., to their nonstandard
counterparts. In particular, ∗ R and ∗ N denote the nonstandard extensions of the reals and natural numbers, respectively. Given a topological space (Y, T ) and a subset
X ⊆ ∗Y , let NS(X ) ⊆ X denote the subset of near-standard elements (defined by
the monadic structure induced by T ) and let st : NS(Y ) → Y denote the standard
part map taking near-standard elements to their standard parts. In both cases, the
notation elides the underlying space Y and the topology T , because the space and
topology will always be clear from context. As an abbreviation, we will write ◦ x for
st(x) for atomic elements x. For functions f , we will write ◦ f for the composition
x 7→ st( f (x)). Finally, given an internal (hyperfinitely additive) probability space
(Ω, F , P), we will write (Ω, F , P) to denote the corresponding Loeb space, i.e.,
the completion of the unique extension of P to σ (F ).
4.1. Nonstandard extension of a statistical decision problem. We will assume
that Θ is a Hausdorff space and adopt its Borel σ -algebra B[Θ].2
One should view the model P as a function from Θ to the space M1 (X ) of probability measures on X . Write ∗ Py for (∗ P)y. For every y ∈ ∗ Θ, the transfer principle
implies that ∗ Py is an internal probability measure on ∗ X (defined on the extension
of its σ -algebra). By Lemma B.1, we know that ∗ (Pθ ) = ∗ Pθ for θ ∈ Θ, as one
would expect from the notation.
Recall that standard decision procedures δ ∈ D have finite risk functions. Therefore, the risk map (θ , δ ) 7→ r(θ , δ ) is a function from Θ × D to R. By the extension
and transfer principles, the nonstandard extension ∗r is an internal function from
∗ Θ × ∗D to ∗ R. and ∗δ ∈ ∗D if δ ∈ D. The transfer principle also implies that every
∆ ∈ ∗D is an internal function from ∗ X to ∗M1 (A). The ∗ risk function of ∆ ∈ ∗D is
the function ∗r(·, ∆) from ∗ Θ to ∗ R. By the transfer of the equation defining risk, the
following statement holds:
Z
i
∗
∗ hZ
∗
(∀θ ∈ ∗ Θ) (∀∆ ∈ ∗D) (∗r(θ , ∆) =
ℓ(θ , a)∆(x, da) ∗ Pθ (dx).
(4.1)
∗X

∗A

As is customary, we will simply write for ∗ , provided the context is clear. (We
will also drop ∗ from the extensions of common functions and relations like addition, multiplication, less-than-or-equal-to, etc.)
R

2

R

In one sense, this is a mild assumption, which we use to ensure that the standard part map st :
NS(∗ Θ) → Θ is well-defined. In another sense, Θ can always be made Hausdorff by, e.g., adopting
the discrete topology. The topology determines the Borel sets and thus determines the set of available
probability measures on Θ (and on ∗ Θ, by extension). Topological considerations arise again in
Section 6, Remark 7.8, and Remark 8.4.

NONSTANDARD BAYES

15

4.2. Nonstandard admissibility. Let δ0 , δ ∈ D, let ε ∈ R≥0 , and assume δ0 is
ε -dominated by δ . Then there exists θ0 ∈ Θ such that
(4.2)

(∀θ ∈ Θ)(r(θ , δ ) ≤ r(θ , δ0 ) − ε ) ∧ (r(θ0, δ ) 6= r(θ0 , δ0 )).

By the transfer principle,
(4.3)

(∀θ ∈ ∗ Θ)(∗r(θ , ∗δ ) ≤ ∗r(θ , ∗δ 0 ) − ε ) ∧ (∗r(θ0 , ∗δ ) 6= ∗r(θ0 , ∗δ 0 )).

Because ∗r(θ0 , ∗δ ) = r(θ0 , δ ) and similarly for ∗r(θ0 , ∗δ 0 ), Lemma A.7.1 implies
that ∗r(θ0 , ∗δ ) 6≈ ∗r(θ0 , ∗δ 0 ). These results motivate the following nonstandard version of domination.
D EFINITION 4.1. Let ∆, ∆′ ∈ ∗D be internal decision procedures, let ε ∈ R≥0 ,
and R, S ⊆ ∗ Θ. Then ∆ is ε -∗ dominated in R/S by ∆′ when
1. ∀θ ∈ S ∗r(θ , ∆′ ) ≤ ∗r(θ , ∆) − ε , and
2. ∃θ ∈ R ∗r(θ , ∆′ ) 6≈ ∗r(θ , ∆).
Write ∗ dominated in R/S for 0-∗ dominated in R/S, and write ε -∗ dominated on S
for ε -∗ dominated in S/S.
The following results are immediate upon inspection of the definition above, and
the fact that (1) implies (2) for R ⊆ S when ε > 0.
L EMMA 4.2. Let ε ≤ ε ′ , R ⊆ R′ , and S ⊆ S′ . Then ε ′ -∗ dominated in R/S′ implies ε -∗ dominated in R′ /S. If ε > 0, then ε -∗ dominated in S/S′ if and only if ε ∗ dominated on S′ , and ε ′ -∗ dominated on S′ implies ε -∗ dominated on S.
The following result connects standard and nonstandard domination.
T HEOREM 4.3.
alent:

Let ε ∈ R≥0 and δ0 , δ ∈ D. The following statements are equiv-

1. δ0 is ε -dominated by δ .
2. ∗δ 0 is ε -∗ dominated in Θ/∗ Θ by ∗δ .
3. ∗δ 0 is ε -∗ dominated on Θ by ∗δ .
If ε > 0, then the following statement is also equivalent:
4. ∗δ 0 is ε -∗ dominated on ∗ Θ by ∗δ .
P ROOF. (1 =⇒ 2) Follows from logic above Definition 4.1. (2 =⇒ 3) Follows
from Lemma 4.2. (3 =⇒ 1) By hypothesis,
(4.4) (∀θ ∈ Θ)(∗r(θ , ∗δ ) ≤ ∗r(θ , ∗δ 0 ) − ε ) ∧ (∃θ0 ∈ Θ)(∗r(θ0 , ∗δ ) 6≈ ∗r(θ0 , ∗δ 0 )).

16

DUANMU AND ROY

Because ∗r(θ0 , ∗δ ) = r(θ0 , δ ), and likewise for δ0 , it follows that
(∀θ ∈ Θ)(r(θ , δ ) ≤ r(θ , δ0 ) − ε ).

(4.5)

Similarly, ◦ (∗r(θ0 , ∗δ )) = r(θ0 , δ ), and likewise for δ0 , hence Lemma A.7.1 implies
(∃θ0 ∈ Θ)(r(θ0, δ ) 6= r(θ0 , δ0 )).

(4.6)

(2 =⇒ 4 =⇒ 3) Follow from Lemma 4.2.
D EFINITION 4.4.

Let ε ∈ R≥0 , R, S ⊆ ∗ Θ, and C ⊆ ∗D, and ∆ ∈ ∗D.

1. ∆ is ε -∗ admissible in R/S among C unless ∆ is ε -∗ dominated in R/S by some
∆′ ∈ C .
2. ∆ is ∗ admissible in R/S among C if ∆ is 0-∗ admissible in R/S among C .
3. ∆ is ε -∗ admissible on S among C if ∆ is ε -∗ admissible in S/S among C .
4. ∆ is ∗ extended admissible on S among C if ∆ is ε -∗ admissible on S among C
for every ε ∈ R>0 .
The following result is immediate upon inspection of the definitions above.
L EMMA 4.5. Let ε ≤ ε ′ , R ⊆ R′ , S ⊆ S′ , and A ⊆ C . Then ε -∗ admissible in
among C implies ε ′ -∗ admissible in R/S′ among A . For ε > 0, ε -∗ admissible
on S among C implies ε ′ -∗ admissible on S′ among A .

R′ /S

The analogous results for ∗ admissible in R/S among C and ∗ extended admissible
on S among C then follow immediately. The following result connects standard and
nonstandard admissibility.
T HEOREM 4.6. Let ε ∈ R≥0 , δ0 ∈ D, and C ⊆ D. Define C ∗ = {∗δ : δ ∈ C }.
Then the following statements are equivalent:
1. δ0 is ε -admissible among C .
2. ∗δ 0 is ε -∗ admissible in Θ/∗ Θ among C ∗ .
3. ∗δ 0 is ε -∗ admissible on Θ among C ∗ .
If ε > 0, then the following statement is also equivalent:
4. ∗δ 0 is ε -∗ admissible on ∗ Θ among C ∗ .
5. ∗δ 0 is ε -∗ admissible on ∗ Θ among ∗ C .
P ROOF. Statement (1) is equivalent to
(4.7)

¬(∃δ ∈ C ) δ0 is ε -dominated by δ .

By Theorem 4.3 and the definition of C ∗ , this is equivalent to both
(4.8)

¬(∃∗δ ∈ C ∗ ) ∗δ 0 is ε -∗ dominated in Θ/∗ Θ by ∗δ

NONSTANDARD BAYES

17

and
¬(∃∗δ ∈ C ∗ ) ∗δ 0 ε -∗ dominated on Θ by ∗δ ,

(4.9)

hence (1 ⇐⇒ 2 ⇐⇒ 3).
Now let ε > 0. Then the above statements are also equivalent to
¬(∃∗δ ∈ C ∗ ) ∗δ 0 is ε -∗ dominated on ∗ Θ by ∗δ ,

(4.10)

hence (1 ⇐⇒ 4). From Lemma 4.5, we see that (5) implies (4). To see that (1)
implies (5), note that, because ε is standard and ε > 0, (1) is equivalent to
¬(∃δ ∈ C )(∀θ ∈ Θ)(r(θ , δ ) ≤ r(θ , δ0 ) − ε ).

(4.11)

By transfer, this statement holds if and only if the following statement holds:
¬(∃∆ ∈ ∗ C )(∀θ ∈ ∗ Θ)(∗r(θ , ∆) ≤ ∗r(θ , ∗δ 0 ) − ε ).

(4.12)

Again, ε > 0 implies ∗r(θ , ∆) 6≈ ∗r(θ , ∗δ 0 ) for all θ ∈ ∗ Θ, hence (5) holds.
The following corollary for extended admissibility follows immediately.
T HEOREM 4.7. Let δ0 ∈ D and C ⊆ D. Define C ∗ = {∗δ : δ ∈ C }. Then the
following statements are equivalent:
1.
2.
3.
4.

δ0 is extended admissible among C .
∗δ is ∗ extended admissible on Θ among C ∗ .
0
∗δ is ∗ extended admissible on ∗ Θ among C ∗ .
0
∗δ is ∗ extended admissible on ∗ Θ among ∗ C .
0

As in the standard universe, the notion of ∗ admissibility lead to notions of complete classes.
D EFINITION 4.8.

Let A , C ⊆ ∗D.

1. A is a ∗ complete subclass of C if for all ∆ ∈ C \ A , there exists ∆′ ∈ A such
that ∆ is ∗ dominated on Θ by ∆′ .
2. A is an ◦ essentially complete subclass of C if for all ∆ ∈ C \ A , there exists
∆′ ∈ A such that ∗r(θ , ∆′ ) / ∗r(θ , ∆) for all θ ∈ Θ.
Near-standard essential completeness allows us to enlarge the set of decision procedures amongst which a decision procedure is extended admissible.
L EMMA 4.9. Suppose A is an ◦ essentially complete subclass of C ⊆ D. Then
admissible on Θ among A implies ∗ extended admissible on Θ among C .

∗ extended

18

DUANMU AND ROY

P ROOF. Let ∆0 ∈ A and suppose ∆0 is not ∗ extended admissible on Θ among
C . Then there exists ∆ ∈ C and ε ∈ R>0 such that ∗r(θ , ∆) ≤ ∗r(θ , ∆0 ) − ε for all
θ ∈ Θ. But then by the ∗ essential completeness of A , there exists some ∆′ ∈ A ,
such that ∗r(θ , ∆′ ) / ∗r(θ , ∆) for all θ ∈ Θ, hence ∗r(θ , ∆′ ) / ∗r(θ , ∆0 ) − ε for all
θ ∈ Θ. But then ∆0 is not ε /2-∗ admissible on Θ among A hence not ∗ extended
admissible on Θ among A .
5. Nonstandard Bayes. We now define the nonstandard counterparts to Bayes
risk and optimality for the class ∗D of internal decision procedures:
D EFINITION 5.1. Let ∆ ∈ ∗D, ε ∈ ∗ R≥0 , and C ⊆ ∗D, and let Π0 be a nonstandard prior, i.e., an internal probability measure on (∗ Θ, ∗ B[Θ]).
1. The internal Bayes risk under Π0 of ∆ is ∗r(Π0 , ∆) = ∗r(θ , ∆)Π0(dθ ).
2. ∆ is ε -∗ Bayes under Π0 among C if ∗r(Π0, ∆) is hyperfinite and, for all ∆′ ∈
C , we have ∗r(Π0, ∆) ≤ ∗r(Π0, ∆′ ) + ε .
3. ∆ is nonstandard Bayes under Π0 among C if there exists an infinitesimal
ε ∈ ∗ R≥0 such that ∆ is ε -∗ Bayes under Π0 among C .
4. ∆ is ε -∗ Bayes among C (resp., nonstandard Bayes among C ) if there exists a
nonstandard prior Π such that ∆ is ε -∗ Bayes under Π among C (resp., nonstandard Bayes under Π among C ).
R

We will sometimes write nonstandard Bayes among C with respect to Π0 to
mean nonstandard Bayes under Π0 among C , and similarly for ε -∗ Bayes among
C . Note that the internal Bayes risk is precisely the extension of the standard
Bayes risk. Similarly, if we consider the relation {(δ , ε , C ) ∈ D × R≥0 × P(D) :
δ is ε -Bayes among C }, then its extension corresponds to {(∆, ε , C ) ∈ ∗D × ∗ R≥0 ×
∗ P(D) : ∆ is ε -∗ Bayes among C }. Note, however, that our definition of “ε -∗ Bayes
among C ” allows the set C ⊆ ∗D to be external, and so it is not simply the transfer
of the definition.
Transfer remains a powerful tool for relating the optimality of standard procedures with that of their extensions. For example, by transfer, δ is ε -Bayes under π
among C if and only if ∗δ is ε -∗ Bayes under ∗ π among ∗ C . (Recall that ∗ ε = ε for
a real ε , by extension.) Transfer also yields the following result:
T HEOREM 5.2. Let C ⊂ D. If δ0 is extended Bayes among C , then ∗δ 0 is nonstandard Bayes among ∗ C .
P ROOF. By hypothesis, the following sentence holds:
(5.1)

(∀ε ∈ R>0 )(∃π ∈ M1 (Θ))(∀δ ∈ C )(r(π , δ0) ≤ r(π , δ ) + ε ).

By the transfer principle,
(5.2)

(∀ε ∈ ∗ R>0 )(∃π ∈ ∗M1 (Θ))(∀δ ∈ ∗ C )(∗r(π , ∗δ 0 ) ≤ ∗r(π , ∗δ ) + ε ).

NONSTANDARD BAYES

19

Taking ε to be infinitesimal implies that the internal Bayes risk of ∗δ 0 is within
an infinitesimal of the minimum Bayes risk among ∗ C with respect to an internal
probability measure on ∗ Θ, hence ∗δ 0 is nonstandard Bayes among ∗ C .
In general, we would not expect an extension ∗δ to be 0-∗ Bayes under Π among C
for a generic nonstandard prior Π and class C ⊆ ∗D. The definition of nonstandard
Bayes provides infinitesimal slack, which suffices to yield a precise characterization
of extended admissible procedures. The follow result shows that, as in the standard
universe, nonstandard Bayes optimality implies nonstandard extended admissibility.
T HEOREM 5.3. Let ∆0 ∈ ∗D, let C ⊆ ∗D, and suppose that ∆0 is nonstandard
Bayes among C . Then ∆0 is ∗ extended admissible on ∗ Θ among C .
P ROOF. Suppose ∆0 is not ∗ extended admissible on ∗ Θ among C . Then for some
standard ε ∈ R>0 , ∆0 is ε -∗ dominated on ∗ Θ by some ∆ ∈ C , i.e.,
(∀θ ∈ ∗ Θ)(∗r(θ , ∆) ≤ ∗r(θ , ∆0 ) − ε ).

(5.3)

Hence, for every nonstandard prior Π, if ∗r(Π, ∆) is not hyperfinite, then neither is
∗r(Π, ∆ ), and if ∗r(Π, ∆) is hyperfinite, then
0
(5.4)
(5.5)

r(Π, ∆0) =

Z

∗

≥

Z

∗

∗

r(θ , ∆0 )Π(dθ )

r(θ , ∆)Π(dθ ) + ε = ∗r(Π, ∆) + ε .

As ε ∈ R>0 , we conclude that ∆0 cannot be nonstandard Bayes under Π among C .
As Π was arbitrary, ∆0 is not nonstandard Bayes among C .
Theorems 4.7 and 5.3 immediately yield the following corollary.
C OROLLARY 5.4. Let δ ∈ D and C ⊆ D. If ∗δ is nonstandard Bayes among
then δ is extended admissible among C .

C ∗,

The above result raises several questions: Are extended admissible decision procedures also nonstandard Bayes? What is the relationship with admissibility and its
nonstandard counterparts?
In this section, we prove that a decision procedure δ is extended admissible if
and only if ∗δ is nonstandard Bayes. In later sections, we give several application of
this equivalence, and then consider the relationship with admissibility, which is far
from settled. It is easy, however, to show that only nonstandard Bayes procedures
can ∗ dominate other nonstandard Bayes procedures: To see this, suppose that ∆ is
nonstandard Bayes among C ⊆ ∗D with respect to some nonstandard prior Π and ∆
is not ∗ admissible on ∗ Θ among C .

20

DUANMU AND ROY

Then ∆ is ∗ dominated on ∗ Θ by some ∆′ ∈ C . Thus weRhave ∗r(θ , ∆′ ) ≤ ∗r(θ , ∆)
for all θ ∈ ∗ Θ. By Definition 5.1, we
have ∗r(Π, ∆) = ∗r(θ , ∆)Π(dθ ) hyperfiR
nite. But then, ∗r(Π, ∆) / ∗r(Π, ∆′ ) = ∗r(θ , ∆′ )Π(dθ ) ≤ ∗r(Π, ∆), hence ∗r(Π, ∆) ≈
∗r(Π, ∆′ ), hence ∆′ is nonstandard Bayes under Π among C . This proves a nonstandard version of a well-known standard result stating that every unique Bayes
procedure is admissible [10, §2.3 Thm. 1]:
T HEOREM 5.5. Suppose ∆ is nonstandard Bayes among C ⊆ ∗D with respect to
a nonstandard prior Π. If ∆ is ∗ dominated on ∗ Θ by ∆′ ∈ C , then ∆′ is nonstandard
Bayes under Π among C . Therefore, if ∗r(θ , ∆′ ) ≈ ∗r(θ , ∆) for all θ ∈ ∗ Θ and for all
∆′ ∈ C such that ∆′ is nonstandard Bayes under Π among C , then ∆ is ∗ admissible
on ∗ Θ among C .
P ROOF. The first statement follows from the logic in the preceding paragraph.
Now suppose that ∆ is ∗ dominated on ∗ Θ by some ∆′ ∈ C . Then ∆′ is nonstandard
Bayes under Π among C . But then, by hypothesis, its risk function is equivalent, up
to an infinitesimal, to that of ∆, a contradiction.
5.1. Hyperdiscretized risk set. In a statistical decision problem with a finite parameter space, one can use a separating hyperplane argument to show that every
admissible decision procedure is Bayes (see, e.g., [10, §2.10 Thm. 1]). In order to
prove our main theorem, we will proceed along similar lines, but with the aid of
extension, transfer, and saturation.
When relating extended admissibility and Bayes optimality for a subclass C ⊆ D,
the set of all risk functions rδ , for δ ∈ C , is a key structure. On a finite parameter
space, the risk set for D is a convex subset of a finite-dimensional vector space
over R. When the parameter space is not finite, one must grapple with infinite dimensional function spaces. However, in a sufficiently saturated nonstandard model,
there exists an internal set TΘ ⊂ ∗ Θ that is hyperfinite and contains Θ. While the
risk at all points in TΘ does not suffice to characterize an arbitrary element of ∗D,
it suffices to study the optimality of extensions of standard decision procedure relative to other extensions. Because TΘ is hyperfinite, the corresponding risk set is a
convex subset of a hyperfinite-dimensional vector space over ∗ R.
Let JΘ ∈ ∗ N be the internal cardinality of TΘ and let TΘ = {t1, . . . ,tJΘ }. Recall
that I(∗ RJΘ ) denotes the set of (internal) functions from TΘ to ∗ R. For an element
x ∈ I(∗ RJΘ ), we will write xk for x(k).
D EFINITION 5.6.
(5.6)

The hyperdiscretized risk set induced by D ⊆ ∗D is the set

S D = {x ∈ I(∗ RJΘ ) : (∃∆ ∈ D) (∀k ≤ JΘ ) xk = ∗r(tk , ∆)} ⊂ I(∗ RJΘ ).

L EMMA 5.7.
convex set.

Let D ⊆ ∗D be an internal convex set. Then S D is an internal

21

NONSTANDARD BAYES

P ROOF. S D is internal by the internal definition principle and the fact that D is
internal. In order to demonstrate convexity, pick p ∈ ∗ [0, 1], and let x, y ∈ S D . Then
there exist ∆1 , ∆2 ∈ D such that xk = ∗r(tk , ∆1 ) and yk = ∗r(tk , ∆2 ) for all k ≤ JΘ .
Because D is convex, p∆1 + (1 − p)∆2 ∈ D. But pxk + (1 − p)yk = ∗r(tk , p∆1 + (1 −
p)∆2 ) for all k ≤ JΘ , and so S D is convex.
D EFINITION 5.8.
C ⊆ ∗D, let

For every A ⊂ D, define A ∗ = {∗δ : δ ∈ A }, and, for every
(C )FC =

(5.7)

[

∗

conv(D)

D∈C [<∞]

be the set of all finite ∗ convex combinations of ∗δ ∈ C .
Note that A ∗ ⊂ ∗ A is an external set unless A is finite. Let δ1 , δ2 ∈ D0 and
∗
let p ∈ ∗ [0, 1]. Then p∗δ 1 + (1 − p)∗δ 2 ∈ D0,FC
if p ∈ [0, 1]. However, p∗δ 1 + (1 −
∗
∗
∗
∗
p) δ 2 ∈ (D0 )FC for all p ∈ [0, 1]. It is easy to see that (D0,FC
)FC = (D0∗ )FC . Thus,
∗
∗
we have D0∗ ⊂ D0,FC
⊂ (D0,FC
)FC = (D0∗ )FC ⊂ ∗D0,FC .
L EMMA 5.9.

For any C ⊆ ∗D, (C )FC is a convex set containing C .

P ROOF. Pick an C ⊆ ∗D. Clearly (C )FC ⊃ C . It remains to show that (C )FC is
a convex set. Pick two elements ∆1 , ∆2 ∈ (C )FC . Then there exist D1 , D2 ∈ C [<∞]
such that ∆1 ∈ ∗conv(D1 ) and ∆2 ∈ ∗conv(D2 ). Let p ∈ ∗ [0, 1]. It is easy to see that
p∆1 + (1 − p)∆2 ∈ ∗conv(D1 ∪ D2 ).
L EMMA 5.10.

∗
D0,FC
is an ◦ essentially complete subclass of (D0∗ )FC .

P ROOF. Let ∆ ∈ (D0∗ )FC . Then ∆ = ∑ni=1 pi ∗δ i for some n ∈ N, δ1 , . . . , δn ∈ D0 ,
and p1 , . . ., pn ∈ ∗ R≥0 , ∑ni=1 pi = 1. Define ∆0 = ∑ni=1 ◦pi ∗δ i and let θ ∈ Θ. For all
i ≤ n, pi ∗r(θ , ∗δ i ) ≈ ◦pi ∗r(θ , ∗δ i ) because ∗r(θ , ∗δ i ) is finite, so ∗r(θ , ∆) ≈ ∗r(θ , ∆0 ).
∗
By Definition 4.8, D0,FC
is an ◦ essentially complete subclass of (D0∗ )FC .
Having defined the (hyperdiscretized) risk set, we now describe a set whose intersection with the risk set captures the notion of 1n -∗ domination, for some standard
n ∈ N. In that vein, for ∆ ∈ ∗D, define the n1 -quantant
(5.8)

1
Q(∆)n = {x ∈ I(∗ RJΘ ) : (∀k ≤ JΘ )(xk ≤ ∗r(tk , ∆) − )},
n

n ∈ ∗ N.

L EMMA 5.11. Fix ∆ ∈ ∗D. The set Q(∆)n is internal and convex and Q(∆)m ⊂
Q(∆)n for every m < n.

22

DUANMU AND ROY

P ROOF. By the internal definition principle, Q(∆)n is internal. Let x, y be two
points in Q(∆)n, let p ∈ ∗ [0, 1], and pick a coordinate k. Then
(5.9)
1
1
1
pxk + (1 − p)yk ≤ p(∗r(tk , ∆) − ) + (1 − p)(∗r(tk , ∆) − ) = (∗r(tk , ∆) − ).
n
n
n
Thus the set is convex. The second statement is obvious.
The following is then immediate from definitions.
L EMMA 5.12. Let C ⊆ ∗D and n ∈ N. Then ∆ is 1n -∗ admissible on TΘ among C
if and only if Q(∆)n ∩ S C = 0.
/
5.2. Nonstandard complete class theorems.
L EMMA 5.13. Let ∆ ∈ ∗D and nonempty D ⊆ ∗D, and suppose there exists a
S
nonzero vector Π ∈ I(∗ RJΘ ) such that hΠ, xi ≤ hΠ, si for all x ∈ n∈N Q(∆)n and s ∈
S D . Then the normalized vector Π/kΠk1 induces an internal probability measure
π on ∗ Θ concentrating on TΘ , and ∆ is nonstandard Bayes under π among D.
P ROOF. We first establish that Π(k) ≥ 0 for all k. Suppose otherwise, i.e., Π(k0 ) <
S
0 for some k0 . Then we can pick a point x0 in n∈N Q(∆)n whose k0 -th coordinate is
arbitrarily large and negative, causing hΠ, x0 i to be arbitrary large, a contradiction
because hΠ, si is hyperfinite for all s ∈ S D . Hence, all coordinates of Π must be
nonnegative.
Define π ∈ I(∗ RJΘ ) by π = Π/kΠk1. Because Π 6= 0 and Π ≥ 0, we have π ≥ 0
and kπ k1 = 1. Therefore, π specifies an internal probability measure on (∗ Θ, ∗ B[Θ]),
concentrating on TΘ , and assigning probability π (k) to tk for every k ≤ JΘ . Because
S
kΠk1 > 0, it still holds that hπ , xi ≤ hπ , si for all x ∈ n∈N Q(∆)n and s ∈ S D .
Let s ∈ S D . Then ◦ (∑k∈JΘ πk (∗r(tk , ∆) − n1 )) ≤ ◦ (∑k∈JΘ πk sk ) for every n ∈ N. The
l.h.s. is simply ◦ (− 1n + ∑k∈JΘ πk ∗r(tk , ∆)), and the limit of this expression as n → ∞
is ◦ (∑k∈JΘ πk ∗r(tk , ∆)). Hence, ∑k∈JΘ πk (∗r(tk , ∆) / ∑k∈JΘ πk sk . This shows that ∆ is
nonstandard Bayes under π among D.
The previous result shows that if a nontrivial hyperplane separates the risk set
from every 1n -quantant, for n ∈ N, then the corresponding procedure is nonstandard
Bayes. In order to prove our main theorem, we require a nonstandard version of the
hyperplane separation theorem, which we give here. For a, b ∈ Rk for some finite
k, let ha, bi denote the inner product. We begin by stating the standard hyperplane
separation theorem:
T HEOREM 5.14 (Hyperplane separation theorem). For any k ∈ N, let S1 and S2
be two disjoint convex subsets of Rk , then there exists w ∈ Rk \ {0} such that, for all
p1 ∈ S1 and p2 ∈ S2 , we have hw, p1 i ≥ hw, p2 i.

NONSTANDARD BAYES

23

Using a suitable encoding of this theorem in first-order logic, the transfer principle yields a hyperfinite version:
T HEOREM 5.15. Fix any K ∈ ∗ N. If S1 , S2 are two disjoint internal convex subsets of I(∗RK ), then there exists W ∈ I(∗ RK ) \ {0} such that, for all P1 ∈ S1 and
P2 ∈ S2 , we have hW, P1i ≥ hW, P2i.
See Appendix A.2.1 for a proof.
Recall that our nonstandard model is κ -saturated for some infinite κ .
T HEOREM 5.16. Let C ⊆ D ∗ be a (necessarily finite or external) set with cardinality less than κ , and suppose that C is a ◦ essentially complete subclass of (C )FC .
Let ∆0 ∈ ∗D and suppose ∆0 is ∗ extended admissible on Θ among C . Then, for every
hyperfinite set T ⊆ ∗ Θ containing Θ, ∆0 is nonstandard Bayes among (C )FC with
respect to some nonstandard prior concentrating on T .
P ROOF. Without loss of generality we may take T = TΘ . By Lemma 4.9 and the
fact that C is an ◦ essentially complete subclass of (C )FC , ∆0 is ∗ extended admissible on Θ among (C )FC . By Lemma 4.5, ∆0 is 1n -∗ admissible on TΘ among (C )FC
for every n ∈ N. Hence, by Lemma 5.12, Q(∆0 )n ∩ S (C )FC = 0/ for all n ∈ N.
∗
By the definition of (C )FC , we have Q(∆0 )n ∩S conv(D) = 0/ for every D ∈ C [<∞] .
∗conv(D)
and Q(∆0 )n are both internal convex sets,
By Lemmas 5.7 and 5.11, S
∗ JΘ
hence, by Theorem 5.15, there is a nontrivial hyperplane ΠD
n ∈ I( R ) that separates them.
For every D ∈ C [<∞] and n ∈ N, let φnD (Π) be the formula
(5.10)
∗
(Π ∈ I(∗ RJΘ )) ∧ (Π 6= 0 ∧ (∀x ∈ Q(∆0 )n ) (∀s ∈ S conv(D) ) hΠ, xi ≤ hΠ, si),
and let F = {φnD (Π) : n ∈ N, D ∈ C [<∞] }. By the above argument and the fact
that C [<∞] is closed under taking finite unions and the sets Q(∆0 )n , for n ∈ N, are
nested, F is finitely satisfiable. Note that F has cardinality no more than κ , yet our
nonstandard extension is κ -saturated by hypothesis. Therefore, by the saturation
principle, there exists a nontrivial hyperplane Π satisfying every sentence in F
simultaneously. That is, there exists Π ∈ I(∗ RJΘ ) such that Π 6= 0 and, for all x ∈
S
S
∗conv(D)
= S (C )FC , we have hΠ, xi ≤ hΠ, si.
n∈N Q(∆0 )n and for all s ∈ D∈C [<∞] S
Hence, by Lemma 5.13, the normalized vector Π/kΠk1 is well-defined and induces a probability measure π on ∗ Θ concentrating on TΘ , and ∆0 is nonstandard
Bayes under π among (C )FC .
T HEOREM 5.17.

For δ0 ∈ D, the following are equivalent statements:

1. δ0 is extended admissible among D0,FC .
∗
2. ∗δ 0 is nonstandard Bayes among D0,FC
.

24

DUANMU AND ROY

3. ∗δ 0 is nonstandard Bayes among (D0∗ )FC .
If (LC) also holds, then the following statements are also equivalent:
4. δ0 is extended admissible among D0 .
5. ∗δ 0 is nonstandard Bayes among D0∗ .
Moreover, statements (2), (3), and (5) can be taken to assert that, for all hyperfinite
sets T ⊆ ∗ Θ containing Θ, Bayes optimality holds with respect to some nonstandard
prior concentrating on T .
P ROOF. From (1) and Theorem 4.7, ∗δ 0 is ∗ extended admissible on Θ among
∗
D0,FC
. It follows from Lemma 5.10 and Theorem 5.16 that, for all hyperfinite sets
T ⊆ ∗ Θ containing Θ, ∗δ 0 is nonstandard Bayes among (D0∗ )FC with respect to some
nonstandard prior π concentrating on T . Hence (3) holds and (2) follows trivially.
From (2) and Theorem 5.3, it follows that ∗δ 0 is ∗ extended admissible on ∗ Θ
∗
among D0,FC
. Then (1) follows from Theorem 4.7.
It is the case that (1) implies (4) by Lemma 4.5, and the other direction follows
from (LC), Lemma 2.13, and Lemma 2.4. Similarly, (2) implies (5). Finally, from
(5) and Theorem 5.3, it follows that ∗δ 0 is ∗ extended admissible on ∗ Θ among D0∗ .
Then (4) follows from Theorem 4.7.
It follows immediately that the class of extended admissible procedures is a complete class if and only if the class of procedures whose extensions are nonstandard
Bayes are a complete class.
6. Applications to compact statistical decision problems. In this section, we
use our nonstandard theory to prove that, under the additional hypotheses that Θ
is compact (and thus normal) and all risk functions are continuous, the class of
extended admissible procedures is precisely the class of Bayes procedures. The
strength of our result lies in the absence of any additional assumptions on the loss
or model.3
Assume ∗δ is nonstandard Bayes with respect to some nonstandard prior π on
∗ Θ. In this section, we will construct a standard probability measure π on Θ from
p
π in such a way that the internal risk of ∗δ under π is infinitesimally close to the
risk of δ under π p . This then implies that π is Bayes with respect to π p , and yields
a standard characterization of extended admissible procedures.
Extension allows us to associate an internal probability measure ∗ π to every standard probability measure π . The next theorem describes a reverse process via Loeb
measures.
3

In Section 5, the Hausdorff condition can be sidestepped by adopting the discrete topology.
Unless Θ is finite, however, Θ will not be compact under the discrete topology. Thus, the topological
hypotheses in this section not only determine the space of priors, but also restrict the set of decision
problems to which the theory applies.

NONSTANDARD BAYES

25

L EMMA 6.1 ([8, Thm. 13.4.1]). Let Y be a compact Hausdorff space equipped
with Borel σ -algebra B[Y ], let ν be an internal probability measure defined on
(∗Y , ∗ B[Y ]), and let C = {C ⊂ Y : st−1 (C) ∈ ∗ B[Y ]ν }. Define a probability measure
ν p on the sets C by ν p (C) = ν (st−1 (C)). Then (Y, C , ν p ) is the completion of a
regular Borel probability space.
Note that st−1 (E) is Loeb measurable for all E ∈ B[Y ] by Theorems A.10 and B.8.
D EFINITION 6.2. The probability measure ν p : C → [0, 1] in Lemma 6.1 is
called the pushdown of the internal probability measure ν .
E XAMPLE 6.3. If a nonstandard prior concentrates on finitely many points in
NS(∗ Θ), then its pushdown concentrates on the standard parts of those points, hence
is a standard measure with support on a finite set.
E XAMPLE 6.4. Suppose S = [K −1 , 2K −1 , . . ., 1 − K −1 , 1] for some nonstandard
natural K ∈ ∗ N \ N. Define an internal probability measure π on ∗ [0, 1] by π {s} =
K −1 for all s ∈ S, and let π p be its pushdown. Then π p is Lebesgue measure on [0, 1].
The following lemma establishes a close link between Loeb integration and integration with respect to the pushdown measure.
L EMMA 6.5. Let Y be a compact Hausdorff space equipped with Borel σ algebra B[Y ], let ν be an internal probability measure on (∗Y , ∗ B[Y ]), let ν p be
the pushdown of ν , and let f : Y → R be Ra boundedR measurable function. Define
g : ∗Y → R by g(s) = f (◦ s). Then we have f dν p = g dν .
P ROOF. For every n ∈ N and k ∈ Z, define Fn,k = f −1 ([ nk , k+1
n )) and Gn,k =
k k+1
−1
∗
/ forms
g ( [ n , n )). As f is bounded, the collection Fn = {Fn,k : k ∈ Z} \ {0}
∗
a finite partition of Y , and similarly for Gn = {Gn,k : k ∈ Z} \ {0}
/ and Y . For every
k
∗
ˆ
ˆ
n ∈ N, define fn : Y → R and ĝn : Y → R by putting fn = n on Fn,k and ĝn = nk
on Gn,k for every k ∈ Z. Thus fˆn (resp., ĝn ) is a simple (resp., ∗ simple) function on
the partition Fn (resp., Gn ). By construction fˆn ≤ f < fˆn + n1 and ĝn ≤ g < ĝn + 1n .
Note that Gn,k = st−1 (Fn,k ) for every n ∈ N and k ∈ Z. Moreover, Y is even regular
Hausdorff, hence Theorem B.8 implies that Gn,k is ν -measurable. It follows that
R
R
R
R
f dν p = limn→∞ fˆn dν p and gdν = limn→∞ ĝn dν . Moreover, by Lemma 6.1,
we have ν (Gn,k ) = ν p (Fn,k ) for every n ∈ N and k ∈ Z. Thus, for every n ∈ N and
R
R
R
R
k ∈ Z, we have fˆn dν p = ĝn dν . Hence we have g dν = f dν p , completing the
proof.
In order to control the difference between the internal and standard Bayes risks
under a nonstandard prior π and its pushdown π p, it will suffice to require that risk
functions be continuous. (Recall that we quoted results listing natural conditions
that imply continuous risk in Theorems 3.4 and 3.5.)

26

DUANMU AND ROY

C ONDITION RC (risk continuity).

r(·, δ ) is continuous on Θ, for all δ ∈ D.

In order to understand the nonstandard implications of this regularity condition,
we introduce the following definition from nonstandard analysis.
D EFINITION 6.6. Let X and Y be topological spaces. A function f : ∗ X → ∗Y is
S-continuous at x ∈ ∗ X if f (y) ≈ f (x) for all y ≈ x.
A fundamental result in nonstandard analysis links continuity and S-continuity:
L EMMA 6.7. Let X and Y be Hausdorff spaces, where Y is also locally compact,
and let D ⊆ X . If a function f : X → Y is continuous on D then its extension ∗ f is
NS(∗Y )-valued and S-continuous on NS(∗ D).
See Appendix A.2 for a proof of this classical result. We are now at the place to
establish the correspondence between internal Bayes risk and standard Bayes risk.
The proof relies on the following technical lemma.
L EMMA 6.8 ([1, Cor. 4.6.1]). Suppose (Ω, F , P) is an internal probability space,
and F : Ω → ∗ R is an internal P-integrable function
such
that ◦ F exists everywhere.
R
R
Then ◦ F is integrable with respect to P and FdP ≈ ◦ FdP.
L EMMA 6.9. Suppose Θ is compact Hausdorff and (RC) holds. Let π be an
internal distribution on ∗ Θ and let π p : C → [0, 1] be its pushdown. Let δ0 ∈ D
be a standard decision procedure. If ∗r(·, ∗δ 0 ) is π -integrable then r(·, δ0 ) is a π p integrable function and r(π p , δ0 ) ≈ ∗r(π , ∗δ 0 ), i.e., the Bayes risk under π p of δ0 is
within an infinitesimal of the nonstandard Bayes risk under π of ∗δ 0 .
P ROOF. Because Θ is compact Hausdorff, ◦t exists for all t ∈ ∗ Θ and Lemma 6.1
implies π p is a probability measure on (Θ, C ), where C is the π p-completion of
B[Θ]. By (RC) and Lemma 6.7, for all t ∈ ∗ Θ, we have
r(t, ∗δ 0 ) ≈ ∗r(◦t, ∗δ 0 ) = r(◦t, δ0 ).

∗

(6.1)

Hence ◦ (∗r(t, δ0)) = r(◦t, δ0 ) exists for all t ∈ ∗ Θ. As ∗r(·, ∗δ 0 ) is π -integrable, by
Lemma 6.8, we know that ◦ (∗r(·, ∗δ 0 )) is π -integrable and
(6.2)

Z

∗

r(t, ∗δ 0 )π (dt) ≈

Z

◦ ∗

( r(t, ∗δ 0 ))π (dt) =

Z

∗ ◦ ∗

r( t, δ 0 )π (dt).

By (RC) andRthe fact that Θ is compact,
it follows that r(·, δ0) is bounded. Thus, by
R
∗
◦
∗
Lemma 6.5, r( t, δ 0 )π (dt) = r(θ , δ0 )π p (dθ ), completing the proof.
L EMMA 6.10. Suppose Θ is compact Hausdorff and (RC) holds. Let δ0 ∈ D
and C ⊆ D. If ∗δ 0 is nonstandard Bayes among C ∗ , then δ0 is Bayes among C .

NONSTANDARD BAYES

27

P ROOF. By Theorem 5.17, we may assume that ∗δ 0 is nonstandard Bayes among
C ∗ with respect to a nonstandard prior π that concentrates on some hyperfinite set T .
Let δ ∈ C . Then ∗δ ∈ C ∗ , hence ∗r(π , ∗δ 0 ) / ∗r(π , ∗δ ). Let π p denote the pushdown
of π . As Θ is compact Hausdorff, we know that π p is a probability measure. As
π concentrates on the hyperfinite set T , we know that ∗r(·, ∗δ 0 ) and ∗r(·, ∗δ ) are π integrable. By Lemma 6.9, we have r(π p , δ0 ) ≈ ∗r(π , ∗δ 0 ) and r(π p , δ ) ≈ ∗r(π , ∗δ ).
Then Lemma A.7.2 implies that r(π p , δ0 ) ≤ r(π p , δ ). As our choice of δ was arbitrary, δ0 is Bayes under π p among C .
T HEOREM 6.11. Suppose Θ is compact Hausdorff and (RC) holds. For δ0 ∈ D,
the following statements are equivalent:
1. δ0 is extended admissible among D0,FC .
2. δ0 is extended Bayes among D0,FC .
3. δ0 is Bayes among D0,FC .
If (LC) also holds, then the equivalence extends to these statements with D0 in place
of D0,FC .
P ROOF. Suppose (1) holds. Then by Theorem 5.17, ∗δ 0 is nonstandard Bayes
∗
among D0,FC
. Then (3) follows from Lemma 6.10. The reverse implications follows
from Theorem 2.8.
The statements with D0,FC imply those for D0 ⊆ D0,FC trivially. When (LC)
holds, we have Lemma 2.13. Hence, the reverse implications follows from Lemma 2.4
and Theorem 2.9.
We conclude this section with a strengthening of Theorem 5.17, showing that
infinitesimal ∗ Bayes risk yields zero ∗ Bayes risk, and that a procedure is optimal
among all extensions if and only if it optimal among all internal estimators:
C OROLLARY 6.12. Suppose Θ is compact Hausdorff and (RC) holds. For δ0 ∈
D, the following statements are equivalent:
1. δ0 is extended admissible among D0,FC .
2. ∗δ 0 is nonstandard Bayes among ∗D0,FC .
3. ∗δ 0 is 0-∗ Bayes among ∗D0,FC .
∗
Moreover, the equivalence extends to these statements with D0,FC
in place of ∗D0,FC .
If (LC) also holds, the equivalence extends to these statement with D0 /D0∗ /∗D0 in
∗
place of D0,FC /D0,FC
/∗D0,FC .

P ROOF. Statement (1) implies that δ0 is Bayes among D0,FC by Theorem 6.11.
This implies (3) by transfer, (3) implies (2) by definition, and (2) implies (1) by
Theorem 5.17.
∗
Statements (2) and (3) with ∗D0,FC imply their counterparts with D0,FC
in place
∗
∗
∗
of D0,FC , trivially. Statement (3) with D0,FC implies (2) with D0,FC which implies
(1) by Theorem 5.17.

28

DUANMU AND ROY

The additional equivalences under (LC) follow by the same logic as above and in
the proof of Theorem 5.17.
7. Admissibility of nonstandard Bayes procedures. Heretofore, we have focused on the connection between extended admissibility and nonstandard Bayes
optimality. In this section, we shift our focus to the admissibility of decision procedures whose extensions are nonstandard Bayes. In all but the final result of this
section, we will assume that Θ is a metric space and write d for the metric.
On finite parameter spaces with bounded loss, it is known that Bayes procedures
with respect to priors assigning positive mass to every state are admissible. Similarly, when risk functions are continuous, Bayes procedures with respect to priors
with full support are admissible. We can establish analogues of these result on general parameter spaces by a suitable nonstandard relaxation of a standard prior having
full support.
D EFINITION 7.1.

For x, y ∈ ∗ R, write x ≫ y when γ x > y for all γ ∈ R>0 .

D EFINITION 7.2. Let X be a metric space with metric d, and let ε ∈ ∗ R>0 . An
internal probability measure π on ∗ Θ is ε -regular if, for every θ0 ∈ Θ and noninfinitesimal r > 0, we have π ({t ∈ ∗ Θ : ∗ d(t, θ0) < r}) ≫ ε .
The following result establishes ∗ admissibility from ∗ Bayes optimality under
conditions analogues to full support and continuity of the risk function.
L EMMA 7.3. Suppose Θ is a metric space. Let ε ∈ ∗ R>0 , ∆0 ∈ ∗D, and C ⊆
∗D, and suppose ∗r(·, ∆) is S-continuous on NS(∗ Θ) for all ∆ ∈ C ∪ {∆ }. If ∆ is
0
0
ε -∗ Bayes among C with respect to an ε -regular nonstandard prior π , then ∆0 is
∗ admissible in Θ/∗ Θ among C .
P ROOF. Suppose ∆0 is not ∗ admissible in Θ/∗ Θ among C . Then, for some ∆ ∈ C
and θ0 ∈ Θ, it holds that
(7.1)
(7.2)

(∀θ ∈ ∗ Θ)(∗r(θ , ∆) ≤ ∗r(θ , ∆0 ))
and ∗r(θ0 , ∆) 6≈ ∗r(θ0 , ∆0 ).

From Eq. (7.2), ∗r(θ0 , ∆0 ) − ∗r(θ0 , ∆) > 2γ for some positive γ ∈ R. Let A be the set
of all a ∈ ∗ R>0 such that
(7.3)

(∀t ∈ ∗ Θ) (∗ d(t, θ0) < a =⇒ ∗r(t, ∆0) − ∗r(t, ∆) > γ ).

By the S-continuity of ∗r on NS(∗ Θ), the set A contains all infinitesimals. By Corollary A.9 and the fact that A is an internal set, A must contain some positive a0 ∈ R.
In summary,
(7.4)

(∀t ∈ ∗ Θ) (∗ d(t, θ0) < a0 =⇒ ∗r(t, ∆0) − ∗r(t, ∆) > γ ).

NONSTANDARD BAYES

29

Let M = {t ∈ ∗ Θ : ∗ d(t, θ0) < a0 }. By the internal definition principle, M is an
internal set. By Eq. (7.1) and the definition and internality of M, the difference in
internal Bayes risk between ∆0 and ∆ satisfies
(7.5)

r(π , ∆0 ) − r(π , ∆) =

Z

(∗r(t, ∆0) − ∗r(t, ∆))π (dt)

≥

Z

(∗r(t, ∆0) − ∗r(t, ∆))π (dt) > γ π (M).

∗

(7.6)

∗

∗Θ

M

But γ π (M) > ε because π is ε -regular, hence ∆0 is not ε -∗ Bayes among C with
respect to π .
The following theorem is an immediate consequence of Lemma 7.3 and is a nonstandard analogue of Blyth’s Method [17, §5 Thm. 7.13] (see also [17, §5 Thm. 8.7]).
In Blyth’s method, a sequence of (potentially improper) priors with sufficient support is used to establish the admissibility of a decision procedure. In contrast, a
single nonstandard prior witnesses the nonstandard admissibility of a nonstandard
Bayes procedure.
T HEOREM 7.4. Suppose Θ is a metric space and (RC) holds. Let δ0 ∈ D and
C ⊂ D. If there exists ε ∈ ∗ R>0 such that ∗δ 0 is ε -∗ Bayes among C ∗ = {∗δ : δ ∈ C }
with respect to an ε -regular nonstandard prior π , then ∗δ 0 is ∗ admissible in Θ/∗ Θ
among C ∗ .
P ROOF. By (RC) and Lemma 6.7, for all δ ∈ D, θ0 ∈ Θ, and t ≈ θ0 , we have
∗
∗
∗
∗
∗
0 , δ ). By Lemma 7.3, δ 0 is admissible in Θ/ Θ among C .

∗r(t, ∗δ ) ≈ ∗r(θ

These theorems have the following consequence for standard decision procedures:
T HEOREM 7.5. Suppose Θ is a metric space and (RC) holds, and let δ0 ∈ D and
C ⊆ D. If there exists ε ∈ ∗ R>0 such that ∗δ 0 is ε -∗ Bayes among C ∗ = {∗δ : δ ∈ C }
with respect to an ε -regular nonstandard prior, then δ0 is admissible among C .
P ROOF. The result follows from Theorem 4.6 and Theorem 7.4.
Theorem 7.5 implies the well-known result that Bayes procedures with respect to
priors with full support are admissible [10, §2.3 Thm. 3] (see also [17, §5 Thm. 7.9]).
T HEOREM 7.6. Suppose Θ is a metric space and (RC) holds and let δ0 ∈ D. If
δ0 is Bayes among D with respect to a prior π with full support, then δ0 is admissible among D.
P ROOF. Note that δ0 is Bayes under π among D if and only if ∗δ 0 is nonstandard Bayes under ∗ π among D ∗ . As π has full support, ∗ π is ε -regular for every
infinitesimal ε ∈ ∗ R>0 . By Theorem 7.5, we have the desired result.

30

DUANMU AND ROY

We close with an admissibility result requiring no additional regularity:
T HEOREM 7.7. Let δ0 ∈ D and C ⊆ D. If there exists ε ∈ ∗ R>0 such that ∗δ 0
is ε -∗ Bayes among ∗ C with respect to a nonstandard prior π satisfying π {θ } ≫ ε
for all θ ∈ Θ, then δ0 is admissible among C .
P ROOF. Suppose δ0 is not admissible among C . Then by Theorem 4.6, ∗δ 0 is
not ∗ admissible in Θ/∗ Θ among C ∗ . Thus there exists δ ∈ C and θ0 ∈ Θ such
that ∗r(θ , ∗δ ) ≤ ∗r(θ , ∗δ 0 ) for all θ ∈ ∗ Θ and ∗r(θ0 , ∗δ 0 ) − ∗r(θ0 , ∗δ ) > γ for some
γ ∈ R>0 . Then ∗r(π , ∗δ 0 ) − ∗r(π , ∗δ ) ≥ π {θ0 }γ > ε . But this implies that ∗δ 0 is not
ε -∗ Bayes under π among C .
R EMARK 7.8. The astute reader may notice that Theorem 7.7 is actually a
corollary of Theorem 7.5 provided we adopt the discrete topology/metric on Θ.
Changing the metric changes the set of available prior distributions and also changes
the set of ε -regular nonstandard priors. See also Remark 8.4.
8. Some Examples. The following examples serve to highlight some of the
interesting properties of our nonstandard theory and its consequences for classical
problems.
E XAMPLE 8.1. Consider any standard statistical decision problem with a finite discrete (hence compact) parameter space. (RC) holds trivially, and so Theorem 6.11 and Corollary 6.12 imply that a decision procedure is extended admissible
if and only if it is extended Bayes if and only if it is Bayes if and only if its extension
is nonstandard Bayes among all internal decision procedures. By Theorem 7.6, we
obtain another classical result: if a procedure is Bayes with respect to a prior with
full support, it is admissible.
E XAMPLE 8.2. Consider the classical problem of estimating the mean of a multivariate normal distribution in d dimensions under squared error when the covariance matrix is known to be the identity matrix. By the convexity of the squared error
loss function, Lemma 2.13 implies the nonrandomized procedures form an essentially complete class (indeed, the loss is strictly convex and so the nonrandomized
procedures are actually a complete class). Theorem 5.17 implies that every extended
∗
admissible estimator among D0 is nonstandard Bayes among D0,FC
. We can derive
further results by noting that risk functions are continuous, which follows from a
general theorem on exponential families:
T HEOREM 8.3 ([17, §5 Ex. 7.10]). Assume P is an exponential family. Then, for
any loss function ℓ such that the risk is always finite, the risk function is continuous.
Thus (RC) holds. Theorem 7.6 then implies that every Bayes estimator with
respect to a prior with full support is admissible. In particular, for every k > 0,

NONSTANDARD BAYES

31

2

the estimator δkB (x) = k2k+1 x is Bayes with respect to the full-support prior πk =
N (0, k2Id ), hence admissible.
Consider now the maximum likelihood estimator δ M (x) = x and let K be an
infinite natural number. Then ∗δ M (x) ≈ (∗δ B )K (x) for all x ∈ NS(∗ Rd ), where ∗δ B
is the extension of the function k 7→ δkB . The normal prior (∗ π )K is “flat” on R in the
sense that, at every near-standard real number, the density is within an infinitesimal
1
of (2π )− 2 K −d . These observations provide a nonstandard interpretation to the idea
that the MLE estimator is a Bayes estimator given a “uniform” prior.
Since (RC) holds, Theorem 7.5 implies that every estimator whose extension is
ε -∗ Bayes among ∗D0 with respect to an ε -regular prior is admissible among D0 .
An easy calculation reveals that the Bayes risk of (∗δ B )K with respect to (∗ π )K is
2
d KK2 +1 , while the Bayes risk of ∗δ M with respect to (∗ π )K is d. Thus, ∗δ M is even
nonstandard Bayes among ∗D, and in particular, ∗δ M is ε -∗ Bayes among ∗D for
ε = (K 2 + 1)−1 . From the density above, it is then straightforward to verify that,
for d = 1 and d = 2, the prior (∗ π )K is ε -regular, but that it fails to be for d ≥ 3.
Therefore, by Theorem 7.5, it follows that δ M is admissible among D0 for d = 1
and d = 2, as is well known. The theorem is silent in this case for d ≥ 3. Indeed,
Stein [27] famously showed that δ M is inadmissible for d ≥ 3.
R EMARK 8.4. Here we have used Theorem 7.5 and the standard metric on
Θ = Rd in order to establish admissibility. Note that the infinite-variance Gaussian
prior is not ε -regular with respect to the discrete metric on Θ, and so a different nonstandard prior would have been needed to establish admissibility via Theorem 7.7.
In Section 6, we established that class of Bayes procedures coincides with the
class of extended admissible estimators under compactness of the parameter space
and continuity of the risk. The next example demonstrates that extended admissibility and Bayes optimality do not necessarily align if we drop the risk continuity
assumption, even when the parameter space is compact. We study a non-Bayes admissible estimator and characterize a nonstandard prior with respect to which it is
nonstandard Bayes.
E XAMPLE 8.5. Let X = {0, 1} and Θ = [0, 1], the latter viewed as a subset
of Euclidean space. Define g : [0, 1] → [0, 1] by g(x) = x for x > 0 and g(0) = 1,
and let Pt = Bernoulli(g(t)), for t ∈ [0, 1], where Bernoulli(p) denotes the distribution on {0, 1} with mean p ∈ [0, 1]. Every nonrandomized decision procedure
δ : {0, 1} → [0, 1] thus corresponds with a pair (δ (0), δ (1)) ∈ [0, 1]2, and so we
will express nonrandomized decision procedures as pairs. Consider the loss function ℓ(x, y) = (g(x) − y)2 . (For every x, the map y 7→ ℓ(x, y) is convex but merely
lower semicontinuous on [0, 1]. It follows from Lemma 2.13 that nonrandomized
procedures form an essentially complete class.)
T HEOREM 8.6.

In Example 8.5, (0, 0) is an admissible non-Bayes estimator.

32

DUANMU AND ROY

P ROOF. Let (a, b) ∈ [0, 1]2 and let c = min{a, b}. For every n ∈ N, we have
r(n−1 , (a, b)) = (1 − n−1 )ℓ(1/n, a) + n−1 ℓ(1/n, b)

(8.1)

and so, for sufficiently large n, we have r(n−1 , (a, b)) ≥ r(n−1 , (c, c)). But, for every d > 0 and sufficiently large n, it also holds that r(n−1 , (d, d)) > r(n−1 , (0, 0)).
Hence, (a, b) does not dominate (0, 0), hence (0, 0) is admissible.
To see that (0, 0) is not Bayes, note that an estimator (a, b) has the same Bayes
risk under π as it would under the (pushforward) prior ν = π ◦ g−1 in the statistical
decision problem with sample space X , parameter space Θ′ = g(Θ) = (0, 1], model
Pt′ = Bernoulli(t), and squared error loss ℓ′ (x, y) = (x, y). However, in this case,
the loss is strictly convex and so the Bayes optimal decision is unique and is the
posterior mean, which is a value in (0, 1], hence (0, 0) cannot be Bayes optimal for
any prior.
The failure of (0, 0) to be Bayes optimal is due to the fact that the posterior mean
cannot be 0. However, in the nonstandard universe, the posterior mean can be made
to be infinitesimal, in which case the Bayes risk of (0, 0) is also infinitesimal.
T HEOREM 8.7. ∗ (0, 0) is nonstandard Bayes with respect to any prior concentrating on some infinitesimal ε > 0.
P ROOF. Pick any positive infinitesimal ε and consider the nonstandard prior π
concentrated on ε . The nonstandard Bayes risk of (0, 0) with respect to π is
(8.2)

∗

r(π , (0, 0)) = ∗r(ε , (0, 0)) = ε (ε − 0)2 + (1 − ε )(ε − 0)2 ≈ 0

Because the loss function in Example 8.5 is nonnegative, (0, 0) must be a nonstandard Bayes estimator with respect to π .
We close by observing that (0, 0) is a generalized Bayesian estimator. In particular, the generalized Bayes risk with respect to the improper prior π (dθ ) = θ −2 dθ is
finite, whereas every other estimator has infinite Bayes risk. The modified statistical decision problem with parameter space Θ′ = (0, 1] under the standard topology,
model P′ and loss ℓ′ meets the hypotheses of Theorem 3.10— indeed, the modified
problem is that of estimating the mean of an exponential family model— hence every extended admissible procedure is generalized Bayes. The original problem does
not meet the hypotheses of Theorem 3.10, since the loss is not jointly continuous.
9. Miscellaneous remarks.
(i) We have required Θ to be Hausdorff in order for the standard part map to
be uniquely defined. Relaxing this assumption would require that we work with a
standard part relation instead. At this moment, we see no roadblocks.

NONSTANDARD BAYES

33

(ii) Assume ∗δ is nonstandard Bayes among C ∗ . Under what conditions can we
conclude that ∗δ is nonstandard Bayes among ∗ C ? 0-∗ Bayes among C ∗ ? Among
∗ C ? In Corollary 6.12, we show that these conclusions follow for C = D
0,FC when
Θ is compact Hausdorff and (RC) holds. Can we weaken these conditions or find
incomparable ones? A related problem is to identify conditions under which δ is
∗ extended admissible on Θ among ∗ C . As a starting point, it is an open problem to
find a procedure δ such that either 1) ∗δ is nonstandard Bayes among ∗D0,FC but ∗δ
∗
∗
is not 0-∗ Bayes among D0,FC
, or 2) ∗δ is nonstandard Bayes among D0,FC
but ∗δ is
not nonstandard Bayes among ∗D0,FC . Note that Example 8.2 demonstrates that δ M
is nonstandard Bayes among ∗D0,FC but not 0-∗ Bayes among ∗D0,FC .
(iii) We restricted our attention to decision procedures whose risk functions are
everywhere finite. However, if we do not make this restriction, it is possible for an
admissible decision procedures to have infinite risk in some state θ ∈ Θ [6, §4A.13
Part (iv)]. We make repeated use of the finite risk property and so it would be an
interesting contribution to relax this assumption. A related issue is our restriction
to nonnegative real-valued loss functions. It would be straightforward to allow loss
functions that are bounded below or above. Allowing arbitrary loss functions, however, raises the possibility that a decision procedure’s risk could be undefined on
some subset of the parameter space.
(iv) It is worth searching for a converse to Theorem 7.7, perhaps with a view to
identifying a nonstandard analogue of Stein’s necessary and sufficient condition for
admissibility [26], but one witnessed by a single (nonstandard) prior distribution.
(v) Our standard result, Theorem 6.11, is similar to Theorem 3.6 of Berger [2]
and Theorem 3.3 of Wald [30]. Our theorem identifies the class of extended admissible procedures and the class of Bayes procedures, and does so by assuming that
risk functions are continuous and the parameter space is compact Hausdorff. These
are weaker assumptions than those of Berger, and more natural than those of Wald.
It would take some work to understand which assumptions of theirs are needed to
show that the extended admissible procedures (equivalently, the Bayes procedures)
form a complete class. In our opinion, it is preferable to understand conditions under which we can identifying extended admissibility and Bayes optimality and then
separately understand conditions under which the former is a complete class. (The
classical textbook by Blackwell and Girschick [4] adopts a similar aesthetic principle.) We believe that the methods developed in this paper may allow us to remove
or generalize regularity conditions in other existing results.
(vi) It would be illuminating to uncover a complete characterization of the relationships between nonstandard Bayes procedures, extended Bayes procedures, limits of Bayes procedures, and generalized Bayes procedures. Some connections can
be identified simply by transfer: e.g., we already know that extended Bayes procedures are nonstandard Bayes by a simple transfer argument. Given our theorems
connecting extended admissibility and nonstandard Bayes optimality, progress on
this question immediately yields new connections between extended admissibility

34

DUANMU AND ROY

and these relaxed notions of Bayes optimality.
(vii) Under compactness and risk continuity, extended admissible procedures are
nonstandard Bayes among all internal decision procedures. In general, however, an
extended admissible procedure is nonstandard Bayes only among the nonstandard
extensions of standard procedures. Can we find an example witnessing the gap between these two results? In particular, can we identify a decision problem for which
the extension of some (all, most) (extended) admissible decision procedure(s) is
(are) not nonstandard Bayes among all internal decision procedures?
Acknowledgments. The authors owe a debt of gratitude to William Weiss for
detailed suggestions, as well as for his assistance with set theoretic and topological issues. We thank Gintarė Džiugaitė, Cameron Freer, and H. Jerome Keisler for
early discussions and insights, and thank Nate Ackerman, Michael Evans, Arno
Pauly, and Aaron Smith for feedback on drafts and helpful discussions. Finally, the
authors would like to thank Peter Hoff for his course notes, which served as our first
introduction to the topic. Work on this publication was made possible through an
NSERC Discovery Grant, Connaught Award, and U.S. Air Force Office of Scientific Research grant #FA9550-15-1-0074. This work was done in part while DMR
was visiting the Simons Institute for the Theory of Computing at UC Berkeley.
References.
[1] A RKERYD , L. O., C UTLAND , N. J. and H ENSON , C. W., eds. (1997). Nonstandard analysis.
NATO Advanced Science Institutes Series C: Mathematical and Physical Sciences 493. Kluwer
Academic Publishers Group, Dordrecht Theory and applications. MR1603227
[2] B ERGER , J. O. (1985). Statistical decision theory and Bayesian analysis, second ed. Springer
Series in Statistics. Springer-Verlag, New York. MR804611
[3] B ERGER , J. O. and S RINIVASAN , C. (1978). Generalized Bayes estimators in multivariate
problems. Ann. Statist. 6 783–801. MR0478426
[4] B LACKWELL , D. and G IRSHICK , M. A. (1954). Theory of games and statistical decisions.
John Wiley and Sons, Inc., New York; Chapman and Hall, Ltd., London. MR0070134
[5] B ROWN , L. D. (1971). Admissible estimators, recurrent diffusions, and insoluble boundary
value problems. Ann. Math. Statist. 42 855–903. MR0286209
[6] B ROWN , L. D. (1986). Fundamentals of statistical exponential families with applications in
statistical decision theory. Institute of Mathematical Statistics Lecture Notes—Monograph Series, 9. Institute of Mathematical Statistics, Hayward, CA. MR882001
[7] C HANG , C. C. and K EISLER , H. J. (1990). Model theory, third ed. Studies in Logic and the
Foundations of Mathematics 73. North-Holland Publishing Co., Amsterdam. MR1059055
[8] C UTLAND , N. J., N EVES , V., O LIVEIRA , F. and S OUSA -P INTO , J., eds. (1995). Developments in nonstandard mathematics. Pitman Research Notes in Mathematics Series 336. Longman, Harlow Papers from the International Colloquium (CIMNS94) held in memory of Abraham Robinson at the University of Aveiro, Aveiro, July 18–22, 1994. MR1394201
[9] DAWID , A. P., S TONE , M. and Z IDEK , J. V. (1973). Marginalization paradoxes in Bayesian
and structural inference. J. Roy. Statist. Soc. Ser. B 35 189–233. With discussion by D. J.
Bartholomew, A. D. McLaren, D. V. Lindley, Bradley Efron, J. Dickey, G. N. Wilkinson, A.
P.Dempster, D. V. Hinkley, M. R. Novick, Seymour Geisser, D. A. S. Fraser and A. Zellner,
and a reply by A. P. Dawid, M. Stone, and J. V. Zidek. MR0365805
[10] F ERGUSON , T. S. (1967). Mathematical statistics: A decision theoretic approach. Probability
and Mathematical Statistics, Vol. 1. Academic Press, New York-London. MR0215390

NONSTANDARD BAYES

35

[11] H EATH , D. and S UDDERTH , W. (1978). On finitely additive priors, coherence, and extended
admissibility. Ann. Statist. 6 333–345. MR0464450
[12] F REMLIN , D. H. (2009). Real-valued-measurable cardinals. Version 19.9.09,
https://www.essex.ac.uk/maths/people/fremlin/rvmc.pdf. Accessed 2017-02-10.
[13] JAMES , W. and S TEIN , C. (1961). Estimation with quadratic loss. In Proc. 4th Berkeley
Sympos. Math. Statist. and Prob., Vol. I 361–379. Univ. California Press, Berkeley, Calif.
MR0133191
[14] K EISLER , H. J. (1984). An infinitesimal approach to stochastic analysis. Mem. Amer. Math.
Soc. 48 x+184. MR732752
[15] L ANDERS , D. and ROGGE , L. (1987). Universal Loeb-measurability of sets and of the standard part map with applications. Trans. Amer. Math. Soc. 304 229–243. MR906814
[16] L E C AM , L. (1955). An extension of Wald’s theory of statistical decision functions. Ann. Math.
Statist. 26 69–81. MR0067443
[17] L EHMANN , E. L. and C ASELLA , G. (1998). Theory of point estimation, second ed. Springer
Texts in Statistics. Springer-Verlag, New York. MR1639875
[18] L OEB , P. A. (1975). Conversion from nonstandard to standard measure spaces and applications
in probability theory. Trans. Amer. Math. Soc. 211 113–122. MR0390154
[19] L UXEMBURG , W. A. J. (1969). A general theory of monads. In Applications of Model Theory
to Algebra, Analysis, and Probability (Inte rnat. Sympos., Pasadena, Calif., 1967) 18–86. Holt,
Rinehart and Winston, New York. MR0244931
[20] P ERLMAN , M. D. (1974). Jensen’s inequality for a convex vector-valued function on an
infinite-dimensional space. J. Multivariate Anal. 4 52–65. MR0362421
[21] R AIFFA , H. and S CHLAIFER , R. (1961). Applied statistical decision theory. Studies in Managerial Economics. Division of Research, Graduate School of Business Administration, Harvard University, Boston, Mass. MR0117844
[22] ROBINSON , A. (1966). Non-standard analysis. North-Holland Publishing Co., Amsterdam.
MR0205854
[23] S ACKS , J. (1963). Generalized Bayes solutions in estimation problems. Ann. Math. Statist. 34
751–768. MR0150908
[24] S CHERVISH , M. J., S EIDENFELD , T. and K ADANE , J. B. (1984). The extent of nonconglomerability of finitely additive probabilities. Z. Wahrsch. Verw. Gebiete 66 205–226. MR749222
[25] S EIDENFELD , T., S CHERVISH , M. J. and K ADANE , J. B. (1998). Non-conglomerability for
finite-valued, finitely additive probability. Sankhyā Ser. A 60 476–491. Bayesian analysis.
MR1718828
[26] S TEIN , C. (1955). A necessary and sufficient condition for admissibility. Ann. Math. Statist.
26 518–522. MR0070929
[27] S TEIN , C. (1956). Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In Proceedings of the Third Berkeley Symposium on Mathematical Statistics
and Probability, 1954–1955, vol. I 197–206. University of California Press, Berkeley and Los
Angeles. MR0084922
[28] S TONE , M. (1967). Generalized Bayes decision functions, admissibility and the exponential
family. Ann. Math. Statist. 38 818–822. MR0226780
[29] WALD , A. (1947a). An essentially complete class of admissible decision functions. Ann. Math.
Statistics 18 549–555. MR0023499
[30] WALD , A. (1947b). Foundations of a general theory of sequential decision functions. Econometrica 15 279–313. MR0024113
[31] WALD , A. (1949). Statistical decision functions. Ann. Math. Statistics 20 165–205.
MR0044802

36

DUANMU AND ROY

A: NONSTANDARD ANALYSIS — BASIC NOTIONS AND KEY RESULTS
In this appendix, we give a brief introduction for those readers not familiar with
nonstandard analysis. The interested reader can find a thorough introduction in [22].
For modern applications of nonstandard analysis, one can read [1] or [8]. Our following introduction of nonstandard analysis owes much to [1].
For a set S, let P(S) denote its power set. Given any set S, define V0 (S) = S and
S
Vn+1 (S) = Vn (S) ∪ P(Vn (S)) for all n ∈ N. Then V(S) = n∈N Vn (S) is called the
superstructure of S, and S is called the ground set of the superstructure V(S). We
treat the members of S as indivisible atomics. The rank of an object a ∈ V(S) is the
smallest k for which a ∈ Vk (S). The members of S have rank 0. The objects of rank
no less than 1 in V(S) are precisely the sets in V(S). The empty set 0/ and S both
have rank 1.
We now formally define the language L (V(S)): For every element c ∈ V(S),
we introduce a constant symbol c̄. We also fix a countable collection of variable
symbols v1 , v2 , . . . , distinct from the constants and the following reserved symbols:
=, ∈, ), (, ∧, ∨, ¬, ∀, and ∃. The (bounded) formulas in L (V(S)) are defined
recursively:
• If u and v are constant/variable symbols, then (u = v) and (u ∈ v) are formulas.
• If φ and ψ are formulas, then (φ ∧ ψ ), (φ ∨ ψ ), and (¬φ ) are formulas.
• If φ is a formula and u is a variable, and v is a constant or variable symbol
distinct from u, then (∀u ∈ v)(φ ) and (∃u ∈ v)(φ ) are formulas.
A variable x is free in a formula φ if it is not within the scope of any quantifiers.
More carefully: x is free in (u = v) if either u or v is the variable x, and similarly
for (u ∈ v); x is free in (φ ∧ ψ ) if it is free in either φ or ψ , and similarly for ∨ and
¬; x is free in (∀u ∈ v)(φ ) if x is not u and x is v or x is free in φ , and similarly
for (∀u ∈ v)(φ ). We will sometimes write φ (x1 , . . . , xn ) to mean that x1 , . . . , xn are
exactly the free variables in φ , and write φ (c1 , . . . , cn ) to mean the formula φ where
we substitute every free occurrence of xi with the constant ci , for every i = 1, . . ., n.
(See [7] for a careful description of substitution.)
A sentence is a formula with no free variables. Informally, sentences in L (V(S))
are either true or false statements about V(S). We formalize this here via statements
of the form φ holds in V(S).
D EFINITION A.1.

Let φ , ψ be sentences in L (V(S)).

• (c̄1 = c̄2 ) holds in V(S) if and only if c1 = c2 ;
• (c̄1 ∈ c̄2 ) holds in V(S) if and only if c2 has rank 1 or higher and c1 ∈ c2 ;
• (φ ∧ ψ ) holds in V(S) if and only if φ holds in V(S) and ψ holds in V(S), and
similarly for ∨ and ¬;
• (∀u ∈ c̄)(φ (u)) holds in V(S) if and only if, for all x ∈ c, the sentence φ (c)
holds in V(S), and similarly for (∃u ∈ v)(φ (u)).

NONSTANDARD BAYES

37

We will also write that φ is true in V(S) to mean φ holds in V(S). When the
structure V(S) is clear from context, we will simply write that φ holds or that φ
is true. We will use the following abbreviations: (φ =⇒ ψ ) for ((¬φ ) ∨ (ψ )) and
(φ ⇐⇒ ψ ) for (φ =⇒ ψ ) ∧ (ψ =⇒ φ ).
It may seem necessary (or at least useful) to include function and relation symbols in our language for every function and relation in C ⊆ V(S). For example, one
would expect (∃x ∈ R̄) (1̄+̄x̄ = 3̄) to be a well-formed sentence in L (V(R)). (Writing¯over every constant is cumbersome and unnecessary in practice, and so we will
drop this symbol from now on.) In fact, every relation and function symbol is, in effect, already available, because there is a mechanical way to translate formulas with
function and relation symbols into the above language (and in a way that commutes
with taking ∗ -transfers as defined below). Informally, the translation works as follows: First, function symbols are removed by rewriting them in terms of their graph
relations using additional quantifiers. Second, because every relation is an element
in V(S) and thus a constant in the language, relation symbols can be replaced with
statements of the form a ∈ b, where a is a tuple. Finally, tuples can be mechanically
encoded once elements of products spaces are encoded as sets of sets in a canonical
way. Thus our language is powerful enough to represent formula with relation and
function symbols, and so these symbols will be used without comment.
Let S and ∗ S be a pair of ground sets, and let ∗ : V(S) → V(∗ S) be a map between
their superstructures that preserves rank. Relative to this map, an element a ∈ V(∗ S)
is internal when there exists b ∈ V(S) such that a ∈ ∗ b, and a is said to be external
otherwise. The language of V(∗ S) is almost the same as L except that we enlarge
the set of constants to include every element in V(∗ S). We denote the language of
V(∗ (S)) by L (V(∗S)). If φ (x1 , . . ., xn ) is a formula in L (V(S)) with free variables
x1 , . . ., xn , then the ∗-transfer of φ is the formula in L (V(∗ S)) obtained by changing
every constant a to ∗ a. Clearly, every constant in ∗ φ (x1 , . . . , xn ) is internal.
Let κ be an uncountable cardinal number. A κ -saturated nonstandard extension
of a superstructure V(S) is a set ∗ S and a map ∗ : V(S) → V(∗ S) satisfying:
• extension: ∗ S is a superset of S and ∗ s = s for all s ∈ S.
• transfer: For every sentence φ in L (V(S)), φ holds in V(S) if and only if its
∗-transfer ∗ φ holds in V(∗ S).
• κ -saturation: For every family F = {Ai : i ∈ I} of internal sets indexed by
a set I of cardinality less than κ , if F has the finite intersection property,
i.e., if every finite intersection of elements in F is nonempty, then the total
intersection of F is nonempty.
A ℵ1 -saturated model can be constructed via an ultrafilter, see [1, Thm. 1.7.13].
However, saturation to any uncountable cardinal number is possible:
T HEOREM A.2 ([19]). For every superstructure V(S) and uncountable cardinal
number κ , there exists a κ -saturated nonstandard extension of V(S).

38

DUANMU AND ROY

From this point on, we shall always assume that our nonstandard extension is at
least ℵ1 saturated.
It is easy to see that every element of ∗ S is internal. It is also clear that A ⊂ ∗ S is
internal if and only if A ∈ ∗ P(S). Most sets are external subsets of ∗ S, and external
subsets play an important role. The main tool for constructing internal sets is the
internal definition principle:
L EMMA A.3 (Internal Definition Principle). Let φ (x) be a formula in the language L (V(∗ S)) with free variable x. Suppose that all constants that occurs in φ
are internal, then {x ∈ V(∗ S) : φ (x) holds in V(∗ S)} is internal in V(∗ S).
Saturation can be equivalently expressed in terms of the satisfiability of families
of formulas. The role of the finite intersection property is played by finite satisfiability:
D EFINITION A.4. Let J be an index set and let A ⊆ V(∗ S). A set of formulas
{φ j (x) | j ∈ J} in L (V(∗S)) is said to be finitely satisfiable in A when, for every
finite subset α ⊂ J, there exists c ∈ A such that φ j (c) holds in V(∗ S) for all j ∈ α .
We provide the following alternative expression of κ -saturation:
T HEOREM A.5 ([1, Thm. 1.7.2]). Let V(∗ S) be a κ -saturated nonstandard extension of the superstructure V(S), where κ is an uncountable cardinal number. Let
J be an index set of cardinality less than κ . Let A be an internal set. For each j ∈ J,
let φ j (x) be a formula in L (V(∗S)) whose constants are internal objects. Further,
suppose that the set of formulas {φ j (x) | j ∈ J} is finitely satisfiable in A. Then there
exists c ∈ A such that φ j (c) holds in ∗ V(S) simultaneously for all j ∈ J.
A.1. The hyperreals. Take S = R. Then any nonstandard extension ∗ R is a
superset of R and ∗ x = x for x ∈ R. For every n ∈ N, let An = {x ∈ R : 0 < x <
1
∗
n }. Then An is internal by definition and, by the transfer principle, we have that
∗A = {x ∈ ∗ R : 0 ∗< x ∗< 1 }, which is clearly seen to be internal by the internal
n
n
definition principle. (For standard functions—like addition and multiplication—and
relations—like the less-than relation—we will drop the ∗ when the context is clear.)
Let F = {∗An : n ∈ N}. Clearly, F has the finite intersection property, which can
be seen to hold by transfer, because each An is nonempty and An ⊇ An+1 . By the
saturation principle, the total intersection of F is nonempty. Therefore, there exists
x ∈ ∗ R such that 0 < x < 1n for all n ∈ N. Such a number is called an infinitesimal.
It follows that there exists y ∈ ∗ R such that y > n for all n ∈ N. Such a number is
called an infinite number. It follows that ∗ R is a proper superset of R.
Write x ≈ y if |x − y| is infinitesimal and write µ (x) = {y ∈ ∗ R : x ≈ y} for the
monad of x. An element x ∈ ∗ R is near-standard if x ≈ a for some a ∈ R. An
element x ∈ ∗ R is finite if |x| is bounded by some standard real number a. By the

NONSTANDARD BAYES

39

compactness of closed intervals of R and the transfer principle, one can show that
an element x ∈ ∗ R is finite if and only if x is near-standard.
It is also straightforward to show that, if x ∈ ∗ R is near-standard, then there exists
a unique real a ∈ R, called the standard part of x, such that x ≈ a. Let NS(∗ R) denote
the collection of all near-standard points in ∗ R, and let st : NS(∗ R) → R denote the
standard part map taking a near-standard point x to its standard part. For a nearstandard real x, we will write ◦ x instead of st(x); for a function f into NS(∗ R), we
will write ◦ f to denote the composition x 7→ st( f (x)); for a set A ⊆ NS(∗ R), we
will write st(A) to denote the set {x ∈ R : (∃a ∈ A) st(a) = x}; and, for a set B ⊆ R,
we will write st−1 (B) to denote the inverse image {x ∈ ∗ R : (∃b ∈ B) x ≈ b}. For
a standard real x, the set st−1 (x) is the set of nonstandard numbers infinitesimally
close to x, i.e., it is the monad µ (x) of x.
The following example is a classical example of external set:
E XAMPLE A.6. Let φ be the sentence: ∀A ∈ P(R), if A is bounded above,
then A has a least upper bound.4 By the transfer principle, it holds that, for all
A ∈ ∗ P(R), i.e., all internal subsets of ∗ R, if A is bounded above, then A has a least
upper bound. Suppose the monad µ (0) is internal. Then there exists a0 ∈ ∗ R such
that a0 is a least upper bound for µ (0). Clearly a0 > 0. Suppose a0 ∈ µ (0). Then
2a0 ∈ µ (0) and 2a0 > a0 , contradicting that a0 is an upper bound, hence a0 6∈ µ (0).
But then a20 6∈ µ (0) and a20 < a0 , contradicting that a0 is the least upper bound.
Hence, µ (0) is external.
The following lemma collects together two simple facts we use repeatedly:
L EMMA A.7.

Let a, b ∈ NS(∗ R).

1. a ≈ b if and only if ◦ a = ◦ b.
2. a / b implies ◦ a ≤ ◦ b.
We conclude this section by introducing two useful properties derived from saturation:
T HEOREM A.8 ([1, Prop. 2.8.2]).

Let A ⊂ ∗ R be an internal set.

1. (Overspill) If A contains arbitrarily large finite positive numbers, then it also
contains an infinite number.
2. (Underspill) If A contains arbitrarily small positive infinite numbers, then it
also contains a finite number.
4 Formally,

φ is the sentence (∀A ∈ P(R))(((∃a ∈ R)(∀x ∈ A)(x < a)) =⇒ ((∃a ∈ R)(((∀x ∈
A)(x < a)) ∧ ((∀b ∈ R)(((∀x ∈ A)(x < a)) =⇒ (a ≤ b)))))), but we will prefer to work informally
for clarity, although caution must be exercised: the “transfer” of the sentence “∀A ⊂ R if A is bounded
above then A has a least upper bound” might lead one to conclude that all bounded sets have least
upper bounds. The error here is that ⊂ is not in the first order language of set theory, and so the
sentence is malformed.

40

DUANMU AND ROY

An immediate consequence of this theorem is the following:
C OROLLARY A.9.

Let A ⊂ ∗ R be an internal set.

1. (Overspill) If A contains arbitrarily large infinitesimals, then A contains some
non-infinitesimal element from ∗ R.
2. (Underspill) If A contains arbitrarily small positive non-infinitesimals, then
A contains some positive infinitesimal element from ∗ R.
A.2. Nonstandard extensions of more general spaces. The concepts from the
previous section can be generalized to a general metric space X by replacing the
standard metric in R by the corresponding metric in the metric space. Indeed, all
the concepts can be defined for an arbitrary topological space (X , T ). In this case,
we will assume that our model is more saturated than the cardinality of T . The
monadic structure is determined by T (although we will elide the topology in our
notation as it will be clear from context). In particular, for x ∈ ∗ X, the monad of x
T
is defined by µ (x) = {∗ G : x ∈ G ∈ T }. In general µ (x) is external, however, for
every x ∈ ∗ X, one can use saturation to prove that there exists an (internal) ∗ open
set B ∈ ∗ T such that x ∈ B ⊆ µ (x). Given a set Y ⊆ ∗ X, the near-standard points
are given by NS(Y ) = {y ∈ Y : (∃x ∈ X ) y ∈ µ (x)}. In general, NS(∗ X) is a proper
subset of ∗ X. However, when X is compact, we have NS(∗ X ) = ∗ X . Indeed, this is
the nonstandard way to characterize a compact space.
T HEOREM A.10 ([1, Thm. 3.5.1]).
compact if and only if ∗ A = NS(∗ A).

Let X be a Hausdorff space. A set A ⊂ X is

It is worth presenting an example:
E XAMPLE A.11. Consider ∗ [0, 1] = {x ∈ ∗ R : 0 ≤ x ≤ 1} under its standard
topology. The closed interval [0, 1] is compact, and so ∗ [0, 1] = NS(∗ [0, 1]). However, the open interval (0, 1) is not compact, hence ∗ (0, 1) 6= NS(∗ (0, 1)). Indeed,
consider a positive infinitesimal ε . Then ε ∈ ∗ (0, 1) but ε 6∈ NS(∗ (0, 1)). If we adopt
the discrete topology on [0, 1], then µ (x) = {x} for every x ∈ ∗ [0, 1] and the set of
near-standard points NS(∗ [0, 1]) is precisely the set of standard points [0, 1] itself.
Nonstandard analysis gives a very succinct characterization of continuity in terms
of S-continuity (Definition 6.6). Here we give a proof of the classical result relating continuity to S-continuity. This proof is generalized from the proof of [1,
Thm. 2.4.1].
P ROOF OF L EMMA 6.7. Let x1 , x2 be two near-standard points in ∗ D such that
x1 ≈ x2 . Let x0 = st(x1 ) = st(x2 ). Then x0 ∈ D. By local compactness, there exists a
compact neighborhood K0 of f (x0 ). Let U0 ⊆ K0 be an open set containing the point
f (x0 ). It is clear that x1 , x2 are elements of ∗ ( f −1 (U0 )) hence ∗ f (x1 ) and ∗ f (x2 ) are

NONSTANDARD BAYES

41

elements of ∗ K0 by transfer. As K0 is compact, both ∗ f (x1 ) and ∗ f (x2 ) are nearstandard.
Suppose ∗ f (x1 ) 6≈ ∗ f (x2 ). For every y ∈ Y , let νY (y) denote the monad of y.
Then ∗ f (x1 ) ∈ νY (y1 ) and ∗ f (x2 ) ∈ νY (y2 ) for distinct y1 , y2 ∈ f (D). Because Y is
Hausdorff, there exists an open set U1 containing y1 but not y2 . Then ∗ f (x2 ) 6∈ ∗U1 ∩
∗ f (D) hence x 6∈ ∗ ( f −1 (U )) ∩ ∗ D. However, x ∈ ∗ ( f −1 (U )) ∩ ∗ D and f −1 (U ) ∩
2
1
1
1
1
D is open in D. This shows that x1 6≈ x2 , a contradiction.
A.2.1. Vector spaces. We conclude this section with a short discussion of vector
spaces over the nonstandard field ∗ R. The notions of convexity of sets and functions
have their ordinary abstract semantics: If X is a vector space over the field ∗ R (where
addition and multiplication are the nonstandard extensions of ordinary addition and
multiplication), a (possibly external) subset A of X is convex if for all a ∈ ∗ [0, 1]
and x1 , x2 ∈ A, we have ax1 + (1 − a)x2 ∈ A. A (possibly external) function f from
X to ∗ R is convex if its graph is a convex set, i.e., for all a ∈ ∗ [0, 1] and x1 , x2 ∈ X ,
f (ax1 + (1 − a)x2) ≤ a f (x1 ) + (1 − a) f (x2). The function is strictly convex if the
inequality is strict. We will make use of several properties that hold when X is an
internal set, i.e., when X is an internal vector space (over the field ∗ R). By transfer,
the first order characterization of internal vector spaces over ∗ R is the same as that
of standard vector spaces over R. An important class of internal convex spaces are
the hyperfinite-dimensional Euclidean spaces I(∗ RN ) for N ∈ ∗ N.
T HEOREM A.12.

I(∗ RN ) is an internal convex space for every N ∈ ∗ N.

P ROOF. Fix any N ∈ ∗ N. For every n ∈ N, let Fn denote the set of functions from
S
{1, . . ., n} to R. Then I(∗ RN ) ∈ ∗ n∈N Fn hence I(∗ RN ) is internal. It is straightforward to demonstrate convexity.
We close with a proof of the hyperfinite separating hyperplane theorem, which
demonstrates the use of transfer.
P ROOF OF T HEOREM 5.15. We first restate the standard hyperplane separation
theorem. We shall view the set RN as the set of functions from N to R. For every
element x ∈ RN , we use x(k) to denote the value of the k-th coordinate of x for any
k ∈ N. The standard hyperplane separation theorem is equivalent to:
For any two disjoint convex S1 , S2 ∈ P(RN ), if ∃k ∈ N such that ∀s ∈
S1 ∪ S2 ∀k′ > k we have s(k′ ) = 0 then ∃a ∈ RN \ {0} with a(k′ ) = 0 for
all k′ > k such that ∀p1 ∈ S1 , p2 ∈ S2 ((∀k′ > k, a(k′) = 0) ∧ (ha, p1 i ≤
ha, p2 i)).
By the transfer principle, we know that ∗ (RN ) denotes the set of all internal functions from ∗ N to ∗ R. We shall view the inner product h·, ·i to be a function from
RN × RN to R. Note that ∀p, s ∈ RN if ∃k ∈ N such that ∀k′ > k we have s(k′ ) = 0

42

DUANMU AND ROY

then hp, si = ∑ki=1 p(i)s(i). Thus the nonstandard extension of h·, ·i is a function
from ∗ (RN ) × ∗ (RN ) to ∗ R satisfying the same property.
Now by the transfer principle we know that:
For any two disjoint convex sets S1 , S2 ∈ ∗ P(RN ). If ∃K ∈ ∗ N such
that ∀s ∈ S1 ∪ S2 ∀K ′ > K we have s(K ′ ) = 0 then ∃W ∈ ∗ (RN ) \ {0}
such that for all p1 ∈ S1 , p2 ∈ S2 we have ((∀K ′ > K,W (K ′ ) = 0) ∧
K
∑K
i=1 W (i)p1 (i) ≤ ∑i=1 W (i)p2 (i)).
In this sentence, it is easy to see that we can view the projections of S1 , S2 as
internal subsets of I(∗ RK ) and the projection of W as an element from I(∗ RK ) \ {0}.
Hence we have that: ∀K ∈ ∗ N, if S1 , S2 are two disjoint internal convex subsets of
I(∗ RK ), then there exists W ∈ I(∗ RK ) \ {0} such that for any P1 ∈ S1 and any P2 ∈ S2 ,
K
∑K
i=1 W (i)P1 (i) ≤ ∑i=1 W (i)P2 (i). Thus we have the desired result.
B: INTERNAL PROBABILITY THEORY
In this section, we give a brief introduction to nonstandard probability theory.
The interested reader can consult [1] and [14] for more details.
Consider a σ -algebra F on a space X , and the space M1 (X , F ) of countably
additive probability measures defined on (X , F ). By the transfer principle, ∗ (X , F )
and the set ∗M1 (X , F ) of internal ∗ countably additive probability measures on ∗F
satisfy (the transfer of) all the first-order properties of their standard counterparts.
Some care is required: e.g., ∗ countably additivity is defined by the behavior of a
measure on internal sequences in ∗F , not arbitrary sequences. In general, the transfer principle is the primary means of relating internal integration/measure theory to
its standard counterpart. Saturation can then be used to control the effect of “small”
perturbations.
As an example of using transfer, consider one of the key structures in this paper:
that of an indexed family of probability measures on a common measurable space
(X , F ). Let Θ be an index set, and, for every θ ∈ Θ, let Pθ be a probability measure
on (X , F ). Equivalently, we can think of P as a function P : Θ × F → [0, 1] defined
by P(θ , A) = Pθ (A) for A ∈ F . By the transfer principle, we know that ∗ P is then
a function from ∗ Θ × ∗F to ∗ [0, 1] and ∗ P(y, ·) is an internal (∗ countably additive)
probability measure on (∗ X, ∗F ) for every y ∈ ∗ Θ. Note that, for each θ ∈ Θ, we
can also take the nonstandard extension of the probability measure Pθ . These two
different nonstandard extensions agree with each other for θ ∈ Θ.
L EMMA B.1.

For every θ ∈ Θ and A ∈ ∗F , ∗ P(θ , A) = ∗ (Pθ )(A).

P ROOF. Fix any θ0 ∈ Θ and define F to be a function from F to [0, 1] by F(B) =
P(θ0 , B). Thus ∗ F is an internal function from ∗F to ∗ [0, 1] given by ∗ P(θ0 , A) for all
A ∈ ∗F . Consider the sentence (∀B ∈ F )(F(B) = Pθ0 (B)). By the transfer principle,
we have (∀A ∈ ∗F )(∗ F(A) = ∗ (Pθ0 )(A)). As our choice of θ0 is arbitrary, we have
∗ P(θ , A) = ∗ P (A) for every θ ∈ Θ and A ∈ ∗F .
θ

NONSTANDARD BAYES

43

Thus, for every y ∈ ∗ Θ, we shall write ∗ Py (·) for ∗ P(y, ·) and keep in mind that
we can view ∗ Py as the y-th fibre of a function of two variables. The next lemma
also follows from a transfer (and extension) argument.
L EMMA B.2. Let (X , F ) be a measurable space, let {Py }y∈Y be a family of
probability measures on (X , F
), and suppose F is F -measurable and Py -integrable
R
for all y ∈ Y . Define r(y) = X F(x)Py(dx) for y ∈ Y . Then ∗ F is ∗F -∗ measurable
and ∗ Py -∗ integrable for all y ∈ ∗Y ,
∗

(B.1)

r(y) =

Z
∗

∗
∗X

F(x)∗ Py (dx)

for all y ∈ ∗Y , and r(y) = (∗ r)(y) for every y ∈ Y .
We will simply write for ∗ , integrable for ∗ integrable, etc., when the context is
clear. Saturation and transfer allow us to study the effects of “small” perturbations:
R

R

L EMMA B.3. Let (X , F , P) be an internal probability space Rand let F,
F ′ be
R ′
′
internal P-integrable functions such that F ≈ F everywhere. Then FdP ≈ F dP.
B.1. Hyperfinitely additive probability measures and Loeb theory. Due to
saturation, the subset of internal ∗ finitely additive probability measures defined on
internal algebras plays a central role in nonstandard probability theory. We can understand ∗ finite (also called hyperfinite) structures by the transfer principle. In particular, a set A ∈ V(∗ S) is hyperfinite if and only if there exists an internal bijection
between A and {1, 2, . . ., N} for some N ∈ ∗ N. If such a number, N, exists, then it is
unique and called the internal cardinality of A.
By definition, hyperfinite sets are themselves internal (otherwise the bijection
would not be internal). By transfer, hyperfinite sets are well behaved like their standard finite counterparts: e.g., hyperfinite sums and products are always convergent.
Internal subsets of hyperfinite sets are hyperfinite. The converse holds as well: a subset of a hyperfinite set is internal if and only if it is hyperfinite [1, Exercise 1.6.17].
The following definitions align with the transfer principle: An internal algebra F ⊂ P(X ) is an internal set containing X and closed under complementation and hyperfinite unions/intersections. A set function P : F → ∗ R is hyperfinitely additive when, for every n ∈ ∗ N and pairwise disjoint family A1 , . . ., An ∈ F ,
S
we have P( i≤n Ai ) = ∑i≤n P(Ai ). An internal (hyperfinitely additive) probability space is a triple (Ω, F , P) composed of an internal set Ω; an internal subalgebra F ⊂ P(Ω); and an internal hyperfinitely additive probability measure
P : F → ∗ [0, 1] on (Ω, F ), i.e., a nonnegative hyperfinitely additive internal function such that P(Ω) = 1 and P(0)
/ = 0. A hyperfinite probability space is an internal probability space (Ω, F , P) such that Ω is a hyperfinite set and F = I [Ω],
where I [Ω] denotes the collection of all internal subsets of Ω. Like finite probability space, we can specify an internal probability measure on I [Ω] by defining the
mass of each ω ∈ Ω.

44

DUANMU AND ROY

One of the key theorems in modern nonstandard measure theory is due to Loeb
[18], who showed that any internal probability space can be extended to a standard
σ -additive probability space.
T HEOREM B.4 ([18]). Let (Ω, F , P) be an internal finitely additive probability
space. Then there is a standard σ -additive probability space (Ω, F P , P) such that:
1. F = F P is a σ -algebra with F ⊂ F ⊂ P(Ω).
2. P(A) = ◦ P(A) for every A ∈ F .
3. For every A ∈ F and standard ε > 0, there exists Ai , Ao ∈ F such that Ai ⊂
A ⊂ Ao and P(Ao \ Ai ) < ε .
4. For every A ∈ F , there exists B ∈ F such that P(A △ B) = 0.
The probability triple (Ω, F , P) is called the Loeb space of (Ω, F , P). The σ algebra F and the probability measure P are called the Loeb extensions of F and
P, respectively. From Loeb’s original proof, we can give the explicit form of F and
P:
1. A ∈ F ⇐⇒ ∀ε ∈ R>0 ∃Ai , Ao ∈ F (Ai ⊂ A ⊂ Ao ) ∧ (P(Ao \ Ai ) < ε ).
2. (∀A ∈ F ) P(A) = inf{P(Ao )|A ⊂ Ao ∈ F } = sup{P(Ai )|A ⊃ Ai ∈ F }.
The next example demonstrates that F may contain external sets.
E XAMPLE B.5 ([1, Exercise 2.2]). Pick any N ∈ ∗ N \ N and let δ t = N1 . Then
δ t is an infinitesimal. Let Ω = {0, δ t, 2δ t, . . ., 1} and F = I [Ω]. Define P on F
by P({ω }) = δ t for all ω ∈ Ω. Then (Ω, F , P) is a hyperfinite probability space.
Let (Ω, F , P) be the corresponding Loeb space, known as the uniform hyperfinite
Loeb space.
C LAIM B.6.

µ (0) ∩ Ω ∈ F .

P ROOF. µ (0) ∩ Ω consists of elements from Ω that are infinitesimally close to 0.
For n ∈ N, let An = {ω ∈ Ω : ω ≤ n1 }, which is internal by the internal definition
T
principle. Thus An ∈ F and F ∋ n∈N An = µ (0) ∩ Ω, completing the proof.
Note that µ (0) ∩ Ω is an external set. This shows that F contains external sets.
In fact, letting ν denote Lebesgue measure, one can show that, for every set A ⊆
[0, 1] that is ν -measurable, st−1 (A) ∩ Ω ∈ F and ν (A) = P(st−1 (A) ∩ Ω). Thus
(Ω, F , P) is a “hyperfinite representation” of Lebesgue measure on [0, 1].
From Loeb’s proof, we know that F P is the P-completion of the σ -algebra generated by F . This suggests that F P will depend on P. However, some sets always
appear in the Loeb σ -algebra:
D EFINITION B.7. A set A ⊂ Ω is called universally Loeb measurable if A ∈ F P
for every internal probability measure P on (Ω, F ).

NONSTANDARD BAYES

45

We denote the collection of all universally Loeb measurable sets on an internal
algebra F by Lu (F ). The following theorem characterizes the universal Loeb measurability of the set of near-standard points and Borel sets under sufficient regularity
conditions:
T HEOREM B.8 ([15, Cor. 3]). Let Y be a Hausdorff space with Borel σ -algebra
B[Y ] and assume our nonstandard model is more saturated than ℵ0 and the cardinality of the topology on Y . Then:
1. NS(∗Y ) ∈ Lu (∗ B[Y ]) for locally compact spaces, for σ -compact spaces, and
for complete metric spaces;
2. st−1 (B) ∈ {A ∩ NS(∗Y ) : A ∈ Lu (∗ B[Y ])}, B ∈ B[Y ], for regular spaces.
Theorem B.8 implies that, if Y is a σ -compact or locally compact Hausdorff
space, then st−1 (B) ∈ Lu (F ) for all B ∈ B[Y ].

D EPT. OF S TATISTICAL S CIENCES
U NIVERSITY OF T ORONTO
100 S T. G EORGE S T
T ORONTO , ON M5S 3G3
E- MAIL : haosui@utstat.toronto.edu
droy@utstat.toronto.edu

