Loss Functions in Restricted Parameter Spaces and Their
Bayesian Applications
P. Mozgunov, T. Jaki and M. Gasparini

arXiv:1706.02104v2 [] 16 Apr 2018

Lancaster University and Politecnico di Torino

Abstract
A squared error loss remains the most commonly used loss function for constructing a
Bayes estimator of the parameter of interest. It, however, can lead to sub-optimal solutions
when a parameter is defined on a restricted space. It can also be an inappropriate choice in
the context when an overestimation and/or underestimation results in severe consequences
and a more conservative estimator is preferred. We advocate a class of loss functions for
parameters defined on restricted spaces which infinitely penalize boundary decisions like
the squared error loss does on the real line. We also recall several properties of loss
functions such as symmetry, convexity and invariance. We propose generalizations of the
squared error loss function for parameters defined on the positive real line and on an
interval. We provide explicit solutions for corresponding Bayes estimators and discuss
multivariate extensions. Three well-known Bayesian estimation problems are used to
demonstrate inferential benefits the novel Bayes estimators can provide in the context of
restricted estimation.

Keywords: Bayesian Estimation; Loss Functions; Restricted Parameter Space.

1

Introduction
In many applications when considering a problem of parameter estimation, the support of

this parameter is either naturally restricted (e.g. probability, variance, exponential distribution
parameter) or an investigator can restrict it based on previously obtained knowledge (e.g. treatment effects on children given the data for adults). The knowledge about restricted space can
carry important information that an experimenter has and can potentially improve the estimation (Mahmoudi and Zakerzadeh, 2011). There are also application areas in which estimates on
bounds of the restricted space are highly undesirable as they can lead to severe consequences.
1

For instance, it can be worse to underestimate the potentiality of an event having disastrous
or life-threatening consequences than to overestimate it (Norstrom, 1996). An erroneously low
estimated risk-level can lead to the absence of initiative to reduce it.
One of the ways to incorporate the information about a restricted space is to employ a
Bayesian approach and to define a (uniform) prior distribution on the restricted space (Hartigan,
2004). However, once a posterior is obtained the squared error loss
Lq (θ, d) = (θ − d)2

(1.1)

where d is a decision the statistician has to take in order to approximate an unknown estimand
θ, called parameter, is often used to summarise a posterior distribution. The squared loss
function (1.1) ignores the information about the restricted parameter space and is recognised
to lead to suboptimal solutions (see e.g. Stein, 1964; Brown, 1968, for alternative loss functions
for a scale parameter). Research on improving the Bayes estimator under the squared loss
function (a posterior mean) for a scale parameter has subsequently attracted a great deal of
attention (Brewster and Zidek, 1974; Kubokawa, 1994).
In addition, since the loss function (1.1) does not penalise boundary values, it was found to
be unacceptable in many application areas: see Norstrom (1996); Soliman (2002) for examples in
reliability analysis, Joseph (2004) in quality control, Karimnezhad et al. (2014) in environmental
sciences and Saint-Hilary et al. (2018) in drug development. To avoid boundary values of a
scale parameter, Norstrom (1996) introduced the precautionary loss function
Lsq (θ, d) =

(d − θ)2
where θ, d ∈ (0, +∞)
d

(1.2)

which was used by many researchers (Kiapour and Nematollahi, 2011; Karimnezhad and Moradi,
2016).
The precautionary loss function covers the case of the scale parameter. There are, however,
many applications in which the parameter of interest is restricted to an interval (a, b) and
similar problems of severe consequences of boundary decisions can appear. The question of an
appropriate loss function choice for a parameter θ defined on the interval (a, b) has received
less attentions in the statistical literature compared to a scale parameter. At the same time,
its importance is acknowledged in many fields (see e.g. Aitchison, 1992; Pawlowsky-Glahn
and Egozcue, 2001; Mateu-Figueras et al., 2013, for examples in compositional data analysis).
Moreover, despite the variety of literature on families of loss functions for parameters restricted

2

to an interval and on corresponding improved Bayes estimators (see e.g. Kubokawa, 1994;
Marchand et al., 2004), they seem to be rarely applied in practice due their complexity and a
lack of closed-form solutions. The choice of loss functions for parameters defined on the interval
and on the positive real line is yet under-represented area in the Bayesian literature and the
usual mean still remains a common summary statistic.
The contribution of this work is twofold. Firstly, we provide a unified approach to define
symmetry of a loss function when a parameter space is restricted to a particular open subset
based on an appropriate definition of the distance. We outline that distances on corresponding
parameter spaces share the same property - an infinite penalization of the bounds which is also
known as the balance property (Mahmoudi and Zakerzadeh, 2011). We also recall two other
desirable properties of loss functions: convexity and invariance. Secondly, we propose several
loss functions which are as simple as the squared loss function (1.1), have explicit solutions for
the corresponding Bayes estimator and include the information about the restricted parameter
space in the corresponding Bayes estimator. In particular, we propose the scale invariant
generalization of the the precautionary loss function for a scale parameter θ ∈ (0, +∞) and the
interval squared loss function
Liq (θ, d) =

(d − θ)2
(d − a)(b − d)

for the parameter θ ∈ (a, b). We show that the Bayes estimator corresponding to the interval
squared loss function includes the Bayes estimator of the squared loss function (1.1) and of the
precautionary loss function as limiting cases. It is found that the interval squared and precautionary loss functions are both symmetric on corresponding parameter spaces and can be useful
in application areas where conservative estimates are preferred. We generalise the approach for
the multivariate parameter space and demonstrate how Bayes estimators obtained using the
proposed loss functions behave in three classic problems of Bayesian estimation compared to
standard approaches.
The rest of the paper is organized as follows. A historical perspective for the scale parameter
estimation and the case of symmetric loss function on the positive real line is given in Section 2.
Section 3 introduces the novel loss function for an interval. The multivariate generalizations
are given in Section 4. Three examples demonstrating novel loss functions and corresponding
Bayes estimators are considered in Section 5.

3

2

Scale Symmetry

2.1

A Historical Anecdote: Galileo on Scale Symmetry

In the Spring of 1627, a peculiar controversy

1

arose in one of Florence intellectual circles,

where noble gentlemen used to entertain erudite talks:
Un cavallo, che vale veramente cento scudi, da uno è stimato mille scudi e da un altro dieci
scudi: si domanda chi abbia di loro stimato meglio, e chi abbia fatto manco stravaganza nello
stimare.
The problem translates into: “A horse, whose true worth is one hundred scudi2 , is estimated
by someone to be one thousand scudi and by someone else to be ten scudi: the question is, who
gave a better estimate, and who instead gave a more extravagant estimate?”. It is formulated in
a letter from Andrea Gerini to Nozzolini, an erudite priest. Gerini wanted Nozzolini’s opinion
on a sentence by Galilei (1627), according to whom
. . . li due stimatori abbiano egualmente esorbitato e commesse eguali stravaganze nello stimare l’uno mille e l’altro dieci quello che realmente val cento,
which translates to: “The two estimators have been equally exorbitant and are responsible for
an equal extravagance by estimating, one thousand the former and ten the latter, what is really
worth one hundred”.
In the intense correspondence following the initial letters, Nozzolini argues that the estimates
should be evaluated according to the arithmetic proportion, whereas Galileo insists that the
correct method of judging is by geometric proportion. The crux of the problem is that the
estimand is a positive quantity, for which the geometric proportion seems more appropriate, as
wittingly argued by Galileo in another letter:
Se uno stimasse alta dugento braccia una torre, che veramente fusse alta cento, con quale
esorbitanza nel meno pareggerà il signor Nozzolini l’altra nel più ?
which translates as: “If one were to overestimate a one-hundred arm high tower as two-hundred
arm high, what underestimate would Nozzolini consider as equally deviating?”
1

the italic is a translation of a commentary to Galilei (1627) appearing in the edition of Galilei’s works
mentioned in the bibliography, from which all of the quotes are taken, following Scàrdovi (1980).
2
a monetary unit, literally, a shield

4

2.2

Scale Symmetry and Scale Invariance

Consider another example to illustrate Galileo’s position in modern statistical terms. Two
inferential procedures are based on two independent experiments:

1. Estimate µ ∈ (−∞, +∞) given i.i.d. Xi ∼ N (µ, σ 2 ), where σ 2 is known;
2. Estimate σ ∈ (0, +∞) given i.i.d. Yi ∼ N (µ, σ 2 ), where µ is known.

Denote the true parameters of interest for both questions by θ = µ = σ and Xi and Yk
independent for all i, k. Using the squared error loss (1.1), the decision µ = 0 in the first
experiment and σ = 0 in the second are equally penalized, while this should not be the case.
The claim of σ = 0 implies that the Y ’s are degenerate random variables, an extremely strong
statement which should be penalized similarly to the decision µ = +∞ or σ = +∞. The
squared error loss function imposes an infinite penalty to a boundary decision in the first
experiment and does not in the second one. While the decision σ = 0 is usually prevented by a
proper choice of the prior, the squared loss function does not imply that it should be avoided
and associated with a severe penalty. In contrast, an appropriate loss function imposes such a
penalty and can be also used to prevent boundary decisions. We define the properties of such
loss function for a scale parameter in this section.
Let us start with the following definition for the parameter define on the whole real line.
Definition 1. A loss function L(θ, d) is symmetric if, for every d1 , d2 and θ ∈ R1
(θ − d1 )2 = (d2 − θ)2

(2.1)

implies L(θ, d1 ) = L(θ, d2 ).

The Definition 1 implies that two decisions defined on the real line should be equally penalized by a symmetric loss function L(·, θ) if they stand on the same squared distance from
θ. Note that for d1 < d2 Equation (2.1) can be rewritten as θ =

d1 +d2
.
2

It follows that if θ is

the arithmetic mean of d1 and d2 then these decisions should be equally penalized. Clearly, the
squared error loss of equation (1.1) is symmetric on the real line by definition.
Then, Galilei’s claim of eguali stravaganze for a positive parameter θ can be expressed in
modern terminology as the requirement of a scale symmetric loss function, as in the following
definition.
5

Definition 2. A loss function L(θ, d) is scale symmetric if, for every d1 , d2 and θ ∈ R+
θ
d1
=
θ
d2

(2.2)

implies L(θ, d1 ) = L(θ, d2 ).

Equation (2.2) can be rewritten as θ =

√
d1 d2 or log(θ) =

log d1 +log d2
.
2

In other words, if θ is

the geometric mean of d1 and d2 , then these decisions should be equally penalized by a scale
symmetric loss function. As in Definition 1 of symmetry for the real line, two decisions are
symmetric if the parameter θ is their appropriate mean – geometric in this case as opposed to
arithmetic. This fact will be used for our proposal of the definition of symmetry on interval in
Section 3.
The definition of squared distance on the positive real line, R+ (Mateu-Figueras et al., 2013)
D+ (θ, d) = (log θ − log d)2

(2.3)

is known in Statistics as Brown’s loss function (Brown, 1968). Its motivation is to rescale the
positive real line to the whole real line via the log transformation and to use the squared error
loss function. Here, the logarithm function is a natural choice for a positive random variable.
Note that D+ (θ, d1 ) = D+ (θ, d2 ) implies either d1 = d2 or Equation (2.2). Therefore, we could
also restate Definition 2 in terms of D+ (·).
The Euclidean distance on the real line and D+ on the positive real line share the same
property - they infinitely penalize boundary values on corresponding parameter space. In case
of θ ∈ R, the squared distance Lq (θ, d) takes an infinite value when d = ±∞. For similar
reasons, we require that an appropriate loss function for a scale parameter should go to infinity
as the decision approaches the natural boundaries of the parameter space, to reproduce the
behaviour at ±∞ of the squared error loss function. A loss function with this property are also
called balanced (Mahmoudi and Zakerzadeh, 2011).
We recall one more property of loss functions for a parameter on the positive real line - the
scale invariance.
Definition 3. Loss function L(θ, d) is scale invariant if for every c > 0 and every pair (θ, d),
L(θ, d) = L(cθ, cd)

Then, the following result can be obtained.
6

Lemma 1. A loss function is scale invariant and scale symmetric if and only if it can be written
as a scalar function g such that g(d/θ) = g(θ/d).

Proof. A loss function L(θ, d) is scale invariant if and only if it is ratio-based, i.e. if and
only if there exists a scalar function g(x), x > 0 such that L(θ, d) = g(d/θ). Scale invariance
therefore implies L(θ, d) = g(d/θ) for some g, whereas by Definition (2) scale symmetry implies
L(θ, d) = L(θ, θ2 /d) and vice versa.

It follows that the squared loss function (1.1) is not scale symmetric, is not scale invariant
and does not penalize all boundaries for a scale parameter.

2.3

Symmetric Loss Functions on the Positive Real Line

In the modern statistical literature, the inadequacy of difference-based loss functions, like the
squared error loss, for estimating certain positive quantities has often been recognized (James
and Stein, 1961; Stein, 1964). Several alternative loss functions have been proposed, the
best-known being the normalized squared loss function proposed by Stein (1964) Lnq (θ, d) =
2

d
d
d
−
1
,
Stein’s
loss
(or
an
entropy
loss
function)
L
(θ,
d)
=
−
1
−
log
and Brown’s loss
S
θ
θ
θ
function (Brown, 1968) itself, D+ (θ, d). One can check that all of functions above are scale
invariant, but only Brown’s loss function is scale symmetric and infinitely penalizes the boundary decisions. Unfortunately, Brown’s loss function is not convex, a feature of loss functions
which is often required to represent risk aversion and for the sake of regularizing the associated
minimization problems. An unpleasant consequences is that, the Bayes estimator associated to
Brown’s loss function is usually difficult to calculate. Below we propose simple alternative loss
functions which share the desirable properties of a loss function on the positive line and have
explicit Bayes estimators.
We propose a family of loss functions defined for k > 0 as
 k  k
d
θ
Lk (θ, d) =
+
−2
θ
d

(2.4)

which are scale symmetric, scale invariant, convex, and which tend to infinity at the boundaries.
In this paper we focus on the case k = 1
(d − θ)2
L1 (θ, d) =
θd

(2.5)

which can be considered as a modification of the squared error loss function. The denominator is
again the squared distance, but the denominator guarantees the infinite penalization for d = 0.
7

It is easy to see that the loss function (2.5) is a scale invariant version of the precautionary loss
function (1.2).
Note that loss functions of similar-looking forms which have been considered in the literature, for example, the normalized squared loss Lnq and the relative squared error loss (Savchuk
and Tsokos, 2011) Lrq (θ, d) =

(d−θ)2
,
θ

do not tend to infinity at the boundary and are not scale

symmetric.

2.4

Scale Means (the Minimizers) and Scale Variances

Within the Bayesian approach, θ is a random parameter with a distribution which conveys
the uncertainty the researcher has in a given state of information (whether prior, posterior,
elicited, objective and so on). In such a scenario, a point summary of the distribution of
θ minimizing the risk (i.e. the expected loss) associated with a given loss function is often
required. Such a minimizer of an expected d is usually called a Bayes estimator. When a scale
symmetric loss function is used, we propose to call such minimizers scale means. In case of
convex loss functions, such as the novel ones listed in the previous section, minimization can
be performed explicitly, as in Theorem 2.1.
Theorem 2.1. Let θ be a positive random variable with a posterior density function f and
such that E(θk ) < ∞ and E(θ−k ) < ∞, where E denotes the posterior mean with respect to f
and k > 0. Then,
(a) Expectation of the loss function Lk (θ, d) (2.4) with respect to f is minimized by the
Bayes estimator (scale mean)
dˆk =



E(θk )
E(θ−k )

 2k1
.

(2.6)

(b) Expectation of the precautionary loss function Lsq (1.2) is minimized by the Bayes
estimator (scale mean)
dˆsq =

p
E(θ2 ),

(2.7)

for which the following bound holds: dˆsq ≥ E(θ).

Proof. The equalities results are proved by differentiating in d, the inequality can be found by
applying Jensen’s inequality (Jensen, 1906).

In a more fundamental Bayesian approach, a Bayes estimator is regarded only as a convenient summary of the posterior, and a loss function as a way to prescribe what kind of summary
8

is appropriate. Typically, a posterior expectation is used as the Bayes estimator, implying that
a squared loss function is being used. A second step is usually taken to accompany the Bayes
estimator with a measure of uncertainty of the posterior. If a posterior mean is used, a posterior
variance is usually presented. However, if a scale symmetric loss function is considered to be
a reasonable criterion for choosing an estimator, i.e. a number which minimizes a posterior
expected loss, then it is also reasonable to present the achieved minimum of the posterior expected loss as a second summary of the posterior. For the given loss functions (2.4) and (1.2),
particularly simple expected posterior losses can be obtained. In particular, for the loss function (2.4) such scale variance of order k of the random variable θ in Theorem 2.1(a) can be
p
written τ̂k (θ) := 2 E(θk )E(θ−k ) − 2, whereas the scale variance for the precautionary loss
p

function (1.2) in Theorem 2.1(b) is τ̂ (θ) = 2
E(θ2 ) − E(θ) .

3

Interval Symmetry
The approach used above for a positive parameter can be generalized to the parameter

defined on the interval (a, b). The issue of a restricted parameter space is not usually discussed
in the choice of the loss function and corresponding Bayes estimator: bounds are taken into
account through the prior specification only, then the squared loss function and posterior mean
(the corresponding Bayes estimator) are used (Marchand et al., 2004). Such solutions can
be suboptimal if boundary decisions are to be avoided. Below, we define the property of the
symmetry on an interval and show that the novel definition generalizes the cases of parameters
on the whole real line and on the positive real line. We provide the loss function with desirable
properties which is, again, a generalization of the squared loss function and the precautionary
loss function.

3.1

Symmetric Loss Functions on Intervals

Let us consider an inferential problem for which the parameter of interest lies in a particular
interval (a, b). Define the following transformation
logit(a,b) (x) = log

x−a
b−x

(3.1)

where a < x < b. Notice that, for a = 0 and b = 1, transformation (3.1) reduces to the common
logit transformation. Following the same line of reasoning as in Section 2.2, we introduce
the following definition of symmetric on the interval (a, b) loss function (or simply interval
symmetric loss function).
Definition 4. A loss function L(θ, d) is symmetric on the interval (a, b) if, for every choice of

9

d1 , d2 ∈ (a, b) and θ ∈ (a, b)
logit(a,b) (θ) =

logit(a,b) (d1 ) + logit(a,b) (d2 )
.
2

(3.2)

implies L(θ, d1 ) = L(θ, d2 ).

In other words, two decisions d1 and d2 should be penalized equally if the mean of their
logit transformation is equal to the logit transformation of θ. Lemma 2 justifies the use of the
logit transformation (3.1).
Lemma 2. Definition 4 is equivalent to Definition 1 when a → −∞ and b → +∞ and
equivalent to Definition 2 when a → 0 and b → +∞.

Proof. Condition (3.2) for a < d1 < d2 < b can be rewritten
θ = f (a, b, d1 , d2 ) ≡

ab − d1 d2 +

p
(d1 − a)(b − d1 )(d2 − a)(b − d2 )
.
a + b − d1 − d2

Obviously, θ is a symmetric function of d1 and d2 . Considering two limits
lim

a→−∞, b→+∞

f (a, b, d1 , d2 ) =

d1 + d2
,
2

lim

a→0, b→+∞

f (a, b, d1 , d2 ) =

p

d1 d2 ,

it can be easily seen that the definitions are equivalent.

It follows from Lemma 2 that Definition 4 is a convenient generalization of the definition of
symmetry and of scale symmetry.

3.2

An Interval Symmetric Loss Function and a Bayes estimator

As in the case of a positive parameter and scale symmetric loss functions, discussed in
Section 2, the approach of Brown (1968) of specifying a squared loss function after rescaling
the interval (a, b) to the real line via, for example, the logit transformation (3.1) provides the
loss function
LiB (θ, d) = logit(a,b) d − logit(a,b) θ

2

.

(3.3)

On the unit interval this loss function is equivalent to so-called Aitchison distance proposed
by Aitchison (1992) for parameters defined on a simplex. However, the loss function (3.3) is
not convex and its minimization problem does not have an explicit solution. As an alternative,

10

we propose the following loss function
Liq (θ, d) =

(d − θ)2
.
(d − a)(b − d)

(3.4)

which is interval symmetric and tends to infinity when the decision d tends to bounds a and b.
Note that loss function (3.4) for a = 0 and b = 1 looks similar to the well-known loss
function (d − θ)2 /(θ(1 − θ)) which, however, does not penalise boundary decisions.
The Bayes estimator corresponding to Liq is given in Theorem 3.1.
Theorem 3.1. Let θ ∈ (a, b) be a random variable with a posterior density function f and
E(θ2 ) < ∞ where E(·) denotes the expectation with respect to f . Then,
(a) the expectation of the interval symmetric loss function Liq (3.4) with respect to f is
minimized by the Bayes estimator
ab − E(θ2 ) +
dˆiq =

p
(E(θ2 ) − ab)2 − (a + b − 2E(θ))(2abE(θ) − (a + b)E(θ2 ))
a + b − 2E(θ)

(3.5)

(b) In the limiting case a → −∞ and b → +∞ estimator (3.5) minimizes the expectation
of squared loss function (1.1), and in the limiting case a → 0 and b → +∞ estimator (3.5)
minimizes the expectation of precautionary loss function (1.2).

Proof. (a) The equality is proved by differentiating in d the expected losses Liq (3.4).
(b) Denote the estimator (3.5) by d ≡ g(a, b, θ); then, taking the limits
lim

a→−∞, b→+∞

g(a, b, θ) = E(θ),

lim

a→0, b→+∞

g(a, b, θ) =

p
E(θ2 ),

it is easy to see that the obtained estimators are equivalent to the minimizers of the squared
loss function (1.1) and of the precautionary loss function (3.4), respectively. Note that dˆiq →
a+b
2

as E(θ) →

a+b
.
2

It follows from Theorem 3.1 that the Bayes estimator dˆiq includes the Bayes estimator under
the squared loss function and the precautionary loss function as special cases. Therefore, it
can be considered as a convenient generalization of these established loss functions. Although
it is not explicit in the form of dˆiq , it will be shown in the example below that it shares the

11

properties implied by the construction of the loss function. Particularly, the Bayes estimator
dˆiq tends to avoid the boundary decisions a and b.

4

Multivariate Generalizations
The definition of symmetry can be generalized to the case of a parameter belonging to a

subset of Rm by applying the same ideas to selected shapes of the parameter space as in the
following definition.
Definition 5. Let θ = (θ(1) , θ(2) , . . . , θ(m) )T be a parameter lying in one of the parameter spaces
(1)

(2)

(m)

Θ ⊂ Rm listed below. Let di = (di , di , . . . , di )T , i = 1, 2 be two vectors of decisions defined
on the same parameter space. A loss function L(θ, d) is a multivariate Θ−symmetric if the
equality
L(θ, d1 ) = L(θ, d2 )
is implied by each triple θ, d1 , d2 ∈ Θ satisfying the following respective equality of distances:

(a) when Θ = Rm (symmetry on Rm itself ):
v
v
uX
uX
m 

2
2
u m  (j)
u
(j)
(j)
t
d1 − θ
=t
d2 − θ(j) ;
j=1

j=1

(i)
(b) when Θ = Rm
> 0, i = 1, . . . , m} (scale symmetry on Rm
+ = {θ : θ
+ ):

v
u m
uX 2
t
log
j=1

(j)
d1
θ(j)

!

v
u m
uX 2
=t
log
j=1

(j)

d2
θj

!
;

(c) when Θ = {θ : (a1 < θ(1) < b1 ), . . . , (am < θ(m) < bm )} (symmetry on an Rm -rectangle):
v
uX
u m 
t
logit(a

(j)
d
j ,bj ) 1

− logit(aj ,bj )

θ(j)

2

j=1

v
uX
u m 
=t
logit(a

(j)
d
j ,bj ) 2

− logit(aj ,bj )

θ(j)

2

;

j=1

(d) when Θ = {θ : θ(1) > 0, θ(2) > 0, . . . , θ(m) > 0;

Pm

i=1

θ(i) = 1} (symmetry on the unit

simplex S m ):
v
!2 v
!2
u
u
(i)
(i)
(i)
(i)
u1 X
u1 X
θ
θ
d
d
1
2
t
log (j)
− log (j)
=t
log (j)
− log (j) .
m i<j
θ
m
θ
d1
d
2
i<j
The definition of the symmetric loss function in each case employs a distance corresponding to
12

the particular restricted space. While the distances in (a) - (c) are natural extensions of the
previously used, the definition in (d) is less straightforward. Definition 5(d) uses the Aitchison distance proposed by Aitchison (1992) and employed in the compositional data analysis.
Regarding properties of the proposed definition, Lemma 3 similar to Lemma 2 holds.
Lemma 3. Let θ = (θ(1) , θ(2) , . . . , θ(m) )T be a vector of parameter of interest such that θ(1) ∈
(1)

(2)

(m)

(a1 , b1 ), θ(2) ∈ (a2 , b2 ), . . . , θ(m) ∈ (am , bm ) and di = (di , di , . . . , di )T be a vector of corresponding decisions lying in corresponding intervals. Definition 5(c) is equivalent to Definition 5(a) when ai → −∞ and bi → ∞ for all i = 1, . . . , m and to Definition 5(b) when ai = 0
and bi → ∞ for all i = 1, . . . , m.

Following Brown (1968), the distances in the definition could be taken as corresponding symmetric loss functions. For example, in case (b) in Definition 5 one could define
(m)
D+

(θ, d) =

m
X
j=1

log

2



d(j)
θ(j)


.

(4.1)

At the same time, some convex alternatives could be considered when leading to simple solutions
of minimization problems. However, the search of the symmetric multivariate generalization
of our proposed loss functions Lk , Lsq and Liq seems to be a non-trivial one. We propose the
following loss functions for parameters with non-negative components.
(1)
(m) T
) ∈ Rm
Proposition 1. Let θ = (θ(1) , . . . , θ(m) )T ∈ Rm
+ . The loss
+ and d = (d , . . . , d

functions
(m)
L1

(θ, d) =

m
X
(d(j) − θ(j) )2
j=1

L(m)
sq

(θ, d) =

(4.2)

d(j) θ(j)

m
X
(d(j) − θ(j) )2
j=1

(4.3)

d(j)

are additive multivariate generalizations of the loss function L1 given in (2.4) and (1.2) respectively, which infinitely penalize each boundary decision d(j) = 0 and d(j) = ∞, j = 1, . . . , m.

(m)

Clearly, loss functions L1

(m)

and Lsq penalize the boundaries as desired and some good per-

formance of the corresponding estimators can be expected. However, the property of symmetry




(m)
(m)
is not satisfied. One can find two decisions d˜1 and d˜2 for which D+ θ, d˜1 = D+ θ, d˜2 ,




(m)
(m)
˜
˜
but L1
θ, d1 6= L1
θ, d2 . Even if the whole loss functions are not symmetric, they are
(2)

(2)

“component-wise” symmetric as shown above. The comparison of loss functions L1 , Lsq and
(2)

D+ for different values of decision d1 and d2 and fixed θ = (1, 2)T is given in the Figure 1. The
13

3.0
2.5
2.0

4

3

1.5

3

2

1.0

2

1

1

0
0.5

1.5

6

Decision 2

4

0.5

3.0
2.5

1

5

2.0

1.0

2

5

1.5

1.5

3

6

1.0

2.0

4

Loss function Lsq

Loss function L1

0.5

2.5

5

Decision 2

3.0

6

0.5

Decision 2

(2)

(2)

Distance

0

2.5

0.5

1.5

0

2.5

0.5

Decision 1

Decision 1

(2)

(2)

1.5

2.5

Decision 1

(2)

Figure 1: Contour plots of loss functions D+ , L1 , Lsq for the case m = 2 and θ = (1, 2)T .
(2)

proposed loss functions perform similar to the distance D+ , but have some more favourable
properties, like convexity.

5

Examples
Below, the performance of the novel Bayes estimators in three classic examples is demon-

strated. The mean squared error (MSE) is chosen to compare the different estimators on
common grounds. In general, this choice is not favourable to the new proposals, since the MSE
is derived from the squared error loss.

5.1

Estimation of a Probability

An important example of a parameter defined on the finite interval [0, 1] is a probability.
In the presence of a binary random sample with an unknown probability of success a uniform
distribution, i.e. a Beta prior distribution B(1, 1) is often assumed, a proposal which dates
back to Laplace. Having observed x successes out of n trials, the posterior distribution is a
conjugate Beta distribution B(x + 1, n − x + 1). The estimator corresponding to the squared
error loss function (posterior mean) has the form p̂q =

x+1
.
n+2

Another widely used estimator is

the so-called “add two successes and two failures” Agresti-Coull estimator (Agresti and Coull,
1998) p̂AC =

x+2
.
n+4

Below we compare these approaches to the newly proposed estimator (3.5).

The symmetric optimal Bayes estimator (3.5) in the case a = 0 and b = 1 can be written
d=

E(θ2 ) −

p
E(θ2 )(1 − 2E(θ) + E(θ2 ))
2E(θ) − 1

where θ is a probability of success lying in the interval (0,1), over which a posterior distribution

14

is given. It is assumed that the extremes of the interval are not possible values for the parameter.
The first and second moments of a Beta distribution can be computed explicitly and plugged
in formula (3.5) to obtain the following interval symmetric optimal Bayes estimator
s
p̂iq =

1+

(n − x + 1)(n − x + 2)
(x + 1)(x + 2)

!−1
.

(5.1)

Simulated trials with sample size n = 15 are considered. On a grid of values θ ∈ (0, 1),
N = 109 trials were simulated. In each simulation the three estimators are computed; for each
value of θ, the MSE is
N
1 X (i)
(p̂k − θ)2 ,
M SEk ≡
N i=1
(i)

where p̂k is a corresponding value in the ith simulation and k = q, iq, AC corresponds to an

0.0

0.2

0.4

0.6

probability

0.8

1.0

Bias

−0.10

0.00 0.05 0.10

0.010
0.006

Variance

0.002

0.008
0.004

MSE

0.012

estimation method. The results are given in Figure 2.

0.0

0.2

0.4

0.6

probability

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

probability

Figure 2: MSE, variance and bias for the restricted symmetric squared error loss function
estimator p̂iq (solid), the squared error loss function estimator p̂q (dashed) and the AgrestiCoull estimator p̂AC (dotted). Results are based on n = 15 observations and 109 simulations.

The proposed estimator p̂iq outperforms (in terms of the MSE) the Bayes estimator obtained
using the squared error loss function p̂q in the interval θ ∈ (0.2, 0.8). The cost of this advantage
is the worse performance on the intervals close to the bounds as the proposed form of the loss
function penalizes boundary decisions and by that drives the final estimate away from them.
However, the proposed estimator outperforms the Agresti-Coull estimator p̂AC at the same
intervals θ ∈ (0, 0.2) and θ ∈ (0.8, 1). Thus, the proposed estimator might be considered as a
trade-off between currently used estimators p̂q and p̂AC , that outperforms p̂AC on bounds and
p̂q away from bounds.

15

In addition to the MSE, the associated confidence intervals and coverage probabilities are
extensively studied in the literature (see e.g. Brown et al., 2001). In particular, coverage
probabilities were shown to have an erratic behaviour and often to go below their nominal
level. Corrections were proposed by Agresti and Coull (1998). Confidence intervals can also be
constructed around our newly proposed point estimator p̂iq . The following confidence intervals
are compared via simulated coverage probabilities in Figure 3:

[1] Normal approximation confidence interval centred around p̂k , k = q, iq, AC as suggested
by Brown et al. (2001)
r
(k)
CIN

= p̂k ± z α2

p̂k (1 − p̂k )
,
n

where 1 − α is the confidence level.
[2] Wilson confidence interval centred around p̂AC (Wilson, 1927)
(AC)

CIW

√ r
x+2
n
x(n − x) 1
=
±2
+ .
n+4
n+2
n2
n

[3] Approximate confidence interval using the delta-method centred around the newly proposed p̂iq
(iq)
CID


2
q
∂f (x)
= p̂iq ± 2 V̂iq , V̂iq =
∂x

x=np̂iq

np̂iq (1 − p̂iq ), f (x) = p̂iq .

(q)

Using the normal approximation confidence interval, the coverage probability of CIN goes
(iq)

(AC)

below the nominal value for several values of θ. The coverage probabilities of CIN and CIN

also fluctuate but do not go below 0.95 for N = 15, which is a desirable property. While one
(iq)

can find combination of N and θ for which coverage probabilities CIN

(AC)

and CIN

might be

below 0.95 (Brown et al., 2001), it would be generally true that their coverage probabilities are
(q)

greater than of CIN for larger intervals of θ and are more robust. In addition, a comparison
between the normal approximation method (left panel of Figure 3) and the Wilson and delta
method intervals (right panel of Figure 3) supports the suggestion by Brown et al. (2001) that
the normal approximation method gives a portmanteau way to construct simple confidence
intervals with - on average - better coverage probabilities than more complicated methods.

5.2

Restricted Estimation of a Normal Distribution Mean

In the following example, it is demonstrated what benefits the proposed form of loss function (3.4) can provide in a Bayesian framework in the presence of the additional information
that the true parameter lies in an interval (a, b).
16

(iq)

(q)

(AC)

Figure 3: Left panel: Coverage probabilities of CIN (blue line), CIN (red line) and CIN
(purple line) using [1], the normal approximation interval. Right panel: Coverage probabilities
(iq)
(AC)
of CID (blue line) and CIW (purple line) using [2], the Wilson, and [3], the delta-method,
confidence intervals respectively. Results are based on n = 15 observations and 109 simulations.

The problem of restricted mean estimation has been known for a long time (e.g. Bartholomew,
1965) and has been extensively studied in the literature (Kumar and Tripathi, 2008). The previously proposed estimators were constructed using the squared error loss function and compared
by the Bayesian risk. For an extensive overview of the problem we refer the reader to Marchand et al. (2004) and for some recent generalizations to Marchand et al. (2011). Interestingly,
despite the variety of the literature on the problem and the fact that the Bayes estimator with
respect to the uniform prior distribution on (−a, a) outperforms uniformly the “unrestricted”
Bayes estimator under squared error loss function (Hartigan, 2004), the sample mean estimator
is still widely used in practice. For this reason, we propose our simple alternative and compare
it to the most commonly used estimators.
Consider the problem of estimating the mean µ of a normal sample of i.i.d. Xi ∼ N (µ, σ 2 ),
i = 1, . . . , n where σ 2 is known. Assume it is known that the parameter µ belongs to the
interval (−a, +a) where a > 0. A possible example is the estimation of the treatment effect
in paediatric studies of a clinical treatment that was already tested on adults. An investigator
might be sure that the same dosage of the drug as given to adults would cause a greater level of
toxicity in children. It is of interest how much one can gain by incorporating this information
in the estimator (3.5) compared to currently used approaches that use the squared error loss.

17

As stated above, the common way to incorporate this information in a Bayesian framework
is restricting the prior distribution for the parameter of interest µ to the interval (−a, +a)
and then using the squared error loss to obtain a point estimate (as a summary of a posterior
distribution). However, the information about the restricted space is used only on the prior
and ignored when choosing summary statistics, while the proposed form of loss function (3.4)
allows to incorporate it again. Moreover, in practice the prior information is often ignored and
the sample mean estimator (corresponding to Jeffrey’s prior) is used. Therefore, a comparison
of the proposed loss function and the currently utilized approaches (with and without the
incorporation the prior information) is of interest.
Consider samples of size n = 15 and alternative Bayes estimators as follows:

• Bayes estimator under Jeffrey’s prior g(µ) ∝ k

pn

σ2

for µ and the squared error loss

function. Denoted by J;
• Bayes estimator under the uniform prior U ∼ (−a, +a) for µ and squared error loss
function. Denoted by U1 .
• Bayes estimator (3.5) under the uniform prior U(−a, +a) for µ and the interval symmetric
squared error loss function. Denoted by U2 .

Alternatively to U2 , the Bayes estimator U20 defined by (3.5), but with a wider interval
(−1.25a, +1.25a) is also applied to investigate how a lower penalization of the bounds influences
the estimation. This wider interval could be considered as a conservative way to incorporate
information about the location of the parameter.
Two cases a = 2 and a = 4 are considered. The parameter σ 2 = 4 is assumed to be known
in both cases. As before, the M SE is used for a fair comparison of methods. The results for
106 replications for each value of µ are given in Figure 4.
Incorporating the interval information in the Bayes estimator allows for improvement if the
true value of the parameter µ does not lie close to the bound. In the case a = 2 estimator U2
outperforms U1 on the interval µ ∈ (−1, +1) and in the case a = 4 on the interval µ ∈ (−3, +3).
The same holds for estimator U20 , however, its MSE never falls below the level of the MSE of
estimator J. Clearly, the wider interval improves estimation on the bounds, at the cost of the
higher MSE in the middle. Note that the wider interval a = 4 corresponds to less amount of the
additional information and to a smaller benefit in the MSE comparing U1 , U2 and U20 against
18

Figure 4: MSE corresponding to different values of the restricted mean parameter µ with (a)
a = 2 and (b) a = 4 and the Bayes estimator U1 (solid), U2 (dashed) and U20 (dotted) and
simple mean estimator J (solid dashed). Results are based on 106 replications.
J. Overall, the Bayes estimator corresponding to the interval symmetric loss function avoids
the boundary decisions, improves the estimation if the parameters lies away from bounds and
can be recommended for the application if boundary decisions lead to severe consequences.

5.3

Bayesian Estimation of the Parameters of a Gamma Distribution

An important example of multidimensional restricted parameter estimation is a Gamma
distribution with positive shape and scale parameters α1 , α2 > 0. Bayesian inference for this
problem has been studied, for example, in Miller (1980) and methods for approximate computation of Bayes estimators were proposed using the Lindley approximation (Lindley, 1980).
(2)

We consider the function Lsq given in (4.3) to obtain Bayes estimators. As the loss function
(4.3) is the sum of the univariate precautionary loss functions (1.2), the following estimators
p
are used α̂i = E(αi2 ), i = 1, 2 where the expectation E is taken with respect to the posterior
density function.
Let us consider an experiment with sample size n = 15. The parameters of the Gamma distribution are varied over a grid, α1 , α2 ∈ (0, 10) and the performance of the different approaches
are compared by simulation. The two approaches compared use the same prior distribution for
parameters α1 and α2 using the following estimators:

19

1) Bayes estimator under the squared error loss function - the posterior mean
2) Bayes estimator under the multivariate precautionary loss function α̂i

We use non-informative Gamma priors with parameter (10−4 , 10−4 ) for each of the parameter
α1 and α2 and an approximate method proposed by Lindley (1980). We emphasize the choice
of the prior distribution is out of the scope of this paper and the goal is to compare two Bayes
estimators when all other parameters are equal. The difference between the MSE of estimators
based on method (1 against method 2) for parameters α1 and α2 in 109 simulations is given in
Figure 5.

Difference in MSE for α1

Difference in MSE for α2
10

0.30

8
6

0.05

0.20
0.15

4

0.10

0.25

0.10

2

6

0.15

α2

8

0.20

4

0.25

2

α2

10

0.30

0.05

0.00
2

4

6

8

0.00

10

2

α1

4

6

8

10

α1

Figure 5: Difference in the MSEs for parameters α1 and α2 for their different true values and
(2)
using Bayes estimator under the squared error loss function and Bayes estimator under Lsq .
Results are based on 109 replications.
The differences in the MSEs for both parameters are positive for all values of the true
parameters α1 and α2 . It means that the Bayes estimator from method 2) is associated with
smaller MSE than the Bayes estimator from method 1). The difference in the MSE increases
as the true value of the parameter increases. This result makes the proposed estimator and
(2)

the associated loss function Lsq good candidates for further investigation in multidimensional
estimation problems.

6

Discussion
The unified definition of a symmetric loss function in a restricted parameter space is in-

troduced in this paper. Scale symmetric and interval symmetric loss functions which share

20

desirable properties are provided. On the basis of three examples, we show that the corresponding Bayes estimators perform well when compared to the squared error loss and other
available estimators, and improve the estimation if the parameter lies away from bounds. The
novel Bayes estimators allow to avoid boundary decisions and can be recommended in applications where more conservative estimates are preferable. We would like to emphasize that
we restrict our choice to some specific loss functions, mainly due to their simplicity and easy
implementation. Alternative loss function sharing the stated properties can and should be
considered.
The proposed definitions were generalized for a subset of Rm , where distances on restricted
spaces could themselves be used as loss functions, which usually are non-convex and result in
non-explicit minimizers. While we have presented the modification of the squared error loss
function for a restricted univariate parameter defined on an interval, the equivalent multidimensional extension seems to be non-trivial and requires further study.

Acknowledgement
This project has received funding from the European Union’s Horizon 2020 research and
innovation programme under the Marie Sklodowska-Curie grant agreement No 633567 and by
the Medical Research Council (MR/M005755/1) and, in part, from Prof Jaki’s Senior Research
Fellowship (NIHR-SRF-2015-08-001) supported by the National Institute for Health Research.
The views expressed in this publication are those of the authors and not necessarily those of
the NHS, the National Institute for Health Research or the Department of Health.

References
Agresti, A. and Coull, B. A. (1998) Approximate is better than “exact” for interval estimation
of binomial proportions. The American Statistician, 52, 119–126.
Aitchison, J. (1992) On criteria for measures of compositional difference. Mathematical Geology,
24, 365–379.
Bartholomew, D. (1965) A comparison of some bayesian and frequentist inferences. Biometrika,
52, 19–36.
Brewster, J.-F. and Zidek, J. (1974) Improving on equivariant estimators. The Annals of
Statistics, 21–38.
Brown, L. (1968) Inadmissibility of the usual estimators of scale parameters in problems with
unknown location and scale parameters. The Annals of Mathematical Statistics, 39, 29–48.
21

Brown, L. D., Cai, T. T. and DasGupta, A. (2001) Interval estimation for a binomial proportion.
Statistical science, 101–117.
Galilei, G. (1627) Lettera (intorno la stima di un cavallo). Le opere di Galileo Galilei, Prima
edizione completa. Societa editrice fiorentina. Firenze.
Hartigan, J. (2004) Uniform priors on convex sets improve risk. Statistics & probability letters,
67, 285–288.
James, W. and Stein, C. (1961) Estimation with quadratic loss. In Proceedings of the fourth
Berkeley symposium on mathematical statistics and probability, vol. 1, 361–379.
Jensen, J. L. W. V. (1906) Sur les fonctions convexes et les inégalités entre les valeurs moyennes.
Acta mathematica, 30, 175–193.
Joseph, V. R. (2004) Quality loss functions for nonnegative variables and their applications.
Journal of Quality Technology, 36, 129.
Karimnezhad, A. and Moradi, F. (2016) Bayes, e-bayes and robust bayes prediction of a future observation under precautionary prediction loss functions with applications. Applied
Mathematical Modelling, 40, 7051–7061.
Karimnezhad, A., Niazi, S. and Parsian, A. (2014) Bayes and robust bayes prediction with an
application to a rainfall prediction problem. Journal of the Korean Statistical Society, 43,
275–291.
Kiapour, A. and Nematollahi, N. (2011) Robust bayesian prediction and estimation under a
squared log error loss function. Statistics & Probability Letters, 81, 1717–1724.
Kubokawa, T. (1994) A unified approach to improving equivariant estimators. The Annals of
Statistics, 290–299.
Kumar, S. and Tripathi, Y. M. (2008) Estimating a restricted normal mean. Metrika, 68,
271–288.
Lindley, D. V. (1980) Approximate bayesian methods. Trabajos de estadı́stica y de investigación
operativa, 31, 223–245.
Mahmoudi, E. and Zakerzadeh, H. (2011) An admissible estimator of a lower-bounded scale
parameter under squared-log error loss function. Kybernetika, 47, 595–611.
Marchand, É., Najafabadi, A. T. P. et al. (2011) Bayesian improvements of a mre estimator of
a bounded location parameter. Electronic Journal of Statistics, 5, 1495–1502.
22

Marchand, E., Strawderman, W. E. et al. (2004) Estimation in restricted parameter spaces: A
review. In A Festschrift for Herman Rubin, 21–44. Institute of Mathematical Statistics.
Mateu-Figueras, G., Pawlowsky-Glahn, V. and Juan José, E. (2013) The normal distribution
in some constrained sample spaces. SORT: statistics and operations research transactions,
37, 29–56.
Miller, R. B. (1980) Bayesian analysis of the two-parameter gamma distribution. Technometrics,
22, 65–69.
Norstrom, J. G. (1996) The use of precautionary loss functions in risk analysis. IEEE Transactions on reliability, 45, 400–403.
Pawlowsky-Glahn, V. and Egozcue, J. J. (2001) Geometric approach to statistical analysis on
the simplex. Stochastic Environmental Research and Risk Assessment, 15, 384–398.
Saint-Hilary, G., Robert, V., Jaki, T., Gasparini, M. and Mozgunov, P. (2018) A novel measure
of drug benefit-risk assessment based on scale loss score (slos). Major Revision in Statistical
Methods in Medical Research.
Savchuk, V. and Tsokos, C. P. (2011) Bayesian theory and methods with applications, vol. 1.
Springer Science & Business Media.
Scàrdovi, I. (1980) Appunti di statistica, vol. 1. Pàtron, Bologna.
Soliman, A. A. (2002) Reliability estimation in a generalized life-model with application to the
burr-xii. IEEE Transactions on Reliability, 51, 337–343.
Stein, C. (1964) Inadmissibility of the usual estimator for the variance of a normal distribution
with unknown mean. Annals of the Institute of Statistical Mathematics, 16, 155–160.
Wilson, E. B. (1927) Probable inference, the law of succession, and statistical inference. Journal
of the American Statistical Association, 22, 209–212.

23

