Order-Sensitivity and Equivariance of
Scoring Functions

arXiv:1711.09628v1 [] 27 Nov 2017

Tobias Fissler∗

Johanna F. Ziegel†

November 28, 2017

Abstract. The relative performance of competing point forecasts is usually measured in terms of loss or scoring functions. It is widely accepted
that these scoring function should be strictly consistent in the sense that the
expected score is minimized by the correctly specified forecast for a certain
statistical functional such as the mean, median, or a certain risk measure.
Thus, strict consistency opens the way to meaningful forecast comparison,
but is also important in regression and M-estimation. Usually strictly consistent scoring functions for an elicitable functional are not unique. To give
guidance on the choice of a scoring function, this paper introduces two additional quality criteria. Order-sensitivity opens the possibility to compare two
deliberately misspecified forecasts given that the forecasts are ordered in a
certain sense. On the other hand, equivariant scoring functions obey similar
equivariance properties as the functional at hand – such as translation invariance or positive homogeneity. In our study, we consider scoring functions for
popular functionals, putting special emphasis on vector-valued functionals,
e.g. the pair (mean, variance) or (Value at Risk, Expected Shortfall).
Keywords: Consistency; Decision theory; Elicitability; Equivariance; M-Estimation;
Order-Sensitivity; Point forecasts; Scoring functions; Translation invariance; Homogeneity
AMS 2010 Subject Classification: 62C99; 62F07; 62G99; 91B06

1. Introduction
From the cradle to the grave, human life is full of decisions. Due to the inherent nature
of time, decisions have to be made today, but at the same time, they are supposed to
∗

Imperial College London, Department of Mathematics, Section Statistics, 180 Queen’s Gate, London
SW7 2AZ, United Kingdom, e-mail: t.fissler@imperial.ac.uk
†
University of Bern, Department of Mathematics and Statistics, Institute of Mathematical Statistics and
Actuarial Science, Sidlerstrasse 5, 3012 Bern, Switzerland, e-mail: johanna.ziegel@stat.unibe.ch

1

account for unknown and uncertain future events. However, since these future events
cannot be known today, the best thing to do is to base the decisions on predictions for
these unknown and uncertain events. The call for and the usage of predictions for future
events is literally ubiquitous and even dates back to ancient times. In those days, dreams,
divination, and revelation were considered as respected sources for forecasts, with the
most prominent example being the Delphic Oracle which was not only consulted for
decisions of private life, but also for strategic political decisions concerning peace and
war. With the development of natural sciences, mathematics, and in particular statistics
and probability theory, the ancient metaphysical art of making qualitative forecasts
turned into a sophisticated discipline of science adopting a quantitative perspective.
Subfields such as meteorology, mathematical finance, or even futurology evolved.
Acknowledging that forecasts are inherently uncertain, two main questions arise:
(i) How good is a forecast in absolute terms?
(ii) How good is a forecast in relative terms?
While question (i) deals with forecast validation, this paper focuses on some aspects of
question (ii) which is concerned with forecast selection, forecast comparison, or forecast
ranking. Specifically, we present results on order-sensitivity and equivariance of consistent scoring functions for elicitable functionals. These results may provide guidance for
choosing a specific scoring function for forecast comparison within the large class of all
consistent scoring functions for an elicitable functional of interest.
We adopt the general decision-theoretic framework following Gneiting (2011); cf. Savage
(1971); Osband (1985); Lambert et al. (2008). For some number n ≥ 1, one has
(a) observed ex post realizations y1 , . . . , yn of a time series (Yt )t∈N , taking values in an
observation domain O with a σ-algebra O;
(b) a family F of probability distributions on (O, O), containing the (conditional) distributions of Yt ;
(i)

(i)

(c) ex ante forecasts x1 , . . . , xn , i ∈ {1, . . . , m} of m ≥ 1 competing experts / forecasters
taking values in an action domain A ⊆ Rk for some k ≥ 1;
(d) a scoring (or loss) function S : A × O → R. The scoring function is assumed to be
negatively oriented, that is, if a forecaster reports the quantity x ∈ A and y ∈ O
materializes, she is assigned the penalty S(x, y) ∈ R.
The observations yt can be real-valued (GDP growth for one year, maximal temperature
of one day), vector-valued (wind-speed, weight and height of persons), functional-valued
(path of the exchange rate Euro–Swiss franc over one day), or also set-valued (area of
rain on a given day, area affected by a flood). In this article, we focus on point forecasts
that may be vector-valued, which is why we assume A ⊆ Rk for some k ≥ 1 and we
equip the Borel set A with the Borel σ-algebra. One is typically interested in a certain
statistical property of the underlying (conditional) distribution Ft of Yt . We assume
that this property can be expressed in terms of a functional T : F → A such as the
mean, a certain quantile, or a risk measure. Examples of vector-valued functionals are
the covariance matrix of a multivariate observation or a vector of quantiles at different

2

levels. Common examples for scoring functions are the absolute loss S(x, y) = |x − y|,
the squared loss S(x, y) = (x − y)2 (for A = O = R), or the absolute percentage loss
S(x, y) = |(x − y)/y| (for A = O = (0, ∞)).
Forecast comparison is done in terms of realized scores
n

S̄(i)
n

1X
(i)
S(xt , yt ),
=
n t=1

i ∈ {1, . . . , m}.

(1.1)

That is, a forecaster is deemed to be the better the lower her realized score is. However,
there is the following caveat: The forecast ranking in terms of realized scores not only
depends on the forecasts and the realizations (as it should definitely be the case), but
also on the choice of the scoring function. In order to avoid impure possibilities of
manipulating the forecast ranking ex post with the data at hand, it is necessary to
specify a certain scoring function before the inspection of the data. A fortiori, for the
sake of transparency and in order to encourage truthful forecasts, one ought to disclose
the choice of the scoring function to the competing forecasters ex ante. But still, the
optimal choice of the scoring function remains an open problem. One can think of two
situations:
(i) A decision-maker might be aware of his actual economic costs of utilizing misspecified forecasts. In this case, the scoring function should reflect these economic
costs.
(ii) The actual economic costs might be unclear and the scoring function might be
just a tool for forecast ranking. However, the directive is given in terms of the
functional T : F → A one is interested in.
For situation (i) described above, one should use the readily economically interpretable
cost or scoring function. Therefore, the only concern is situation (ii). In this paper,
we consider predictions in a one-period setting, thus, dropping the index t. This is
justified by our objectives to understand the properties of scoring functions S which
do not change over time and is common in the literature (Murphy and Daan, 1985;
Diebold and Mariano, 1995; Lambert et al., 2008; Gneiting, 2011).
Assuming the forecasters are homines oeconomici and adopting the rationale of expected utility maximization, given a concrete scoring function S, the most sensible action
consists in minimizing the expected score EF S(x, Y ) with respect to the forecast x, where
Y follows the distribution F , thus issuing the Bayes act arg minx∈A EF S(x, Y ). Hence,
a scoring function should be incentive compatible in that it encourages truthful and
honest forecasts. In line with Murphy and Daan (1985) and Gneiting (2011), we make
the following definition.
Definition 1.1 (Consistency and elicitability). A scoring function is a map S : A × O →
R that is F-integrable.1 It is F-consistent for a functional T : F → A if
S̄(T (F ), F ) ≤ S̄(x, F )
1

(1.2)

We say that a function a : O → R is F-integrable if it is F -integrable for each F ∈ F. A function
g : A × O → R is F-integrable if g(x, ·) is F-integrable for each x ∈ A.

3

for all F ∈ F and for all x ∈ A, where S̄(x, F ) := EF S(x, Y ). It is strictly F-consistent
for T if it is F-consistent for T and if equality in (1.2) implies x = T (F ). A functional
T : F → A is called elicitable, if there exists a strictly F-consistent scoring function for
T.
Clearly, elicitability and consistent scoring functions are naturally linked also to estimation problems, in particular, M-estimation (Huber, 1964; Huber and Ronchetti, 2009)
and regression with prominent examples being ordinary least squares, quantile, or expectile regression (Koenker, 2005; Newey and Powell, 1987).
The necessity of utilizing strictly consistent scoring functions for meaningful forecast
comparison is impressively demonstrated in terms of a simulation study in Gneiting
(2011). However, for a given functional T : F → A, there is typically a whole class
of strictly consistent scoring functions for it, such as all Bregman functions in case of
the mean (Savage, 1971); further examples are given below. Patton (2017) shows that
the forecast ranking based on (1.1) may depend on the choice of the strictly consistent
scoring function for T in finite samples, and even at the population level if we compare
two imperfect forecasts with each other.
Therefore, we naturally have a threefold elicitation problem:
(i) Is T elicitable?
(ii) What is the class of strictly F-consistent scoring functions for T ?
(iii) What are distinguished strictly F-consistent scoring functions for T ?
Even though the denomination and the synopsis of the described problems under the
term ‘elicitation problem’ are novel, there is a rich strand of literature in mathematical
statistics and economics concerned with the threefold elicitation problem. Foremost,
one should mention the pioneering work of Osband (1985), establishing a necessary
condition for elicitability in terms of convex level sets of the functional, and a necessary representation of strictly consistent scoring functions, known as Osband’s principle
(Gneiting, 2011). Whereas the necessity of convex level sets holds in broad generality,
Lambert (2013) could specify sufficient conditions for elicitability for functionals taking
values in a finite set, and Steinwart et al. (2014) showed sufficiency of convex level sets
for real-valued functionals satisfying certain regularity conditions. Moments, ratios of
moments, quantiles, and expectiles are in general elicitable, whereas other important
functionals such as variance, Expected Shortfall or the mode functional are not (Savage,
1971; Osband, 1985; Weber, 2006; Gneiting, 2011; Heinrich, 2014).
Concerning subproblem (ii) of the elicitation problem, Savage (1971), Reichelstein and Osband
(1984), Saerens (2000), and Banerjee et al. (2005) gave characterizations for strictly consistent scoring functions for the mean functional of a one-dimensional random variable
in terms of Bregman functions. Strictly consistent scoring functions for quantiles have
been characterized by Thomson (1979) and Saerens (2000). Gneiting (2011) provides a
characterization of the class of strictly consistent scoring functions for expectiles. The
case of vector-valued functionals apart from means of random vectors has been treated
substantially less than the one-dimensional case (Osband, 1985; Banerjee et al., 2005;
Lambert et al., 2008; Frongillo and Kash, 2015a,b; Fissler and Ziegel, 2016a).

4

The strict consistency of S only justifies a comparison of two competing forecasts if
one of them reports the true functional value. If both of them are misspecified, it is
per se not possible to draw a conclusion which forecast is ‘closer’ to the true functional
value by comparing the realized scores. To this end, some notions of order-sensitivity
are desirable. According to Lambert (2013) we say that a scoring function S is F-ordersensitive for a one-dimensional functional T : F → A ⊆ R if for any F ∈ F and any
x, z ∈ A such that either z ≤ x ≤ T (F ) or z ≥ x ≥ T (F ), then S̄(x, F ) ≤ S̄(z, F ). This
means, if a forecast lies between the true functional value and some other forecast, then
issuing the forecast in-between should yield a smaller expected score than issuing the
forecast further away. In particular, order-sensitivity implies consistency. Vice versa,
under weak regularity conditions on the functional, strict consistency also implies ordersensitivity if the functional is real-valued; see Nau (1985, Proposition 3), Lambert (2013,
Proposition 2), Bellini and Bignozzi (2015, Proposition 3.4).
This article is dedicated to a thorough investigation of order-sensitive scoring functions for vector-valued functionals, thus contributing to a discussion of subproblem (iii)
of the elicitation problem. Furthermore, we investigate to which extent invariance or
equivariance properties of elicitable functionals are reflected in their respective consistent
scoring functions.
Lambert et al. (2008) introduced a notion of componentwise order-sensitivity for the
case of A ⊆ Rk . Friedman (1983) and Nau (1985) considered similar questions in the
setting of probabilistic forecasts, coining the term of effectiveness of scoring rules which
can be described as order-sensitivity in terms of a metric. In Section 3, we consider three
notions of order-sensitivity in the higher-dimensional setting: metrical order-sensitivity,
componentwise order-sensitivity, and order-sensitivity on line segments. We discuss their
connections and give conditions when such scoring functions exist and of what form they
are for the most relevant functionals, such as vectors of quantiles, expectiles, ratios of
expectations, the pair of mean and variance, and the pair consisting of Value at Risk
and Expected Shortfall, two important risk measures in banking and insurance.
Complementing our results on order-sensitivity, in Section 2, we consider the analytic
properties of the expected score x 7→ S̄(x, F ), x ∈ A ⊆ Rk , for some scoring function
S and some distribution F ∈ F. The (strict) consistency of S for some functional T is
equivalent the expected score having a (unique) global minimum at x = T (F ). Ordersensitivity ensures monotonicity properties of the expected score. As a technical result,
we show that under weak regularity assumptions on T , the expected score of a strictly
consistent scoring function has a unique local minimum – which, of course, coincides
with the global minimum at x = T (F ). Accompanied with a result on self-calibration, a
continuity property of the inverse of the expected score, which ensures that the minimum
of the expected score is well-separated in the sense of van der Vaart (1998), these two
findings may be of interest on their own right in the context of M-estimation.
In Section 4, we consider functionals that have an invariance or equivariance property such as translation invariance or homogeneity. It is a natural question whether a
functional T that is, for example, translation equivariant has a consistent scoring function that respects this property in the sense that if we evaluate forecast performance of

5

translated predictions and observations, the ranking of predictive performance remains
the same as that of the original data. In parametric estimation problems, such a scoring
functions may allow to translate the data without affecting the estimated parameter values. For one-dimensional functionals, invariance of the scoring function often determines
it uniquely up to equivalence while this is not necessarily the case for higher-dimensional
functionals (Proposition 4.7 and Corollary 4.12).

2. Analytic properties of expected scores
2.1. Monotonicity
Definition 2.1 (Mixture-continuity). Let F be convex. A functional T : F → A ⊆ Rk
is called mixture-continuous if for all F, G ∈ F the map
[0, 1] → R,

λ 7→ T ((1 − λ)F + λG)

is continuous.
It is appealing that one does not have to specify a topology on F to define mixturecontinuity because it suffices to work with the induced Euclidean topology on [0, 1] and
on A ⊆ Rk .
It turns out that mixture-continuity of a functional is strong enough to imply ordersensitivity in the case of one-dimensional functionals (see Nau (1985, Proposition 3),
Lambert (2013, Proposition 2), Bellini and Bignozzi (2015, Proposition 3.4)), and desirable monotonicity properties of the expected scores also in higher dimensions (Propositions 2.5 and 2.7). At the same time, numerous functionals of applied relevance are
mixture-continuous, and we start by giving examples and a sufficient condition (Proposition 2.2).
It is straight forward to see that the ratio of expectations is mixture-continuous.
Moreover, by the implicit function theorem, one can verify the mixture-continuity of
quantiles and expectiles directly under appropriate regularity conditions (e.g., in the
case of quantiles, all distributions in F should be C 1 with non-vanishing derivatives).
Generalizing Bellini and Bignozzi (2015, Proposition 3.4c), we give a sufficient criterion
for mixture-continuity in the next proposition. Our version is not restricted to distributions with compact support (however, the image of the functional must be bounded),
and we formulate the result for k-dimensional functionals.
Proposition 2.2. Let T : F → Rk be an elicitable functional with a strictly F-consistent
scoring function S : Rk × O → R such that S̄(·, F ) is continuous for all F ∈ F. Then
T is mixture-continuous on any F0 ⊆ F such that F0 is convex and the image T (F0 ) is
bounded.
Proof. Let F0 ⊆ F be convex such that T (F0 ) ⊆ [−C, C]k for some C > 0. Let
F, G ∈ F0 . Define hF,G : [−C, C]k × [0, 1] → R via
hF,G (x, λ) = S̄(x, (1 − λ)F + λG) = (1 − λ)S̄(x, F ) + λS̄(x, G).

6

Then hF,G is jointly continuous, and due to the strict consistency
T ((1 − λ)F + λG) = arg min hF,G (x, λ).
x∈[−C,C]k

By virtue of the Berge Maximum Theorem (Aliprantis and Border, 2006, Theorem 17.31
and Lemma 17.6), the function λ 7→ arg minx∈[−C,C]k hF,G (x, λ) is continuous.
Similarly to the original proof of Bellini and Bignozzi (2015), a sufficient criterion
for the continuity of S̄(·, F ) for any F ∈ F is that for all y ∈ O, the score S(x, y) is
quasi-convex and continuous in x.2
Recall that, under appropriate regularity conditions on F, the asymmetric piecewise
linear loss Sα (x, y) = (1{y ≤ x} − α)(x − y) and the asymmetric piecewise quadratic
loss Sτ (x, y) = |1{y ≤ x} − τ |(x − y)2 are strictly consistent scoring functions for the
α-quantile and the τ -expectile, respectively, and both, Sα as well as Sτ , are continuous
in their first argument and convex. Hence, Proposition 2.2 yields that both quantiles
and expectiles are mixture-continuous.
Steinwart et al. (2014) used Osband’s principle (Osband, 1985) and the assumption
of continuity of T with respect to the total variation distance to show order-sensitivity.
Bellini and Bignozzi (2015) showed that the weak continuity of a functional T implies
its mixture-continuity. Consequently, one can also derive the order-sensitivity in the
framework of Steinwart et al. (2014) directly using only mixture-continuity.
Lambert (2013) showed that it is a harder requirement to have order-sensitivity if
T (F) is discrete. Then both approaches, invoking Osband’s principle or using mixturecontinuity, do not work because the interior of the image of T is empty. Moreover,
mixture-continuity implies that the functional is constant (such that only trivial cases
can be considered). Furthermore, it is proven in Lambert (2013) that for a functional
T with a discrete image, all strictly consistent scoring functions are order-sensitive if
and only if there is one order-sensitive scoring function for T .In particular, there are
functionals admitting strictly consistent scoring functions that are not order-sensitive,
one such example being the mode functional.3
Let us turn attention to vector-valued functionals now. To understand the monotonicity properties of the expected score of a mixture-continuous elicitable functional T : F →
A ⊆ Rk , it is useful to consider paths γ : [0, 1] → A ⊆ Rk , γ(λ) = T (λF + (1 − λ)G)
for F, G ∈ F. If T is elicitable, a classical result asserts that T necessarily has convex
level sets (Gneiting, 2011, Theorem 6). This implies that the level sets of γ can only be
2

We remark that for A ⊆ R, if a scoring function S is strictly Fp -consistent for some functional
T : Fp → A where Fp = {δy : y ∈ O} consists of all point measures on O, then the quasi-convexity of
x 7→ S(x, y) for all y ∈ O is equivalent to the Fp -order-sensitivity of S for T .
3
Note that due to Proposition 1 in Heinrich (2014), the mode functional is elicitable relative to the class
of probability measure F containing unimodal discrete measures. Moreover, interpreting the mode
functional as a set-valued functional, it is elicitable in the sense of Gneiting (2011, Definition 2). A
strictly F-consistent scoring function is given by S(x, y) = 1{x 6= y}. The main result of Heinrich
(2014) is that the mode functional is not elicitable relative to the class F of unimodal probability
measures with Lebesgue densities.

7

closed intervals including the case of singletons and the empty set. This rules out loops
and some other possible pathologies of γ. Furthermore, under the assumption that T
is identifiable as defined below, one can even show that the path γ is either injective or
constant.
Definition 2.3 (Identifiability). Let A ⊆ Rk . An F-integrable function V : A × O → Rk
is said to be an F-identification function for a functional T : F → A ⊆ Rk if
V̄ (T (F ), F ) = 0
for all F ∈ F. Furthermore, V is a strict F-identification function for T if V̄ (x, F ) = 0
implies x = T (F ) for all F ∈ F and for all x ∈ A. A functional T : F → A ⊆ Rk is said
to be identifiable, if there exists a strict F-identification function for T .
In line with Gneiting (2011, Section 2.4), one can often obtain an identification function as the gradient of a sufficiently smooth scoring function. However, the converse
intuition is not so clear – at least in the higher dimensional setting k > 1: Not all strict
identification functions can be integrated to a strictly consistent scoring function. They
have to satisfy the usual integrability conditions (Königsberger, 2004, p. 185); see also
Fissler and Ziegel (2016a, Corollary 3.3) and the discussion thereafter.
Lemma 2.4. Let F be convex and T : F → A ⊆ Rk be identifiable with a strict Fidentification function V : A × O → Rk . Then for any F, G ∈ F, the path γ : [0, 1] → A,
γ(λ) = T (λF + (1 − λ)G), is either constant or injective.
Proof. Let F, G ∈ F such that t = T (F ) = T (G). For any λ ∈ [0, 1], one has V̄ (t, λF +
(1 − λ)G) = λV̄ (t, F ) + (1 − λ)V̄ (t, G) = 0. Since V is a strict F-identification function
for T , t = γ(λ) for all λ ∈ [0, 1].
Now let T (F ) 6= T (G) and let 0 ≤ λ < λ′ ≤ 1. Since V is a strict F-identification
function, V̄ (T (F ), G) 6= 0 (and symmetrically V̄ (T (G), F ) 6= 0.) Assume that γ(λ) =
γ(λ′ ). Define Hλ = λF + (1 − λ)G, Hλ′ = λ′ F + (1 − λ′ )G. There are µ, µ′ ∈ R such
that F = µHλ + (1 − µ)Hλ′ and G = µ′ Hλ + (1 − µ′ )Hλ′ . Hence,
V̄ (γ(λ), F ) = µV̄ (γ(λ), Hλ ) + (1 − µ)V̄ (γ(λ), Hλ′ ) = 0,
and similarly V̄ (γ(λ), G) = 0. Consequently, T (F ) = γ(λ) = T (G), which is a contradiction to the assumption that T (F ) 6= T (G). This implies that γ(λ) 6= γ(λ′ ).
Proposition 2.5. Let F be convex and T : F → A ⊆ Rk be mixture-continuous and
surjective. Let S : A × O → R be strictly F-consistent for T . Then for each F ∈ F,
t = T (F ) and each x ∈ A, x 6= t there is a continuous path γ : [0, 1] → A such that
γ(0) = x, γ(1) = t, and the function [0, 1] ∋ λ 7→ S̄(γ(λ), F ) is decreasing. Additionally,
for 0 ≤ λ < λ′ ≤ 1 such that γ(λ) 6= γ(λ′ ) it holds that S̄(γ(λ), F ) > S̄(γ(λ′ ), F ).
Proof. Let F ∈ F, t = T (F ) and x 6= t. Then there is some G ∈ F with x = T (G).
Define γ : [0, 1] → A, λ →
7 T (λF +(1−λ)G). Clearly, γ(0) = x and γ(1) = t. Due to the

8

mixture-continuity of T , the path γ is also continuous. The rest follows along the lines
of the proof of Nau (1985, Proposition 3). Let 0 ≤ λ < λ′ ≤ 1. If γ(λ) = γ(λ′ ), there is
nothing to show. So assume that γ(λ) 6= γ(λ′ ). Define Hλ = λF + (1 − λ)G, and Hλ′
analogously. Then, for µ := (λ′ − λ)/(1 − λ) ∈ (0, 1], it holds that Hλ′ = µF + (1 − µ)Hλ .
The strict consistency of S implies that
µS̄(γ(λ′ ), F ) + (1 − µ)S̄(γ(λ′ ), Hλ ) = S̄(γ(λ′ ), Hλ′ )
< S̄(γ(λ), Hλ′ ) = µS̄(γ(λ), F ) + (1 − µ)S̄(γ(λ), Hλ ) ,
which is equivalent to

1−µ
S̄(γ(λ′ ), Hλ ) − S̄(γ(λ), Hλ ) < S̄(γ(λ), F ) − S̄(γ(λ′ ), F ) .
µ

By strict consistency of S, the left-hand side is non-negative yielding the assertion.
Remark 2.6. (i) Proposition 2.5 remains valid if S is only F-consistent. Then, we
merely have that the function [0, 1] ∋ λ 7→ S̄(γ(λ), F ) is decreasing, so the last
inequality in Proposition 2.5 is not necessarily strict.
(ii) If one assumes in Proposition 2.5 that T is also identifiable, one can use the injectivity of γ implied by Lemma 2.4 to see that the function [0, 1] ∋ λ 7→ S̄(γ(λ), F )
is strictly decreasing.
Under certain (weak) regularity conditions, the expected scores of a strictly consistent
scoring function has no other local minimum apart from the global one at x = T (F ).
Proposition 2.7. Let F be convex and T : F → A ⊆ Rk be mixture-continuous and
surjective. If S : A × O → R is strictly F-consistent for T , then for all F ∈ F the
expected score S̄(·, F ) : A → R has only one local minimum which is at x = T (F ).
Proof. Let F ∈ F with t = T (F ). Due to the strict F-consistency of S, the expected
score S̄(·, F ) has a local minimum at t. Assume there is another local minimum at some
x 6= t. Then there is a distribution G ∈ F with x = T (G). Consider the path γ : [0, 1] →
A, λ 7→ T (λF + (1 − λ)G). Due to Proposition 2.5 the function λ 7→ S̄(γ(λ), F ) is
decreasing and strictly decreasing when we move on the image of the path from x to t.
Hence S̄(·, F ) cannot have a local minimum at x = γ(0).

2.2. Self-calibration
With Proposition 2.5 it is possible to prove that, under mild regularity conditions, strictly
consistent scoring functions are self-calibrated which turns out to be useful in the context
of M-estimation.

9

Definition 2.8 (Self-calibration). A scoring function S : A × O → R is called F-selfcalibrated for a functional T : F → A ⊆ Rk with respect to a norm4 k · k on A if for all
ε > 0 and for all F ∈ F there is a δ = δ(ε, F ) > 0 such that for all x ∈ A and t = T (F )
S̄(x, F ) − S̄(t, F ) < δ

=⇒

kt − xk < ε.

The notion of self-calibration was introduced by Steinwart (2007) in the context of
machine learning. In a preprint version of Steinwart et al. (2014),5 the authors translate
this concept to the setting of scoring functions as follows (using our notation):
“For self-calibrated S, every δ-approximate minimizer of S̄(·, F ), approximates the
desired property T (F ) with precision not worse than ε. [. . . ] In some sense order
sensitivity is a global and qualitative notion while self-calibration is a local and
quantitative notion.”

In line with this quotation, self-calibration can be considered as the continuity of the
inverse of the expected score S̄(·, F ) at the global minimum x = T (F ) – and as such, it is
a local property of the inverse. This property ensures that convergence of the expected
score to its global minimum implies convergence of the forecast to the true functional
value. On the other hand, self-calibration of a scoring function S is equivalent to the
fact that the argmin T (F ) of the expected score S̄(·, F ) is a well-separated point of
minimum in the sense of van der Vaart (1998, p. 45) – as such being a global property
of the expected score itself. That means that for any ε > 0
inf{S̄(x, F ) : kT (F ) − xk ≥ ε} > S̄(T (F ), F ).
It is relatively straight forward to see that self-calibration implies strict consistency:
Let S be F-self-calibrated for T , F ∈ F, t = T (F ) and x ∈ A with x 6= t. Then for
ε := kx − tk/2 > 0 there is a δ > 0 such that S̄(x, F ) − S̄(t, F ) ≥ δ > 0.
In the preprint version of Steinwart et al. (2014) it is shown for k = 1 that ordersensitivity implies self-calibration. The next Proposition shows that the kind of ordersensitivity given by Proposition 2.5 also implies self-calibration for k ≥ 1.
Proposition 2.9. Let F be convex, A ⊆ Rk be closed, and T : F → A be a surjective
and mixture-continuous functional. If S : A × O → R is strictly F-consistent for T and
S̄(·, F ) : A → R is continuous for all F ∈ F, then S is F-self-calibrated for T .
Proof. Let F ∈ F, t = T (F ) and ε > 0. Define
δ := min{S̄(z, F ) − S̄(t, F ) : z ∈ A, kz − tk = ε}.
Due to the continuity of S̄(·, F ), the minimum is well-defined and, as a consequence of
the strict F-consistency of S for T , δ is positive. Let x ∈ A. If kx − tk = ε, we have,
4

It is straight forward to use a metric instead of a norm on A but in this article we only consider
A ⊆ Rk , so we did not see any benefit in considering this more general case. See also the discussion
before Definition 3.4.
5
Available at http://users.cecs.anu.edu.au/~ williams/papers/P196.pdf

10

by the definition of δ, that S̄(x, F ) − S̄(t, F ) ≥ δ. Assume that kx − tk > ε. Then there
is a distribution G ∈ F with T (G) = x. Due to Proposition 2.5 there is a continuous
path γ : [0, 1] → A such that γ(0) = x, γ(1) = t and such that S̄(γ(λ), F ) is decreasing
in λ. Moreover, if λ < λ′ such that γ(λ) 6= γ(λ′ ) it holds that S̄(γ(λ), F ) > S̄(γ(λ′ ), F ).
Due to the continuity of γ there is some x′ ∈ γ([0, 1]) with kx′ − tk = ε. Then we obtain
S̄(x, F ) − S̄(t, F ) > S̄(x′ , F ) − S̄(t, F ) ≥ δ.
We end this subsection about self-calibration by demonstrating its applicability in the
context of M-estimation.
Theorem 2.10. Let S : A × O → R be an F-self-calibrated scoring function for a functional T : F → A ⊆ Rk . Then, the following assertion holds for all F ∈ F. If Y1 , Y2 , . . .
is a sequence of random variables with distribution F ∈ F such that
n

1X
P
sup
S(x, Yi ) − S̄(x, F ) −→ 0,
x∈A n
i=1

then

n

1X
P
S(x, Yi ) −→ T (F ).
arg min
n
x∈A
i=1

Proof. This is a direct consequence of van der Vaart (1998, Theorem 5.7).

3. Order-sensitivity
3.1. Different notions of order-sensitivity
The idea of order-sensitivity is that a forecast lying between the true functional value
and some other forecast is also assigned an expected score lying between the two other
expected scores. If the action domain is one dimensional, there are only two cases to
consider: both forecasts are on the left-hand side of the functional value or on the righthand side. However, if A ⊆ Rk for k ≥ 2, the notion of ‘lying between’ is ambiguous.
Two obvious interpretations for the multidimensional case are the componentwise interpretation and the interpretation that one forecast is the convex combination of the true
functional value and the other forecast.
Definition 3.1 (Componentwise order-sensitivity). A scoring function S : A × O → R
is called componentwise F-order-sensitive for a functional T : F → A ⊆ Rk , if for all
F ∈ F, t = T (F ) and for all x, z ∈ A we have that:
For all m ∈ {1, . . . , d} : zm ≤ xm ≤ Tm (F ) or zm ≥ xm ≥ Tm (F )
=⇒ S̄(x, F ) ≤ S̄(z, F ). (3.1)
Moreover, S is called strictly componentwise F-order-sensitive for T if S is componentwise F-order-sensitive and if x 6= z in (3.1) implies that S̄(x, F ) < S̄(z, F ).

11

Remark 3.2. In economic terms, a strictly componentwise order-sensitive scoring function rewards Pareto improvements 6 in the sense that improving the prediction performance in one component without deteriorating the prediction ability in the other
components results in a lower expected score.
Definition 3.3 (Order-sensitivity on line segments). Let k · k be the Euclidean norm
on Rk . A scoring function S : A × O → R is F-order-sensitive on line segments for a
functional T : F → A ⊆ Rk , if for all F ∈ F, t = T (F ), and for all v ∈ Sk−1 := {x ∈
Rk : kxk = 1} the map
ψ : D = {s ∈ [0, ∞) : t + sv ∈ A} → R,

s 7→ S̄(t + sv, F )

is increasing. If the map ψ is strictly increasing, we call S strictly F-order-sensitive on
line segments for T .
These two notions of order-sensitivity do not allow for a comparison of any two misspecified forecasts, no matter where they are relative to the true functional value. An
intuitive requirement could be ‘the closer to the true functional value the smaller the
expected score’, thus calling for the notion of a metric. Since, for a fixed functional
T and some fixed distribution F , we always have a fixed reference point T (F ) and we
have the induced vector-space structure of Rk on A, we
shall only work with ℓp -norms
P
k
k · kp , p ∈ [1, ∞]. Recall that for x ∈ Rk , kxkp := ( i=1 |xi |p )1/p for p ∈ [1, ∞) and
kxk∞ := supi=1,...,k |xi |. If the assertion does not depend on the choice of p, we shall
usually omit the p in the notation. For other choices of A, it would be also interesting
to replace the norm by a metric in the following definition.
Definition 3.4 (Metrical order-sensitivity). Let p ∈ [1, ∞]. A scoring function S : A ×
O → R is metrically F-order-sensitive for a functional T : F → A ⊆ Rk relative to the
ℓp -norm, if for all F ∈ F, t = T (F ) and for all x, z ∈ A we have that
kx − tkp ≤ kz − tkp =⇒ S̄(x, F ) ≤ S̄(z, F ).

(3.2)

If additionally the inequalities in (3.2) are strict, we say that S is strictly metrically
F-order-sensitive for T relative to k · kp .
Similarly to (strict) consistency, all three notions of (strict) order-sensitivity are preserved when considering two scoring functions that are equivalent.7
6

The definition of the Pareto principle according to Scott and Marshall (2009): “A principle of welfare economics derived from the writings of Vilfredo Pareto, which states that a legitimate welfare
improvement occurs when a particular change makes at least one person better off, without making
any other person worse off. A market exchange which affects nobody adversely is considered to be
a ‘Pareto-improvement’ since it leaves one or more persons better off. ‘Pareto optimality’ is said
to exist when the distribution of economic welfare cannot be improved for one individual without
reducing that of another.”
7
Two scoring functions S1 , S2 : A × O → R are equivalent if there is a positive constant λ > 0 and an
F-integrable function a : O → R such that S2 (x, y) = λS1 (x, y) + a(y), for all (x, y) ∈ A × O.

12

The notion of componentwise order-sensitivity corresponds almost literally to the notion of accuracy-rewarding scoring functions introduced by Lambert et al. (2008). Metrically order-sensitivity scoring functions have their counterparts in the field of probabilistic forecasting in effective scoring rules introduced by Friedman (1983) and further
investigated by Nau (1985). Actually, the latter paper has also given the inspiration
for the notion of order-sensitivity on line segments. It is obvious that any of the three
notions of (strict) order-sensitivity implies (strict) consistency. The next lemma formally
states this result and gives some logical implications concerning the different notions of
order-sensitivity. The proof is standard and therefore omitted.
Lemma 3.5. Let T : F → A ⊆ Rk be a functional and S : A × O → R a scoring function.
(i) Let p ∈ [1, ∞). If S is (strictly) metrically F-order-sensitive for T relative to the
ℓp -norm, then S is (strictly) componentwise F-order-sensitive for T .
(i’) If
S
(i”) If
S

S
is
S
is

is (strictly) metrically F-order-sensitive for T relative to the ℓ∞ -norm, then
componentwise F-order-sensitive for T .
is (strictly) metrically F-order-sensitive for T relative to the ℓ∞ -norm, then
(strictly) F-consistent for T .

(ii) If S is (strictly) componentwise F-order-sensitive for T , then S is (strictly) Forder-sensitive on line segments for T .
(iii) If S is (strictly) F-order-sensitive on line segments for T , then S is (strictly) Fconsistent for T .

3.2. Componentwise order-sensitivity
Under restrictive regularity assumptions, Lambert et al. (2008, Theorem 5) claim that
whenever a functional has a componentwise order-sensitive scoring function, the components of the functional must be elicitable. Moreover, assuming that the measures in F
have finite support, they assert that any componentwise order-sensitive scoring function
is the sum of strictly consistent scoring functions for the components. Lemma 3.6 shows
the first claim under less restrictive smoothness assumptions on the scoring function.
For many common examples of functionals, the second claim can be shown relaxing the
restrictive condition on F; see Proposition 3.7 and the discussion before.
Lemma 3.6. Let T = (T1 , . . . , Tk ) : F → A ⊆ Rk be a k-dimensional functional with
components Tm : F → Am ⊆ R where A = A1 × · · · × Ak . If there is a strictly componentwise F-order-sensitive scoring function S : A × O → R for T , then the components Tm ,
m ∈ {1, . . . , k}, are elicitable.
Proof. Fix m ∈ {1, . . . , k}. Let F ∈ F and x, z ∈ A such that Tm (F ) = xm , xi = zi for
all i 6= m and xm 6= zm . Due to the strict componentwise F-order-sensitivity of S this
implies that S̄(x, F ) < S̄(z, F ). This in turn means that for any z = (z1 , . . . , zk ) ∈ A
the map Sm,z : Am × O → R,
(xm , y) 7→ Sm,z (xm , y) := S(z1 , . . . , zm−1 , xm , zm+1 , . . . , zk , y)

13

(3.3)

is a strictly F-consistent scoring function for Tm .
If Tm : F → Am ⊆ R, m ∈ {1, . . . k}, are mixture-continuous and elicitable with
strictly F-consistent scoring functions Sm : Am × O → R, then they are order-sensitive
according to Lambert (2013,
PkProposition 2) and Bellini and Bignozzi (2015, Proposition
3.4). Therefore, the sum m=1 Sm (xm , y) is strictly componentwise F-order-sensitive
for (T1 , . . . , Tk ). More interestingly, one can establish the reverse of the last assertion.
Any strictly componentwise order-sensitive scoring function must necessarily be additively separable. In Fissler and Ziegel (2016a, Section 4), we established a dichotomy for
functionals with elicitable components: In most relevant cases, the functional (the corresponding strict identification function, respectively) satisfies Assumption (V4) therein
(e.g., when the functional is a vector of different quantiles and / or different expectiles
with the exception of the 1/2-expectile), or it is a vector of ratios of expectations
with the same denominator, or it is a combination of both situations. Under some
regularity conditions, Fissler and Ziegel (2016a, Propositions 4.2 and 4.4) characterize
the form of strictly consistent scoring functions for the first two situations, whereas
Fissler and Ziegel (2016a, Remark 4.5) is concerned with the third situation. For this
latter situation, any strictly consistent scoring function must be necessarily additive for
the respective blocks of the functional. And for the first situation, Fissler and Ziegel
(2016a, Proposition 4.2) yields the additive form of S automatically. It remains to consider the case of Fissler and Ziegel (2016a, Proposition 4.4), that is, a vector of ratios of
expectations with the same denominator.
Proposition 3.7. Let T : F → A ⊆ Rk be a ratio of expectations with the same denominator, that is, T (F ) = EF [p(Y )]/EF [q(Y )] for some F-integrable functions p : O → Rk ,
q : O → R such that EF [q(Y )] > 0 for all F ∈ F.8 Assume that T is surjective, and
that int(A) 6= ∅ is simply connected. Moreover, consider the strict F-identification function V : A × O → Rk , V (x, y) = q(y)x − p(y) and some strictly F-consistent scoring
function S : A × O → R such that the Assumptions (V1), (S2), (F1), and (VS1) in
Fissler and Ziegel (2016a) hold. If S is strictly componentwise F-order-sensitive for T ,
then S is of the form
k
X
Sm (xm , y),
(3.4)
S(x1 , . . . , xk , y) =
m=1

for almost all (x, y) ∈ A × O, where Sm : Am × O → R, m ∈ {1, . . . , k}, are strictly
F-consistent scoring functions for Tm : F → Am , Am := Tm (F) ⊆ R, and Tm (F ) =
EF [pm (Y )]/EF [q(Y )].
Proof. Due to the fact that for fixed y ∈ O, V (x, y) is a polynomial in x, Assumption
8

It is no loss of generality to assume that q̄(F ) > 0 for all F ∈ F in Proposition 3.7. In order to ensure
that T is well-defined, necessarily q̄(F ) 6= 0 for all F ∈ F. However, Assumption (V1) implies that
F is convex. So if there are F1 , F2 ∈ F such that q̄(F1 ) < 0 and q̄(F2 ) > 0 then there is a convex
combination G of F1 and F2 such that q̄(G) = 0. Consequently, either q̄(F ) > 0 for all F ∈ F or
q̄(F ) < 0 for all F ∈ F, and by possibly changing the sign of p one can assume that the first case
holds.

14

(V3) in Fissler and Ziegel (2016a) is automatically satisfied. Let h : int(A) → Rk×k be
the matrix-valued function given in Osband’s principle; see Fissler and Ziegel (2016a,
Theorem 3.2). By Fissler and Ziegel (2016a, Proposition 4.4(i)) we have that
∂l hrm (x) = ∂r hlm (x),

hrl (x) = hlr (x)

(3.5)

for all r, l, m ∈ {1, . . . , k}, l 6= r, where the first identity holds for almost allx ∈ int(A)
and the second identity for all x ∈ int(A). Moreover, the matrix hrl (x) l,r=1,...,k is
positive definite for all x ∈ int(A). If we can show that hlr = 0 for l 6= r, we can use
the first part of (3.5) and deduce that for all m ∈ {1, . . . , k} there are positive functions
gm : A′m → R, where A′m = {xm ∈ R : ∃(z1 , . . . , zk ) ∈ int(A) and zm = xm }, such that
hmm (x1 , . . . , xk ) = gm (xm )
for all (x1 , . . . , xk ) ∈ int(A). Then, we can conclude like in the proof of Fissler and Ziegel
(2016a, Proposition 4.2(ii)).9
Fix l, r ∈ {1, . . . , k} with l 6= r and F ∈ F such that T (F ) ∈ int(A). Due to the strict
F-consistency of Sl,z defined at (3.3) we have that
0=

d
S̄l,z (xl , F ) = ∂ S̄l,z (xl , F ) = ∂l S̄(z1 , . . . , zl−1 , xl , zl+1 , . . . , zk , F )
dxl

whenever xl = Tl (F ) and for all z ∈ int(A). This means the map int(A) ∋ z 7→
∂ S̄l,z (Tl (F ), F ) is constantly 0. Hence, for all x ∈ int(A)
∂r ∂l S̄(x, F ) = 0
whenever xl = Tl (F ). Using the special form of V and Fissler and Ziegel (2016a, Corollary 3.3), we have for x = t = T (F ) that
0 = ∂r ∂l S̄(t, F ) = hlr (t)∂r V̄r (t, F ) = hlr (t)q̄(F )
and by assumption q̄(F ) > 0. Using the surjectivity of T we obtain that hlr (t) = 0 for
all t ∈ int(A), which ends the proof.
The notion of componentwise order-sensitivity has an appealing interpretation in the
sense that it rewards Pareto improvements of the predictions; see Remark 3.2. The
results of Lemma 3.6 and Proposition 3.7 give a clear understanding of the concept including its limitations to the case of functionals only consisting of elicitable components.
Ehm et al. (2016) introduced Murphy diagrams for forecast comparison of quantiles
and expectiles. Murphy diagrams have the advantage that forecasts are compared simultaneously with respect to all consistent scoring functions for the respective functional.
9

The arguments in Fissler and Ziegel (2016a, Proposition 4.2(ii)) use Fissler and Ziegel (2016a, Proposition 3.4). There is a flaw in the latter result which has been pointed out in Brehmer (2017). We
present a corrected version of the result in Appendix A.

15

For many multivariate functionals such as ratios of expectations, the methodology cannot be readily extended because there are no mixture representations available for the
class of all consistent scoring functions. Proposition 3.7 shows that when considering
only componentwise order-sensitive consistent scoring functions, the situations is different and mixture representations (and hence Murphy diagrams) are readily available for
forecast comparison.

3.3. Metrical order-sensitivity
We start with an equivalent formulation of metrical order-sensitivity.
Lemma 3.8. Let F be convex and T : F → A ⊆ Rk be mixture-continuous and surjective.
Let S : A × O → R be (strictly) F-consistent for T . Then S is (strictly) metrically Forder-sensitive for T relative to k · k if and only if for all F ∈ F, t = T (F ) and x, z ∈ A
we have the implication
kx − tk = kz − tk =⇒ S̄(x, F ) = S̄(z, F ).

(3.6)

Proof. Let S be metrically F-order sensitive for T relative to d. Let F ∈ F, t = T (F ),
x, z ∈ A such that kx − tk = kz − tk. Then we have both S̄(x, F ) ≤ S̄(z, F ) and
S̄(z, F ) ≤ S̄(x, F ).
Assume that (3.6) holds and S is (strictly) F-consistent. Let F ∈ F with t = T (F )
and x, z ∈ A. Suppose that kx − tk ≤ kz − tk. If kx − tk = kz − tk, (3.6) implies
that S̄(x, F ) = S̄(z, F ) and there is nothing to show. If kx − tk < kz − tk, we can
apply Proposition 2.5. There is a continuous path γ : [0, 1] → A such that γ(0) = z
and γ(1) = t, and the function [0, 1] ∋ λ 7→ S̄(γ(λ), F ) is decreasing. Due to continuity
there is a λ′ ∈ [0, 1] such that kγ(λ′ ) − tk = kx − tk. Invoking (3.6) it holds that
S̄(x, F ) = S̄(γ(λ′ ), F ) ≤ S̄(z, F ). If S is strictly F-consistent then the latter inequality
is strict.
For a real-valued functional T there can be at most one strictly metrically ordersensitive scoring function, up to equivalence. To show this, we use Osband’s principle
and impose the corresponding regularity conditions.
Proposition 3.9. Let T : F → A ⊆ R be a surjective, elicitable and identifiable functional with an oriented strict F-identification function V : A × O → R. If int(A) 6= ∅
is convex and S, S ∗ : A × O → R are two strictly metrically F-order-sensitive scoring
functions for T such that the Assumptions (V1), (V2), (S1), (F1) and (VS1) from
Fissler and Ziegel (2016a) (with respect to both scoring functions) hold, then S and S ∗
are equivalent almost everywhere.
Proof. We apply Osband’s principle, that is, Fissler and Ziegel (2016a, Theorem 3.2) to
S. Consequently, there is a function h : int(A) → R such that
d
S̄(x, F ) = h(x)V̄ (x, F )
dx

16

(3.7)

for all F ∈ F, x ∈ int(A). Due to the strict F-consistency of S and the orientation of V ,
it holds that h ≥ 0. We show that actually h > 0. Applying Lemma 3.8, one has that
S̄(T (F ) + x, F ) = S̄(T (F ) − x, F )

(3.8)

for all F ∈ F, x ∈ R such that T (F ) + x, T (F ) − x ∈ int(A). Hence, also the derivative
with respect to x of the left-hand side of (3.8) must coincide with the derivative on the
right-hand side. This yields, using (3.7),
h(T (F ) + x)V̄ (T (F ) + x, F ) = −h(T (F ) − x)V̄ (T (F ) − x, F )

(3.9)

for all F ∈ F, x ∈ R such that T (F ) + x, T (F ) − x ∈ int(A). Assume h(z) = 0 for some
z ∈ int(A). Then, by surjectivity of T and convexity of int(A), for all z ′ ∈ int(A) \ {z}
there exists an F ∈ F and x ∈ R\{0} such that z = T (F )+x and z ′ = T (F )−x. Since V
is a strict F-identification function for T , both V̄ (T (F )+x, F ) 6= 0 and V̄ (T (F )−x, F ) 6=
0. Hence, (3.9) implies that h(z ′ ) = 0. This implies that h identically vanishes on int(A)
which contradicts the strict F-consistency of S.
Therefore, V ∗ (x, y) := h(x)V (x, y) is an oriented strict F-identification function for
T . Applying Osband’s principle to S ∗ , one obtains a function h∗ : int(A) → R such that
d/(dx)S̄ ∗ (x, F ) = h∗ (x)V̄ ∗ (x, F ) for all F ∈ F, x ∈ R such that T (F ) + x, T (F ) − x ∈
int(A). Due to the analogue of (3.8) for S ∗ and (3.9), one obtains
h∗ (T (F ) + x)V̄ ∗ (T (F ) + x, F ) = −h∗ (T (F ) − x)V̄ ∗ (T (F ) − x, F )
= h∗ (T (F ) − x)V̄ ∗ (T (F ) + x, F ).
for all F ∈ F, x ∈ R with T (F ) + x, T (F ) − x ∈ int(A). By a similar reasoning as
above, one can deduce that h∗ must be constant and positive. Now, the claim follows
by Fissler and Ziegel (2016a, Proposition 3.4); see Appendix A.
For the higher-dimensional setting we can show a slightly more limited version of
Proposition 3.9. Two scoring functions that are additively separable as in (3.4) and
that are strictly metrically order-sensitive for the same functional must necessarily be
equivalent. For most practically relevant cases – namely when we consider an ℓp -norm
with p ∈ [1, ∞) and when the functional possesses an identification function satisfying
Assumption (V4) in Fissler and Ziegel (2016a) or that are ratios of expectations with the
same denominator – Lemma 3.5, Proposition 3.7 and Fissler and Ziegel (2016a, Proposition 4.2) yield that any metrically order-sensitive scoring function – presuming there
is one – is additively separable. Hence, for these situations, metrically order-sensitive
scoring functions are unique, up to equivalence.
Proposition 3.10. Let S : A × O → R be a strictly metrically F-order-sensitive scoring
function for a surjective functional T = (T1 , . . . , Tk ) : F → A ⊆ Rk of the form
S(x1 , . . . , xk , y) =

k
X

m=1

17

Sm (xm , y)

for all (x, y) ∈ A × O where Sm : Am × O → R, m ∈ {1, . . . , k}, Am = {xm ∈
R : ∃(z1 , . . . , zk ) ∈ A and zm = xm }, are strictly F-consistent scoring functions for Tm .
Assume that int(A) 6= ∅. Then, the following assertions hold:
(i) The scoring functions Sm , m ∈ {1, . . . , k}, are strictly metrically F-order-sensitive
for Tm .
(ii) Let λ1 , . . . , λk > 0 and define the scoring function S ∗ : A × O → R via
S ∗ (x1 , . . . , xk ) =

k
X

λm Sm (xm , y).

m=1

Then S ∗ is strictly metrically F-order-sensitive (with respect to the same ℓp -norm
as S) if and only if λ1 = · · · = λk .
Proof. (i) Let m ∈ {1, . . . , k}, F ∈ F with t = T (F ) ∈ int(A). Let µ ∈ R and
x, z ∈ int(A) with xi = zi = ti for all i 6= m and with xm = tm + µ and zm = tm − µ,
such that |x − t| = |z − t|. Due to Lemma 3.8 and due to the particular additive
form of S, we have
0 = S̄(x, F ) − S̄(z, F ) = S̄m (xm , F ) − S̄m (zm , F )
= S̄m (tm + µ, F ) − S̄m (tm − µ, F ).
Again with Lemma 3.8 one obtains the assertion.
(ii) The only interesting direction is to assume that S ∗ is strictly metrically F-ordersensitive (with respect to the same ℓp -norm as S). We will show that λ1 = λm for
all m ∈ {2, . . . , k}. Let F ∈ F, t = T (F ) ∈ int(A), x, z ∈ int(A) with kx − tkp =
kz −tkp > 0 and xi = zi = ti for all i ∈ {2, . . . , k}\{m}. Moreover, let x1 6= z1 = t1 .
Due to Lemma 3.8 we have that S̄(x, F ) − S̄(z, F ) = S̄ ∗ (x, F ) − S̄ ∗ (z, F ) = 0.
Moreover,
0 = S̄(x, F ) − S̄(z, F ) =

k
X

S̄i (xi , F ) − S̄i (zi , F )

i=1

= S̄1 (x1 , F ) − S̄1 (z1 , F ) + S̄m (xm , F ) − S̄m (zm , F ).
Setting ε := S̄1 (x1 , F ) − S̄1 (z1 , F ) > 0, one obtains with the same calculation
0 = S̄ ∗ (x, F ) − S̄ ∗ (z, F )


= λ1 S̄1 (x1 , F ) − S̄1 (z1 , F ) + λm S̄m (xm , F ) − S̄m (zm , F ) = ε(λ1 − λm ).
Next, we use the derived theoretical results to examine when some popular functionals
admit strictly metrically order-sensitive scoring functions, and if so, of what form they
are.

18

3.3.1. Ratios of expectations with the same denominator
We start with the one-dimensional characterization.
Lemma 3.11. Let F be convex and p, q : O → R two F-integrable functions such that
q̄(F ) > 0 for all F ∈ F. Define T : F → A ⊆ R, T (F ) = p̄(F )/q̄(F ) and assume that T
is surjective and int(A) 6= ∅ is convex. Then the following two assertions are true:
(i) Any scoring function which is equivalent to
S : A × O → R,

S(x, y) =

1
q(y)x2 − p(y)x
2

(3.10)

is strictly metrically F-order-sensitive for T .
(ii) If F is such that Assumptions (V1), (F1) in Fissler and Ziegel (2016a) are satisfied
with V (x, y) = q(y)x − p(y), then any scoring function S ∗ : A × O → R, which is
strictly metrically F-order-sensitive and satisfies Assumptions (S1) and (VS1), is
equivalent to S defined at (3.10) almost everywhere.
Proof.

(i) We can apply Lemma 3.8. Let F ∈ F. Then
R ∋ x 7→ S̄(T (F ) + x, F ) =

1 p̄(F )2
1
q̄(F )x2 −
2
2 q̄(F )

is an even function in x. Moreover, equivalence of scoring functions preserves
(strict) metrical order-sensitivity.
(ii) The convexity of A is implied by the mixture-continuity of T and the convexity of
F. Then, the claim follows with Proposition 3.9.

Now, we turn to the multivariate characterization.
Proposition 3.12. Let k ≥ 2, F be convex and p : O → Rk , q : O → R two F-integrable
functions such that q̄(F ) > 0 for all F ∈ F. Define T : F → A ⊆ Rk , T (F ) = p̄(F )/q̄(F )
and assume that T is surjective and int(A) 6= ∅. Then, the following assertions are true:
(i) Any scoring function which is equivalent to
S : A × O → R,

S(x1 , . . . , xk , y) =

k
X
1
q(y)x2m − pm (y)xm
2

(3.11)

m=1

is strictly metrically F-order-sensitive for T with respect to the ℓ2 -norm.
(ii) If F is such that Assumptions (V1), (F1) in Fissler and Ziegel (2016a) are satisfied with V (x, y) = q(y)x − p(y), then any scoring function S ∗ : A × O → R, which
is strictly metrically F-order-sensitive with respect to the ℓ2 -norm and satisfies Assumptions (S1) and (VS1), is equivalent to S defined at (3.11) almost everywhere.

19

(iii) If F is such that Assumptions (V1), (F1) in Fissler and Ziegel (2016a) are satisfied
with V (x, y) = q(y)x − p(y), then there is no scoring function S ∗ : A × O → R
which satisfies Assumptions (S1) and (VS1) and which is strictly metrically Forder-sensitive with respect to an ℓp -norm with p ∈ [1, ∞) \ {2}.
k
Proof. To show (i) we apply again Lemma 3.8.
Pk For any2 F ∈ F, x ∈ R , we have
2
S̄(T (F ) + x, F ) = (1/2)q̄(F )kxk2 − 1/(2q̄(F )) m=1 p̄m (F ) which only depends on the
ℓ2 -norm of x.
We prove (ii) and (iii) together. Assume there is a scoring function S ∗ satisfying the
conditions above, so in particular, it is strictly metrically F-order-sensitive with respect
to the ℓp -norm for p ∈ [1, ∞). Invoking Lemma 3.5(i), S ∗ is strictly componentwise Forder-sensitive for T . Thanks to Proposition 3.7, S ∗ is additively separable. By Lemma
3.11(i), it is of the form

∗

S (x1 , . . . , xk ) =

k
X

m=1

λm



1
q(y)x2m − pm (y)xm
2



+

k
X

am (y).

m=1

If p = 2, part (i) and Proposition 3.10(ii) yield that λ1 = · · · = λk , andPhence, S and
S ∗ are equivalent. For p 6= 2, we obtain S̄(T (F ) + x, F ) = (1/2)q̄(F ) km=1 λm x2m −
P
1/(2q̄(F )) km=1 p̄m (F )2 . It is not hard to see that there are always x, x′ with kxkp =
kx′ kp but S̄(T (F ) + x, F ) 6= S̄(T (F ) + x′ , F ).
Savage (1971, Section 5) has already shown that in case of the mean, the squared
loss is essentially the only symmetric loss in the sense that it is the only metrically
order-sensitive loss for the mean. See also Patton (2017, Section 2.1) for a discussion
that symmetry – or metrical order-sensitivity – is not necessary for strict consistency of
scoring functions with respect to the mean.
3.3.2. Quantiles
Since we treat only point-valued functionals in this article, we shall assume that the
α-quantile of F is a singleton and identify the set with its unique element (henceforth,
we shall refer to this assumption as F having a unique α-quantile).10 Furthermore,
note that assuming the identifiability of the α-quantile with the canonical identification
function Vα (x, y) = 1{y ≤ x} − α on a class F amounts to assuming that F (qα (F )) = α
for all F ∈ F.11
10

Recall that the α-quantile of a distribution F consists of all points x ∈ R satisfying limt↑x F (t) ≤ α ≤
F (x).
11
Actually, assuming F is convex and rich enough, this holds for any identification function for the
α-quantile. Indeed, consider some distribution function F0 ∈ F and some level α ∈ (0, 1). Fix some
x0 ∈ R such that F0 (x0 ) < α, implying that qα (F0 ) > x0 . Assume that for any λ ∈ [0, 1], the
distribution
(
F0 (x),
x < x0
Fλ (x) =
(1 − λ)F0 (x) + λ, x ≥ x0

20

Proposition 3.13. Let α ∈ (0, 1) and F be a family of distribution functions F on R
with unique α-quantiles qα (F ) satisfying F (qα (F )) = α for all F ∈ F. Assume that there
is an F0 ∈ F, such that its translation Fλ (·) = F0 (· − λ) is also an element of F for all
λ ∈ R. Consequently, Tα = qα : F → A = R is surjective. Under assumptions (V1) in
Fissler and Ziegel (2016a) with respect to the strict identification function Vα : R × R →
R, Vα (x, y) = 1{y ≤ x} − α, there is no strictly metrically F-order-sensitive scoring
function for Tα satisfying Assumption (S1) in Fissler and Ziegel (2016a).
Proof. Assume that there exists a strictly metrically F-order-sensitive scoring function
Sα : R×R → R satisfying Assumption (S1) in Fissler and Ziegel (2016a). Due to Lemma
3.8, for any F ∈ F and any x ∈ R
S̄α (Tα (F ) + x, F ) = S̄α (Tα (F ) − x, F ).
Using Osband’s principle (Fissler and Ziegel, 2016a, Theorem 3.2) and taking the derivative with respect to x on both sides, this yields
h(Tα (F ) + x)V̄α (Tα (F ) + x, F ) = −h(Tα (F ) − x)V̄α (Tα (F ) − x, F )

(3.12)

for some positive function h : R → R (the fact that h ≥ 0 follows from the strict consistency of Sα and the surjectivity of Tα , and h > 0 follows like in the proof of Proposition
3.9). Assume that Tα (F0 ) = 0. For λ ∈ R, we have Tα (F0 (· − λ)) = λ. Therefore, (3.12)
implies
V̄α (λ − x, F0 (· − λ))
F0 (−x) − α
h(λ + x)
=−
=−
.
(3.13)
h(λ − x)
F0 (x) − α
V̄α (λ + x, F0 (· − λ))
Setting λ = ±x, one can see that h(±∞) := limx→±∞ h(x) exists and that h(+∞) =
h(0)α/(1 − α), h(−∞) = h(0)(1 − α)/α, hence, h(+∞)/h(−∞) = 1. On the other hand,
for fixed λ ∈ R, we obtain
α
h(λ + x)
h(+∞)
= lim
=
.
h(−∞) x→∞ h(λ − x)
1−α
As a consequence, the only remaining possibility is α = 1/2. For fixed x ∈ R, we have
1=

h(+∞)
F0 (−x) − 1/2
h(λ + x)
= lim
=−
h(+∞) λ→∞ h(λ − x)
F0 (x) − 1/2

is an element of F. Then, there is some λ′ ∈ (0, 1) such that Fλ′ (x0 ) = α implying that Fλ (x0 ) > α
for all λ ∈ (λ′ , 1] and qα (Fλ ) = x0 for all λ ∈ [λ′ , 1]. Assume that V is a strict F-identification
function for qα . That means V̄ (x0 , Fλ ) = 0 for all λ ∈ [λ′ , 1] and V̄ (x0 , Fλ ) 6= 0 for all λ ∈ [0, λ′ ).
Consider some λ ∈ [λ′ , 1]. Then,
V̄ (x0 , Fλ ) = (1 − λ)V̄ (x0 , F0 ) + λV̄ (x0 , F1 ) = (1 − λ)V̄ (x0 , F0 ) 6= 0.
This is a contradiction to V being a strict F-identification function for qα .

21

implying that h must be constant using (3.12), and that F0 must be symmetric around
its median, i.e. F0 (x) = 1 − F0 (−x) for all x ∈ R.12 Moreover, since h is constant,
(3.12) implies that also any other distribution F ∈ F must be symmetric around its
median, i.e. F (T1/2 (F ) + x) = 1 − F (T1/2 (F ) − x) for all x ∈ R. However, if F0 is
symmetric around its median, then any translation Fλ of F0 is symmetric around its
median. But then, there is a convex combination of F0 and Fλ with mixture-parameter
β ∈ (0, 1), β 6= 1/2, such that βF0 + (1 − β)Fλ is not symmetric around its median if
λ 6= 0. Consequently, the conditions of the proposition are violated such that a strictly
metrically F-order-sensitive function for the median does not exist in this setting.
The reasons for the non-existence of a strictly metrically order-sensitive scoring function for the α-quantile are of different nature in the two cases that α 6= 1/2 and that
α = 1/2 in the proof of Proposition 3.13. In both cases, we used Osband’s principle to derive a representations of the derivative of the expected score. Assuming that
the derivative has the form as stated in Osband’s principle, one can directly derive a
contradiction for α 6= 1/2. However, for α = 1/2, this form merely implies that the
distributions in F must be symmetric around their medians. This is not contradictory
to the form of the gradient derived via Osband’s principle, but only to the assumption
that F is convex. Dropping this assumption, we can derive the following Lemma. The
proof is straight forward from Lemma 3.8.
Lemma 3.14. Let F be a family of distribution functions on R with unique medians
T1/2 : F → R and finite first moments. If all distributions in F are symmetric around
their medians in the sense that
F (T1/2 (F ) + x) = 1 − F ((T1/2 (F ) − x)−)

(3.14)

for all F ∈ F, x ∈ R, then any scoring function that is equivalent to the absolute loss
S : R × R → R, S(x, y) = |x − y|, is strictly metrically F-order-sensitive with respect to
the median.
As mentioned above, under the conditions of Lemma 3.14, the necessary characterization of strictly consistent scoring functions via Osband’s principle is not available. In
particular, this means that we cannot use Proposition 3.9. Indeed, if the distributions in
F are symmetric around their medians in the sense of (3.14) and under the integrability
condition that all elements in F have a finite first moment, the median and the mean
coincide. Hence, any convex combination of a strictly consistent scoring function for
the mean and the median provides a strictly consistent scoring function. A fortiori, any
scoring function which is equivalent to S(x, y) = (1 − λ)|x − y| + λ|x − y|2 , λ ∈ [0, 1] is
12

This equation implies that F0 is necessarily continuous. This fact also follows directly from Assumption
(S1) in Fissler and Ziegel (2016a) and the assumption that F is closed under translations of F0 .
Indeed, assume that F0 is discontinuous at some point x0 . Then h has to be discontinuous at that
point. But since F0 has at most countably many points of discontinuity, there is some λ0 ∈ R such
that Fλ0 is continuous at x0 . But this would imply that the derivative of S̄(·, Fλ0 ) is discontinuous
at x0 , which contradicts the assumptions.

22

strictly metrically F-order-sensitive. However, the class of strictly metrically F-ordersensitive scoring functions is even bigger – Lehmann and Casella (1998, Corollary 7.19,
p. 50) show that (subject to integrability conditions) for an even and strictly convex
function Φ : R → R, the score S(x, y) = Φ(x − y) is strictly metrically F-order-sensitive
for the median. Note that if the distributions in F are symmetric, their center of symmetry, which is the functional solving (3.14), is unique (Fissler, 2017, Lemma 4.1.34), even
if the median is not unique. The result of Lehmann and Casella (1998, Corollary 7.19,
p. 50) holds for this center of symmetry. Acknowledging that some popular choices for Φ
are not strictly convex (see Example 3.16), the following proposition gives a refinement
of their result.
Proposition 3.15. Let F be a class of symmetric distributions on R with center of
symmetry C : F → R, that is, F (C(F ) + x) = 1 − F ((C(F ) − x)−) for all F ∈ F, x ∈ R.
Let Φ : R → R be a convex and even function, and S : R × R → R, S(x, y) = Φ(x − y).
For any x ∈ R, define the function Ψx : R → R, Ψx (y) = 21 (Φ(x − y) + Φ(−x − y)), and
for x, z ∈ R the set Mx,z = {y ∈ R : Ψx (y) − Ψz (y) > 0}. If for all F ∈ F and for all
x, z ∈ R with |x| > |z| one has that P(Y − C(F ) ∈ Mx,z ) > 0, Y ∼ F , then S is strictly
metrically F-order-sensitive for C. In particular, if for all F ∈ F and for all x 6= 0 it
holds that P(Y − C(F ) ∈ Mx,0 ) > 0, Y ∼ F , then S is strictly F-consistent for C.
Proof. Let |x| > |z|. Note that due to the convexity of Φ, it holds that Ψx ≥ Ψz . Let
F ∈ F with center of symmetry c = C(F ) and let Y ∼ F . Then, using the fact that Φ
d

is even and that Y − c = c − Y , one obtains
S̄(c + x, F ) − S̄(c + z, F ) = EF [Φ(x − (Y − c)) − Φ(z − (Y − c))]
= EF [Ψx (Y − c) − Ψz (Y − c)] > 0 .
This shows the strict metrical F-order-sensitivity. The strict F-consistency follows upon
taking z = 0.
If Φ is strictly convex then Mx,z = R for all |x| > |z|.
Example 3.16. Let F be a class of symmetric distributions and S(x, y) = Φ(x − y).
(i) If Φ(t) = |t|2 , the squared loss arises. Since Φ is strictly convex, the squared loss
is strictly metrically F-order-sensitive.
(ii) For Φ(t) = |t|, S takes the form of the absolute loss. Then S is strictly metrically
F-order-sensitive (and strictly F-consistent) if and only if C(F ) ∈ supp(F ) for all
F ∈ F.13
(iii) Another prominent example of a metrically order-sensitive scoring function for the
center of a symmetric distribution besides the absolute or the squared loss is the
so-called Huber loss which was presented in Huber (1964) and arises upon taking
13

With the support of F supp(F ) we denote the support of the measure induced by F . In this context,
C(F ) ∈ supp(F ) is equivalent to F having a unique median.

23

S(x, y) = Φ(x − y) with
Φ(t) =

(

1 2
2t ,

k|t| −

1 2
2k ,

for |t| < k,
for |t| ≥ k,

where k ∈ R, k ≥ 0 is a tuning parameter. The Huber loss is strictly metrically Forder-sensitive (strictly F-consistent) if and only if [C(F )−k, C(F )+k]∩supp(F ) 6=
∅ for all F ∈ F.
We emphasize that there are not only metrically-order sensitive strictly consistent
scoring functions for the center of symmetric distributions. One can also use asymmetric
scoring functions, for example those for the median or the mean, to elicit the center of
symmetry.
Due to the negative result of Proposition 3.13 we dispense with an investigation of
scoring functions that are metrically order-sensitive for vectors of different quantiles.
3.3.3. Expectiles
The special situation of the 1/2-expectile, which coincides with the mean functional, was
already considered in Subsection 3.3.1, so let τ 6= 1/2. It is obvious that the canonical
scoring function for the τ -expectile, that is, the asymmetric squared loss
Sτ (x, y) = |1{y ≤ x} − τ |(x − y)2
is not metrically order-sensitive since x 7→ Sτ (x + y, y) is not an even function. A
fortiori, it turns out that (under some assumptions) there is no strictly metrically Forder-sensitive scoring function for the τ -expectile for τ 6= 1/2.
Proposition 3.17. Let τ ∈ (0, 1), τ 6= 1/2, and Tτ = µτ : F → A ⊆ R, int(A) 6= ∅
convex, be the τ -expectile. Assume that Tτ is surjective, and that Assumption (V1)
in Fissler and Ziegel (2016a) holds with respect to the strict F-identification function
Vτ (x, y) = 2|1{y ≤ x} − τ | (x − y). Suppose that V̄ (·, F ) is twice differentiable for all
F ∈ F and that there is a strictly F-consistent scoring function S : A × R → R such
that S̄(·, F ) is three times differentiable for all F ∈ F. In particular, let each F ∈ F be
differentiable with derivative f = F ′ .
If there is a t ∈ A and F1 , F2 ∈ F such that Tτ (F1 ) = Tτ (F2 ) = t, F1 (t) = F2 (t), but
F1′ (t) = f1 (t) 6= f2 (t) = F2′ (t), then S is not metrically F-order-sensitive.
Proof. Under the assumptions, Osband’s principle yields the existence of a function
h : int(A) → R, h > 0 (by an argument like in the proof of Proposition 3.9) such that
for all F ∈ F, x ∈ int(A)
d
S̄(x, F ) = h(x)V̄ (x, F ).
dx
Using the same argument as in the proof of Osband’s principle (Fissler and Ziegel, 2016a,
Theorem 3.2), h is twice differentiable. Assume that S is metrically F-order sensitive. Then, due to Lemma 3.8, for any F ∈ F the function gF : A ∋ x 7→ gF (x) =

24

S̄(Tτ (F ) + x, F ) is an even function. Hence, invoking the smoothness assumptions, the
third derivative of gF must be odd. So necessarily gF′′′ (0) = 0. Denoting tF = Tτ (F ),
some tedious calculations lead to

gF′′′ (0) = 2h′ (tF ) F (tF )(1 − 2τ ) + τ + 2h(tF )f (tF )(1 − 2τ ).
(3.15)

Recalling that h > 0 and τ 6= 1/2 implies gF′′′1 (0) 6= gF′′′2 (0). So S cannot be metrically
F-order-sensitive.
Inspecting the proof of Proposition 3.17, equation (3.15) yields for τ = 1/2
gF′′′ (0) = h′ (tF )

for any F ∈ F, tF = Tτ (F ). With the surjectivity of Tτ this proves that h′ = 0, such that
h is necessarily constant. Hence, we get an alternative proof that the squared loss is the
only strictly metrically order-sensitive scoring function for the mean, up to equivalence.

3.4. Order-sensitivity on line segments
Recalling Lemma 3.5, every componentwise order-sensitive scoring function is also ordersensitive on line segments. However, for the particular class of linear functionals, the
following corollary shows that any strictly consistent scoring function is already strictly
componentwise order-sensitive on line segments.14
Corollary 3.18. If F is convex and T : F → A ⊆ Rk is linear and surjective, then any
strictly F-consistent scoring function for T is strictly F-order-sensitive on line segments.
Proof. The linearity of T implies that T is mixture-continuous. Then the assertion
follows directly by Proposition 2.5 and the special form of the image of the path γ in
the proof therein, which is a line segment.
Corollary 3.18 immediately leads the way to the result that the class of strictly ordersensitive scoring functions on line segments is strictly bigger than the class of strict
componentwise order-sensitive scoring functions (for some functionals with dimension
k ≥ 2.) E.g. consider a vector of expectations satisfying the conditions of Proposition 3.7
which are the same as the ones in Fissler and Ziegel (2016a, Proposition 4.4). Due to the
latter result, there are strictly consistent scoring functions – and hence, with Corollary
3.18, strictly order-sensitive on line segments – which are not additively separable. By
Proposition 3.7 they cannot be strictly componentwise order-sensitive.
We can extend the result of Corollary 3.18 to the case of ratios of expectations with
the same denominator.
14

According to Abernethy and Frongillo (2012), we call a functional T : F → A linear, if it behaves
linearly for mixtures of distributions. That is, for any F, G ∈ F such that (1 − λ)F + λG ∈ F for
λ ∈ [0, 1] it holds that T ((1 − λ)F + λG) = (1 − λ)T (F ) + λT (G). Examples of linear functionals are
expectations of transformations, that is, T (F ) = EF [p(Y )] for some F-integrable function p : O → Rk .

25

Lemma 3.19. Let T : F → A ⊆ Rk be a ratio of expectations with the same denominator,
that is, T (F ) = p̄(F )/q̄(F ) for some F-integrable functions p : O → Rk , and q : O → R
where we assume that q̄(F ) > 0 for all F ∈ F and that A is open and convex. Any
scoring function of the form
S(x, y) = −φ(x)q(y) + ∇φ(x)(q(y)x − p(y))

(3.16)

is strictly F-order sensitive on line segments, where φ is strictly convex differentiable
function on A.
Proof. Let F ∈ F, t = T (F ), v ∈ Sk−1 and 0 ≤ s < s′ such that t + sv, t + s′ v ∈ A. Then
S̄(t + sv, F ) = q̄(F )(−φ(t + sv) + s∇φ(t + sv)v). The subgradient inequality yields
S̄(t + sv, F ) − S̄(t + s′ v, F ) < q̄(F ) (s′ − s)∇φ(t + s′ v)v

+ s∇φ(t + sv)v − s′ ∇φ(t + s′ v)v ≤ 0.
Fissler and Ziegel (2016a, Proposition 4.4) shows that essentially all strictly consistent
scoring functions for T in the above Lemma 3.19 are of the form at (3.16); see also
Frongillo and Kash (2015a, Theorem 13).
Order-sensitivity on line segments is stable under applying an isomorphism via the
revelation principle (Gneiting, 2011, Theorem 4). However, dropping the linearity assumption on the bijection in the revelation principle, order-sensitivity on line segments
is generally not preserved; see Subsection 3.4.1.
Lemma 3.20. Let S : A × O → R be a (strictly) F-order-sensitive scoring function on
line segments for a functional T : F → A ⊆ Rk . Let g : A → A′ ⊆ Rk be an isomorphism
where A′ is the image of A under g. Then Sg : A′ × O → R defined as Sg (x′ , y) =
S(g−1 (x′ ), y) is a (strictly) F-order-sensitive scoring function on line segments for the
functional Tg = g ◦ T : F → A′ .
Proof. Let F ∈ F, t = T (F ) and tg = Tg (F ) = g(t). Let v ∈ Sk−1 and s ∈ [0, ∞). Using
the linearity of g−1 we get

S̄g (tg + sv, F ) = S̄ g−1 (g(t) + sv), F = S̄(t + sg −1 (v), F ).

Since also g is an isomorphism, we have that g−1 (v)/kg−1 (v)k ∈ Sk−1 . Hence, the map
s 7→ S̄g (tg + sv, F ) is (strictly) increasing for all v ∈ Sk−1 if S is (strictly) order-sensitive
on line segments.
3.4.1. The pair (mean, variance)
The pair (mean, variance) is of importance not only from an applied point of view
but it is also an interesting example in the theory about elicitability. Due to the lack

26

of convex level sets, variance is not elicitable (Gneiting, 2011, Theorem 6). However,
the pair (mean, variance) is a bijection of the (elicitable) pair (mean, second moment),
and, invoking the revelation principle (Gneiting, 2011, Theorem 4), variance is jointly
elicitable with the mean. The revelation principle provides an explicit link between
the class of strictly consistent scoring functions for the first two moments which are of
Bregman-type (Fissler and Ziegel, 2016a, Proposition 4.4) and the respective class for
mean and variance.
As the pair (mean, variance) has of a non-elicitable component, if fails to be componentwise order-sensitive (Lemma 3.6) and therefore, it is also not metrically order-sensitive.
A priori, order-sensitivity on line segments is not ruled out. Corollary 3.18 implies that
any strictly consistent scoring function for the pair of the first and second moment is
order-sensitive on line segments. Even though the bijection connecting (mean, variance)
with the pair of the first two moments is not linear, and, hence, we cannot apply Lemma
3.20, the following proposition gives necessary and sufficient conditions for scoring functions to be order-sensitive on line segments for (mean, variance). Example 3.22 shows
the existence of order-sensitive scoring functions on line segments for (mean, variance).
Proposition 3.21. Let F be a class of distributions on R with finite second moments
such that the functional T = (mean, variance) : F → A is surjective on A = R × (0, ∞).
Let Assumptions (F1) and (V1) from Fissler and Ziegel (2016a) be satisfied with the
⊤
strict F-identification function V : A × R → R2 , V (x1 , x2 , y) = x1 − y, x2 + x21 − y 2 .
Let S : A × R → R be a scoring function that is (jointly) continuous and for any y ∈ R,
the function A ∋ x 7→ S(x, y) be twice continuously differentiable. Then S is F-ordersensitive on line segments for T if and only if S is of the form


x1 − y
2
2
+ a(y),
(3.17)
S(x1 , x2 , y) = −φ(x1 , x2 + x1 ) + ∇φ(x1 , x2 + x1 )
x2 + x21 − y 2
where a : R → R is some F-integrable function and φ : A′ → R, A′ = {(x1 , x2 + x21 ) ∈
R2 | x ∈ A} = {(m1 , m2 ) ∈ R2 | m21 < m2 }, is a convex, three times continuously differentiable function such that the second order partial derivatives φij := ∂i ∂j φ satisfy
φ12 (m1 , m2 ) = −2m1 φ22 (m1 , m2 )
φ11 (m1 , m2 ) ≥ (m2 +

3m21 )φ22 (m1 , m2 )

(3.18)
(3.19)

for all (m1 , m2 ) ∈ A′ .
Proof. Let S be F-order-sensitive on line segments. This implies that S is F-consistent.
Using the revelation principle, S ′ : A′ × R → R,
S ′ (m1 , m2 , y) = S(m1 , m2 − m21 , y)

(3.20)

is an F-consistent scoring function for T ′ = (T1 , T2 + T12 ) : F → A′ , the pair of the
first and second moment. Moreover, S ′ fulfils the same regularity conditions as S.
Fissler and Ziegel (2016a, Proposition 4.4) holds mutatis mutandis also for consistent

27

scoring functions with φ convex. It is straight forward to check that the conditions for
Fissler and Ziegel (2016a, Proposition 4.4) are fulfilled for S ′ and T ′ with the canonical
⊤
identification function V ′ : A′ × R → R2 , V ′ (m1 , m2 , y) = m1 − y, m2 − y 2 . Hence, S ′
is necessarily of the form


m1 − y
S ′ (m1 , m2 , y) = −φ(m1 , m2 ) + ∇φ(m1 , m2 )
+ a(y),
m2 − y 2
where a : R → R is some F-integrable function and φ : A′ → R is a convex C 3 -function
with gradient ∇φ (considered as a row vector) and Hessian ∇2 φ = (φij )i,j=1,2 . In
summary, (3.20) yields the form at (3.17).
Now, we verify conditions (3.18) and (3.19). Let F ∈ F, with (t1 , t2 ) = T (F ). For
v ∈ R2 , kvk = 1, s ∈ R with s̄ := t + sv ∈ A, it holds that



d
v1
2
2
S̄(t + sv, F ) = s (v1 , v2 + 2v1 s̄1 )∇ φ s̄1 , s̄2 + s̄1
v2 + 2v1 s̄1 − sv12
ds



v1
2
2
= s (v1 , v2 + 2v1 s̄1 )∇ φ s̄1 , s̄2 + s̄1
(3.21)
v2 + 2v1 s̄1


− s2 v13 φ12 s̄1 , s̄2 + s̄21 + 2s̄1 φ22 s̄1 , s̄2 + s̄21
(3.22)

2
2 2
(3.23)
− s v1 v2 φ22 s̄1 , s̄2 + s̄1 .

Since (φij )i,j=1,2 is positive semi-definite, the term at (3.21) is non-negative and the
term at (3.23) has the sign of −v2 . Consider v = (1, 0)⊤ . Due to the surjectivity of T
it holds that for all c1 ∈ R, c2 > 0, s ∈ R there exists a distribution F + ∈ F such that
T1 (F + ) + s = c1 and T2 (F + ) = c2 . Hence,
s

−1

 
 1
d
+
2
2
S̄(t + sv, F ) = (1, 2c1 )∇ φ c1 , c2 + c1
2c1
ds



− s φ12 c1 , c2 + c21 + 2c1 φ22 c1 , c2 + c21 . (3.24)

Due to the F-order-sensitivity of S, the term on the left-hand side of (3.24) is non
negative for all s ∈ R. Since |s| can be arbitrarily large, the term φ12 c1 , c2 + c21 +
2c1 φ22 c1 , c2 + c21 must vanish and we obtain (3.18).

Finally, let v be such that v1 , v2 6= 0 and w.l.o.g. v2 > 0. Then




v1
2
2
−1 d
S̄(s̄, F ) = (v1 , v2 + 2v1 s̄1 )∇ φ s̄1 , s̄2 + s̄1
− sv12 v2 φ22 s̄1 , s̄2 + s̄21
s
v2 + 2v1 s̄1
ds


2
2
2
2
= v1 φ11 s̄1 , s̄2 + s̄1 − sv1 v2 φ22 s̄1 , s̄2 + s̄1




+ (v2 + 2v1 s̄1 ) 2v1 φ12 s̄1 , s̄2 + s̄21 + 4v1 s̄1 φ22 s̄1 , s̄2 + s̄21 + (v2 − 2v1 s̄1 )φ22 s̄1 , s̄2 + s̄21




= v12 φ11 s̄1 , s̄2 + s̄21 − (sv2 + 4s̄21 )φ22 s̄1 , s̄2 + s̄21 + v22 φ22 s̄1 , s̄2 + s̄21 .

Due to the surjectivity of T it holds that for all c1 ∈ R, c2 > 0, s < c2 /v2 there exists a
distribution F + ∈ F such that T1 (F + ) + sv1 = c1 and T2 (F + ) + sv2 = c2 . Consequently,

28

one obtains the lower bound




d
s−1 S̄(s̄, F + ) ≥ v12 φ11 c1 , c2 + c21 − (c2 + 4c21 )φ22 c1 , c2 + c21 + v22 φ22 c1 , c2 + c21
ds

and this bound is asymptotically attained for s ↑ c2 /v2 . As v2 can be arbitrarily small,
it is necessary and sufficient
for order sensitivity on

 line segments that the map A ∋
2
2
2
(c1 , c2 ) 7→ φ11 c1 , c2 + c1 − (c2 + 4c1 )φ22 c1 , c2 + c1 is non-negative which is equivalent
to (3.19). The reverse direction follows with analogous considerations.

Example 3.22. An example for a class of strictly convex C 3 -function φ : A′ → R satisfying (3.18) and (3.19) with equality is given by
φ(m1 , m2 ) = m2 − m21

−1

+ b1 m 1 + b2 m 2 + b3 ,

b1 , b2 , b3 ∈ R.

For the case b1 = b2 = b3 = 0, the resulting scoring function of the form at (3.17) is

x21 − 2x2 − 2x1 y + y 2 .
S(x1 , x2 , y) = x−2
(3.25)
2

Interestingly, this results not only in an order-sensitive scoring function on line segments
for the pair (mean, variance), but it is also a mixed positively homogeneous scoring
function of degree −2; see Section 4.2.
3.4.2. The pair (Value at Risk, Expected Shortfall)

Value at Risk (VaR) and Expected Shortfall (ES) are popular risk measures in banking
and insurance. For a financial position Y with distribution F and a level α ∈ (0, 1), they
are defined as
VaRα (F ) := F −1 (α) = inf{x ∈ R : F (x) ≥ α},
Z
1 α
VaRβ (F ) dβ
ESα (F ) :=
α 0

1
1
= EF [Y 1{Y ≤ VaRα (F )}] + VaRα (F ) α − F (VaRα (F )) .
α
α

Note that if F is continuous at VaRα (F ), that means, if F (VaRα (F )) = α, one can
write ESα (F ) equivalently as EF [Y | Y ≤ VaRα (F )]. Our sign convention implies that
risky positions yield large negative values of VaRα or ESα . Intuitively, VaRα gives the
worst loss out of the best (1 − α) × 100% of all cases, whereas ESα gives the average
loss given one exceeds VaRα . Merits and pitfalls of these two important risk measures
are discussed in Embrechts et al. (2014); Embrechts and Hofert (2014) where numerous
further references are given.
VaRα , as a quantile, is elicitable under mild regularity conditions, whereas ESα fails
to be elicitable (Gneiting, 2011). However, recently it was shown in Fissler and Ziegel

29

(2016a, Theorem 5.2 and Corollary 5.5) that the pair (VaRα , ESα ) is elicitable and the
class of strictly convex scoring functions was characterized to be of the form (3.26) (under
the conditions of Osband’s principle, Fissler and Ziegel (2016a, Theorem 3.2, Corollary
3.3)). Note that the proof of Fissler and Ziegel (2016a, Theorem 5.2(ii) and Corollary
5.5) is imprecise for the case that a distribution F ∈ F is not continuous at its α-quantile.
However, the arguments are easily adapted and the result holds as stated.
Proposition 3.23. Let α ∈ (0, 1), F be a class of continuously differentiable distribution
functions on R with finite first moments and unique α-quantiles. Let A ⊆ {(x1 , x2 ) ∈
R2 : x1 ≥ x2 } be convex. Define A2 as the projection of A onto the second coordinate
axis and let S : A × R → R be a scoring function of the form

S(x1 , x2 , y) = 1{y ≤ x1 } − α g(x1 ) − 1{y ≤ x1 }g(y)
(3.26)



y
x
1
− φ(x2 ),
− 1{y ≤ x1 }
+ φ′ (x2 ) x2 + 1{y ≤ x1 } − α
α
α
with g : R → R differentiable and increasing and φ : A2 → R twice differentiable, and
φ′ > 0, φ′′ > 0. If
φ′ (x) + (x − z)φ′′ (x) ≥ 0,

for all x, z ∈ A2 ,

(3.27)

then S is strictly F-order-sensitive on line segments for (VaRα , ESα ).
Proof. Let F ∈ F with density f , t = (t1 , t2 ) = T (F ), v = (v1 , v2 ) ∈ S2 , and s > 0 such
that t + sv ∈ A. Then, after some calculation, we find

1
d
S̄(t1 + sv1 , t2 + sv2 , F ) = (F (t1 + sv1 ) − α)v1 g′ (t1 + sv1 ) + φ′ (t2 + sv2 )
ds
α

 1 Z t1 +sv1
+ sv22 φ′′ (t2 + sv2 ) + v2 φ′′ (t2 + sv2 )
F (y)dy − sv1 . (3.28)
α t1
We have
αsv1 ≤

Z

t1 +sv1

F (y)dy ≤ F (t1 + sv1 )sv1 .

(3.29)

t1

Note that by assumption g′ ≥ 0, φ′ > 0, φ′′ > 0, and, furthermore (F (t1 +sv1 )−α)v1 > 0
for v1 6= 0. Therefore, if v2 ≥ 0, the first two summands on the right-hand side of (3.28)
are strictly positive and the last one is non-negative using the first inequality in (3.29).
For v2 < 0, we find using the second inequality in (3.29)
1
d
S̄(t1 + sv1 , t2 + sv2 , F ) > (F (t1 + sv1 ) − α)v1 φ′ (t2 + sv2 )
ds
α

1 ′′
+ v2 φ (t2 + sv2 ) F (t1 + sv1 ) − α sv1
α

1
= (F (t1 + sv1 ) − α)v1 φ′ (t2 + sv2 ) + sv2 φ′′ (t2 + sv2 ) ≥ 0,
α

where the last inequality is due to the assumption at (3.27).

30

Example 3.24. Consider the action domain A = {x ∈ R2 : x1 ≥ x2 , x2 < 0}, so A2 =
(−∞, 0). For all φ in the family {φb : (−∞, 0) → R : b ∈ (0, 1]} where φ1 (x) = − log(|x|),
x < 0 and for b ∈ (0, 1)
1
φb (x) =
|x|1−b , x < 0,
b−1
condition (3.27) is satisfied.
A strict F-identification function V : A × O → R for a functional T : F → A ⊆ R is
oriented for T if
V̄ (x, F ) > 0 ⇐⇒ x > T (F )
(3.30)
for all F ∈ F, x ∈ A (Lambert et al., 2008; Steinwart et al., 2014). One possible
generalization of orientation for higher-dimensional functionals is the following. Let
T : F → A ⊆ Rk be a functional with a strict F-identification function V : A × O → Rk .
Then V is called an oriented strict F-identification function for T if
v ⊤ V̄ (T (F ) + sv, F ) > 0

⇐⇒

s>0

for all v ∈ Sk−1 := {x ∈ Rk : kxk = 1}, for all F ∈ F and for all s ∈ R such that
T (F ) + sv ∈ A.
Our notion of orientation differs from the one proposed by Frongillo and Kash (2015a).
In contrast to their definition, our definition is per se independent of a (possibly nonexisting) strictly consistent scoring function for T . Moreover, whereas their definition
has connections to the convexity of the expected score, our definition shows strong ties
to order-sensitivity on line segments.
If the gradient of an expected score induces an oriented identification function, then
the scoring function is strictly order-sensitive on line segments, and vice versa. However,
the existence of an oriented identification function is not sufficient for the existence of
a strictly order-sensitive scoring function on line segments. The reason is that – due
to integrability conditions – the identification function is not necessarily the gradient of
some (scoring) function.

4. Equivariant functionals and order-preserving scoring
functions
Many statistical functionals have an invariance or equivariance property. For example,
the mean is a linear functional, and hence, it is equivariant under linear transformations.
So E[ϕ(X)] = ϕ(E[X]) for any random variable X and any linear map ϕ : R → R (of
course, the same is true for the higher-dimensional setting). On the other hand, the
variance is invariant under translations, that is Var(X − c) = Var(X) for any c ∈ R, but
scales quadratically, so Var(λX) = λ2 Var(X) for any λ ∈ R. The next definition strives
to formalize such notions.

31

Definition 4.1 (π-equivariance). Let F be a class of probability distributions on O and
A be an action domain. Let Φ be a group of bijective transformations ϕ : O → O, Φ∗ a
group of bijective transformations ϕ∗ : A → A, and π : Φ → Φ∗ be a map. A functional
T : F → A is π-equivariant if for all ϕ ∈ Φ
T (L(ϕ(Y ))) = (πϕ)(T (L(Y )))
for all random variables Y such that L(Y ) ∈ F.
Example 4.2. (i) For A = O = R, the mean functional is π-equivariant for Φ =
Φ∗ = {x 7→ x + c, c ∈ R} the translation group and π the identity map, or for
Φ = Φ∗ = {x 7→ λx, λ ∈ R \ {0}} the multiplicative group and again π the identity
map.
(ii) For A = O = Rk , the multivariate mean functional is π-equivariant for Φ = Φ∗ =
{x 7→ x + c, c ∈ Rk } the translation group and π the identity map.
(iii) For A = O = R, Value at Risk at level α, Expected Shortfall at level α and the
τ -expectile are π-equivariant for Φ = Φ∗ = {x 7→ x + c, c ∈ R} the translation
group and π the identity map, or for Φ = Φ∗ = {x 7→ λx, λ > 0} the multiplicative
group and again π the identity map.
(iv) For A = [0, ∞) and O = R, the variance is π-equivariant for Φ = {x 7→ x+ c, c ∈ R}
the translation group and Φ∗ = {idA } the trivial group consisting only of the
identity on A, such that π is the constant map.
(v) For A = [0, ∞) and O = R, the variance is π-equivariant for Φ = Φ∗ = {x 7→
λx, λ ∈ R \ {0}} the multiplicative group, and π((x 7→ λx)) = (x 7→ λ2 x).
(vi) Let A = Rk , O = R and T be the functional whose mth component is the mth
moment. Then T is π-equivariant with Φ = {y 7→ λy, λ ∈ R \ {0}}, Φ∗ = {x 7→
(λm xm )km=1 , λ ∈ R \ {0}}, and π((y 7→ λy)) = (x 7→ (λm xm )km=1 ).
If a functional T is elicitable, π-equivariance can also be expressed in terms of strictly
consistent scoring functions; see also Gneiting (2011, p. 750).
Lemma 4.3. Let S : A × O → R be a strictly F-consistent scoring function for a functional T : F → A and let π : Φ → Φ∗ . Then, T is π-equivariant if and only if for all
ϕ∈Φ
arg min S̄((πϕ)(x), L(ϕ(Y ))) = arg min S̄(x, L(Y ))
x∈A

x∈A

for all random variables Y such that L(Y ) ∈ F.
The proof of Lemma 4.3 is direct. It implies that the scoring function
Sπ,ϕ : A × O → R,

(x, y) 7→ Sπ,ϕ (x, y) = S((πϕ)(x), ϕ(y))

(4.1)

is also strictly F-consistent for T . Similarly to the motivation of order-sensitivity of
scoring functions, for fixed π : Φ → Φ∗ , it is a natural requirement on a scoring function
S that for all ϕ ∈ Φ the ranking of any two forecasts is the same in terms of S and in
terms of Sπ,ϕ .

32

Definition 4.4 (π-order-preserving). Let π : Φ → Φ∗ . A scoring function S : A × O → R
is π-order-preserving with respect to F if for all ϕ ∈ Φ one has


sgn S̄(x, F ) − S̄(x′ , F ) = sgn S̄π,ϕ (x, F ) − S̄π,ϕ (x′ , F )

for all F ∈ F and for all x, x′ ∈ A, where Sπ,ϕ is defined at (4.1). S is linearly π-orderpreserving if for all ϕ ∈ Φ and for all x, x′ ∈ A there is a λ > 0 such that

λ S(x, y) − S(x′ , y) = Sπ,ϕ (x, y) − Sπ,ϕ (x′ , y)
(4.2)

for all y ∈ O. If S is linearly π-order-preserving with a λ > 0 independent of x, x′ ∈ A,
then we call S uniformly linearly π-order-preserving.
The following lemma is immediate.

Lemma 4.5. Let π : Φ → Φ∗ . If a scoring function S : A × O → R is linearly π-orderpreserving, it is π-order-preserving with respect to any class F of probability distributions
on O.
The two practically most relevant examples of uniform linear π-order preservingness
are translation invariance and positive homogeneity of scoring functions, or, to be more
precise, of score differences. They are described in the two subsequent subsections.

4.1. Translation invariance
Consider a translation equivariant functional such as the mean treated in Example 4.2
(ii). Then, a scoring function S : Rk × Rk → R is said to have translation invariant score
differences if it is uniformly linearly π-equivariant with λ = 1 for all ϕ ∈ Φ. In formulae,
we require S to satisfy
S(x − z, y − z) − S(x′ − z, y − z) = S(x, y) − S(x′ , y)

(4.3)

for all x, x′ , y, z ∈ Rk . Note that what is particularly appealing is that the action domain
and the observation domain coincide and, in particular, have the same dimension. However, there are also other functionals such as vectors of different quantiles or expectiles,
or the vector (VaRα , ESα ) satisfying properties one can naturally call translation equivariant, but that have the drawback that A 6= O (typically, O is of lower dimension than
A). Then, translation invariance means that the score is invariant under a simultaneous
translation of the observation and the forecast along respective linear subspaces of A
and O.
Let A ⊆ Rk , O = Rd and m ∈ {1, . . . , min{k, d}}. Let MO ∈ Rd×m and MA ∈ Rk×m
be two matrices with rank m. Define the transformation groups
Φ := ΦMO := {y 7→ y − MO z, z ∈ Rm },
Φ∗ := Φ∗MA := {x 7→ x − MA z, z ∈ Rm },

33

where we impose that x − MA z ∈ A for all x ∈ A, z ∈ Rm . Then, the map π =
πMO ,MA : ΦMO → Φ∗MA naturally induced by MO and MA is given as
πMO ,MA ((y 7→ y − MO z)) = (x 7→ x − MA z).
We say that a functional T : F → Rk is linearly equivariant if there are such matrices
MO , MA such that T is πMO ,MA -equivariant.
Example 4.6. (i) Let O = R, A = {(x1 , x2 ) ∈ R2 : x2 ≤ x1 } and T = (VaRα , ESα )
with some generic F. Then T is πMO ,MA -equivariant with MO = idR and MA =
(1, 1)⊤ .
(ii) Let O = R, A = R × [0, ∞) and T = (mean, variance) : F → A where all F ∈ F
have finite second moments. Then T is πMO ,MA -equivariant with MO = idR and
MA = (1, 0)⊤ .
(iii) Let O = R, A = Rk and T be a vector of k different quantiles. Let M ∈ Rk×k
have rank at least 1 and consider the functional TM = M (T ). Then TM is πMO ,MA equivariant with MO = idR and MA = M (1, . . . , 1)⊤ .
Adopting this notion, we say that a scoring function S : A × Rd → R is linearly
(MO , MA )-invariant for two matrices MO ∈ Rd×m , MA ∈ Rk×m with rank(MO ) =
rank(MA ) = m ∈ {1, . . . , min{k, d}} if
S(x − MA z, y − MO z) = S(x, y)
for all x ∈ A, y ∈ Rd , z ∈ Rm . Similarly, we will speak about linearly (MO , MA )-invariant
identification functions and score differences.
Given a certain functional T : F → Rk and some MO ∈ Rd×m , MA ∈ Rk×m with
rank(MO ) = rank(MA ) = m ∈ {1, . . . , min{k, d}}, one can wonder about the class
of strictly consistent scoring functions that are linearly (MO , MA )-invariant. Clearly,
with respect to Lemma 4.3 and Lemma 4.5, this class is empty if the functional T is
not πMO ,MA -equivariant. In the situation that A = O = Rk and MO = MA = idRk the
following proposition characterizes the gradients of linearly (idRk , idRk )-invariant strictly
consistent scoring function (if such scoring functions exist).
Proposition 4.7. Let T : F → Rk be a surjective, identifiable functional with a linearly
(idRk , idRk )-invariant strict F-identification function V : Rk × Rk → Rk . Then, the
following assertions hold.
(i) T is πidRk ,idRk -equivariant.
(ii) Assume there is a strictly F-consistent scoring function S : Rk × Rk → R for T
with linearly (idRk , idRk )-invariant score differences. Then, under Assumptions
(V1) and (S1) in Fissler and Ziegel (2016a), there is a constant matrix h ∈ Rk×k
such that
∇S̄(x, F ) = h V̄ (x, F )
(4.4)
for all x ∈ Rk and for all F ∈ F.

34

Proof. If a random variable Y has distribution F with F ∈ F, we write F − z for the
distribution of Y − z where z ∈ Rk . To show the first part, consider any F ∈ F and
z ∈ Rk . Then
0 = EF [V (T (F ), Y )] = EF [V (T (F ) − z, Y − z)].
Since V is a strict F-identification function for T , T (F − z) = T (F ) − z.
For the second part, Fissler and Ziegel (2016a, Theorem 3.2) implies that there exists
a matrix-valued function h : Rk → Rk×k such that
∇S̄(x, F ) = h(x)V̄ (x, F )
for all x ∈ Rk and for all F ∈ F. We will show that h is constant. Since S̄(x, F ) −
S̄(x′ , F ) = S̄(x − z, F − z) − S̄(x′ − z, F − z) for all x, x′ , z ∈ Rk and F ∈ F, we obtain
by taking the gradient with respect to x
h(x)V̄ (x, F ) = h(x − z)V̄ (x − z, F − z) = h(x − z)V̄ (x, F ),

(4.5)

where the second identity is due to the linear (idRk , idRk )-invariance of V . So (4.5) is
equivalent to

V̄ (x, F ) ∈ ker h(x − z) − h(x) .

Now, one can use Assumption (V1) and Fissler and Ziegel (2016a, Remark 3.1), which
implies that

ker h(x − z) − h(x) = Rk .

Since x, z ∈ Rk were arbitrary, the function h is constant.

Using Fissler and Ziegel (2016a, Proposition 4.4) one can establish the converse of
Proposition 4.7: If V is a linearly (idRk , idRk )-invariant strict F-identification function,
then (4.4) implies that S has linearly (idRk , idRk )-invariant score differences. The following lemma shows how to normalize scores with translation invariant score differences to
obtain a translation invariant score.
Lemma 4.8. Let S : Rk × Rk → R a strictly F-consistent scoring function for T : F →
Rk with linearly (idRk , idRk )-invariant score differences. If for all y ∈ Rk , the point
measures δy are in F and the function y 7→ S(T (δy ), y) is F-integrable, then
S0 (x, y) := S(x, y) − S(T (δy ), y)

(4.6)

is a linearly (idRk , idRk )-invariant, non-negative, strictly F-consistent scoring function
for T .
Proof. If S has linearly (idRk , idRk )-invariant score differences, S satisfies (4.3) for all
x, x′ , y, z ∈ Rk . Due to Lemma 4.3, T must be πidRk ,idRk -equivariant, hence, T (δy ) − z =
T (δy−z ). This yields that S0 defined at (4.6) is linearly (idRk , idRk )-invariant. Since S
and S0 are of equivalent form, also S0 is strictly F-consistent for T . The non-negativity
follows directly from the fact that F contains all point measures and from the strict
consistency.

35

In case of the mean functional on R, Proposition 4.7 has already been shown by Savage
(1971) who showed that the squared loss is the only strictly consistent scoring function
for the mean that is of prediction error form, up to equivalence.15 Furthermore it implies
that general τ -expectiles and α-quantiles have essentially one linearly (idR , idR )-invariant
strictly consistent scoring function only, namely the canonical choices Sτ (x, y) = |1{y ≤
x} − τ |(x − y)2 and Sα (x, y) = (1{y ≤ x} − α)(x − y).
The uniqueness – up to equivalence – disappears for k > 1. For example, for the
the 2-dimensional mean functional, the previous results yield that any scoring function
S : R2 × R2 → R of the form
S(x, y) =

h22
h11
(x1 − y1 )2 +
(x2 − y2 )2 + h12 y2 (y1 − x1 ) + h12 x2 (x1 − y1 )
2
2

is strictly consistent for the 2-dimensional mean functional and linearly (idR2 , idR2 )invariant, for any h11 > 0 and h11 h22 − h212 > 0.
Due to the additive separability of strictly consistent scoring functions for vectors
consisting of different quantiles and expectiles (Fissler and Ziegel, 2016a, Proposition
4.2), strictly consistent scoring functions that are linearly (idR , idRk )-invariant for these
vectors are not unique. However, the only flexibility in that class consists in choosing
different weights for the respective summands of the scores.
The pair (mean, variance) is a πMO ,MA -equivariant functional with MO and MA as in
Example 4.6(ii). Curiously, it has a linearly (MO , MA )-invariant identification function
⊤
V (x1 , x2 , y) = x1 − y, x2 − (x1 − y)2 but does not possess a strictly consistent linearly
(MO , MA )-invariant scoring function.
Proposition 4.9. Let F be a class of distributions on R with finite second moments such
that the functional T = (mean, variance) : F → A is surjective on A = R × I,where I ⊆
[0, ∞) is an interval. Let Assumptions (F1) and (V1) from Fissler and Ziegel (2016a)
be satisfied with the strict F-identification functions V : A × R → R2 , V (x1 , x2 , y) =
⊤
⊤
x1 − y, x2 − (x1 − y)2
and V ∗ : A × R → R2 , V ∗ (x1 , x2 , y) = x1 − y, x2 + x21 − y 2 .
Let S : A × R → R be a F-consistent scoring function for T that is (jointly) continuous,
and for any y ∈ R, the function A ∋ x 7→ S(x, y) is twice continuously differentiable.
If S has linearly (MO , MA )-invariant score differences, then there is a λ ≥ 0 and an
F-integrable functional a : R → R such that
S(x1 , x2 , y) = λ(x1 − y)2 + a(y).
In particular, S cannot be strictly F-consistent for T .
Proof. Fissler and Ziegel (2016a, Theorem 3.2) asserts that there is a matrix-valued
function h : int(A) → R2×2 such that for all (x1 , x2 ) ∈ int(A) and for all F ∈ F we have
∇S̄(x, F ) = h(x1 , x2 )V̄ (x1 , x2 , F ).
15

That means that the scoring function is a function in x − y only.

36

(4.7)

Due to the special form of V and Assumption (F1), this equation holds also pointwise
for all y ∈ R. Moreover, the function h is continuously differentiable. Assume that S has
linearly (MO , MA )-invariant score differences. This implies, combined with the previous
result, that for all (x1 , x2 ) ∈ int(A), and for all y, z ∈ R
h(x1 , x2 )V (x1 , x2 , y) = ∇x S(x1 , x2 , y)
= ∇x S(x1 + z, x2 , y + z)
= h(x1 + z, x2 )V (x1 + z, x2 , y + z)
= h(x1 + z, x2 )V (x1 , x2 , y).
An application of Assumption (V1) (similarly to the proof of Proposition 4.7) yields that
h must be necessarily constant in its first argument.
On the other hand, arguing as in the proof of Proposition 3.21 with identification
function V ∗ , the revelation principle yields that


x1 − y
2
2
S(x1 , x2 , y) = −φ(x1 , x2 + x1 ) + ∇φ(x1 , x2 + x1 )
+ a(y),
(4.8)
x2 + x21 − y 2
where a : R → R is some F-integrable function and, due to our assumptions and
Fissler and Ziegel (2016a, Proposition 4.4), φ : A′ → R is C 3 and convex with gradient
∇φ and Hessian (φij )i,j=1,2 . Using the representation at (4.8), one obtains ∂2 S(x1 , x2 , y) =
φ22 (x1 , x2 + x21 )(x2 + x21 − y 2 ). A comparison to the form at (4.7) yields that
h22 (x1 , x2 ) = φ22 (x1 , x2 + x21 )
h21 (x1 , x2 ) = 2x1 φ22 (x1 , x2 + x21 ).
Since ∂1 h22 (x1 , x2 ) vanishes, we obtain that 0 = ∂1 h21 (x1 , x2 ) = 2φ22 (x1 , x2 + x21 ). As
the Hessian of φ must be positive semi-definite φ11 ≥ 0 and φ12 = φ21 = 0. Since φ
is C 3 , we have that ∂2 φ11 = ∂1 φ12 = 0, hence φ11 is constant in the first argument.
Equating the first component of (4.7) and (4.8) and using that h12 = h21 = 0, we find
that φ11 (x1 , x2 + x21 ) = h11 (x1 , x2 ). As h11 is constant in x1 , this implies that φ11 is also
constant in its second argument which yields the claim.
The functional (VaRα , ESα ), α ∈ (0, 1), is also a relevant πMO ,MA -equivariant functional with MO and MA as in Example 4.6(i). However, scoring functions with linearly
(MO , MA )-invariant score differences only exist for restricted classes of distribution functions F which may not be natural choices in risk management applications.
Proposition 4.10. Let α ∈ (0, 1). Let F be a class of distribution functions on R
with finite first moments and unique α-quantiles. Consider T = (VaRα , ESα ) : F →
{(x1 , x2 ) ∈ R2 : x2 ≤ x1 }. Then, the following assertions hold:
(i) Suppose there is some c > 0 such that
ESα (F ) + c > VaRα (F )

37

for all F ∈ F.

(4.9)

That is, T (F) ⊆ Ac := {(x1 , x2 ) ∈ R2 : x2 ≤ x1 < x2 + c}. Then, any scoring
function S : Ac × R → R, which is equivalent to
Sc (x1 , x2 , y) = (1{y ≤ x1 } − α)c(x1 − y) + α(x22 /2 + x21 /2 − x1 x2 )
+ 1{y ≤ x1 }(−x2 (y − x1 ) + y 2 /2 − x21 /2), (4.10)
is strictly F-consistent for T and has linearly (MO , MA )-invariant score differences
with MO = idR , MA = (1, 1)⊤ .
(ii) Under the conditions of Fissler and Ziegel (2016a, Theorem 5.2(iii)), there are
strictly F-consistent scoring functions for T with linearly (MO , MA )-invariant score
differences if and only if there is some c > 0 such that (4.9) holds. Then, any such
scoring function is necessarily equivalent to Sd defined at (4.10) almost everywhere,
with d ≥ c.
Proof. The scoring function Sc is of equivalent form as given at (3.26) with g(x1 ) =
−x21 /2 + cx1 and φ(x) = (α/2)x22 . This means that φ is strictly convex and the function
x1 7→ x1 φ′ (x2 )/α+g(x1 ) is strictly increasing in x1 if and only if x2 +c > x1 , that is, if and
only if (x1 , x2 ) ∈ Ac , such that we obtain the F-consistency of Sc with Fissler and Ziegel
(2016a, Theorem 5.2(ii)). A direct computation yields that Sc (x1 + z, x2 + z, y + z) =
Sc (x1 , x2 , y) for all (x1 , x2 ) ∈ Ac , y, z ∈ R. This proves the first part.
Under the conditions of Fissler and Ziegel (2016a, Theorem 5.2(iii)), any strictly Fconsistent scoring function S : A × R → R, where A = T (F), is almost everywhere of
the form given at (3.26) with g continuously differentiable and φ twice continuously
differentiable. By translation invariance of score differences the function Ψ : R × A × A ×
R → R,
Ψ(z, x1 , x2 , x′1 , x′2 , y) = S(x1 + z, x2 + z, y + z) − S(x′1 + z, x′2 + z, y + z)
− S(x1 , x2 , y) + S(x′1 , x′2 , y)
constantly vanishes. Let z, y ∈ R and (x1 , x2 ), (x′1 , x′2 ) ∈ A. Then
0=


d
1
Ψ(z, x1 , x2 , x′1 , x′2 , y) = x2 −x1 + 1{y ≤ x1 }(x1 −y) (φ′′ (x2 +z)−φ′′ (x2 )),
dx2
α

hence φ′′ is constant, that is, φ(x2 ) = d1 x22 + d2 x2 + d3 with d1 > 0 (ensuring the strict
convexity of φ) and d2 , d3 ∈ R. Similarly, the derivative of Ψ with respect to z must
vanish for all z, y ∈ R and (x1 , x2 ), (x′1 , x′2 ) ∈ A. A calculation yields
0=


d
Ψ(z, x1 , x2 , x′1 , x′2 , y) = 1{y ≤ x1 } − α g ′ (x1 + z) − 1{y ≤ x1 }g′ (y + z)
dz

− 1{y ≤ x′1 } − α g′ (x′1 + z) + 1{y ≤ x′1 }g′ (y + z)

2d1
+
1{y ≤ x1 }(x1 − y) − x1
α

2d1
1{y ≤ x′1 }(x′1 − y) − x′1 .
−
α
38

This implies that necessarily g′ (x1 ) = (−2d1 /α)x1 + d4 for some d4 ∈ R. Hence, g(x1 ) =
(−d1 /α)x21 + d4 x1 + d5 for some d5 ∈ R. Now, by Fissler and Ziegel (2016a, Theorem
5.3(iii)), the function
ψx2 (x1 ) = x1 φ′ (x2 )/α + g(x1 ) = x1 (2d1 x2 + d2 )/α − d1 x21 /α + d4 x1 + d5
must be strictly increasing in x1 which holds if and only if
x2 +

d2 + d4 α
> x1 .
2d1

This condition is satisfied for all (x1 , x2 ) ∈ A = T (F) if and only there is a c > 0 such
that T (F) ⊆ Ac and d := (d2 + d4 α)/(2d1 ) ≥ c. The scoring function at (3.26) with
φ(x2 ) = d1 x22 + d2 x2 + d3 , d1 > 0, d2 , d3 ∈ R, g(x1 ) = (−d1 /α)x21 + d4 x1 + d5 , d4 , d5 ∈ R
is equivalent to Sd defined at (4.10), which concludes the proof.
The scoring function Sc has a close relationship to the class of scoring functions S W
proposed in Acerbi and Szekely (2014); see Fissler and Ziegel (2016a, Equation (5.6)).
Indeed, Sc (x1 , x2 , y) = c 1{y ≤ x1 }−α (x−y)+S W (x1 , x2 , y) with W = 1. That means
it is the sum of the standard α-pinball loss for VaRα – which is translation invariant – and
S 1 . In the same flavor, the condition at (4.9) is similar to the one at Fissler and Ziegel
(2016a, Equation (5.7)). Since ESα ≤ VaRα , the maximal action domain where Sc is
strictly consistent is the stripe Ac = {(x1 , x2 ) ∈ R2 : x2 ≤ x1 < x2 + c}. Of course, by
letting c → ∞, one obtains the maximal sensible action domain {(x1 , x2 ) ∈ R2 : x1 ≥ x2 }
for the pair (VaRα , ESα ). However, considering the properly normalized version Sc /c,
this converges to a strictly consistent scoring function for VaRα as c → ∞, but which
is independent of the forecast for ESα . Hence, there is a caveat concerning the tradeoff
between the size of the action domain and the sensitivity in the ES-forecast. This might
cast doubt on the usage of scoring functions with translation invariant score differences
for (VaRα , ESα ) in general.
Interestingly, the scoring function Sc at (4.10) has positively homogeneous score differences if and only if c = 0. However, A0 = ∅, which means that the requirement of
translation invariance and homogeneity for score differences are mutually exclusive in
case of strictly consistent scoring functions for (VaRα , ESα ).

4.2. Homogeneity
If one is interested in a positively homogeneous functional of degree one such as the
mean, expectiles, quantiles, or ES, a scoring function S : R × R → R is said to have
positively homogeneous score differences of degree b ∈ R for this functional if the scoring
function is uniformly linearly π-equivariant with Φ = {R ∋ x 7→ cx ∈ R, c > 0} the
multiplicative group, π the identity on Φ and λ = cb in (4.2). This means that S needs
to satisfy

S(cx, cy) − S(cz, cy) = cb S(x, y) − S(z, y)
(4.11)

39

for all x, z, y ∈ R and c > 0. Since positive homogeneity of score differences is equivalent
to invariance of forecast rankings under a change of unit, it has been argued that it is
important in financial applications (Acerbi and Szekely, 2014). Nolde and Ziegel (2017)
give a characterization of scoring functions with positively homogeneous score differences
for many risk measures of applied interest, such as VaR / quantiles, expectiles, and the
pair (VaR, ES); cf. Patton (2011) for results concerning the mean functional.
If the functional T is vector-valued, the degree of homogeneity can be different in the
respective components, e.g. in case of the pair (mean, variance) or the vector consisting
of the first k moments; cf. Example 4.2(vi). One can denote this property by mixed
positive homogeneity, which means in case of the vector of the first k moments that
T (L(cY )) = Λ(c)T (L(Y ))

(4.12)

for all c > 0, where Λ(c) is the k×k-diagonal matrix with diagonal elements c, c2 , . . . , ck .16
In this situation, an interesting instant for uniformly linearly π-order-preserving scoring
functions S : A × R → R are those with mixed positively homogeneous score differences
of degree b ∈ R. That is,

S(Λ(c)x, cy) − S(Λ(c)z, cy) = cb S(xy) − S(z, y)
(4.13)

for all x, z ∈ A, y ∈ R, and for all c > 0. With k = 2, corresponding assertions hold for
the pair (mean, variance) and the respecitve scoring functions.

Proposition 4.11. Let A ⊆ Rk such that Λ(c)x ∈ A for all c > 0, x ∈ A. Let S : A×R →
R be a consistent scoring function for the vector of the first k moments of the form


(4.14)
S(x, y) = −φ(x) + ∇φ(x) x − (y, y 2 , . . . , y k )⊤ + a(y),
where φ : A → R is convex and differentiable with gradient ∇φ (considered as a row
vector). Then S has mixed positively homogeneous score differences of degree b ∈ R if
and only if for all c > 0 the map
x 7→ ∇φ(Λ(c)x)Λ(c) − cb ∇φ(x)

(4.15)

is constant.
Proof. Suppose φ satisfies (4.15). This implies that for any c > 0 the map z 7→ φ(Λ(c)z)−
cb φ(z) is an affine function. Moreover, a Taylor expansion yields that for all x, z ∈ A

φ(Λ(c)z) − cb φ(z) = ∇φ(Λ(c)x)Λ(c) − cb ∇φ(x) (z − x) + φ(Λ(c)x) − cb φ(x).

Then, a direct calculation yields the result.
Now, suppose (4.13) is satisfied. Its left-hand side equals

16

−φ(Λ(c)x) + ∇φ(Λ(c)x)Λ(c)x + φ(Λ(c)z) − ∇φ(Λ(c)z)Λ(c)z


+ ∇φ(Λ(c)z)Λ(c) − ∇φ(Λ(c)x)Λ(c) (y, y 2 , . . . , y k )⊤ ,

Of course, we tacitly assume that for all x ∈ A and for all c > 0, we have Λ(c)x ∈ A.

40

whereas the right-hand side is


−cb φ(x) + cb ∇φ(x)x + cb φ(z) − cb ∇φ(z)z + cb ∇φ(z) − ∇φ(x) (y, y 2 , . . . , y k )⊤ .

Both terms are polynomials in y of degree k, which leads to the identity


∇φ(Λ(c)z)Λ(c) − ∇φ(Λ(c)x)Λ(c) = cb ∇φ(z) − ∇φ(x) .
This is exactly condition (4.15).

Recall that the scoring functions of the form at (4.14) are essentially all consistent
scoring functions for the vector of different moments (Fissler and Ziegel, 2016a, Proposition 4.4). Using Proposition 4.11 it is straight forward to derive consistent scoring
functions for (mean, variance) with mixed positively homogeneous score differences.
Corollary 4.12. Let F be a class of distributions on R with finite second moments
such that the functional T = (mean, variance) : F → A ⊆ R × [0, ∞) is surjective,
where for all (x1 , x2 ) ∈ A and c > 0, (cx1 , c2 x2 ) ∈ A. Let Assumptions (F1) and
(V1) from Fissler and Ziegel (2016a) be satisfied with the strict F-identification function
⊤
V : A × R → R2 , V (x1 , x2 , y) = x1 − y, x2 + x21 − y 2 . Let S : A × R → R be a strictly
F-consistent scoring function for T that is (jointly) continuous and for any y ∈ R,
the function A ∋ x 7→ S(x, y) be twice continuously differentiable. Then S has mixed
positively homogeneous score differences of degree b ∈ R if and only if


x1 − y
2
2
S(x1 , x2 , y) = −φ(x1 , x2 + x1 ) + ∇φ(x1 , x2 + x1 )
+ a(y),
(4.16)
x2 + x21 − y 2
where φ : A → R is strictly convex, twice continuously differentiable, and moreover for
all c > 0 the map


c 0
2
2 2
A ∋ (x1 , x2 ) 7→ ∇φ(cx1 , c x2 + c x1 )
− cb ∇φ(x1 , x2 + x21 )
(4.17)
0 c2
is constant.
Proof. The form at (4.16) follows as in the proof of Proposition 3.21. The rest follows
by Proposition 4.11.
It appears that the class of (strictly) convex functions φ satisfying (4.15) is rather
flexible. One subclass is the class of additively separable functions φ. That is,
φ(x) =

k
X

φm (xm ),

(4.18)

m=1

where each φm needs to be convex and xm 7→ cm φ′m (cm xm )−cb φ′m (xm ) constant. Reviewing Nolde and Ziegel (2017, Theorem 5) and restricting attention to the case A ⊆ (0, ∞)k ,

41

φm can be an element of the class Ψb/m , where Ψb
of the form

b

d0 + d1 y /(b(b − 1)),
ψb (y) = d0 + d1 y log(y) + d2 y,


d0 − d1 log(y) + d2 y,

consists of functions ψb : (0, ∞) → R
for b ∈ R \ {0, 1}
for b = 1
for b = 0

with constants d1 > 0, d0 , d2 ∈ R. On the other hand, there are choices of φ not
satisfying such a additive decomposition as in (4.18). One such example can be found
in Example 3.22 for b = −2, and is of the form φ(x1 , x2 ) = (x2 − x21 )−1 for x2 > x21 .

Acknowledgement
Tobias Fissler would like to thank Tilmann Gneiting, Werner Ehm, Sebastian Lerch,
Alexander Jordan, Fabian Krüger, and Jonas Brehmer for stimulating and insightful
discussions during a research visit at the Heidelberg Institute for Theoretical Studies.
He is grateful to the Department of Mathematics at Imperial College London for funding
his Chapman Fellowship.

References
J. D. Abernethy and R. Frongillo. A characterization of scoring rules for linear properties. In
Proceedings of the 25th Annual Conference on Learning Theory, volume 23 of Proceedings of
Machine Learning Research, pages 27.1–27.13, Edinburgh, Scotland, 2012. PMLR.
C. Acerbi and B. Szekely. Backtesting expected shortfall. Risk Magazine, 27:76–81, 2014.
C. D. Aliprantis and K. C. Border. Infinite Dimensional Analysis: A Hitchhiker’s Guide.
Springer, Berlin Heidelberg New York, 3rd edition, 2006.
A. Banerjee, X. Guo, and H. Wang. On the optimality of conditional expectation as a bregman
predictor. IEEE Transactions on Information Theory, 51:2664–2669, 2005.
F. Bellini and V. Bignozzi. On elicitable risk measures. Quantitative Finance, 15:725–733, 2015.
J. R. Brehmer. Elicitability and its application in risk management. Master’s thesis, University
of Mannheim, 2017. URL http://arxiv.org/abs/1707.09604.
F. X. Diebold and R. S. Mariano. Comparing predictive accuracy. Journal of Business and
Economic Statistics, 13:253–263, 1995.
W. Ehm, T. Gneiting, A. Jordan, and F. Krüger. Of quantiles and expectiles: Consistent scoring
functions, Choquet representations and forecast rankings. Journal of the Royal Statistical
Society: Series B (Statistical Methodology), 78(3):505–562, 2016.
P. Embrechts and M. Hofert. Statistics and quantitative risk management for banking and
insurance. Annual Review of Statistics and Its Application, 1:493–514, 2014.
P. Embrechts, G. Puccetti, L. Rüschendorf, R. Wang, and A. Beleraj. An academic response to
Basel 3.5. Risks, 2:25–48, 2014.
T. Fissler. On Higher Order Elicitability and Some Limit Theorems on the Poisson and Wiener
Space. PhD thesis, University of Bern, 2017.

42

T. Fissler and J. F. Ziegel. Higher order elicitability and Osband’s principle. Annals of Statistics,
44:1680–1707, 2016a.
T. Fissler and J. F. Ziegel. Supplement to “Higher order elicitability and Osband’s principle”.
Annals of Statistics, 44:1680–1707, 2016b.
D. Friedman. Effective scoring rules for probabilistic forecasts. Management Science, 29:447–454,
1983.
R. Frongillo and I. Kash. Vector-valued property elicitation. In Proceedings of The 28th Conference on Learning Theory, volume 40 of Proceedings of Machine Learning Research, pages
710–727, Paris, France, 2015a. PMLR.
R. Frongillo and I. Kash. On elicitation complexity. In C. Cortes, N. D. Lawrence, D. D. Lee,
M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems
28, pages 3258–3266. Curran Associates, Inc., 2015b.
T. Gneiting. Making and evaluating point forecasts. Journal of the American Statistical Association, 106:746–762, 2011.
C. Heinrich. The mode functional is not elicitable. Biometrika, 101:245–251, 2014.
P. J. Huber. Robust estimation of a location parameter. Annals of Mathematical Statistics, pages
73–101, 1964.
P. J. Huber and E. M. Ronchetti. Robust Statistics. John Wiley & Sons, Inc., Hoboken, New
Jersey, 2nd edition, 2009.
R. Koenker. Quantile Regression. Cambridge University Press, Cambridge, 2005.
K. Königsberger. Analysis 2. Springer-Verlag, Berlin Heidelberg New York, 5th edition, 2004.
N. Lambert. Elicitation and Evaluation of Statistical Functionals. Preprint, 2013. URL
http://web.stanford.edu/~nlambert/papers/elicitation.pdf.
N. Lambert, D. M. Pennock, and Y. Shoham. Eliciting properties of probability distributions.
In Proceedings of the 9th ACM Conference on Electronic Commerce, pages 129–138, Chicago,
Il, USA, 2008. ACM.
E. L. Lehmann and G. Casella. Theory of Point Estimation. Springer Science & Business Media,
New York, 2nd edition, 1998.
A. H. Murphy and H. Daan. Forecast evaluation. In A. H. Murphy and R. W. Katz, editors, Probability, Statistics and Decision Making in the Atmospheric Sciences, pages 379–437. Westview
Press, Boulder, Colorado, 1985.
R. F. Nau. Should scoring rules be ‘effective’ ? Management Science, 31:527–535, 1985.
W. K. Newey and J. L. Powell. Asymmetric Least Squares Estimation and Testing. Econometrica,
55:819–847, 1987.
N. Nolde and J. F. Ziegel. Elicitability and backtesting: Perspectives for banking regulation.
Annals of Applied Statistics, 2017. To appear.
K. H. Osband. Providing Incentives for Better Cost Forecasting. PhD thesis, University of
California, Berkeley, 1985.
A. J. Patton. Data-based ranking of realised volatility estimators. Journal of Econometrics, 161:
284–303, 2011.
A. J. Patton. Comparing possibly misspecified forecasts. Working paper, Duke University, 2017.
URL http://public.econ.duke.edu/~ap172/.
S. Reichelstein and K. Osband. Incentives in government contracts. Journal of Public Economics,
24:257–270, 7 1984.

43

M. Saerens. Building cost functions minimizing to some summary statistics. IEEE Transactions
on Neural Networks, 11:1263–1271, Nov 2000.
L. J. Savage. Elicitation of personal probabilities and expectations. Journal of the American
Statistical Association, 66:783–801, 1971.
J. Scott and G. Marshall. A Dictionary of Sociology. Oxford University Press, 3 rev. edition,
2009.
I. Steinwart. How to compare different loss functions and their risks. Constructive Approximation,
26:225–287, 2007.
I. Steinwart, C. Pasin, R. Williamson, and S. Zhang. Elicitation and identification of properties. In Proceedings of The 27th Conference on Learning Theory, volume 35 of Proceedings of
Machine Learning Research, pages 482–526, Barcelona, Spain, 2014. PMLR.
W. Thomson. Eliciting production possibilities from a well-informed manager. Journal of Economic Theory, 20:360–380, 6 1979.
A. W. van der Vaart. Asymptotic Statistics. Cambridge University Press, Cambridge, 1998.
S. Weber. Distribution-invariant risk measures, information, and dynamic consistency. Mathematical Finance, 16:419–441, 2006.

Appendix
A. Refinements of Fissler and Ziegel (2016a)
As detailed Brehmer (2017) there are two technicalities that need to be resolved in
Fissler and Ziegel (2016a, Proposition 3.4): Firstly, due to the particular choice of the
integration path in the original version of Fissler and Ziegel (2016a, Proposition 3.4),
the image of the integration path is not necessarily contained in int(A). Secondly, one
needs to assume that the identification function V is locally bounded jointly in the
two components. Proposition A.1 gives a refined version of Fissler and Ziegel (2016a,
Proposition 3.4).
Proposition A.1. Assume that int(A) ⊆ Rk is simply connected and let T : F → A be
a surjective, elicitable and identifiable functional with a strict F-identification function
V : A × O → Rk and a strictly F-consistent scoring function S : A × O → R. Suppose
that Assumption (V1), (V2), (S1) from Fissler and Ziegel (2016a) are satisfied. Let h
be the matrix-valued function appearing at Fissler and Ziegel (2016a, Equation (3.2)).
For any F ∈ F and any points x, z ∈ int(A) such that γ : [0, 1] → int(A) is an
integration path with γ(0) = x, γ(1) = z the score difference is necessarily of the form
S̄(x, F ) − S̄(z, F ) =

Z

γ

dS̄(·, F ) =

Z

1
0



h γ(λ) V̄ γ(λ), F γ ′ (λ) dλ .

(A.1)

Moreover, if Assumptions (F1) and (VS1) from Fissler and Ziegel (2016a) are satisfied
and V is locally bounded, then there is a Lebesgue null set N ⊆ A × O such that for all

44

(x, y) ∈ N c , (z, y) ∈ N c it necessarily holds that
S(x, y) − S(z, y) =

Z

dS(·, y) =
γ

Z

0

1



h γ(λ) V γ(λ), y γ ′ (λ) dλ ,

(A.2)

where again γ : [0, 1] → int(A) is an integration path with γ(0) = x, γ(1) = z.
Proof. Equation (A.1) follows from Fissler and Ziegel (2016a, Theorem 3.2) and Königsberger
(2004, Satz 2, p. 183). The proof of (A.2) follows the lines of the original proof in
Fissler and Ziegel (2016b); cf. Brehmer (2017, Theorem 1.31) for details.

45

