arXiv:1212.4966v3 [] 27 Nov 2017

Combining p-values via averaging
Vladimir Vovk
v.vovk@rhul.ac.uk
http://vovk.net
November 28, 2017
Abstract
This note discusses the problem of multiple testing of a single hypothesis, with a standard goal of combining a number of p-values without
making any assumptions about their dependence structure. An old result
by Rüschendorf shows that the p-values can be combined by scaling up
their arithmetic mean by a factor of 2 (but no smaller factor is sufficient in
general). More recent results in mathematical finance show, in addition,
that K p-values can be combined by scaling up their geometric mean by
a factor of e and by scaling up their harmonic mean by a factor of e ln K.
This leads to a generalized version of the Bonferroni–Holm method.
The version of this paper at http://alrw.net (Working Paper 21) is updated most often.

1

Introduction

Suppose we are testing the same hypothesis using K ≥ 2 different statistical
tests and obtaining p-values p1 , . . . , pK . How can we combine them into a single
p-value?
One of the earliest papers answering this question was Fisher’s [4]. However,
Fisher’s paper assumes that the p-values are independent, whereas we would like
to avoid any assumptions besides all pk , k = 1, . . . , K, being bona fide p-values.
Fisher’s method has been extended to dependent p-values in, e.g., [2, 12], but
the combined p-values obtained in those papers are approximate; in this note
we are interested only in precise or conservative p-values.
The simplest way of combining p-values is the Bonferroni method:
F (p1 , . . . , pK ) := K min(p1 , . . . , pK )

(1)

(when F (p1 , . . . , pK ) exceeds 1 it can be replaced by 1, but we usually ignore
this trivial step). Albeit F (p1 , . . . , pK ) is a p-value, it has been argued that in
many cases it is overly conservative. Rüger [18] extends the Bonferroni method

1

by showing that, for any fixed k ∈ {1, . . . , K},
F (p1 , . . . , pK ) :=

K
p(k)
k

(2)

is a p-value, where p(k) is the kth smallest p-value among p1 , . . . , pK ; see [16]
for a simpler exposition. Hommel [8] develops this by showing that


1
K
1
min
p(k)
(3)
F (p1 , . . . , pK ) := 1 + + · · · +
2
K k=1,...,K k
is also a p-value. (Simes [20] improves (3) by removing the first factor on the
right-hand side of (3), but he assumes the independence of p1 , . . . , pK .)
Intuitively, the most natural way to combine K numbers is to average them;
my interest in this topic started from this way of combining p-values being used
in the method of cross-conformal prediction ([21], (11)), which uses arithmetic
mean. None of the functions F in (1), (2), and (3) involves the arithmetic mean
1
p̄ := K
(p1 + · · · + pK ), although the Bonferroni method (1) can be considered as
a scaled version of a generalized mean, min(p1 , . . . , pK ) (see, e.g., [6], (2.3.1)).
Section 2 of this note draws the reader’s attention to a result by Rüschendorf
([19], Theorem 1) showing that p̄ is not always a p-value but 2p̄ is; moreover,
the factor of 2 cannot be improved in general. That section also gives a simple
proof of the part of Rüschendorf’s result stating that 2p̄ is a bona fide p-value
(perhaps conservative). Section 3 considers the case K = 2, in which it is very
easy to see that the factor of 2 is optimal. Section 4 discusses implications for
cross-conformal prediction. Section 5 considers a generalized notion of mean
as axiomatized by Andrei Kolmogorov [11] and adapts various results of robust
mathematical finance to combining p-values by averaging in Kolmogorov’s wider
sense. In particular, to obtain a p-value from given p-values p1 , . . . , pK , it is
sufficient to multiply their geometric mean by e and to multiply their harmonic
mean by e ln K.
It is often possible to automatically transform results about multiple testing of a single hypothesis into results about testing multiple hypotheses; the
standard procedures are Marcus et al.’s [15] closed testing procedure and its
modification by Hommel [9]. In particular, when applied to the Bonferroni
method the closed testing procedure gives the well-known method due to Holm
[7], which we will refer to as the Bonferroni–Holm method; see, e.g., [9, 10] for
its further applications. In Section 6 we briefly discuss a similar application to
one of the procedures of Section 5.

Some notation and terminology
If E is a property of elements of a set X, 1E : X → [0, ∞) is the indicator
function of E: 1E (x) = 1 if x satisfies E and 1E (x) = 0 if not. A function
F : [0, 1] → [0, ∞) is increasing (resp. decreasing) if F (x1 ) ≤ F (x2 ) (resp.
F (x1 ) ≥ F (x2 )) whenever x1 ≤ x2 . A function F : [0, 1]K → [0, ∞) is increasing
(resp. decreasing) if it is increasing (resp. decreasing) in each of its arguments. A
2

set in [0, 1]K is increasing (resp. decreasing) if its indicator function is increasing
(resp. decreasing).

2

Combining p-values by scaled arithmetic
mean

A p-value function is a random variable P that satisfies
P(P ≤ ) ≤ ,

∀ ∈ [0, 1].

(4)

The values taken by a p-value function are p-values (allowed to be conservative). (In Section 1 the expression “p-value” was loosely used to refer to
p-value functions as well.) A merging function is an increasing Borel function F : [0, 1]K → [0, ∞) such that F (U1 , . . . , UK ) is a p-value function, where
U1 , . . . , UK are random variables distributed uniformly on [0, 1].
Remark. The requirement that a merging function be Borel does not follow
automatically from the requirement that it be increasing: see the remark after
Theorem 4.4 in [5] (Theorem 4.4 itself says that every increasing function on
[0, 1]K is Lebesgue measurable).
Notice that, for any merging function F , F (P1 , . . . , PK ) is a p-value function
whenever P1 , . . . , PK are p-value functions. Indeed, for each k ∈ {1, . . . , K} we
can define a uniformly distributed random variable Uk ≤ Pk by
Uk (ω) := P(Pk < Pk (ω)) + θ P(Pk = Pk (ω)), ω ∈ Ω,
where θ is a random variable distributed uniformly on [0, 1] and independent of
P1 , . . . , Pk , and Ω is the underlying probability space extended (if required) to
carry such a θ; we then have
P(F (P1 , . . . , PK ) ≤ ) ≤ P(F (U1 , . . . , UK ) ≤ ) ≤ ,

∀ ∈ [0, 1].

The following proposition states Rüschendorf’s [19] result in terms of merging functions.
Proposition 1. The function M : [0, 1]K → [0, ∞) defined by
M (p1 , . . . , pK ) := 2

p1 + · · · + pK
K

(5)

is a merging function.
The rest of this section is devoted to a self-contained proof of Proposition 1.
A copular probability measure is a probability measure on [0, 1]K all of whose
marginals are uniform probability measures on [0, 1]. The upper copular probability C(E) of a Borel set E ⊆ [0, 1]K is defined to be the supremum of x(E),
x ranging over the copular probability measures. In terms of C, an increasing Borel function F : [0, 1]K → [0, ∞) is a merging function if and only if
3

C(F ≤ ) ≤  for all  ∈ [0, 1]. We say that a merging function F is precise if
C(F ≤ ) =  for all  ∈ [0, 1].
For m ∈ [0, 1], define


K u1 + · · · + uK
Em := (u1 , . . . , uK ) ∈ [0, 1]
≤ m ⊆ [0, 1]K .
K
Proposition 1 can be strengthened: in fact, M is a precise merging function.
The original statement of this result, in our notation, is as follows.
Lemma 1 ([19], Theorem 1). For any m ∈ [0, 1],
C(Em ) = min(2m, 1).
Remark. In Section 1 we already alluded to an example of a set with a known
upper copular probability: the set

(u1 , . . . , uK ) ∈ [0, 1]K 1u1 ≤α + · · · + 1uK ≤α ≥ k ,
where α ∈ [0, 1] and k ∈ {1, . . . , K}, has upper copular probability of (K/k)α;
this is equivalent to (2) being a precise merging function. Another well-known
example is H := [0, u1 ] × · · · × [0, uK ], where u1 , . . . , uK ∈ [0, 1]. The upper copular probability of H is min(u1 , . . . , uK ). This is known as one of the
Fréchet–Hoeffding bounds in the theory of copulas, and is known to be a copula (see, e.g., [17], Exercise 2.35). Lemma 1 is one more example of this kind.
Lemma 2 below will give a simple characterization of upper copular probability
in the easy case K = 2.
Given Lemma 1, the proof of Proposition 1 is trivial: for any  ∈ [0, 1],



U1 + . . . + UK
≤
= min(, 1) ≤ .
P(M (U1 , . . . , UK ) ≤ ) = P
K
2
Notice that for the proof of Proposition 1 we only need the inequality ≤ in the
lemma. The rest of this section is devoted to the proof of this inequality.
Let K[0, 1] be the sum of K disjoint copies of the interval [0, 1]. A (somewhat
arbitrary) concrete representation of K[0, 1] is the set ∪K
k=1 [2(k − 1), 2k − 1]. We
will sometimes use the notation K[0, 1]k for the kth copy of [0, 1] in K[0, 1]; so
that K[0, 1]k = [2(k − 1), 2k − 1] in the concrete representation (but K[0, 1]k is
always identified with [0, 1], via the bijection u 7→ u − 2(k − 1) in the concrete
representation). If x is a measure on [0, 1]K , we define xk to be the projection
of x onto the kth coordinate of [0, 1]K ,
xk (E) := x([0, 1]k−1 × E × [0, 1]K−k ), E ⊆ [0, 1] is Borel,
and we define Ax to be the measure on K[0, 1] that coincides with xk on K[0, 1]k
(so that Ax’s total mass is K when x is a probability measure). The uniform
measure on K[0, 1] is the measure on the Borel σ-algebra on K[0, 1] that coincides with the uniform probability measure on each of its components K[0, 1]k
4

(so that Ax is the uniform measure on K[0, 1] if and only if x is a copular
probability measure).
Lemma 1 can be interpreted as a statement about the following infinitedimensional problem of linear programming:
cx → sup

subject to Ax = b, x ≥ 0,

(6)

where c is the indicator function of the set REm , the variable x ranges over all
measures on [0, 1]K , cx is understood to be cdx, Ax is as defined above, and
b is the uniform measure on K[0, 1]. The condition x ≥ 0 is an embellishment
without a formal meaning (and emphasizes the fact that measures take only
nonnegative values). Lemma 1 says that the value of (6) is 2m when m ≤ 1/2.
The formal dual problem to (6) is
λb → inf

subject to

λA ≥ c,

(7)

which we will interpret as follows: the dual
R variable λ ranges over all Borel
functions on K[0, 1], λb is understood to be λdb, λA is the function on [0, 1]K
defined by
(λA)(u1 , . . . , uK ) := λ1 (u1 ) + · · · + λK (uK ),
where λk is the restriction of λ to K[0, 1]k , and ≥ stands, as usual, for the
pointwise inequality.
It is easy to see that the operators x 7→ Ax and λ 7→ λA are dual, in the
sense that (λA)x = λ(Ax):
Z
Z
Z
(λA)x = λ1 dx1 + · · · + λK dxK = λdAx = λ(Ax).
(This justifies using the same letter for both operators.) As usual, the value
of the original problem (6) does not exceed the value of the dual problem (7):
indeed, if x satisfies the constraints in (6) and λ satisfies the constraint in (7),
cx ≤ (λA)x = λ(Ax) = λb.
Now we have all components for the proof of the inequality ≤ in Lemma 1.
Proof of the inequality ≤ in Lemma 1. It suffices to prove that the value of the
dual problem (7) does not exceed 2m. Define λ : K[0, 1] → [0, ∞) by λk (u) :=
(2/K − u/Km)+ for all k ∈ {1, . . . , K}, where t+ is t if t ≥ 0 and 0 otherwise.
(In other words, assuming m ≤ 1/2, λk : [0, 1] → [0, ∞) is the function with the
subgraph of the smallest area among all functions that are linear when positive
and whose graph passes through (m, 1/K).) Since
Z
Z
λb = λ1 + · · · + λK ≤ 2m
(with = in place of the last ≤ when m ≤ 1/2), it remains to prove that the
constraint in (7) is satisfied. This is accomplished by the following chain of
5

inequalities:
+
K 
X
2
uk
≥
−
λA(u1 , . . . , uK ) =
K
Km
k=1

!+
K 
X
2
uk
−
K
Km

k=1
+

= (2 − (u1 + · · · + uK )/Km) ≥ 12−(u1 +···+uK )/Km≥1
= 1(u1 +···+uK )/K≤m = c(u1 , . . . , uK ).

3

Case K = 2

In the case K = 2 upper copular probability admits a simple characterization.
Lemma 2. If a nonempty Borel set E ⊆ [0, 1]2 is decreasing, its upper copular
probability is


C(E) = min inf u1 + u2 (u1 , u2 ) ∈ [0, 1]2 \ E , 1 .
(8)
Lemma 2 implies that the factor 2 in (5) is optimal for K = 2: indeed, it shows
that the function Mα (p1 , p2 ) := α(p1 + p2 ), where α > 0, satisfies C(Mα ≤ ) =
min(/α, 1) for all  ∈ [0, 1]; therefore, Mα is a merging function if and only if
α ≥ 1. It is clear that M1 = M is the only precise merging function among Mα .
Proof of Lemma 2. Let E be a nonempty decreasing Borel set in [0, 1]2 ; suppose
C(E) is strictly less than the right-hand side of (8). Let t be any number strictly
between C(E) and the right-hand side of (8). The copular probability measure
concentrated on
[(t, 0), (0, t)] ∪ [(t, 1), (1, t)]
has a value of at least t on E since E contains [(t, 0), (0, t)]. Therefore, C(E) ≥ t.
This contradiction proves the inequality ≥ in (8).
As for the opposite inequality, we will check

C(E) ≤ inf u1 + · · · + uK (u1 , . . . , uK ) ∈ [0, 1]K \ E
for an arbitrary K ≥ 2. Let us assume that E does not contain the set of all
(u1 , . . . , uK ) with u1 + · · · + uK = 1 (the case when it does is trivial). Choose
 > 0 and (p1 , . . . , pK ) ∈ [0, 1]K \ E such that t := p1 + · · · + pK ∈ [, 1] and E
contains all (u1 , . . . , uK ) ∈ [0, 1]K satisfying u1 + . . . + uK = t − . Since E is
decreasing, we have
E⊆

K
[


(u1 , . . . , uK ) ∈ [0, 1]K uk ≤ pk ,

k=1

and the subadditivity of C further implies
C(E) ≤

K
X

C



(u1 , . . . , uK ) ∈ [0, 1]K uk ≤ pk

k=1

6



=

K
X


pk = t ≤ inf u1 + · · · + uK (u1 , . . . , uK ) ∈ [0, 1]K \ E + .

k=1

It remains to notice that  can be chosen arbitrarily small.
A merging function F1 dominates a merging function F2 if F1 ≤ F2 . The
following corollary of Lemma 2 says that, in the case K = 2, the merging
function (5) is dominated by all precise merging functions. This is not true when
K > 2: for example, for the Bonferroni function (1) we have M (p, . . . , p) = 2p <
Kp = F (p, . . . , p). (The Bonferroni merging function being precise, C(F ≤ ) =
, is witnessed by the probability distribution on [0, 1]K that is uniform inside
the K-dimensional rectangles [0, /K] × [/K, 1] × · · · × [/K, 1],. . . , [/K, 1] ×
· · · × [/K, 1] × [0, /K], [/K, 1] × · · · × [/K, 1] and assigns to them probabilities
/K,. . . , /K, 1 − , respectively.)
Corollary 1. When K = 2, any precise merging function dominates M .
Proof. Let F : [0, 1]2 → [0, ∞) be a merging function that does not dominate
M . Choose (u1 , u2 ) ∈ [0, 1]2 such that F (u1 , u2 ) > u1 + u2 and choose  ∈
(u1 + u2 , F (u1 , u2 )). Since {F ≤ } does not contain (u1 , u2 ), we have C(F ≤
) ≤ u1 + u2 < , and so F is not precise.

4

Connections with conformal prediction

This section assumes the knowledge of basic definitions of conformal prediction
(see, e.g., [22]). The method of cross-conformal prediction, already mentioned
in Section 1, defines putative p-values as, essentially, the arithmetic means of
the p-values computed from K folds (cf. [21], (11)). The experiments reported
in [21] confirm the empirical validity of the arithmetic means in the sense of
approximately satisfying (4), although Appendix A of [21] gives examples showing that in general (4) may be violated. In their recent paper [13] Linusson
et al. give examples of particularly unstable randomized underlying algorithms
which make the arithmetic means used in the method of conformal prediction
violate (4) in their experimental studies. This makes the phenomenon discussed
theoretically in Appendix A of [21] a practical issue. Proposition 1 shows that
there is a limit to the degree to which the validity of cross-conformal predictors
can be violated, even for the most unstable underlying algorithms: namely, we
always have
P(P ≤ ) ≤ 2, ∀ ∈ [0, 1],
(9)
where P is the arithmetic mean of p-values.
In fact, cross-conformal predictors output only approximate arithmetic
means of p-values; the precise expression ([21], (11)) for their putative p-values
p is
K −1
p = p̄ +
(p̄ − 1),
l+1

7

where l is the size of the training set, K is the number of equal-sized folds,
and p̄ is the arithmetic mean of the p-values computed from the different folds.
Expressing p̄ via p,


l+1
K −1
p̄ =
p+
l+K
l+1
and using Proposition 1, we obtain




K −1
l+1
p+
≤ δ ≤ 2δ
P
l+K
l+1
for any δ > 0. Rewriting the last inequality as


l+K
K −1
P p≤
δ−
≤ 2δ
l+1
l+1
and setting
 :=

l+K
K −1
δ−
,
l+1
l+1

we finally obtain
K −1
(1 − )
(10)
l+K
for any  > 0. The last equation, (10), is the precise version of (9) in the case
where p is the putative p-value output by a cross-conformal predictor; we can
see that (10) is marginally weaker.
P (p ≤ ) ≤ 2 + 2

5

More general averages

So far we have only considered averaging in the sense of arithmetic mean. A
much more general notion of averaging, axiomatized by Kolmogorov [11], is


φ(p1 ) + · · · φ(pK )
Mφ (p1 , . . . , pK ) := ψ
,
(11)
K
where φ : [0, 1] → [−∞, ∞] is a continuous strictly monotonic function and
ψ is its inverse (with the domain φ([0, 1])). For example, arithmetic mean
corresponds to the identity function φ(p) = p, geometric mean corresponds to
φ(p) = ln p, and harmonic mean corresponds φ(p) = 1/p. In this section we will
extend Proposition 1 to the general class of means (11).
The main results of this section, Theorems 1, will be very simple corollaries
of known results in the field of mathematical finance dealing with robust versions
of the Value at Risk (VaR); the origin of this field lies in a problem posed by
Kolmogorov (see, e.g., [14]).
Theorem 1. Suppose a continuous strictly monotonic φ : [0, 1] → [−∞, ∞] is
R1
integrable, i.e., 0 |φ(u)| du < ∞. Then, for any  > 0,

 Z 

1
P Mφ (p1 , . . . , pK ) ≤ ψ
φ(u)du
≤ .
(12)
 0
8

Proof. Without loss of generality we can, and will, assume that φ is strictly
increasing. Indeed, if φ is strictly decreasing, we can redefine φ := −φ and
ψ(u) := ψ(−u) and notice that the statement of Proposition 1 for new φ and ψ
will imply the analogous statement for the original φ and ψ. In the rest of this
proof, φ and, therefore, ψ are assumed monotonically increasing.
Theorem 1 of [1] implies
K
X

LTVaR (Xk ) ≤ VaR (S) ≤

k=1

K
X

TVaR (Xk ),

(13)

k=1

where S := X1 + · · · + XK , all Xk and S are assumed to have continuous and
strictly decreasing distribution functions and to possess the first moment,
Z 1
1
VaRu (X)du,
TVaR (X) :=
1− 
Z 
1
VaRu (X)du,
LTVaR (X) :=
 0
−1
VaR (X) ∈ FX
()
−1
(the set FX
() is a singleton and, therefore, VaR (X) is well-defined), and
FX is the distribution function of X. We will only use the lower bound in (13),
applying it to the random variables Xk := φ(pk ). They possess the first moment
R1
since we assumed 0 |φ(u)| du < ∞.
We have

FXk (x) = P(φ(pk ) ≤ x) = P(pk ≤ ψ(x)) = ψ(x)
and, therefore, VaRu (Xk ) = ψ −1 (u) = φ(u). This gives
Z
Z
1 
1 
VaRu (Xk )du =
φ(u)du.
LTVaR (Xk ) =
 0
 0
Plugging this into the lower bound in (13) gives
Z
K 
φ(u)du ≤ FS−1 (),
 0
where S = φ(p1 ) + · · · + φ(pK ) and the singleton FS−1 () is identified with its
only element. The last inequality can be rewritten as


Z
K 
φ(u)du ≤ ,
P S≤
 0
which is equivalent to

 Z 

1
P ψ(S/K) ≤ ψ
φ(u)du
≤ ,
 0
which in turn is equivalent to (12).
9

A special case of (11) is

Mr (p1 , . . . , pK ) :=

pr1 + · · · + prK
K

1/r
,

where r ∈ R \ {0} and the following standard conventions are used: 0c := ∞ for
c < 0, 0c := 0 for c > 0, ∞ + c := ∞ for c ∈ R ∪ {∞}, and ∞c := 0 for c < 0.
The case r = 0 is treated separately (as the limit as r → 0):


ln p1 + · · · ln pK
M0 (p1 , . . . , pK ) := exp
,
K
where, as usual, ln 0 := −∞, −∞+c := −∞ for c ∈ R∪{−∞}, and exp(−∞) :=
0. It is also natural to set
M∞ (p1 , . . . , pK ) := max(p1 , . . . , pK ),
M−∞ (p1 , . . . , pK ) := min(p1 , . . . , pK ).
The most important special cases of Mr are perhaps those corresponding to
r = −∞ (minimum), r = −1 (harmonic mean), r = 0 (geometric mean), r = 1
(arithmetic mean), and r = ∞ (maximum); the cases r ∈ {−1, 0, 1} are known
as Platonic means.
Corollary 2. Let r ∈ (−1, ∞]. Then the function
M (p1 , . . . , pK ) := (r + 1)1/r Mr (p1 , . . . , pK )
is a merging function.
The expression (r + 1)1/r is understood to be e = limr→0 (r + 1)1/r when
r = 0 and 1 = limr→∞ (r + 1)1/r when r = ∞.
Proof. Evaluating the term
ψ


 Z 
1
φ(u)du
 0

(14)

in (12), we obtain:
• when r = 0,
 Z 

 Z 



1
1
1

ψ
φ(u)du = exp
ln udu = exp
[u ln u − u]0 = /e;
 0
 0

• when r 6= 0,
ψ

 Z 
  Z 
1/r   r+1  1/r
1
1
1 u
φ(u)du =
ur du
=
= (r + 1)−1/r .
 0
 0
 r+1 0

10

The condition r > −1 in Corollary 2 ensures that the term (14) is finite,
R1
and also that the condition 0 |φ(u)| du < ∞ in Theorem 1 is satisfied. For
arithmetic mean it gives the same result as before: namely, Proposition 1 is a
special case of Corollary 2 corresponding to r = 1. However, the condition rules
out harmonic mean (for which r = −1) and the minimum (r = −∞). The next
simple corollary of another known result will cover those cases as well.
Theorem 2. Suppose φ : [0, 1] → [−∞, ∞] is a strictly decreasing continuous
function satisfying φ(0) = ∞. Then, for any  ∈ [0, 1] such that φ() ≥ 0,
R φ()+(K−1)t
P (Mφ (p1 , . . . , pK ) ≤ ) ≤

φ()−t

inf

ψ(u)du
.

t

t∈(0,φ()]

(15)

Proof. We will apply Theorem 4.2 of [3] in our situation where the function
φ (and, therefore, ψ as well) in (11) is decreasing. Letting Xk := φ(pk ) and
dropping “+” in m+ (in the notation of Theorem 4.2 of [3]), we have, by the
definition of m,
!
K
X
P
Xk < s ≥ m(s),
k=1

P

K
1 X
φ(pk ) < s/K
K

!
≥ m(s),

k=1

P (Mφ (p1 , . . . , pK ) > ψ(s/K)) ≥ m(s),
P (Mφ (p1 , . . . , pK ) ≤ ψ(s/K)) ≤ 1 − m(s).
The lower bound on m(s) given in Theorem 4.2 of [3] involves 1 − F (x), where
F is the common distribution function of Xk , and in our current context we
have:
1 − F (x) = P(Xk > x) = P(φ(pk ) > x) = P(pk < ψ(x)) = ψ(x).
The last inequality and chain of equalities in combination with Theorem 4.2 of
[3] give
R s−(K−1)r
P (Mφ (p1 , . . . , pK ) ≤ ψ(s/K)) ≤ K

inf
r∈[0,s/K)

r

ψ(x)dx
.
s − Kr

Setting  := ψ(s/K) ∈ [ψ(∞), ψ(0)] (so that it is essential that ψ(∞) = 0), we
obtain, using s = Kφ(),
R Kφ()−(K−1)r
P (Mφ (p1 , . . . , pK ) ≤ ) ≤ K

inf
r∈[0,φ())

r

ψ(x)dx
.
(φ() − r)K

Setting t := φ() − r and renaming x to u, this can be rewritten as (15).

11

As t → 0, the upper bound in (15) is not informative since, for t ≈ 0,
R φ()+(K−1)t
φ()−t

ψ(u)du

t

≈

Ktψ(φ())
= K,
t

which is dominated by the Bonferroni bound. On the other hand, the upper
bound is informative when t = φ() provided the integral is convergent. For
example, we have the following corollary.
Corollary 3. For r < −1, we have, for any  ∈ [0, 1],
P (Mr (p1 , . . . , pK ) ≤ ) ≤

r
K 1+1/r .
r+1

(16)

Proof. By Theorem 2 applied to φ(u) := ur , r < −1, we have:
R Kr 1/r
u du
ψ(u)du
= 0
P (Mφ (p1 , . . . , pK ) ≤ ) ≤
φ()
r
 1/r+1 Kr
r
u
−r
=
K 1+1/r .
=
1/r + 1 0
r+1
R Kφ()
0

Corollary 3 includes the Bonferroni bound (1): for r := −∞, the right-hand
side of (16) is K. On the other hand, it does not cover the case r = −1 directly,
although it easily implies it.
Corollary 4. Suppose K > 2. For any  ∈ [0, 1],
P (M−1 (p1 , . . . , pK ) ≤ ) ≤ (e ln K).

(17)

r
Proof. Let us find the smallest value of the coefficient r+1
K 1+1/r in front of 
in (16). Setting the derivative in r of the logarithm of this coefficient to 0, we
obtain a linear equation whose solution is

r=

ln K
.
1 − ln K

(18)

Plugging this into the coefficient gives
ln K
1−ln K
ln K
1−ln K +

1

K 1+

1−ln K
ln K

=

1
ln K
K ln K = e ln K.
ln K + 1 − ln K

It remains to notice that r defined by (18) satisfies r < −1 and apply the
inequality Mr ≤ M−1 ([6], Theorem 16).
We stated Corollary 4 in a weakened form because of the importance of
harmonic mean as one of the three Platonic means, but the proof shows that in
fact M−1 in (17) can be replaced by Mr ≤ M−1 for
r :=

ln K
< −1
1 − ln K
12

Figure 1: The parts of the spheres Mr =  for K = 2,  = 0.4, and r ∈
{∞, 2, 1, 0, −1, −∞} inside the square [0, 1]2 . The innermost (orange) sphere
is M∞ = , the next one is M2 =  (blue, Euclidean sphere, corresponding to
quadratic mean), then M1 =  (black and straight, corresponding to arithmetic
mean), then M0 =  (magenta, corresponding to geometric mean), then M−1 = 
(red, corresponding to harmonic mean), and finally M−∞ =  (outermost and
green, corresponding to minimum).
(cf. (18); the inequality assumes K > 2).
The discussion in Section 3 shows that the bounds in Corollaries 2, 3, and 4
are far from optimal in the case K = 2. Those corollaries can be restated as
saying that the function

1/r

if r > −1
(r + 1) Mr (p1 , . . . , pK )
M (p1 , . . . , pK ) := (e ln K)Mr (p1 , . . . , pK )
if r = −1

 r
1+1/r
K
M
(p
,
.
.
.
,
p
)
if
r < −1
r 1
K
r+1
(the first case becoming eM0 (p1 , . . . , pK ) when r = 0) is a merging function. Applying Lemma 2 to the balls E := {(p1 , p2 ) | Mr (p1 , p2 ) ≤ } (whose boundaries
are shown in Figure 1), we can see that in fact the smaller function
(
21/r Mr (p1 , p2 ) if r ≥ 1
M (p1 , p2 ) :=
2Mr (p1 , p2 )
if r ≤ 1
is a merging function when K = 2.

13

Algorithm 1 Generalized Bonferroni–Holm procedure
Require: A significance level  > 0 and parameter r < −1 (or, w.l.o.g., (19)).
Require: A sequence of p-values p1 , . . . , pK ordered as pk1 ≤ · · · ≤ pkK .
for k = 1, . . . , K do
reject := true
I := {k}
for i = K, . . . , 1, 0 do
1+1/r
r
|I|
Mr (pI ) >  then
if r+1
reject := false
end if
I := I ∪ {ki }
end for
if reject = true then
reject Hk
end if
end for

6

Application to testing multiple hypotheses

In this section we apply the results of the previous sections, concerning multiple
testing of a single hypothesis, to testing multiple hypotheses. Namely, we will
arrive at a generalization of the Bonferroni–Holm method [7]. Fix a parameter
r≤

ln K
1 − ln K

(19)

(cf. (18)); the Bonferroni–Holm case is r := −∞.
Suppose pk is a p-value for testing a composite null hypothesis Hk (meaning
that, for any  > 0, P(pk ≤ ) ≤  under Hk ). For I ⊆ {1, . . . , K}, let HI be
the hypothesis

HI := (∩k∈I Hk ) ∩ ∩k∈{1,...,K}\I Hkc ,
where Hkc is the complement of Hk . Fix a significance level . Let us reject HI
when
r
1+1/r
|I|
Mr (pI ) ≤ ,
r+1
where pI is the vector of pk for k ∈ I; by Corollary 3, the probability of error
will be at most . If we now reject Hk when all HI with I ⊇ {k} are rejected, the
family-wise error rate will be at most . This gives the procedure described as
Algorithm 1, in which (k1 , . . . , kK ) stands for a fixed permutation of {1, . . . , K}
such that pk1 ≤ · · · ≤ pkK .

Acknowledgments
I am grateful to Dave Cohen, Alessio Sancetta, and Wouter Koolen for their
advice. This work was partially supported by the Cyprus Research Promotion
Foundation.
14

References
[1] Carole Bernard, Ludger Rüschendorf, and Steven Vanduffel. Value-at-Risk
bounds with variance constraints. Journal of Risk and Insurance, 84:923–
959, 2017.
[2] Morton B. Brown. A method for combining non-independent, one-sided
tests of significance. Biometrics, 31:987–992, 1975.
[3] Paul Embrechts and Giovanni Puccetti. Bounds for functions of dependent
risks. Finance and Stochastics, 10:341–352, 2006.
[4] Ronald A. Fisher. Combining independent tests of significance. American
Statistician, 2:30, 1948.
[5] Benjamin T. Graham and Geoffrey R. Grimmett. Influence and sharpthreshold theorems for monotonic measures. Annals of Probability, 34:1726–
1745, 2006.
[6] G. H. Hardy, John E. Littlewood, and George Pólya. Inequalities. Cambridge University Press, Cambridge, England, second edition, 1952.
[7] Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6:65–70, 1979.
[8] Gerhard Hommel. Tests of the overall hypothesis for arbitrary dependence
structures. Biometrical Journal, 25:423–430, 1983.
[9] Gerhard Hommel. Multiple test procedures for arbitrary dependence structures. Metrika, 33:321–336, 1986.
[10] Gerhard Hommel. A stagewise rejective multiple test procedure based on
a modified Bonferroni test. Biometrika, 75:383–386, 1988.
[11] Andrei N. Kolmogorov. Sur la notion de la moyenne. Atti della Reale
Accademia Nazionale dei Lincei. Classe di scienze fisiche, matematiche, e
naturali. Rendiconti Serie VI, 12(9):388–391, 1930.
[12] James T. Kost and Michael P. McDermott. Combining dependent p-values.
Statistics and Probability Letters, 60:183–190, 2002.
[13] Henrik Linusson, Ulf Norinder, Henrik Boström, Ulf Johansson, and Tuve
Löfström. On the calibration of aggregated conformal predictors. Proceedings of Machine Learning Research, 60:154–173, 2017.
[14] G. D. Makarov. Estimates for the distribution function of the sum of two
random variables with given marginal distributions. Theory of Probability
and its Applications, 26:803–806, 1981.
[15] Ruth Marcus, Eric Peritz, and K. Ruben Gabriel. On closed testing procedures with special reference to ordered analysis of variance. Biometrika,
63:655–660, 1976.
15

[16] Dietrich Morgenstern. Berechnung des maximalen Signifikanzniveaus des
Testes “Lehne H0 ab, wenn k unter n gegebenen Tests zur Ablehnung
führen”. Metrika, 27:285–286, 1980.
[17] Roger B. Nelsen. An Introduction to Copulas. Springer, New York, second
edition, 2006.
[18] Bernhard Rüger. Das maximale Signifikanzniveau des Tests “Lehne H0
ab, wenn k unter n gegebenen Tests zur Ablehnung führen”. Metrika,
25:171–178, 1978.
[19] Ludger Rüschendorf. Random variables with maximum sums. Advances in
Applied Probability, 14:623–632, 1982.
[20] R. John Simes. An improved Bonferroni procedure for multiple tests of
significance. Biometrika, 73:751–754, 1986.
[21] Vladimir Vovk.
Cross-conformal predictors.
Technical Report
arXiv:1208.0806v1 [stat.ML], arXiv.org e-Print archive, August 2012.
[22] Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning
in a Random World. Springer, New York, 2005.

16

