arXiv:1712.03305v1 [math.ST] 8 Dec 2017

FALSE DISCOVERY CONTROL FOR PAIRWISE
COMPARISONS—AN ASYMPTOTIC SOLUTION TO WILLIAMS,
JONES AND TUKEY’S CONJECTURE
WEIDONG LIU, DENNIS LEUNG, AND QIMAN SHAO
Abstract. Under weak moment and asymptotic conditions, we offer an affirmative answer to whether the BH procedure (Benjamini and Hochberg, 1995)
can control the false discovery rate in testing pairwise comparisons of means
under a one-way ANOVA layout. Specifically, despite the fact that the two
sample t-statistics do not exhibit positive regression dependency (Benjamini
and Yekutieli, 2001), our result shows that the BH procedure can asymptotically control the directional false discovery rate as conjectured by Williams,
Jones, and Tukey (1999). Such a result is useful for most general situations
when the number of variables is moderately large and/or when idealistic assumptions such as normality and a balanced design are violated.

1. Introduction
Suppose we have m independent groups of observations Xi = {Xki , 1 ≤ k ≤ ni },
1 ≤ i ≤ m, where for each i, Xki ’s are independent and identically distributed
random variables with mean µi and variance σi2 . The pairwise comparison problem
(1.1)

H0ij : µi = µj

against H1ij : µi 6= µj , 1 ≤ i < j ≤ m

has been widely studied since Tukey’s early work on multiple comparisons. With the
goal of controlling the family-wise error rate (FWER), Tukey (1953) and Kramer
(1956) independently proposed their famous procedure for testing (1.1) based on
the studentized range distribution. Such developments culminated in the work of
Hayter (1984) who established the conservativeness of the Tukey-Kramer procedure
in the affirmative. In modern applications when m can be large, the number of

hypotheses q := m
2 = m(m−1)/2 to consider is even larger, making any procedure
aiming to control FWER too conservative to be useful. As a more scalable error
measure for modern large scale testing problems, the false discovery rate (FDR),
defined as the expectation of the false discovery proportion (FDP),
#hypotheses incorrectly rejected
,
#hypotheses rejected
was proposed by Benjamini and Hochberg (1995), in which it was demonstrated
that their step-up procedure (widely known as the BH procedure, or “BH” for
short) can control FDR at a pre-specified level 0 < α < 1 when the test statistics
are independent.
2000 Mathematics Subject Classification. 62H05.
1

2

W. LIU, D. LEUNG, AND Q. SHAO

However, the test statistics for the pairwise comparisons problem (1.1), namely,
the two-sample t-statistics,
(1.2)

X̄i − X̄j
, 1≤i<j≤m
Tij = q
σ̂i2 /ni + σ̂j2 /nj

Pni
Pni
where X̄i = n1i k=1
Xki is the sample mean and σ̂i2 = ni1−1 k=1
(Xki − X̄i )2 is
the sample variance, are not independent. To put it into context, let
(1.3)

pij,U = 1 − Fij (Tij ) and pij,L = Fij (Tij ),

be the one-sided upper and lower tailed (approximate) p-values for each Tij , where
Fij can be an exact or approximate cumulative distribution function for Tij under
the null µi = µj . Typically, Fij ’s are taken to be the distribution functions for the
standard normal or some t-distributions. The BH procedure would stipulate that
we first sort all the q two sided p-values, defined as
(1.4)

p|ij| = 2 min(pij,L , pij,U ) for each pair (i, j),

as their order statistics
p(1) ≤ p(2) ≤ . . . ≤ p(q) ,
and reject a hypothesis Hij whenever p|ij| ≤ p(k̂) for k̂ = max{i : p(i) ≤ iα/q}.
Conjecturing that the BH procedure can still control the FDR at level α, Williams,
Jones, and Tukey (1999) conducted an extensive simulation study to verify this
with satisfactory results, to quote their ending remark: “Each of the three authors
believes that the B-H procedure is the best available choice”. Again, the same problem has been mentioned by subsequent work of Benjamini and Yekutieli (2001,
p.1882):
“Another important open question is whether the same procedure
controls the FDR when testing pairwise comparisons of normal
means, either Studentized or not”,
in which they established the validity of the “BH” in controlling FDR under positive
regression dependence of the test statistics. Nevertheless, such a condition is also
not satisfied by the two-sample t-statistics.
In this paper, we pursue an asymptotic treatment, for the regime where m, n −→
∞ with n := maxi ni , and show that “BH” is valid for (1.1). In many applications,
m and q can be large, for example, (m, q) = (10, 45) in Oishi et al. (1998), (m, q) =
(41, 820) in Williams, et al. (1999) and (m, q) = (72, 2556) in Pawluk-Kolc et al.
(2006). Hence, it is important to prove the conjecture when q → ∞. Moreover,
the normality conditions Xki ∼ N (µi , σi2 ), alongside a balanced design, are always
assumed in the ANOVA literature . In real applications any of these assumptions
can be easily violated, and an asymptotic treatment renders us the flexibility to
address more general situations.
Williams et al. (1999) actually took the view that in reality, none of the null
hypotheses in (1.1) can be true, i.e., µi − µj can at best be close to zero to many
decimal points for any given pair (i, j). Thus, a type 1 error of “rejecting a true
null” could never occur. Nevertheless, one can still make sense of the BH procedure:

3

Following the rejection of a hypothesis Hij , depending on which of pij,U and pij,L
is smaller, a declaration of the sign of the quantity µi − µj will be made by the
practitioner. If pij,U is less than pij,L , or equivalently, Tij > 0, then sgn(µi − µj )
will be declared as positive, and vice versa. In this new context, the false discovery
rate can be modified as


#{(i, j) : sgn(µi − µj ) is incorrectly declared}
.
E
#{(i, j) : Hij is rejected}

In numerical studies where none of the µi ’s are set to be equal, Williams et al.
(1999) showed that this modified FDR can be controlled at α/2 by applying “BH”
with the nominal level α followed by sign declarations of µi − µj . This is intuitive
because in an extremely error-prone situation with all differences µi − µj being very
close to zero, every Tij stochastically behaves almost as if µi − µj = 0, in which case
upon rejecting Hij there is approximately 1/2 chance of making false declaration
about sgn(µi − µj ). Such is known as a directional error. In this paper, following
work such as Benjamini and Hochberg (2000) and Benjamini and Yekutieli (2005),
we acknowledge the possibility that some null hypotheses in (1.1) could in fact be
true, and define directional error more generally as follows:
Definition 1.1. A directional error for a pair (i, j) is made, when either sgn(µi −µj )
is incorrectly declared after rejection of Hij when µi −µj 6= 0, or the null hypothesis
Hij is rejected when µi − µj = 0.
With Definition 1.1, we formally define the directional false discovery proportion
(dFDP) as
(1.5)

dF DP =

#{(i, j) : a directional error is made for (i, j)}
#{(i, j) : Hij is rejected}

and the directional false discovery rate (dFDR) is the expectation thereof. Note
that a directional error boils down to the usual type 1 error for the two sided tests
when µi − µj = 0. Under weak conditions, our main results imply that the “BH”
followed by sign declarations can asymptotically control both the dFDP and dFDR
at (i) the usual nominal level α when all the nulls are true, and (ii) at the level α/2
as demonstrated in the simulations of Williams et al. (1999) when none of the nulls
are true.
The rest of the paper is organized as follows. Section 2 give the main results
showing Williams, Jones, and Tukey (1999)’s conjecture holds under some weak
conditions. Section 3 proves a uniform weak law of large numbers that is crucial
for our main result. Section 4 concludes with some numerical studies. Throughout,
C, c are positive constants whose values, unless otherwise specified, are understood
to be independent of (m, n), and may vary from place to place. The dependence of
m = mn on n is implicit. For two sequences of real numbers {an } and {bn }, an . bn
if there exists a constant C such that |an | ≤ C|bn | holds for all sufficiently large n.
The notation Card(A) denotes the cardinality of a set A. Φ(·) is the cumulative
distribution function of the standard normal and Φ̄(·) = 1 − Φ(·). φ is the standard
normal density. logd means taking log for d times and for two functions f and g,
f (x) ∼ g(x), x → ∞ means f (x)/g(x) −→ 1 as x goes to infinity.

4

W. LIU, D. LEUNG, AND Q. SHAO

2. Theoretical results
We formally restate Williams et al. (1999)’s version of the BH procedure with
declaration of signs for the µi − µj ’s after rejections of null hypotheses below.
Definition 2.1 (Williams et al. (1999)’s BH procedure for the testing (1.1) at level
α).
(i) Obtain the two sided p-values p|ij| according to (1.2), (1.3) and (1.4),
and sort them as the order statistics p(1) ≤ p(2) ≤ . . . ≤ p(q) , where
q = m(m − 1)/2.
(ii) Let k̂ = max{k : p(k) ≤ kα/q}, and reject Hij whenver p|ij| ≤ p(k̂) .
(iii) For each rejected Hij , declare sgn(µi − µj ) > 0 if Tij > 0, and declare
sgn(µi − µj ) < 0 if Tij < 0.
Our main result, Theorem 2.1 below, is proved under the following assumptions:
Assumption 1. There exist 0 < cL ≤ 1 ≤ cU such that for all 1 ≤ i 6= j ≤ m,
cL ≤ σi2 /σj2 , ni /nj ≤ cU .

Assumption 2. Suppose that for some constants r, C, ǫ, K > 0,
m ≤ Cnr and max E|(X1i − µi )/σi |4r+2+ǫ ≤ K
1≤i≤m

Assumption 3. For all i < j, Fij are monotone increasing distribution functions,
with the properties that Fij (t) = 1 − Fij (−t) for t ≥ 0 and
max

sup

1≤i<j≤m 0≤t≤2√log m

1 − Fij (t)
− 1 = O(n−c ) as m, n −→ ∞
Φ̄(t)

for some c > 0, where the dependence of the distribution functions Fij on (m, n1 , . . . , nm )
is suppressed for notational simplicity.
Theorem 2.1. Let H0 := {(i, j) : i < j, µi − µj = 0} and q0 = |H0 |. Under
Assumptions 1 to 3 and the signal size condition




p
|µi − µj |
(2.1)
Card (i, j) : 1 ≤ i < j ≤ m, q
≥ 8c2U log m ≥ 1,


σ 2 /n + σ 2 /n
i

i

j

j

we have
(2.2)

lim sup
m,n

and for any ǫ > 0,
(2.3)
as m, n −→ ∞.

P

α
2

dF DR

 ≤1
1 + qq0





α
q0
dF DP <
1+
+ ǫ −→ 1
2
q

Our theorem is an asymptotic analog to Corollary 3 and 6 in Benjamini and
Yekutieli (2005), although these previous results do not cover the types of dependency exhibited among the two sample t-statistics (1.2). One can see that the term

5

+ qq0 ) becomes α when all hypotheses are true (q0 = q) and α/2 when no hypotheses are true (q0 = 0), validating our claim in Section 1. The theorem is also
proven under very modest assumptions. Assumption 1 only requires our setup to
be approximately balanced, while Assumption 2 allows the growth of m in n to be
of a polynomial order that depends on the variables’ moments. In particular, 4r + ǫ
can in fact be less than 1, despite the commensurate limited growth rate of m in
that case. Assumption 3 says that we only need to form our p-values with reference
distribution functions that are approximately normal in a uniform way. Trivially,
it will hold when Fij ’s are taken to be Φ(·). By the Cramér-type moderate deviation for t-statistics (Jing et al., 2003), it will also hold when the Fij ’s are taken
as t-distribution functions with, say, min(ni , nj ) − 1 degrees of freedom (Scheffé,
1943), under Assumptions 1 and 2. Lastly, (2.1) is a very minimal condition that is
needed for control of the dFDP in (2.3), in particular only one (i, j) pair is required
to give a prominent signal; the reader can compare to condition (2) in Liu and Shao
(2014) and Corollary 2.2 therein to see that such condition is almost necessary for
FDP-type control.
Now we begin to prove Theorem 2.1. In fact, we only need to prove (2.3),
since it implies that lim supm,n E[dF DP ]/ α2 (1 + qq0 ) ≤ 1 + ǫ for any ǫ > 0, and
the arbitrariness of ǫ will give (2.2). These notations will be used throughout this
section and the next: Let H+ := {(i, j) : i < j, µi − µj > 0} and H− := {(i, j) :
i < j, µi − µj < 0}. Define q+ := |H+ | and q− = |H− |, and hence q = q+ + q− + q0 .
P
For notational brevity, we will also use H+ to denote a summation over all pairs
P
P
P
P
(i, j) in the set H+ , and use H− , H0 , H+ ∪H0 and H− ∪H0 similarly. Finally,
for each pair i < j, we define the centered versions of the two-sample t statistics
α
2 (1

T̄ij =

(X̄i − X̄j ) − (µi − µj )
q
.
σ̂i2 /ni + σ̂j2 /nj

Note that the rejection rule in Definition 2.1 (ii) is equivalent to the classical
BH procedure, so by Theorem 2 in Benjamini and Hochberg (1995) it is equivalent
to a procedure that rejects Hij whenever |p|ij| | < α̂, where




αX
1(p|ij| ≤α̃) .
α̂ = sup 0 ≤ α̃ ≤ 1 : α̃ ≤


q i<j
P
Since, with probability one, αq i<j 1(p|ij| ≤α̃) is right continuous as a function in α̃
on the interval [0, 1], by elementary arguments it can be shown that
αX
1(p|ij| ≤ α̂).
(2.4)
α̂ =
q i<j
If, for every ǫ > 0, we can show that, as m, n −→ ∞,

(2.5)
P

sup

√
2Φ̄( 2 log m)≤α̃≤1

P

H+

1(pij,L ≤α̃/2) +

P

H−

1(pij,U ≤α̃/2) +

α̃(q + q0 )/2

P

H0

1(p|ij| ≤α̃)

≤1+ǫ

−→p 1

!

6

W. LIU, D. LEUNG, AND Q. SHAO

and
(2.6)

p
P (2Φ̄( 2 log m) ≤ α̂ ≤ 1) −→p 1,

then (2.3) is proven since, by (2.4), the expression
P
P
P
2q[ H+ 1(pij,L ≤α̂/2) + H− 1(pij,U ≤α̂/2) + H0 1(p|ij| ≤α̂) ]
2q
P
dF DP =
α(q + q0 )
α(q + q0 ) i<j 1(p|ij| ≤α̂)

is less than 1 + ǫ with probability tending to one. (2.5) essentially amounts to
proving a uniform law of large number, which will be done in Section 3. We will
focus on showing the probabilistic bound for the cutoff point α̂ in (2.6) for the rest
of this section.
We claim that, under Assumption 1, there exists a subset Hs ⊂ {(i, j) : 1 ≤ i <
j ≤ m} such that,

p
|µi − µj |
(2.7) Card{Hs } = ⌊m/2⌋ and q
≥ 4 log m for each (i, j) ∈ Hs .
σi2 /ni + σj2 /nj
By (2.1), without loss of generality we can assume
(2.8)

p
|µ1 − µ2 |
2
p
log m.
≥
8c
U
σ12 /n1 + σ22 /n2

|µ1 −µj |
σ12 /n1 +σj2 /nj

There can only be two cases: If Card{j : 3 ≤ j ≤ m, √

√
≥ 4 log m} ≥

m/2, then (2.7) is satisfied; otherwise, we have




p
|µ1 − µj |
(2.9)
Card j : 3 ≤ j ≤ m, q
< 4 log m ≥ m/2.


σ12 /n1 + σj2 /nj

Since |µ2 − µj | ≥ |µ2 − µ1 | − |µ1 − µj |, by Assumption 1,

|µ2 − µ1 |
|µ2 − µj |
cU |µj − µ1 |
q
≥ p 2
.
−q
2
cU σ2 /n2 + σ1 /n1
σ22 /n2 + σj2 /nj
σj2 /nj + σ12 /n1

Recalling that cU ≥ 1, together with (2.8) and (2.9) we have




p
|µ2 − µj |
Card j : 3 ≤ j ≤ m, q
≥ 4 log m ≥ m/2


σ22 /n2 + σj2 /nj

and hence (2.7) also holds. In Appendix A.1 we will also prove that


σ̂i2
−2
− 1 > (log m)
= O(n−r−c ).
(2.10)
P max
1≤i≤m σi2
for some c > 0, which, together with (2.7), gives


p
|µ
−
µ
|
i
j
≥ 3.9 log m −→ 1.
(2.11)
P  min q
(i,j)∈Hs
σ̂i2 /ni + σ̂j2 /nj

7

On the other hand, by the Cramér-type moderate deviation for two-sample tstatistics (see Jing et al. (2003), also see Cao (2007)[Theorem 1.2] with a slight
change to the moment conditions), under Assumptions 1 and 2 we have

4r+2+ǫ
1+s
P (T̄ij ≥ s)
= 1 + O(1)
(2.12)
Φ̄(s)
n1/2−1/(4r+2+ǫ)
uniformly in s ∈ [0, o(n1/2−1/(4r+2+ǫ) )) and 1 ≤ i < j ≤ m. A union bound then
implies


p
p
(2.13) P
max T̄ij ≥ 2 log m ≤ ⌊m/2⌋(2 + o(1))Φ̄( 2 log m)
(i,j)∈Hs

= O((log m)−1/2 ) −→ 0

using the fact that Φ̄(x) ≤ (2π)−1/2 x−1 exp(−x2 /2) for all x > 0.
In view of (2.11) and (2.13), we have that


X
P
1(|Tij |≥2√log m) ≥ ⌊m/2⌋ −→ 1.
1≤i<j≤m

√
By Assumption 3, |Tij | ≥ 2 log m implies p|ij| = o(m−2 ), so recognizing q = O(m2 )
we have


X
P
1(p|ij| ≤1/q) ≥ ⌊m/2⌋ → 1,
1≤i<j≤m

which in turn gives

p
α
2Φ̄( 2 log m) ≤

P

P

1≤i<j≤m

1(p|ij| ≤2Φ̄(√2 log m))
q

!

→1

√
since 2Φ̄( 2 log m) = O((log m)−1/2 m−1 ). By the definition of α̂ this implies (2.6).

3. Proof of the uniform law of large numbers
This section proves (2.5). We claim that it suffices to show
(3.1)
P
P
H− ∪H0 1(Φ̄−1 (p̄ij,U )≥t)
H+ ∪H0 1(Φ̄−1 (p̄ij,L )≥t) +
− 1 −→p 0,
sup
√
Φ̄(t)(q + q0 )
0≤t≤ 2 log m
where p̄ij,L = Fij (T̄ij ) and p̄ij,U = 1 − Fij (T̄ij ). This can be easily seen to be true
by observing that for each (i, j), the events
{pij,L ≤ α̃/2},

{pij,U ≤ α̃/2} and {p|ij| ≤ α̃}

are identical to
{Φ̄−1 (pij,L ) ≥ Φ̄−1 (α̃/2)}, {Φ̄−1(pij,U ) ≥ Φ̄−1 (α̃/2)} and {Φ̄−1 (p|ij| ) ≥ Φ̄−1 (α̃)}

8

W. LIU, D. LEUNG, AND Q. SHAO

respectively, and that p̄ij,L ≤ pij,L for (i, j) ∈ H+ ∪ H0 , p̄ij,U ≤ pij,U for (i, j) ∈
H− ∪ H0 .
√
Let 0 = t0 < t1 < ... < tg = 2 log m satisfying tl − tl−1 = vm for 1 ≤ l ≤ g − 1
√
and tg − tg−1 ≤ vm , where vm = ( 2 log m log4 m)−1 . By the mean value theorem,
for some c ∈ (tl−1 , tl ),
(3.2)

Φ̄(tl )
Φ̄(tl−1 ) − vm φ(c)
=
= 1 + o(1).
Φ̄(tl−1 )
Φ̄(tl−1 )

The o(1) term in the above display is uniform in all 0 ≤ l ≤ g − 1, and it comes
from the fact that Φ̄(tl−1 ) ∼ t−1
l−1 φ(tl−1 ) as tl−1 −→ ∞ and hence
vm φ(c)/Φ̄(tl−1 ) ∼ vm tl−1 φ(c)/φ(tl−1 )

which goes to 0 uniformly for all l = 0, . . . , g − 1 since vm tl−1 ≤ (log4 m)−1 . For
any tl−1 ≤ t ≤ tl , note that
(3.3)
P
P
P
H+ 1(Φ̄−1 (p̄ij,L )≥t)
H+ 1(Φ̄−1 (p̄ij,L )≥tl−1 ) Φ̄(tl−1 )
H+ 1(Φ̄−1 (p̄ij,L )≥tl ) Φ̄(tl )
≤
≤
Φ̄(tl )(q + q0 )
Φ̄(tl−1 )
Φ̄(t)(q + q0 )
Φ̄(tl−1 )(q + q0 )
Φ̄(tl )
while the preceding display also hold if we replace H+ with H− /H0 , or p̄ij,L with
p̄ij,U . This, together with (3.2), implies that for proving (3.1) it suffices to show
P
P
H− ∪H0 1(Φ̄−1 (p̄ij,U )≥tl )
H+ ∪H0 1(Φ̄−1 (p̄ij,L )≥tl ) +
(3.4)
max
− 1 −→p 0.
0≤l≤g
Φ̄(tl )(q + q0 )
By integral comparison (Appendix B.3), for any ǫ > 0,
!
P
P
H+ ∪H0 1(Φ̄−1 (p̄ij,L )≥tl ) +
H− ∪H0 1(Φ̄−1 (p̄ij,U )≥tl )
(3.5) P max
−1 >ǫ ≤
0≤l≤g
Φ̄(tl )(q + q0 )
!
P
P
Z tg
1
1
+
−1 (p̄
−1 (p̄
(
Φ̄
)≥t)
(
Φ̄
)≥t)
H
∪H
H
∪H
ij,U
ij,L
−
0
+
0
−1
P
− 1 > ǫ/2 dt+
vm
Φ̄(t)(q + q0 )
0
!
P
P
X
H− ∪H0 1(Φ̄−1 (p̄ij,U )≥tl )
H+ ∪H0 1(Φ̄−1 (p̄ij,L )≥tl ) +
P
−1 >ǫ
Φ̄(tl )(q + q0 )
l=0,g
√
In fact for each t ∈ [0, 2 log m], for large enough m the probabilities in the display
above satisfy
!
P
P
H− ∪H0 1(Φ̄−1 (p̄ij,U )≥t)
H+ ∪H0 1(Φ̄−1 (p̄ij,L )≥t) +
−1 >ǫ =
(3.6) P
Φ̄(t)(q + q0 )
!
P
P
P
H− 1(T̄ij ≥tij )
H+ 1(−T̄ij ≥tij ) +
H0 1(|T̄ij |≥tij ) +
−1 >ǫ ,
P
Φ̄(t)(q + q0 )
where
tij = Φ̄−1 (Φ̄(t)(1 + ǫ(T̄ij ))),
for some numbers ǫ(T̄ij ) that may depend on T̄ij but have the property that for a
sequence ǫn > 0,
(3.7)

|ǫ(T̄ij )| ≤ ǫn for all (i, j) a.s.

and ǫn = O(n−c ) for some c > 0.

9

The argument leading to (3.6) is a bit delicate and is deferred to Appendix B.1. To
finish the proof we need the following lemma, where we define
Tij∗ =

(X̄i − µi ) − (X̄j − µj )
q
.
σi2 /ni + σj2 /nj

Lemma 3.1. Under our assumptions, we have for large enough n,
(i)

(ii)

P (T̄ij ≥ tij )
= 1 + O(n−c )
Φ̄(t)
√
uniformly in 0 ≤ t ≤ 2 log m and all (i, j), and
P Ti∗1 j1 ≥ (1 − (log m)−2 )ti1 j1 , Ti∗2 j2 ≥ (1 − (log m)−2 )ti2 j2



≤ C(1 + t)−2 exp(−t2 /(1 + δ)) + O(n−c )
√
uniformly in 0 ≤ t ≤ 2 log m and all (i1 , j1 ), (i2 , j2 ) such that |{i1 , j1 } ∩
{i2 , j2 }| = 1, for some constant 1 > δ > 0.
The proof of the lemma is given in Appendix B.2. Continuing from (3.6), we get
√
that for 0 ≤ t ≤ 2 log m and suffciently large n,
!
P
P
H+ ∪H0 1(Φ̄−1 (p̄ij,L )≥t) +
H− ∪H0 1(Φ̄−1 (p̄ij,U )≥t)
P
−1 >ǫ
Φ̄(t)(q + q0 )
(3.8)
.

(

X

(i,j)∈H0 ∪H+
(i′ ,j ′ )∈H0 ∪H−
|{i,j}∩{i′ ,j ′ }|=1




P −Tij∗ ≥ 1 − (log m)−2 tij , Ti∗′ j ′ ≥ 1 − (log m)−2 ti′ j ′

(3.9)
+

X

2

(i,j)∈H0 ∪H+
(i′ ,j ′ )∈H0 ∪H−
|{i,j}∩{i′ ,j ′ }|=0

(i,j)∈H0

P |T̄ij | ≥ tij

)



(ǫΦ̄(t)(q + q0 ))2 + O(m−1 Φ̄(t)−2 n−r−c )

(3.10)
.m

E[(1(−T̄ij ≥tij ,T̄i′ j′ ≥ti′ j′ ) − Φ̄(t)) ] +

X

−2

−1

(Φ̄(t))

+m

−1

−1 2

exp(δ(1 + δ)

t )+O




log m
−c
.
+n
nc

An application of Chebyshev’s inequality to (3.6), together with (2.10), will give
us the first inequality. We obtain the second inequality in the following way: Since
√
Φ̄(t) ∼ t−1 φ(t) as t → ∞, Φ̄( 2 log m) = (4π log m)−1/2 m−1 (1 + o(1)), and hence
the last “big-O” term in (3.9) becomes O((log m)mn−r−c ) = O((log m)n−c ) by
Assumption 2. Application of Lemma 3.1(i) to the two terms on (3.9) give the
terms m−2 (Φ̄(t))−1 and O(n−c ). Lastly, applying Lemma 3.1(ii) to the sum in
(3.8), together with the fact that (1 + t)−1 φ(t2 ) . Φ̄(t) for all t ≥ 0, gives the terms

10

W. LIU, D. LEUNG, AND Q. SHAO

m−1 exp(δ(1 + δ)−1 t2 ) and O(m−1 n−c ), the later of which can be absorbed into
O(n−c ).
√
Putting t = 2 log m in (3.10), up to constants, we can bound (3.5) by

√

δ−1
2 log m
(log m)
−c
(2 log m log4 m)
+
n
+ m 1+δ + O
m
nc
which converges to zero by Assumption 2, and (3.4) is proved.
4. Discussion and Simulation Study
In an asymptotic setting with minimal assumptions, we established, as conjectured in Williams, Jones, and Tukey (1999), that the BH procedure can control
the false discovery proportion (FDP) and the false discovery rate (FDR) for the
pairwise comparison problem, where inferential errors are defined in the directional
sense. While it is widely believed to be true that, for Gaussian test statistics with
arbitrary dependence structure, the BH procedure can control FDR at the nominal
level in the finite sample setting, proving it in the affirmative remains a hard open
problem in mathematical statistics. For most practical purposes, our treatment is
flexible enough that it established the asymptotic conservativeness of “BH” in the
pairwise comparison setting even when the data can be heavy-tailed or the ANOVA
design can be unbalanced. It is conceivable the techniques in this paper can also be
used to prove the conservativeness of “BH” when it is applied for pairwise comparisons in large scale genomic studies (Yekutieli, 2008). It also came to our attention
that finite sample results on F DP control has appeared in recent literature (Li and
Barber, 2017, Theorem 1), and it will be interesting to explore how their proof
techniques will play out when some dependence exists among the test statistics
considered.
Here we conduct a numeric experiment that parallels our results in Theorem 2.1.
For simplicity, we only consider balanced setups where the same number of samples n are observed for each variable. For each experiment we first generate m
means µ1 , . . . , µm that are i.i.d. realizations of a mean-zero normal distribution
N (0, effect size), where effect size is a chosen value for the standard deviation,
given such a name since the larger effect size is, the more dispersed are the values
µ1 , . . . , µm and hence the pairwise differences µi − µj , 1 ≤ i < j ≤ m will generally
be larger. Notice that we are in the scenario described by Williams, Jones, and
Tukey (1999) where none of the differences µi − µj can be exactly zero. Then for
each 1 ≤ i ≤ m, we generate n i.i.d t-distributed error terms ǫki , 1 ≤ k ≤ n with 6
degrees of freedom, and let our observations be Xki = µi + ρi ǫki , where ρ2i are m
numbers generated from N (mean = 1, sd = .1) to give the Xi ’s different variances.
At level α = 0.2, the BH-procedure in Definition 2.1 is applied using Φ to calibrate
the p-values, and each experiment is repeated 500 times for different combinations
of m, n and effect size. The estimated probability of dFDP being less than α/2
and estimated FDR are reported in Tables 1 and 2. Evidently in Tables 1 , the
estimated P (dF DP ≤ α/2) tends to becomes close to one as m and n grow. Moreover, the higher the signal size, the larger are these probabilities, pointing to the

11

fact that condition (2.1) is required to guaranteed control of false discovery proportions. On the other hand, false discovery rate is always controlled for all effect sizes
in Table 2, and when effect size is smaller, i.e. when it is more easy to commit a
directional error, the estimated dFDR’s are closer to the upper bound α/2 = 0.1,
as expected.
Table 1. Estimation of P (dF DP ≤ α/2) based on 500 repetitions
of experiments, at level α = 0.2
m\n
5
15
30
40

40
100 200 400
effect size = 0.01
0.88 0.90 0.91 0.90
0.87 0.91 0.90 0.89
0.88 0.90 0.91 0.90
0.88 0.89 0.91 0.90

40
100 200 400
effect size = 0.05
0.90 0.93 0.93 0.95
0.89 0.93 0.91 0.89
0.87 0.92 0.91 0.90
0.89 0.90 0.92 0.93

5
15
30
40

effect size = 0.15
0.92 0.93 0.95 0.97
0.92 0.96 1.00 1.00
0.88 0.97 1.00 1.00
0.91 0.99 1.00 1.00

effect size = 0.25
0.93 0.94 0.96 0.99
0.98 1.00 1.00 1.00
0.95 1.00 1.00 1.00
0.99 1.00 1.00 1.00

Table 2. Estimation of dF DR based on 500 repetitions of experiments, at level α = 0.2
m\n
5
15
30
40

40
100 200 400
effect size = 0.01
0.08 0.07 0.06 0.06
0.08 0.05 0.06 0.07
0.07 0.06 0.05 0.05
0.08 0.06 0.05 0.05

40
100 200 400
effect size = 0.05
0.05 0.04 0.02 0.02
0.05 0.03 0.03 0.03
0.06 0.03 0.03 0.03
0.05 0.03 0.02 0.02

5
15
20
40

effect size = 0.15
0.03 0.02 0.01 0.00
0.02 0.02 0.01 0.01
0.03 0.02 0.02 0.02
0.03 0.02 0.02 0.01

effect size = 0.25
0.01 0.01 0.00 0.00
0.02 0.01 0.01 0.01
0.02 0.02 0.01 0.01
0.02 0.02 0.01 0.01

12

W. LIU, D. LEUNG, AND Q. SHAO

Appendix A. Supporting lemmas
A.1. Proof of probability bound in (2.10). Note that for all 1 ≤ i ≤ m,
σ̂i2
=
σi2

Pn

− X̄i )2
=
(n − 1)σi2

k=1 (Xki

Pn

− µi )2
n(X̄i − µi )2
−
,
2
(n − 1)σi
(n − 1)σi2

k=1 (Xki

which in turn gives
σ̂i2
−1 ≤
σi2

Pn

− µi )2 − σi2 ]
n
n(X̄i − µi )2
+
.
−
1
+
(n − 1)σi2
n−1
(n − 1)σi2

k=1 [(Xki

Hence for sufficiently large n, given that m ≤ cnr we have


σ̂i2
−2
(A.1) P max 2 − 1 > (log m)
i
σi
!
r

n 
X
Xki − µi
n(n − 1)
≤ P max
(log m)−1 +
>
i
σi
2
k=1
!

n 
X
(log m)−2 (n − 1)
(Xki − µi )2
.
−1 >
P max
i
σi2
2
k=1

It suffices to show the two terms on the right hand side of (A.1) is of that order.
The first term, for large enough n, under Assumption 2 can be bounded as

P

max

1≤i≤m


n 
X
Xki − µi
σi

k=1

(

≤m P

!
n(n − 1)
−1
(log m)
>
2
!)
r
n
X
n(n − 1)
Xki − µi
−1
>
(log m)
σi
2
r

k=1

.m

n1+2r+ǫ/2 (log m)2+4r+ǫ
. n−r−c ,
n2+4r+ǫ

for some c > 0, where the second last inequality follows from Rosenthal’s inequality
(Rosenthal, 1970, Lemma 1). For the second term, we once again have

(A.2) P

max
i

n 
X
(Xki − µi )2

k=1

σi2

mP

!
(log m)−2 (n − 1)
≤
−1 >
2
!

n 
X
(log m)−2 (n − 1)
(Xki − µi )2
.
−1 >
σi2
2


k=1

If 2r+1+ǫ/2 ≤ 2, by von Bahr and Esseen (1965, Theorem 2), E[|

Pn

k=1 (

(Xki −µi )2
−
σi2

1)|2r+1+ǫ/2 ] . n, and hence with the right hand side of (A.2) can be bounded by
m(log m)4r+2+ǫ n−2r−ǫ/2 = O(n−r−c ). If 2r + 1 + ǫ/2 ≥ 2, then we can apply Liu

13

et al. (2008)[Lemma 6.1, (6.2)] to give
!

n 
X
(Xki − µi )2
(log m)−2 (n − 1)
P
−1 >
σi2
2
k=1




n
X
n(n − 1)
c(n − 1)
(Xki − µi )2
+
exp
−
+ Cn− max(2r+ǫ,2)
−
1
>
≤
P
σi2
2(log m)2
c(log m)2
k=1


(log m)2r+1+ǫ/2
n(n − 1)
+ Cn− max(2r+ǫ,2) . n−2r−c1
.
+ exp −
c(log m)2
n2r+ǫ/2
for some small enough c, c1 > 0 and large enough C > 0, which implies the right
hand side of (A.2) can be bounded by n−r−c1 .
Appendix B. Proof for Section 3
We first introduce a lemma that will be used to finish the proofs.
√
Lemma B.1. Under our assumptions, for any constant c > 2 and a sequence
√
bn = o(1), it must be true for all t ∈ [0, 2 log m]
p
Φ̄−1 (Φ̄(t)(1 + bn )) ≤ c log m

for a sufficiently large n.

Proof. Let t̃ = Φ̄−1 (Φ̄(t)(1 + bn )). It must be true that t̃ ≤ am where am =
√
Φ̄−1 (Φ̄( 2 log m)(1 + bn )). Since sΦ̄(s)/φ(s) → 1 as s −→ ∞, it must be that
√
φ(am ) 2 log m
√
= (1 + o(1))(1 + bn ) = 1 + o(1).
am φ( 2 log m)
By taking log on both side of the preceding display it is easy to see
q
p
p
t̃ ≤ am ≤ 2 log m + 2 log( 2 log m) + o(1) ≤ c log m
fo large enough n (and hence m).



B.1. Proof for (3.6). We will first show that for a sufficiently large n, the equivalence of the events
p̄ij,U ≤ Φ̄(t) ⇐⇒ T̄ij ≥ Φ̄−1 (Φ̄(t)(1 + ǫT̄ij ))
with ǫT̄ij having the property in (3.7). To show this we first note the following two
events are identical, by their definitions:
(B.1)

{p̄ij,L ≤ Φ̄(t)} = {1 − Fij (−T̄ij ) ≤ Φ̄(t)}.
√
For a fixed constant 2 < c′ < 2, for a sufficiently large n the event in (B.1) can
√
√
be realized for all 0 ≤ t ≤ 2 log m if −T̄ij ≥ c′ log m since, with Assumption 3,
p
p
p
1 − Fij (−T̄ij ) ≤ 1 − Fij (c′ log m) = Φ̄(c′ log m)(1 + o(1)) ≤ Φ̄( 2 log m) ≤ Φ̄(t)

when m is large enough, using the fact that Φ(s) ∼ s−1 φ(s) as s → ∞. But since
√
the event in (B.1) may also be true for some −T̄ij < c′ log m, using Assumption 3

14

W. LIU, D. LEUNG, AND Q. SHAO

√
again and the identity (B.1) we can conclude, conditioning on −T̄ij < c′ log m,
the equivalence of events
p̄ij,L ≤ Φ̄(t) ⇐⇒

1 − Fij (−T̄ij )
Φ̄(−T̄ij ) ≤ Φ̄(t)
Φ̄(−T̄ij )

⇐⇒ Φ̄(−T̄ij )(1 + ǫT̄ij ) ≤ Φ̄(t) ⇐⇒ −T̄ij ≥ Φ̄−1 (Φ̄(t)(1 + ǫT̄ij ).
for large enough n, where ǫT̄ij has the property in (3.7). However, in light of
√
Lemma B.1, for large enough n, −T̄ij ≥ c′ log m will necessarily imply −T̄ij ≥
Φ̄−1 (Φ̄(t)(1 + ǫT̄ij ), and hence the train of equivalence in the preceding display is
√
also true without conditioning on −T̄ij < c′ log m. By a completely analogous
argument we also have the equivalence of events
p̄ij,U ≤ Φ̄(t) ⇐⇒ T̄ij ≥ Φ̄−1 (Φ̄(t)(1 + ǫT̄ij ))
for large enough n with same property for ǫT̄ij .
B.2. Proof of Lemma 3.1. For notational simplicity we let tU = Φ̄−1 (Φ̄(t)(1 −
ǫn )) and tL = Φ̄−1 (Φ̄(t)(1 + ǫn )), where the subscripts are suggestive of the fact
that tL is always less than tU . We also note that
p
(B.2)
tL ≤ tij ≤ tU for all (i, j) and t ∈ [0, 2 log m].

Proof of Lemma 3.1(i). In fact, it suffices to show that for sufficiently large m,
(B.3)

P (T̄ij ≥ tU )
= 1 + O(n−c )
Φ̄(t)

(B.4)

P (T̄ij ≥ tL )
= 1 + O(n−c )
Φ̄(t)

√
both uniformly in t ∈ [0, 2 log m] and all (i, j), in light of (B.2).
(B.3): By Lemma B.1, we note that
√ for large enough n, it is true that tL , tU ≤
√
c′ log m for a fixed constant c′ > 2. Since it must be that 0 ≤ tU , by Assumption 2 and the Cramér-type moderate deviation (2.12), we have

4r+2+ǫ !
P (T̄ij ≥ tU )
1 + tU
= (1 − ǫn ) 1 + O(1)
Φ̄(t)
n1/2−1/(4r+2+ǫ)
= 1 + O(n−c )
√
uniformly in all (i, j) and 0 ≤ t ≤ 2 log m, for sufficiently large n.
(B.4): Since tL can be a negative number, (2.12) is not directly applicable.
√
However, we can do a separation argument: For any t ∈ [1, 2 log m], for large
enough n it must be that tL ≥ 0, hence with the same argument for proving (B.3),
we have
p
P (T̄ij ≥ tL )
= 1 + O(n−c ) uniformly in t ∈ [1, 2 log m] and all (i, j).
Φ̄(t)

15

For any t ∈ [0, 1), we have
P (T̄ij ≥ tL )
P (T̄ij ≥ t) P (T̄ij ≥ tL ) − P (T̄ij ≥ t)
=
+
Φ̄(t)
Φ̄(t)
Φ̄(t)
P (T̄ij ≥ tL ) − P (T̄ij ≥ t)
= 1 + O(n−c ) +
Φ̄(t)

(B.5)

uniformly in (i, j) and t ∈ [0, 1), where the last equality is based on Assumption 2
and (2.12). But we also note that
(B.6)

|P (T̄ij ≥ tL ) − P (T̄ij ≥ t)|
≤
Φ̄(t)
|P (T̄ij ≥ tL ) − Φ̄(tL )| |P (T̄ij ≥ t) − Φ̄(t)| |Φ̄(t) − Φ̄(tL )|
+
+
.
Φ̄(t)
Φ̄(t)
Φ̄(t)

Given that Φ̄(t) > C > 0 for some C on interval [0, 1), on the right hand side of
(B.6) the first two terms are of order O(n−c ) by (2.12), and the last term is so due
to bounded first derivative of Φ̄−1 on the interval [Φ̄(1), Φ̄(−1)], all uniformly in
(i, j) and t ∈ [0, 1).

Proof of Lemma 3.1(ii). Given (B.2), it suffices to show, for sufficiently large n,

P Ti∗1 j1 ≥ t̃, Ti∗2 j2 ≥ t̃ ≤ C(1 + t)−2 exp(−t2 /(1 + δ))
√
uniformly in 0 ≤ t ≤ 2 log m and |{i1 , j1 } ∩ {i2 , j2 }| = 1 for some constant δ > 0,
where t̃ := (1 − (log m)−2 )tL .
We first recognize that for any fixed pair i 6= j, Tij∗ can be rewritten as the
standardized sum
Pnj
Pni
Pni ∨nj
−1
ηk,ij
k=1 ni nj (Xkj − µj )
k=1 (Xki − µi ) −
∗
q
Tij =
,
= k=1
√
aij
ni σi2 + n2i σj2 /nj
where

ηk,ij :=

(

[(Xki − µi ) − ni nj−1 (Xkj − µj )]/σi
[(Xki − µi )1{ni >nj } −

ni n−1
j (Xkj

− µj )1{ni <nj } ]/σi

if k ≤ min(ni , nj )

if k > min(ni , nj )

,

1 ≤ k ≤ ni ∨ nj , are independent random variables with mean 0, and aij :=
ni + (n2i σj2 )/(nj σi2 ).
Without loss of generality we will prove the lemma by assuming that i1 = i2 ,
and make the identification i1 = i2 = i, j1 = j, j2 = l with three distinct indices
i, j, l from now on. Denote the Euclidean norm on the plane with k · k, and for each
√
√
k = 1, . . . , ni ∨ nj ∨ nl , let ηk = (ηk,ij 1(k≤ni ∨nj ) / aij , ηk,il 1(k≤ni ∨nl ) / ail )′ and
Pni ∨nj ∨nl
Σ = Cov( k=1
η k η ′k ). We suppressed the dependence of η k on the particular
choice of the triple (i, j, l) for brevity. Later we will use the properties that
(B.7)

E[η k ] = (0, 0)′ ,

(B.8)

Σ is a 2 × 2 matrix with 1’s on the diagonal

16

W. LIU, D. LEUNG, AND Q. SHAO

and
(B.9)
(1+c2U )−1 ≤ Σ1,2

v
!−1 s
u
−1
2n
u
σ
σl2 ni
i
j
t
∗
∗
≤ (1+c2L )−1
= Cov(Tij , Til ) =
1+ 2
1+ 2
σi nj
σi nl

under Assumption 1. For 1 ≤ k ≤ ni ∨ nj ∨ nl , by defining the the truncations
η̂ k = (η̂k,ij , η̂k,il )′ := η k 1(kηk k≤(log m)−4 ) − E[η k 1(kηk k≤(log m)−4 ) ]

and
η̃ k = ηk − η̂ k = η k 1(kηk k>(log m)−4 ) − E[η k 1(kηk k>(log m)−4 ) ]

(again the reliance of η̃ k and η̂ k on (i, j, l) is implicit), we get that
!
ni ∨nj ∨nl
X

∗
∗
−2
(B.10) P Tij ≥ t̃, Til ≥ t̃ ≤ P k
η̃k k ≥ (log m)
+
k=1

P

ni ∨nj

X

k=1

−2

η̂k,ij ≥ t̃ − (log m)

,

nX
i ∨nl
k=1

−2

η̂k,il ≥ t̃ − (log m)

!

.

Then, by Assumptions 2 and 1, there exists some constant κ, such that
max E[kη k k4r+2+ǫ ] ≤ κ/n2r+1+ǫ/2,

k,(i,j,l)

and hence
(B.11)

(B.12)

(B.13)
(B.14)
(B.15)

E
≤
≤

"ni ∨nj ∨nl
X

k=1
ni ∨nj ∨nl

X

k=1
ni ∨nj ∨nl

X

η k 1(kηk k>(log m)−4 )

#

E[kη k k1(kηk k>(log m)−4 ) ]
−4

(log m)

k=1

E

"

ηk
(log m)−4

4r+2+ǫ

#

≤ κn−2r−ǫ/2 (log m)16r+4+4ǫ
= o((log m)−2 ).

P
uniformly for all (i, j, l). Hence for a sufficiently large n, in order for k k η̃ k k to
be greater than (log m)−2 , from the definition of η̃ k it can be seen that at least one
of η k 1(kηk k>(log m)−4 ) , k = 1, . . . , ni ∨ nj ∨ nl , must be non-zero, which gives
!
ni ∨nj ∨nl
X
−2
P
η̃k > (log m)
≤ n max P (kη k k > (log m)−4 )
k

k=1

≤ κ(log m)16r+8+4ǫ n−2r−ǫ/2 = O(n−c ),

again uniformly for all (i, j, l). From (B.10) it remains to show
!
ni ∨nj
nX
i ∨nl
X
−2
−2
≤ C(t+1)−2 exp(−t2 /(1+δ))
P
η̂k,ij ≥ t̃ − (log m) ,
η̂k,il ≥ t̃ − (log m)
k=1

k=1

√
uniformly in 0 ≤ t ≤ 2 log m for some δ > 0.

17

Since kη̂k k are bounded by 2(log m)−4 , it satisfies the condition of Bernstein’s
inequality, by applying Zaı̆tsev (1987, Theorem 1.1) (Taking τ = 2(log m)−4 for τ
in that theorem), we have
!
X
X
−2
−2
P
η̂k,ij ≥ t̃ − (log m) ,
η̂k,il ≥ t̃ − (log m)
k

k




(log m)2
≤ P W1 > t̃ − 2(log m)−2 , W2 > t̃ − 2(log m)−2 + c1 exp −
c2

for some absolute constants c1 , c2 > 0, where (W1 , W2 )′ is a bivariate Gaussian
Pn1 ∨nj ∨nl
vector with mean zero
structure Σ̂ := E[ k=1
η̂ k η̂ ′k ]. Note

 and covariance
(log m)2
is less than C(1 + t)−2 exp(−t2 /(1 + δ)) for some
that the term c1 exp − c2
√
C, δ > 0 on the range t ∈ [0, 2 log m], and we are left with bounding the first term
on the right hand side of the preceding display.
By (B.7), (B.15) and Assumption 1,
"ni ∨nj ∨nl
#
X
E
η k 1(kηk k≤(log m)−4 ) = o((log m)−2 ),
k=1

hence, by letting k · k∞ denote the matrix max norm,
#
"ni ∨nj ∨nl
X
′
kΣ − Σ̂k∞ ≤ E
ηk η k 1(kηk k>(log m)−4 )
k=1

≤
(B.16)

ni ∨nj ∨nl

X

k=1

+ o((log m)−4 )
∞

E[kηk k2 1(kηk k>(log m)−4 ) ] + o((log m)−4 )

≤ (log m)16r+4ǫ

κ
+ o((log m)−4 ) ≤ o((log m)−2 ),
n2r+ǫ/2

again uniformly in all (i, j, l). By Berman (1962, Lemma 2), for 0 ≤ t ≤
and large enough n,

(B.17) P W1 > t̃ − 2(log m)−2 , W2 > t̃ − 2(log m)−2 ≤


 min
C



2 exp −

−2
−2
√ m) , t̃−2(log
√ m)
1 + min t̃−2(log
Σ̂11

Σ̂22

t̃−2(log m)−2

√

Σ̂11

,

√
2 log m

t̃−2(log m)−2

1 + √ Σ̂12

√

Σ̂11 Σ̂22

Σ̂22

2 



.


By the bound in (B.16), (B.8), (B.9) and elementary calculations, the right hand
side can further be bounded by
C
exp(−t2 /(1 + δ))
(1 + t)2
for some other constants C > 0 and 1 > δ > 0, uniformly in (i, j, l) and 0 ≤ t ≤
√
2 log m. These elementary calculations are left to the readers, but the fact that
√
tL = Φ̄−1 (Φ̄(t)(1 + ǫn )) converges to t uniformly in t ∈ [0, 2 log m] is helpful: It
is obvious how tL → t uniformly for t ∈ [0, 2], since Φ̄−1 has bounded derivative

18

W. LIU, D. LEUNG, AND Q. SHAO

√
on [Φ̄(2), Φ̄(0)]. On the interval t ∈ [2, 2 log m], by inverse function theorem and
with mean value theorem,
ǫn
|tL − t| ≤ |φ(t)|−1 |Φ(t)|ǫn ≤
→0
t
uniformly since ǫn = O(n−c ).

B.3. Comparison argument. By a union bound,
(B.18)
P
≤

max

0≤l≤g

X

P

H+ ∪H0

I(Φ̄−1 (p̄ij,L ) ≥ tl ) +

P

H− ∪H0

I(Φ̄−1 (p̄ij,U ) ≥ tl )

Φ̄(tl )(q + q0 )
P
P
−1
(p̄ij,L ) ≥ tl ) + H− ∪H0 I(Φ̄−1 (p̄ij,U ) ≥ tl )
H+ ∪H0 I(Φ̄

Φ̄(tl )(q + q0 )
P
P
−1
(p̄ij,L ) ≥ tl ) + H− ∪H0 I(Φ̄−1 (p̄ij,U ) ≥ tl )
H+ ∪H0 I(Φ̄

0≤l≤g

X

P

P

Φ̄(tl )(q + q0 )

0≤l≤g

−1 >ǫ

!

!

>1+ǫ +
!

<1−ǫ .

By (3.3), for large enough n,
(B.19)
X

P

1≤l≤g−1

≤

−1
vm

Z

tg

P
0

P

H+ ∪H0

I(Φ̄−1 (p̄ij,L ) ≥ tl ) +

P

H− ∪H0

I(Φ̄−1 (p̄ij,U ) ≥ tl )

Φ̄(tl )(q + q0 )
P
P
−1
(p̄ij,L ) ≥ t) + H− ∪H0 I(Φ̄−1 (p̄ij,U ) ≥ t)
H+ ∪H0 I(Φ̄
Φ̄(t)(q + q0 )

>1+ǫ

!
!

> 1 + ǫ/2 dt

and similarly,
(B.20)
X

P

1≤l≤g−1

≤

−1
vm

Z

tg

P
0

P

H+ ∪H0

I(Φ̄−1 (p̄ij,L ) ≥ tl ) +

P

H− ∪H0

I(Φ̄−1 (p̄ij,U ) ≥ tl )

Φ̄(tl )(q + q0 )
P
P
−1
(p̄ij,L ) ≥ t) + H− ∪H0 I(Φ̄−1 (p̄ij,U ) ≥ t)
H+ ∪H0 I(Φ̄
Φ̄(t)(q + q0 )

<1−ǫ

!
!

< 1 − ǫ/2 dt.

References
Benjamini, Y. and Hochberg, Y. (1995). “Controlling the false discovery rate: a practical
and powerful approach to multiple testing.” J. Roy. Statist. Soc. Ser. B , 57(1): 289–300.
— (2000). “On the adaptive control of the false discovery rate in multiple testing with
independent statistics.” Journal of educational and Behavioral Statistics, 25(1): 60–83.
Benjamini, Y. and Yekutieli, D. (2001). “The control of the false discovery rate in multiple
testing under dependency.” Ann. Statist., 29(4): 1165–1188.
— (2005). “False discovery rate-adjusted multiple confidence intervals for selected parameters.” J. Amer. Statist. Assoc., 100(469): 71–93. With comments and a rejoinder by
the authors.
Berman, S. M. (1962). “A law of large numbers for the maximum in a stationary Gaussian
sequence.” Ann. Math. Statist., 33: 93–97.

19

Cao, H. (2007). “Moderate deviations for two sample t-statistics.” ESAIM Probab. Stat.,
11: 264–271.
Hayter, A. J. (1984). “A proof of the conjecture that the Tukey-Kramer multiple comparisons procedure is conservative.” Ann. Statist., 12(1): 61–75.
Jing, B.-Y., Shao, Q.-M., and Wang, Q. (2003). “Self-normalized Cramér-type large
deviations for independent random variables.” Ann. Probab., 31(4): 2167–2215.
Kramer, C. Y. (1956). “Extension of multiple range tests to group means with unequal
numbers of replications.” Biometrics, 12: 307–310.
Li, A. and Barber, R. F. (2017). “Accumulation tests for FDR control in ordered hypothesis testing.” J. Amer. Statist. Assoc., 112(518): 837–849.
Liu, W. and Shao, Q.-M. (2014). “Phase transition and regularized bootstrap in large-scale
t-tests with false discovery rate control.” Ann. Statist., 42(5): 2003–2025.
Liu, W.-D., Lin, Z., and Shao, Q.-M. (2008). “The asymptotic distribution and BerryEsseen bound of a new test for independence in high dimension with an application to
stochastic optimization.” Ann. Appl. Probab., 18(6): 2337–2366.
Oishi, S., Schimmack, U., Diener, E., and Suh, E. M. (1998). “The measurement of values and individualism-collectivism.” Personality and social psychology bulletin, 24(11):
1177–1189.
Pawluk-Kolc, M., Zieba-Palus, J., and Parczewski, A. (2006). “Application of false discovery rate procedure to pairwise comparisons of refractive index of glass fragments.”
Forensic science international , 160(1): 53–58.
Rosenthal, H. P. (1970). “On the subspaces of Lp (p > 2) spanned by sequences of
independent random variables.” Israel J. Math., 8: 273–303.
Scheffé, H. (1943). “On solutions of the Behrens-Fisher problem, based on the tdistribution.” Ann. Math. Statistics, 14: 35–44.
von Bahr, B. and Esseen, C.-G. (1965). “Inequalities for the rth absolute moment of a
sum of random variables, 1 ≤ r ≤ 2.” Ann. Math. Statist, 36: 299–303.
Williams, V. S., Jones, L. V., and Tukey, J. W. (1999). “Controlling error in multiple
comparisons, with examples from state-to-state differences in educational achievement.”
Journal of educational and behavioral statistics, 24(1): 42–69.
Yekutieli, D. (2008). “Hierarchical false discovery rate-controlling methodology.” J. Amer.
Statist. Assoc., 103(481): 309–316.
Zaı̆tsev, A. Y. (1987). “On the Gaussian approximation of convolutions under multidimensional analogues of S. N. Bernstein’s inequality conditions.” Probab. Theory Related
Fields, 74(4): 535–566.
Institute of Natural Sciences, Shanghai Jiao Tong University
E-mail address: weidongl@sjtu.edu.cn
Department of Data Sciences and Operations, University of Southern California
E-mail address: dmhleung@uw.edu
Department of Statistics, Chinese University of Hong Kong
E-mail address: qmshao@sta.cuhk.edu.hk

