Where are the logs?

Abstract: The commonly quoted error rates for QMC integration with an infinite low discrepancy sequence is $O(n^{-1}\log(n)^r)$ with $r=d$ for extensible sequences and $r=d-1$ otherwise. Such rates hold uniformly over all $d$ dimensional integrands of Hardy-Krause variation one when using $n$ evaluation points. Implicit in those bounds is that for any sequence of QMC points, the integrand can be chosen to depend on $n$. In this paper we show that rates with any $r<(d-1)/2$ can hold when $f$ is held fixed as $n\to\infty$. This is accomplished following a suggestion of Erich Novak to use some unpublished results of Trojan from the 1980s as given in the information based complexity monograph of Traub, Wasilkowski and Woźniakowski. The proof is made by applying a technique of Roth with the theorem of Trojan. The proof is non constructive and we do not know of any integrand of bounded variation in the sense of Hardy and Krause for which the QMC error exceeds $(\log n)^{1+\epsilon}/n$ for infinitely many $n$ when using a digital sequence such as one of Sobol's. An empirical search when $d=2$ for integrands designed to exploit known weaknesses in certain point sets showed no evidence that $r>1$ is needed. An example with $d=3$ and $n$ up to $2^{100}$ might possibly require $r>1$.

\section{Introduction}

In this article, we study the asymptotic
error rates for integration by quasi-Monte Carlo (QMC) as $n\to\infty$
while $f$ is fixed.
Most of the error upper bounds in QMC are based on fooling functions
$f_n$ that, given $n$ integration points, are poorly integrated.
By contrast, most of the
published empirical results follow the integration
error for a single integrand $f$ as $n$ increases.
The upper bounds have us play against an adaptive adversary choosing
an unfavorable $f_n$ at each sample size $n$
instead of keeping $f$  fixed as $n\to\infty$.
The error bounds, that we describe in more detail below,
are typically $O(\log(n)^r/n)$ where $r$ can be as large as the
dimension of the integrand's domain.
These bounds can be enormous and, to our knowledge, there has never
been an integrand exhibited where a standard QMC point set
is shown to need $r>1$.
That raises the question of whether $r>1$ is simply
a consequence of the adversarial formulation.
The alternative is that some function $f$ is a `persistent fooling function'
causing large errors for infinitely many $n$.
In an earlier version of this article we posed a question about whether
{\sl any} integrand of bounded variation in the sense of Hardy and Krause (BVHK)
on $[0,1]^d$ for any $d\geqslant1$
has an integration error above $c\log(n)^r/n$ for infinitely many $n$
with $r>1$ and $c>0$ using a digital sequence such as Sobol's for the integration points.
For background on bounded variation in the sense of Hardy and Krause
or in the sense of Vitali, we refer the reader to \cite{variation}.

We owe a great debt to Erich Novak who pointed us to some unpublished
work of Trojan described in detail in Chapter 10 of
the information based complexity monograph of Traub, Wasilkowski
and Wo\'zniakowsk \cite{trau:wasi:wozn:1988}.
Trojan's work is about very general problems of computing
linear operators on Banach spaces based on the values of $n\to\infty$ linear functionals.
He shows that the adversarial worst case convergence rate is also very nearly the attained
rate for some specific problem instances.
In the QMC context, that work pertains to a single integrand $f$
as the number $n$ of evaluation points diverges to infinity.
A consequence of that work is that for any infinite
sequence of integration points, there are indeed integrands
in BVHK$[0,1]^d$ with an absolute error larger than $c\log(n)^{(d-1)/2}/(n\log\log(n))$
infinitely often, for any $c>0$.
Furthermore, those integrands are present within a reproducing kernel Hilbert space (RKHS)
on a certain unanchored space.  They are dense in that space, though
this does not mean that the usual Gaussian processes on such spaces
give them positive measure.
We only get $r=(d-1)/2$ logarithmic factors instead of $d$ or $d-1$
of them.  The explanation is that we use an $L^2$ bound
just like Roth \cite{roth:1954} used in getting a lower bound on star discrepancy.
A different analysis might yield larger $r$.
The $\log\log(n)$ factor in the denominator
can be replaced by a sequence that diverges more slowly.

We have not been able to construct a function in BVHK$[0,1]^d$
that provably needs $r>1$ powers of $\log(n)$
for a Sobol' sequence \cite{sobo:1967:tran}
or the Halton \cite{halt:1960}
sequence, even when exploiting known weaknesses of commonly used QMC sequences.
So, we are left to wonder: where are the logs?

An outline of this paper is as follows.
Section~\ref{sec:back} presents some results from the QMC literature
and introduces notation on some QMC sequences.
Section~\ref{sec:proofofbound} proves our main result
described above on existence of persistent fooling functions.
Section~\ref{sec:d=1} looks at the case $d=1$
to exhibit some example functions requiring $r=1$
for the van der Corput sequence: $f(x)=1\{x<2/3\}$
and $f(x)=x$.
Section~\ref{sec:d=2} computes error for some $d=2$
dimensional problems. The Halton and Sobol'
points there are closely related to van der Corput
points, yet two dimensional generalizations of the
problematic integrands from Section~\ref{sec:d=1}
fail to show a need for $r>1$.
In fact some of the empirical results are more consistent
with an $O(1/n)$ error.
Section~\ref{sec:bigm} computes a local discrepancy
$\delta({\boldsymbol{z}}$)
for  Sobol' nets with $d=2,3$ and $1\leqslant m\leqslant100$ where all components of ${\boldsymbol{z}}$ equal $2/3$
chosen because $2/3$ is difficult to approximate by dyadic rationals.
It also includes $d=4$ for $1\leqslant m\leqslant 50$.
The cases with $d>2$ are the closest we have found to needing $r>1$
but are inconclusive.
Section~\ref{sec:discussion} discusses these results.

\section{Background}\label{sec:back}

From the Koksma-Hlawka inequality \cite{hick:2014}
combined with convergence rates for the star discrepancy \cite{nied:1992},
we get the widely quoted convergence rates
for the error in quasi-Monte Carlo integration
of a function $f:[0,1]^d\to{\mathbb{R}}$.
An integrand $f$ of bounded variation in the
sense of Hardy and Krause, written $f\in\mathrm{BVHK}[0,1]^d$,
can be integrated with error $O(n^{-1}(\log n)^{d-1})$
using $n$ function evaluations.  If we must use the first $n$ points
of an infinite sequence, then the rate
$O(n^{-1}(\log n)^{d})$ is attainable.
This article is mostly about the infinite sequence version.
Both of these rates are often written $O(n^{-1+\epsilon})$
where $\epsilon$ can be any positive constant
but $\log(n)^d\gg n^\epsilon$ for many use cases of interest.

For high dimensional problems, such powers of
$\log(n)$ are enormous and then
there is genuine uncertainty about whether
$O(n^{-1}(\log n)^{d})$ is better than the
root mean squared error (RMSE) of $O(n^{-1/2})$ from plain
Monte Carlo (MC) at practically relevant  $n$.
These rates omit three implied constants: one in
the star discrepancy (see \cite{faur:lemi:2014} for information),
one in the total variation of $f$
and the third one is the standard deviation of $f$.
These unknown constants contribute to
uncertainty about the $n$ at which QMC would
outperform MC.
A further complication is that the Koksma-Hlawka
bound is for a worst case integrand.
The situation is quite different in Monte Carlo
(MC) where the rate $\sigma n^{-1/2}$ holds for all finite $n$ making
it simultaneously a guide to how accuracy progresses
for a single integrand of variance $\sigma^2$ and
the RMSE formula (upper and lower bound)
for all integrands of variance~$\sigma^2$.

That observed errors for realistic $n$ and large $d$
do not follow a trend like $\log(n)^d/n$
was reported by Schlier \cite{schl:2004} among others.
That work also found that the variance of $f({\boldsymbol{x}})$
was more useful than its total variation in explaining the empirical accuracy
of QMC integration on test functions, despite the fact
that proved theoretical bounds for QMC error use total
variation and variance does not require any of the smoothness that
QMC relies on.
Many papers include empirically estimated convergence
rates for individual $f$ found by fitting a regression model
for log error versus $\log(n)$. See for instance L'Ecuyer \cite{lecu:2018}.
We do not see results that look like a large power of $\log(n)$
is present.

This mismatch between empirical results and theoretical ones
is troubling.  Empirical results alone don't give enough confidence
that they will apply to future problems.  Similarly, bounds that are
favorable (but asymptotic) or unfavorable (but worst case) could
also fail to provide a reliable guide to attained accuracy.
This mismatch has brought practical difficulties.
For instance, the logarithmic powers
in the Koksma-Hlawka bound led Bratley, Fox and Niederreiter \cite{brat:fox:nied:1992}
to limit their software to $d\leqslant 12$.

For some randomizations of digital nets
the RMSE is $O(n^{-1/2})$ whenever $f\in L^2[0,1]^d$ \cite{snxs}
and is also $O(\log(n)^{(d-1)/2}/n)$ under further smoothness
conditions \cite{smoovar,localanti,yue:mao:1999}.
In such cases the large powers of $\log(n)$ are
subject to a simultaneous $O(n^{-1/2})$ bound that
limits how much worse randomized QMC can be compared to MC
for finite $n$.
It would be interesting to know whether something like that also
holds for plain QMC. Perhaps the coefficient of $\log(n)^r/n$ is ordinarily
very small, or the effect is only relevant for impractically large $n$ or
perhaps not even present for most commonly investigated integrands.
For a survey of randomized QMC see L'Ecuyer and Lemieux \cite{lecu:lemi:2002}.

We conclude this section by describing $(t,m,d)$-nets and $(t,d)$-sequences
using the formulation from Niederreiter \cite{nied:1987}.
Let $b\geqslant2$ be an integer.
For ${\boldsymbol{k}} = (k_1,\dots,k_d)\in{\mathbb{N}}} % natural numbers {1, 2, ..._0^d$ and
${\boldsymbol{c}} = (c_1,\dots,c_d)\in\Z_0^d$
with $0\leqslant c_j<b^{k_j}$ the half open hyper-rectangle
\begin{align}\label{eq:elemint}
E({\boldsymbol{k}},{\boldsymbol{c}}) = \prod_{j=1}^d\Bigl[\frac{c_j}{b^{k_j}}, \frac{c_j+1}{b^{k_j}}\Bigr)
\end{align}
is called an elementary interval in base $b$.
It has volume $b^{-|{\boldsymbol{k}}|}$ where $|{\boldsymbol{k}}|=\sum_{j=1}^dk_j$.
We define its indicator function as
$$
I_{{\boldsymbol{k}},{\boldsymbol{c}}}({\boldsymbol{x}})
=I_{{\boldsymbol{k}},{\boldsymbol{c}}}({\boldsymbol{x}};b)
= 1\{{\boldsymbol{x}}\in E({\boldsymbol{k}},{\boldsymbol{c}})\}.
$$

For integers $m\geqslant t\geqslant0$, $b\geqslant2$, $n=b^m$ and $d\geqslant1$
the points ${\boldsymbol{x}}_0,\dots,{\boldsymbol{x}}_{n-1}\in[0,1]^d$ are
a $(t,m,d)$-net in base $b$ if
$$
\frac1n\sum_{i=0}^{n-1} I_{{\boldsymbol{k}},{\boldsymbol{c}}}({\boldsymbol{x}}_i) =
\int_{[0,1]^d}I_{{\boldsymbol{k}},{\boldsymbol{c}}}({\boldsymbol{x}}){\mathrm{d}}{\boldsymbol{x}}=b^{-|{\boldsymbol{k}}|}
$$
holds for all elementary intervals with $|{\boldsymbol{k}}|\leqslant m-t$.
Other things being equal, smaller $t$ are better and
$t=0$ is best, but the choices of $d$, $b$, and $m$
impose a lower bound on the possible $t$ which
may rule out $t=0$.
The minT project of \cite{schu:schm:2009}
tracks the best known values of $t$ as well as some
lower bounds on $t$.

In this paper we emphasize infinite sequences.
The sequence ${\boldsymbol{x}}_i\in[0,1]^d$ for integers $i\geqslant0$
is a $(t,d)$-sequence in base $b$ if
${\boldsymbol{x}}_{rb^m},\dots,{\boldsymbol{x}}_{(r+1)b^m-1}$ is a $(t,m,d)$-net
in base $b$ for all $m\geqslant t$ and all integers $r\geqslant0$.
These are extensible $(t,m,d)$-nets in that
the first $b^\ell$ points of a $(t,d)$-sequence form a $(t,m+\ell,d)$-net
for any integer $\ell\geqslant1$.
The most used $(t,d)$-sequences are the
$(0,d)$-sequences in prime bases $p\geqslant d$
of Faure \cite{faur:1982} and the $(t,d)$-sequences
in base $2$ of Sobol' \cite{sobo:1967:tran}.

We will make special use of the
van der Corput sequences in base $b\geqslant2$.
These are $(0,1)$-sequences in base $b$.
If we write the natural number $i=\sum_{k=1}^\infty i_k b^{k-1}$
with digits $i_k\in\{0,1,\dots,b-1\}$, then the sum only has $K(i)<\infty$
nonzero terms and we then set
$x_i = \sum_{k=1}^{K(i)}b^{-k}i_k$.
This sequence has star discrepancy $D^*_n=O(\log(n)/n)$.
For $n=b^m$ it is a left endpoint rule containing points
$i/n$ for $0\leqslant i< n$ and so it has $D^*_n = 1/n$
by \cite[Theorem 2.6]{nied:1992}.
The original van der Corput sequence in base $b=2$
is from \cite{vand:1935:I}.

\section{Proof of the lower bound}\label{sec:proofofbound}

We begin with a general theorem on worst-case errors.
Later we specialize it to the QMC setting.

\begin{theorem}\label{thm:asymptoticrate}
Let $(F,\Vert\cdot\Vert)$ be a Banach space and $S$ be a linear functional on $F$. For a sequence of continuous linear functionals $L_n$ on $F$, define
\begin{align*}
N_n(f)&=(L_1(f),\ldots,L_n(f)),\quad\text{and}\\
r_n&=\sup\bigl\{|S(f)|\bigm| f\in F, N_n(f)=\bszero, \Vert f\Vert\leqslant 1\bigr\}.
\end{align*}
Then for any sequence of mappings $\phi_n$ from $\mathbb{R}^n$ to $\mathbb{R}$, there exists $f\in F$ such that
\begin{equation}\label{eqn:target}
    \limsup_{n\to \infty}\frac{ |S(f)-\phi_n(N_n(f))|}{(\log\log n)^{-1}r_n}=+\infty.
\end{equation}
\end{theorem}
\begin{proof} This follows from Theorem 2.1.1 in Chapter 10 of
\cite{trau:wasi:wozn:1988}
who cite unpublished work by Trojan.
\end{proof}

We will set $F$ to be some reproducing kernel Hilbert space (RKHS) contained in BVHK$[0,1]^d$, $S(f)$ to be $\int_{[0,1]^d}f({\boldsymbol{y}})\,{\mathrm{d}}{\boldsymbol{y}}$ and $N_n(f)$ to be $(f({\boldsymbol{x}}_0),\ldots,f({\boldsymbol{x}}_{n-1}))$. We note that evaluation at ${\boldsymbol{x}}_i$ is a continuous linear functional in an RKHS. As the theorem suggests, we can use $r_n/\log\log n$ as a lower bound on the asymptotic convergence rate achievable by all functions in BVHK$[0,1]^d$, so it remains to determine a lower bound on $r_n$.

To derive such a lower bound, we will apply the proof techniques used in Roth's lower bound on the $L^2$ discrepancy.
See Chen and Travaglini \cite{chen:trav:2009} for a nice summary.
Dick, Hinrichs and Pillichshammer \cite{dick:hinr:pill:2015}
use this strategy to prove that the worst-case error of any equal-weight quadrature rule is
lower bounded by $\Omega(n^{-1}(\log n)^{(d-1)/2})$ when $F$ is the RKHS with kernel $K({\boldsymbol{x}},{\boldsymbol{y}})=\prod_{j=1}^d (1+\min(x_j,y_j))$. Wozniakowski \cite{wozn:1991} points out that the same strategy still works if the equal weight requirement is removed.
Below we illustrate Roth's technique by showing that $r_n=\Omega(n^{-1}(\log n)^{(d-1)/2})$ if $F$ is chosen to be the  RKHS with kernel
 \begin{align}\label{eqn:kernel}
     K({\boldsymbol{x}},{\boldsymbol{y}})=\prod_{j=1}^d \frac{4}{3}+\frac{1}{2}\Bigl(x_j^2+y_j^2-x_j-y_j-|x_j-y_j|\Bigr).
 \end{align}
This is the unanchored space introduced in
\cite{dick:sloa:wang:wozn:2004}.
It has the inner product
\begin{align}\label{eqn:innerproduct}
    (f,g)=\sum_{u\subseteq\{1,\ldots,d\}}\int_{[0,1]^{|u|}}\bigg(\int_{[0,1]^{d-|u|}}\frac{\partial^{|u|}f}{\partial {\boldsymbol{y}}_u}({\boldsymbol{y}})\,{\mathrm{d}}{\boldsymbol{y}}_{-u}\bigg) \bigg(\int_{[0,1]^{d-|u|}}\frac{\partial^{|u|}g}{\partial {\boldsymbol{y}}_u}({\boldsymbol{y}})\,{\mathrm{d}}{\boldsymbol{y}}_{-u}\bigg)\, {\mathrm{d}}{\boldsymbol{y}}_u
\end{align}
where ${\partial^{|u|}f}/{\partial {\boldsymbol{y}}_u}$ is the partial derivative of $f$ taken once with respect to
each $y_j$ with $j\in u$.
Any $f$ belonging to this RKHS has mixed partial derivative
${\partial^{|u|}f}/{\partial {\boldsymbol{y}}_u}\in L^2$ for any $u\subseteq\{1,\ldots,d\}$, so $f$ belongs to BVHK$[0,1]^d$
using equation (5) and Proposition~13 of \cite{variation}.

Letting $\bsone\in F$ be the function equal to $1$ for all ${\boldsymbol{y}}\in[0,1]^d$,
it is straightforward to verify that
$$(f,\bsone)=\int_{[0,1]^{d}}f({\boldsymbol{y}}) \,{\mathrm{d}}{\boldsymbol{y}}=S(f).$$
In other words, the function $\bsone$ is the Riesz representation of integration over $[0,1]^d$. Moreover, by the reproducing property
$(f,K({\boldsymbol{x}}_i,\cdot))=f({\boldsymbol{x}}_i).$
Therefore
\begin{align*}
    r_n&=\sup\bigl\{|S(f)|\mid f\in F, f({\boldsymbol{x}}_0)=\cdots=f({\boldsymbol{x}}_{n-1})=0, \Vert f\Vert\leqslant 1\bigr\}\\
    &=\sup\bigl\{|(f,\bsone)|\mid f\in F, (f,K({\boldsymbol{x}}_0,\cdot))=\cdots=(f,K({\boldsymbol{x}}_{n-1},\cdot))=0, \Vert f\Vert\leqslant 1\bigr\}.
\end{align*}
This is the well-known least squares projection problem. The maximizer is proportional to the projection of $\bsone$ into the orthogonal complement of the linear span of $\{K({\boldsymbol{x}}_0,\cdot),\ldots,K({\boldsymbol{x}}_{n-1},\cdot)\}$.
Therefore
\begin{align*}
    r_n=\min_{a_0,\ldots,a_{n-1}} \Bigl\Vert\bsone-\sum_{i=0}^{n-1} a_i K({\boldsymbol{x}}_i,\cdot)\Bigr\Vert.
\end{align*}
Now we prove that $\Vert\bsone-\sum_{i=0}^{n-1} a_i K({\boldsymbol{x}}_i,\cdot)\Vert=\Omega(n^{-1}(\log n)^{(d-1)/2})$
for any choice of $a_0,\dots,a_{n-1}$, including  $a_i=1/n$ as used in QMC.
\begin{theorem}\label{thm:Roth}
Let $K$ be the kernel~\eqref{eqn:kernel} for the unanchored RKHS.
For any points  ${\boldsymbol{x}}_0,\ldots,{\boldsymbol{x}}_{n-1}\in[0,1]^d$ and any  weights $a_0,\ldots,a_{n-1}\in{\mathbb{R}}$
\begin{align}\label{eq:Roth}
\Bigl\Vert1-\sum_{i=0}^{n-1} a_i K({\boldsymbol{x}}_i,\cdot)\Bigr\Vert\geqslant A_d\frac{(\log n)^{(d-1)/2}}{n}
\end{align}
holds for some positive number $A_d$ independent of $n$.
\end{theorem}
\begin{proof}
Because the function $\bsone$ is the Riesz representation of integration over $[0,1]^d$,
$(\bsone,\bsone)=\int_{[0,1]^{d}}\bsone \,{\mathrm{d}}{\boldsymbol{y}}=1,$ and
$$(K({\boldsymbol{x}},\cdot),1)=\prod_{j=1}^d \int_0^1 \frac{4}{3}+\frac{1}{2}\Bigl(x_j^2+y_{j}^2-x_j-y_{j}-|x_j-y_{j}|\Bigr)\,{\mathrm{d}} y_j=1.$$
Therefore
$$\Bigl(\bsone-\sum_{i=0}^{n-1} a_i K({\boldsymbol{x}}_i,\cdot),\bsone\Bigr)
=(\bsone,\bsone)-\sum_{i=0}^{n-1} a_i (K({\boldsymbol{x}}_i,\cdot),\bsone)=1-\sum_{i=0}^{n-1} a_i.$$
On the other hand,
$$\Bigl|\Bigl(\bsone-\sum_{i=0}^{n-1} a_i K({\boldsymbol{x}}_i,\cdot),\bsone\Bigr)\Bigr|
\leqslant \Vert\bsone\Vert\times \Bigl\Vert\bsone-\sum_{i=0}^{n-1} a_i K({\boldsymbol{x}}_i,\cdot)\Bigr\Vert=\Bigl\Vert\bsone-\sum_{i=0}^{n-1} a_i K({\boldsymbol{x}}_i,\cdot)\Bigr\Vert,$$
so we get the first lower bound
\begin{equation}\label{eqn:bound1}
    \Bigl\Vert1-\sum_{i=0}^{n-1} a_i K({\boldsymbol{x}}_i,\cdot)\Bigr\Vert\geqslant \Bigl|1-\sum_{i=0}^{n-1} a_i\Bigr|.
\end{equation}
Next we evaluate $\Vert1-\sum_{i=0}^{n-1} a_i K({\boldsymbol{x}}_i,\cdot)\Vert^2$ directly.
If we ignore all summands except for $u=\{1,\ldots,d\}$ in \eqref{eqn:innerproduct}, we get
\begin{align}\label{eqn:ainorm}
\Bigl\Vert1-\sum_{i=0}^{n-1} a_i K({\boldsymbol{x}}_i,\cdot)\Bigr\Vert^2&\geqslant \int_{[0,1]^d} \bigg(\sum_{i=0}^{n-1} a_i\frac{\partial^{d}K({\boldsymbol{x}}_i,{\boldsymbol{y}})}{\partial y_1\cdots\partial y_d}\bigg)^2 \,{\mathrm{d}}{\boldsymbol{y}} \nonumber\\
   &= \int_{[0,1]^d} \bigg(\sum_{i=0}^{n-1} a_i\prod_{j=1}^d (y_j-1\{y_j>x_{ij}\})\bigg)^2 \,{\mathrm{d}}{\boldsymbol{y}}
\end{align}
where $x_{ij}$ is the $j$th component of ${\boldsymbol{x}}_i$.

The left hand side of~\eqref{eqn:ainorm} is a squared norm in the RKHS
while the right hand side is a plain $L^2$ squared norm.
To provide a lower bound, we construct a function $h$ satisfying
\begin{align*}
&\int_{[0,1]^d} h({\boldsymbol{y}})^2 \,{\mathrm{d}}{\boldsymbol{y}}=O((\log n)^{d-1}),\quad\text{and}\\
&\int_{[0,1]^d} h({\boldsymbol{y}})\bigg(\sum_{i=0}^{n-1} a_i\prod_{j=1}^d (y_j-1\{y_j>x_{ij}\})\bigg) \,{\mathrm{d}}{\boldsymbol{y}}=\Omega\Bigl(\frac{(\log n)^{d-1}}{n}\Bigr),
\end{align*}
neither of which involve the RKHS inner product, so the function $h$ does not have to be in the RKHS.

Define $E({\boldsymbol{k}},{\boldsymbol{c}})$ to be the $d$-dimensional interval
from~\eqref{eq:elemint} with $b=2$, that is
$$
E({\boldsymbol{k}},{\boldsymbol{c}})=\prod_{j=1}^d\Bigl[
\frac{c_j}{2^{k_j}},
\frac{c_j+1}{2^{k_j}}
\Bigr)
$$
where ${\boldsymbol{k}} = (k_1,\dots,k_d)\in{\mathbb{Z}}^d$
and ${\boldsymbol{c}} = (c_1,\dots,c_d)\in{\mathbb{Z}}^d$
satisfy $k_j\geqslant0$ and $0\leqslant c_j <2^{k_j}$. Given ${\boldsymbol{k}}$, we define $|{\boldsymbol{k}}|=\sum_{j=1}^dk_j$.
For a given vector ${\boldsymbol{k}}$,
the $2^{|{\boldsymbol{k}}|}$ elementary intervals $E({\boldsymbol{k}},{\boldsymbol{c}})$
partition $[0,1)^d$ into congruent sub-intervals.

For each $E({\boldsymbol{k}},{\boldsymbol{c}})$, define $U_{{\boldsymbol{k}},{\boldsymbol{c}}}({\boldsymbol{y}})$ by
$$
U_{{\boldsymbol{k}},{\boldsymbol{c}}}({\boldsymbol{y}})=
\begin{cases}
(-1)^{\sum_{j=1}^d 1\{2^{k_j}y_j-c_j<1/2\}}, & {\boldsymbol{y}}\in E({\boldsymbol{k}},{\boldsymbol{c}})\\
0, &\text{else}.
\end{cases}
$$
If we divide $E({\boldsymbol{k}},{\boldsymbol{c}})$ into $2^d$ sub-intervals, the value of $U_{{\boldsymbol{k}},{\boldsymbol{c}}}({\boldsymbol{y}})$ is constant on each sub-interval and it alternates between $1$ and $-1$.

It is straightforward to verify that $\int_{[0,1]^d}U_{{\boldsymbol{k}},{\boldsymbol{c}}}({\boldsymbol{y}})U_{{\boldsymbol{k}}',{\boldsymbol{c}}'}({\boldsymbol{y}})\,{\mathrm{d}}{\boldsymbol{y}}=0$ if ${\boldsymbol{k}}\neq {\boldsymbol{k}}'$ or ${\boldsymbol{c}}\neq {\boldsymbol{c}}'$. It is trivially true if $E({\boldsymbol{k}},{\boldsymbol{c}})\cap E({\boldsymbol{k}}',{\boldsymbol{c}}')=\varnothing$.
If instead $E({\boldsymbol{k}},{\boldsymbol{c}})\cap E({\boldsymbol{k}}',{\boldsymbol{c}}')\neq \varnothing$, then there must be some $j$ with $k_j<k'_j$ and observe that as a function of $y_j$, $U_{{\boldsymbol{k}},{\boldsymbol{c}}}({\boldsymbol{y}})U_{{\boldsymbol{k}}',{\boldsymbol{c}}'}({\boldsymbol{y}})$ equals $1$ on a $1/2^{k_j+1}$-length interval and equals $-1$ on an adjacent $1/2^{k_j+1}$-length interval, so the integration over variable $j$ always returns $0$. Then for any set $P$ of $({\boldsymbol{k}},{\boldsymbol{c}})$ pairs,
\begin{equation}\label{eqn:Unorm}
   \int_{[0,1]^d}\bigg(\sum_{({\boldsymbol{k}},{\boldsymbol{c}})\in P}U_{{\boldsymbol{k}},{\boldsymbol{c}}}({\boldsymbol{y}})\bigg)^2 \,{\mathrm{d}}{\boldsymbol{y}}=\sum_{({\boldsymbol{k}},{\boldsymbol{c}})\in P}\int_{[0,1]^d}U_{{\boldsymbol{k}},{\boldsymbol{c}}}({\boldsymbol{y}})^2 \,{\mathrm{d}}{\boldsymbol{y}}=\sum_{({\boldsymbol{k}},{\boldsymbol{c}})\in P} 2^{-|{\boldsymbol{k}}|}.
\end{equation}

Now let $\mathcal{P}=\{{\boldsymbol{x}}_0,\ldots,{\boldsymbol{x}}_{n-1}\}$ and choose $m$ so that $2n\leqslant 2^m<4n$. For any ${\boldsymbol{k}}$ with $|{\boldsymbol{k}}|=m$, define the set $P_{\boldsymbol{k}}$ to be
$$P_{{\boldsymbol{k}}} =\{{\boldsymbol{c}}\mid E({\boldsymbol{k}},{\boldsymbol{c}})\cap \mathcal{P}=\varnothing\}.$$
For each ${\boldsymbol{c}}\in P_{\boldsymbol{k}}$,
$$\int_{[0,1]^d}U_{{\boldsymbol{k}},{\boldsymbol{c}}}({\boldsymbol{y}})\prod_{j=1}^d (y_j-1\{y_j>x_{ij}\}) \,{\mathrm{d}}{\boldsymbol{y}}=\frac{1}{4^{m+d}}.$$
Because there are $2^m$ intervals associated with ${\boldsymbol{k}}$, the cardinality of $P_{{\boldsymbol{k}}}$ is at least $2^m-n\geqslant n$. Hence
\begin{equation}\label{eqn:Uprod}
    \int_{[0,1]^d} \bigg( \sum_{{\boldsymbol{c}}\in P_{{\boldsymbol{k}}}}  U_{{\boldsymbol{k}},{\boldsymbol{c}}}({\boldsymbol{y}})\bigg)\prod_{j=1}^d (y_j-1\{y_j>x_{ij}\})\,{\mathrm{d}}{\boldsymbol{y}}\geqslant \frac{n}{4^{m+d}}.
\end{equation}
Now we define
$$h({\boldsymbol{y}})=\sum_{{\boldsymbol{k}}:|{\boldsymbol{k}}|=m} \sum_{{\boldsymbol{c}}\in P_{{\boldsymbol{k}}}} U_{{\boldsymbol{k}},{\boldsymbol{c}}}({\boldsymbol{y}}).$$
The number of ${\boldsymbol{k}}$ with $|{\boldsymbol{k}}|=m$ is the number of ways to partition $m$ into $d$ nonnegative ordered integers,
which equals ${m+d-1\choose d-1}$. Equation~\eqref{eqn:Unorm} and $2n\leqslant 2^m<4n$ imply that
$$\int_{[0,1]^d} h({\boldsymbol{y}})^2 \,{\mathrm{d}}{\boldsymbol{y}}=\sum_{{\boldsymbol{k}}:|{\boldsymbol{k}}|=m} \sum_{{\boldsymbol{c}}\in P_{{\boldsymbol{k}}}} 2^{-m}\leqslant \sum_{{\boldsymbol{k}}:|{\boldsymbol{k}}|=m}1={m+d-1\choose d-1}\leqslant C_d (\log n)^{d-1}$$
for some positive number $C_d$ independent of $n$. On the other hand, equation~\eqref{eqn:Uprod} implies that
$$    \int_{[0,1]^d} h({\boldsymbol{y}})\prod_{j=1}^d (y_j-1\{y_j>x_{ij}\})\,{\mathrm{d}}{\boldsymbol{y}}\geqslant {m+d-1\choose d-1}\frac{n}{4^{m+d}}\geqslant  \frac{c_d(\log n)^{d-1}}{n}$$
for another positive number $c_d$ independent of $n$.

By the Cauchy-Schwarz inequality and equation~\eqref{eqn:ainorm}
\begin{align*}
   &\int_{[0,1]^d} h({\boldsymbol{y}})\bigg(\sum_{i=0}^{n-1} a_i\prod_{j=1}^d (y_j-1\{y_j>x_{ij}\})\bigg) \,{\mathrm{d}}{\boldsymbol{y}}\\
   &\leqslant \bigg(\int_{[0,1]^d} h({\boldsymbol{y}})^2 \,{\mathrm{d}}{\boldsymbol{y}}\bigg)^{\frac{1}{2}}\bigg(\int_{[0,1]^d} \bigg(\sum_{i=0}^{n-1} a_i\prod_{j=1}^d (y_j-1\{y_j>x_{ij}\})\bigg)^2 \,{\mathrm{d}}{\boldsymbol{y}}\bigg)^{\frac{1}{2}}\\
   &\leqslant \bigg(\int_{[0,1]^d} h({\boldsymbol{y}})^2 \,{\mathrm{d}}{\boldsymbol{y}}\bigg)^{\frac{1}{2}}\Bigl\Vert\bsone-\sum_{i=0}^{n-1} a_i K({\boldsymbol{x}}_i,\cdot)\Bigr\Vert
\end{align*}
which provides the lower bound
\begin{align*}
    \Bigl\Vert1-\sum_{i=0}^{n-1} a_i K({\boldsymbol{x}}_i,\cdot)\Bigr\Vert
&\geqslant (C_d (\log n)^{d-1})^{-\frac{1}{2}} \frac{c_d(\log n)^{d-1}}{n} \sum_{i=0}^{n-1} a_i=\bigg(\frac{c_d}{C_d^{1/2}}\sum_{i=0}^{n-1} a_i\bigg)\frac{(\log n)^{(d-1)/2}}{n} .
\end{align*}

Combining the above lower bound with equation~\eqref{eqn:bound1} we get
$$\Bigl\Vert\bsone-\sum_{i=0}^{n-1} a_i K({\boldsymbol{x}}_i,\cdot)\Bigr\Vert
\geqslant \max\Biggl(\Bigl|1-\sum_{i=0}^{n-1} a_i\Bigr|,\bigg(\frac{c_d}{C_d^{1/2}}\sum_{i=0}^{n-1} a_i\bigg)\frac{(\log n)^{(d-1)/2}}{n} \Biggr).$$
For $\lambda>0$,
$$
\min_{a\in{\mathbb{R}}}\max( |1-a|,\lambda a) = \frac{\lambda}{\lambda+1}
$$
so that
\begin{align*}
    \Bigl\Vert1-\sum_{i=0}^{n-1} a_i K({\boldsymbol{x}}_i,\cdot)\Bigr\Vert
&\geqslant
\frac{{c_d(\log n)^{(d-1)/2}}/{(nC_d^{1/2})}}{1+{c_d(\log n)^{(d-1)/2}}/{(nC_d^{1/2})}}
\end{align*}
and we let $A_d =(c_d/C_d^{1/2})/(1+c_dM_d/C_d^{1/2})$
for $M_d = \sup_{n\in{\mathbb{N}}}\log(n)^{(d-1)/2}/n$.
\end{proof}
\begin{corollary}
For any sequence of points $({\boldsymbol{x}}_i)_{i\geqslant0}$ in $[0,1]^d$, there exists a function $f$ in the RKHS with kernel defined by \eqref{eqn:kernel} such that
$$\limsup_{n\to \infty}\frac{\big|\int_{[0,1]^{d}}f({\boldsymbol{x}}) \,{\mathrm{d}}{\boldsymbol{x}}-\frac{1}{n}\sum_{i=0}^{n-1} f({\boldsymbol{x}}_i)\big|}{(n\log\log n)^{-1}(\log n)^{(d-1)/2}}=+\infty$$
\end{corollary}
\begin{proof}
Apply Theorem~\ref{thm:asymptoticrate} with the lower bound on $r_n$ from Theorem~\ref{thm:Roth}.
\end{proof}

\section{Discrepancy and the case of $d=1$}\label{sec:d=1}

Let $x_0,x_1,\dots,x_{n-1}\in[0,1]$.
The local discrepancy of these points at $\alpha\in[0,1]$ is
$$
\delta_n(\alpha) = \frac1n\sum_{i=0}^{n-1}1\{x_i<\alpha\}-\alpha
$$
and the star discrepancy is $D_n^*=\sup_{0\leqslant\alpha\leqslant1}|\delta_n(\alpha)|$.
No infinite sequence $x_i$ can have $D_n^*=o(\log(n)/n)$.
Using results from discrepancy theory we see below that there are specific
values of $\alpha$ for which $D_n^*=\Omega(\log(n)/n)$.
For those values $1\{x<\alpha\}-\alpha$ is a persistent
fooling function.  We show below that $f(x)=x$ is also a persistent
fooling function for the van der Corput sequence.

The set of $\alpha\in[0,1]$
with $|\delta_n(\alpha)|=o(\log(n)/n)$ has Hausdorff dimension 0
for any sequence $(x_i)_{i\geqslant0}\subset[0,1]$. See \cite{hala:1981}.
So $r=1$ is not just available for $d=1$ it is the usual
rate for functions of the form $f(x) = 1\{x<\alpha\}$.

For $x_i$ taken from the van der Corput sequence,
and $\alpha = \sum_{k=1}^\infty a_k/2^k$ for bits $a_k\in\{0,1\}$,
Drmota, Larcher and Pillichshammer \cite{drmo:larc:pill:2005} note
that $n|\delta_n(\alpha)|$
is bounded as $n\to\infty$, if and only if $\alpha$ has a representation
with only finitely many nonzero $a_k$.  Further, letting
$$h_\alpha(m) = \#\{ k<m\mid a_k\ne a_{k+1}\}$$
their Corollary 1 in our notation has
\begin{align}\label{eq:theircor1}
\lim_{m\to\infty}\frac1{2^m}\#\bigl\{
1\leqslant n \leqslant 2^m \mid n\delta_n > (1-\epsilon) h_\alpha(m)
\bigr\}=1
\end{align}
for any $\epsilon >0$.
The base $2$ representation of $2/3$ is $0.10101\cdots$
and so $h_{2/3}(m)=m$.
It follows that $f(x) = 1\{x<2/3\}$ has
$|\hat\mu_n-\mu|>c\log(n)/n$ infinitely often for some $c>0$.
Even more, the fraction of such $n$ among the first $N=2^m$
sample sizes becomes ever closer to $1$ as $m\to\infty$.

If we average the local discrepancy over $\alpha$ we get
$$
\int_0^1\delta_n(\alpha)\,{\mathrm{d}}\alpha = \frac1n\sum_{i=0}^nx_i-\frac12
$$
which is the integration error for the function $f(x)=x$
that we study next.
In our study of $f(x)=x$, we use sample sizes $n$
with base $2$ expansion $10101\cdots101$.
That is, for some $L\geqslant1$
$$n=n_L = \sum_{\ell=0}^L4^{\ell}.$$
The first few values of $n_L$ are
$1$, $5$, $21$, $85$, $341$, $1365$, and $5461$.

\begin{proposition}\label{prop:sumx}
For integers $0\leqslant i<n$, let $x_i$ be the $i$'th van der Corput
point in base $2$.
Then
\begin{align}\label{eq:sumx}
\sum_{i=0}^{n-1}x_i =
\sum_{k=1}^\infty
\lfloor 2^{-k-1}n\rfloor+(2^{-k}n-2\lfloor 2^{-k-1}n\rfloor-1)_+.
\end{align}
\end{proposition}
\begin{proof}
As $i$ increases from $0$, the $k$'th digit of $x_i$
comes in alternating blocks of $2^k$ zeros and $2^k$ ones,
starting with zeros.  For $0\leqslant i<r2^{k+1}$ with
an integer $r\geqslant0$, the $k$'th digits sum to $r2^k$
because there are $r$ complete blocks of $2^k$ ones.
The number of ones among $i_k$ for $0\leqslant i<n$ is then
$$
2^k\lfloor 2^{-k-1}n\rfloor+(n-2^{k+1}\lfloor 2^{-k-1}n\rfloor-2^k)_+
$$
where $z_+=\max(z,0)$.  The first term counts ones
from $\lfloor 2^{-k-1}n\rfloor$ complete blocks of $2^{k+1}$
indices.  That leaves an incomplete block of
$n-2^{k+1}\lfloor 2^{-k-1}n\rfloor$ indices $i$
of which the first $2^k$, should there be that many,
must be $0$s.  Any indices past the first $2^k$ are
ones providing the second term above.
To complete the proof, the $k$'th digits have a coefficient
of $2^{-k}$ in $x_i$ and summing over digits yields~\eqref{eq:sumx}.
\end{proof}

The sum over $k$ in~\eqref{eq:sumx} only needs to go
as far as the number of nonzero  binary digits
in $n-1$.
For larger $k$, both parts of the $k$'th term are zero.

\begin{proposition}
For $L\geqslant0$,  let $n_L = \sum_{\ell=0}^L4^\ell$.
For integers $i\geqslant0$ let $x_i$ be the van der Corput points in base $2$
and for $n\geqslant1$ let $\hat\mu_{n}=(1/n)\sum_{i=0}^{n-1}x_i$.
Then
$$
\limsup_{L\to\infty} \frac{n_L}{\log(n_L)} |\hat\mu_{n_L}-1/2|
>c
$$
if $0<c<1/(8\log(2))$.
\end{proposition}
\begin{proof}
We need $K(n_L)=2L-1$ base $2$ digits to represent $n_L$.
We can write $n_L = \sum_{\ell=0}^L2^{2\ell}$.
Then for $1\leqslant k\leqslant 2L-1$,
\begin{align*}
\lfloor 2^{-k-1}n_L\rfloor
&
=\Bigl\lfloor2^{-k-1}\sum_{\ell=0}^L2^{2\ell}\Bigr\rfloor
=2^{-k-1}n_L-\sum_{\ell=0}^{\lfloor (k-1)/2\rfloor}2^{2\ell-k-1}.
\end{align*}
Next let
\begin{align*}
\theta_k\equiv
\sum_{\ell=0}^{\lfloor (k-1)/2\rfloor}2^{2\ell-k-1}
=2^{-k-1}\sum_{\ell=0}^{\lfloor (k-1)/2\rfloor}2^{2\ell}
= 2^{-k-1}\frac{4^{\lfloor (k+1)/2\rfloor}-1}3
\end{align*}
and then
\begin{align*}
(2^{-k}n_L-2\lfloor 2^{-k-1}n_L\rfloor-1)_+
&=
(2^{-k}n_L-2^{-k}n_L+2\theta_k-1)_+
=(2\theta_k-1)_+
\end{align*}
Because $\theta_k<1/3$ we have
$(2\theta_k-1)_+=0$.
Using Proposition~\ref{prop:sumx},
\begin{align*}
\frac1{n_L}\sum_{i=0}^{n_L-1}x_i
&=\frac1{n_L}\sum_{k=1}^{K(n_L)}\lfloor 2^{-k-1}n_L\rfloor
=\frac1{n_L}\sum_{k=1}^{K(n_L)}
2^{-k-1}n_L-\theta_k\\
&=
\frac12-2^{-K(n_L)-1}-\frac1{n_L}\sum_{k=1}^{K(n_L)}\theta_k.
\end{align*}
Now because $\theta_k\geqslant\theta_2=1/8$,
$|\hat\mu_{n_L}-1/2| \geqslant %
K(n_L)/(8n_L)$
and so
\begin{align*}
\frac{n_L}{\log(n_L)}|\hat\mu_{n_L}-1/2|
&\geqslant \frac{K(n_L)}{8\log(n_L)}
> \frac18\frac{2L-1}{(L+1)\log(4)}\to\frac1{4\log(4)},
\end{align*}
completing the proof.
\end{proof}

We can see why the sample sizes $n_L$ give unusually innaccurate estimates
of the integral of $x$ in the van der Corput sequence.
Those values of $n$ consistently miss getting into the `ones block'
for digit $k$.

Figure~\ref{fig:vdcempiricals} shows some empirical
behavior of the scaled errors for the two integrands
we considered in this section.
The scaled error there is essentially the number of observations
by which the count of points in $[0,2/3)$ differs from
$2n/3$.  It is $n|\delta_n(2/3)|$ which is the customary
scaling in the discrepancy literature.

\begin{figure}[t]
\centering
\includegraphics[width=.9\hsize]{figvdcempiricals}
\caption{\label{fig:vdcempiricals}
Both panels show the scaled error $n|\hat\mu_n-\mu|$ versus $n$
for the first $2^{14}$ points of the van der Corput sequence.
The top panel has the integrand $f(x)=x$ and the values
for $n=n_L$ are indicated with open circles.  The bottom panel has the integrand $f(x) = 1\{x<2/3\}$
and the values $n=\tilde n_L$ are indicated.
}
\end{figure}

The integrands $x$ and $1\{x<2/3\}$ both have
total variation one.  It would be interesting to know
what integrand of total variation one has the largest
value for $\limsup_{n\to\infty} n|\hat\mu_n-\mu|/\log(n)$
in the van der Corput sequence.

For integration, it is advisable to use $n=b^m$ for $m\geqslant0$
in the van der Corput sequence. Then the Koksma-Hlawka inequality
gives us $|\hat\mu_{b^m}-\mu|=O(1/b^m)$ as $m\to\infty$
for any $f$ of bounded variation on $[0,1]$ because
the van der Corput sequence has $D^*_{b^m} = b^{-m}=1/n$.
Any $(t,d)$-sequence for $d=1$ has $D^*_{b^m}=O(1/n)$.
For $d=1$, bounded variation in the sense of Hardy
and Krause reduces to the familiar one dimensional notion
of bounded variation.
As a result a log power $r>0$ can apply to
the limit as $n\to\infty$ through positive
integers but not as $n=b^m$ for $m\to\infty$.

\section{Empirical investigations for $d=2$}
\label{sec:d=2}

This section reports on an empirical search for
an integrand with errors that are $\Omega( \log(n)^r/n)$
for some $r>1$ when using a sequence of points
with $D_n^*=O(\log(n)^2/n)$.  We know from Section~\ref{sec:proofofbound}
that this is attainable for $1<r<3/2$
and not ruled out for $3/2\leqslant r<2$.
We search using some knowledge of the
weaknesses of the van der Corput points for the functions
from Section~\ref{sec:d=1}.

The search must take place with $d\geqslant2$ and so $d=2$
is the natural first place to look for a problematic integrand.
It is clear that the integrand should not be additive because
then integrating it involves a sum of two one dimensional
integration problems where we know that $r>1$ is not needed.
We look at two generalizations of the van der Corput
sequence.  The first is the Halton sequence for $d=2$.
In that sequence, ${\boldsymbol{x}}_{i1}$ is the van der Corput sequence
in base $2$ and ${\boldsymbol{x}}_{i2}$ is the van der Corput sequence
in base $3$.
The second sequence is the Sobol' sequence in base $2$.
It has $t=0$ and like the Halton points, the first
component ${\boldsymbol{x}}_{i1}$ is the van der Corput sequence
in base $2$.  For $n=2^m$, the second component
${\boldsymbol{x}}_{i2}$ of the Sobol' sequence is a permutation
of the first $n$ elements of the van der Corput sequence.

We look first at the scaled error $n|\hat\mu_n-\mu|$
for $f({\boldsymbol{x}}_i) = (x_{i1}-1/2) (x_{i2}-1/2)$.
This integrand has integral zero and two dimensional Vitali variation of one.
It integrates to zero over each component of ${\boldsymbol{x}}$ when the other
component is fixed at any value.  It has no nonzero ANOVA component
of dimension smaller than $2$.  Each of the factors is a persistent
fooling function for the van der Corput points.

For the Halton sequence, the scaled error equals $1/4$ when $n=1$
and it remains below $1/4$ for $2\leqslant n\leqslant 2^{20}$.  It repeatedly approaches $1/4$
from below.
Figure~\ref{fig:d2prodempirical} shows the first $2^{19}$ scaled errors
for this product integrand for both Halton and Sobol' points.
For both constructions we see no evidence that the error is
$\Omega(\log(n)^r/n)$ for $r>1$.  In fact, what is quite surprising
there is the apparent $O(1/n)$ error rate, which is then
{\sl better} than what the van der Corput sequence attains for $f(x)=x$.

By plotting only $2^{19}$ points
instead of all $2^{20}$ it becomes possible to see something of
the structure within the triangular peaks for the Sobol' points.
The linear scale for $n$ (versus a logarithmic one) shows
a clear repeating pattern consistent with an $O(1/n)$ error rate.
Whether or not the errors are $O(1/n)$, this integrand is
not a promising one to investigate further in the
search for an integrand needing $r>1$.

\begin{figure}[t]\centering
\includegraphics[width=.9\hsize]{figd2prodempirical.png}
\caption{\label{fig:d2prodempirical}
The panels show scaled errors $n|\hat\mu_n-\mu|$ versus $n$
for the integrand $f({\boldsymbol{x}})=(x_1-1/2) (x_2-1/2)$ on $[0,1]^2$.
The top panel is for Halton points and the bottom one is
for Sobol' points.
}
\end{figure}

Another challenging integrand for van der Corput
points was $1\{x<2/3\}$.
For the Sobol' points we then take
$f({\boldsymbol{x}})=\prod_{j=1}^2( 1\{x_j<2/3\}-2/3)$
for $d=2$.  Once again we have removed
the additive contribution to the integrand
that we know is integrated at the $\log(n)/n$
rate making it easier to discern the effect
of the bivariate component.
For Halton points, we do not use $2/3$
as that has a terminating expansion in base $3$
that defines the second component of the ${\boldsymbol{x}}_i$.
We use
$f({\boldsymbol{x}})=(1\{x_1<2/3\}-2/3) (1\{x_2<3/5\}-3/5)$
for Halton points because $3/5$ does not have a terminating
expansion in base $3$ that could make the problem
artificially easy.  Both of these functions have
two dimensional Vitali variation equal to one, just
like $\prod_{j=1}^2(x_j-1/2)$.
Figure~\ref{fig:d2rectempirical} shows the scaled errors
$n|\hat\mu-\mu|/\log(n)$. The logarithmic scaling
in the denominator is there so that
a value of $r>1$ would give an infinite series of new records.
We don't see many such new records for $n\leqslant 2^{22}$
for either of the two test cases. These integrands were designed
to exploit weaknesses in the Sobol' and Halton sequences
and they did not produce the appearance of errors growing
faster than $\log(n)/n$.  It remains possible that errors
grow like $\log(n)^r/n$ for $r>1$ but with the records being set very sparsely.
The situation is quite different from Figure~\ref{fig:vdcempiricals}
in the one dimensional case where we see a steady sequence of
record values with a fractal pattern in the empirical errors.

\begin{figure}[t]\centering
\includegraphics[width=.9\hsize]{figd2rectempirical.png}
\caption{\label{fig:d2rectempirical}
The panels show scaled errors $n|\hat\mu_n-\mu|/\log(n)$ versus $n>1$.
The top panel has $f({\boldsymbol{x}})=(1\{x_1<2/3\}-2/3)\times(1\{x_2<2/3\}-2/3)$
for Sobol' points.
The bottom panel has
$f({\boldsymbol{x}})=(1\{x_1<2/3\}-2/3)\times(1\{x_2<3/5\}-3/5)$
for Halton points.
}
\end{figure}

There are some other integrands that could be difficult for
QMC.  A badly approximable irrational number $\theta$
is one where the distance between $n\theta$
and $\Z$ is above $c/n$ for some $c>0$ and all integers $n\geqslant1$.
For instance, $\theta=\sqrt{2}-1$ is badly approximable.
An integrand like $\prod_{j=1}^d (1_{x_j<\theta}-\theta)$
could pose a challenge to QMC methods, but some exploration
with Halton and Sobol' points was inconclusive: there was
no empirical indication that $r>1$ would be required.
Another potentially difficult integrand is $\prod_{j=1}^d (x_j^\theta-1/(1+\theta))$
for $0<\theta<1$. Partial derivatives of this integrand with respect to a subset
of components $x_j$ are unbounded.  That would make them fail the sufficient
conditions for scrambled nets to attain $O(n^{-3/2+\epsilon})$ RMSE
in \cite{smoovar,localanti,yue:mao:1999}.
These integrands did not show a need for $r>1$.

Next, we consider the function $f({\boldsymbol{x}}) = 1\{x_1+x_2<1\}$.
This integrand has infinite Vitali variation
over $[0,1]^2$. Therefore it also has infinite Hardy-Krause
variation  and so the Koksma-Hlawka bound degenerates to $+\infty$.
Because $f$ is Riemann integrable we know that $\hat\mu_n\to\mu_n$
if $D_n^*\to0$.
Finding that $r>1$ for this $f$ would not provide
an example of a BVHK function needing that $r>1$.
It remains interesting to see what happens because the case is
not covered by QMC theory without randomization.

Figure~\ref{fig:d2notbvempirical} shows scaled errors
for this integrand, using an exponent of $r=1$  for $\log(n)$.
There is a striking difference between the results for Sobol'
points versus Halton points.  We see a few approximate doublings of the
scaled error for Sobol' points even when using $r=1$.
This does not prove that
we need $r>1$ here; perhaps we saw the last doubling
or perhaps similar jumps for larger $n$ arise at extremely sparse intervals.
It does serve to raise some additional
questions.  For instance, why are Halton points so much more
effective on this integrand, and to which other integrands
of unbounded variation might that apply?
For smooth integrands, Sobol' points commonly perform
much better when $n$ is a power of two than otherwise.
Here, those sample sizes did not bring much better outcomes.

\begin{figure}[t]\centering
\includegraphics[width=.9\hsize]{figd2notbvempirical.png}
\caption{\label{fig:d2notbvempirical}
This shows
scaled errors $n|\hat\mu_n-\mu|/\log(n)$ versus $n>1$.
The integrand is $1\{x_1+x_2<1\}$ which has infinite
Vitali variation on $[0,1]^2$. The upper curve is for Sobol' points.
The lower curve is for Halton points. There is a reference
line at scaled error of zero.
}
\end{figure}

\section{Very large $m$ for Sobol' nets}\label{sec:bigm}

If the need for $r>1$ is only evident for very large $n$
then we might fail to detect it by computing $\hat\mu_n$.
If we restrict to sample sizes $n=2^m$  for $m\geqslant0$ then
we can use properties of the generating matrices of Sobol'
sequences to compute the scaled error $n|\hat\mu-\mu|$
for very large $n$ when $f$ is the indicator function
of a set $[\bszero,{\boldsymbol{a}})$.
The first $n=2^m$ Sobol' points form a $(t,m,s)$-net in
base $2$ for which the Koksma-Hlawka bound gives
a rate of $O(n^{-1}\log(n)^{d-1})$.  As a result we must
look to $d=3$ or larger for a problem that needs $r>1$
for these values of $n$.
We choose ${\boldsymbol{a}} = (2/3,2/3,\dots,2/3)$ of length $d$
as $2/3$ is difficult to approximate in base $2$.

We can partition $[0,2/3)^d$ into a countable number
of elementary intervals in base $2$.
The number of points of the digital net
${\boldsymbol{x}}_0,\dots,{\boldsymbol{x}}_{b^m-1}$
that are in $E({\boldsymbol{k}},{\boldsymbol{c}})$ equals the number of solutions
$\vec{i}\in \{0,1\}^m$ to a linear equation
$C\vec{i}=\vec{a} \mathrm{\,mod\,} 2$
for a matrix $C\in\{0,1\}^{|{\boldsymbol{k}}|\times m}$
that takes the first $k_j$ rows from the $j$'th
generating matrix of the digital net, for $j=1,\dots,d$
and some $\vec{a}\in\{0,1\}^m$.
See \cite{pan:owen:2021:tr} for a description and
a discussion of how to find the number of such points.
For our Sobol' points we use  the direction
numbers from Joe and Kuo \cite{joe:kuo:2008}.

To make the computation finite,
we replace $[0,2/3)^d$ by $[0,a_m)^d$
where $a_m=\lfloor 2^m (2/3)\rfloor/2^m$.
For $a\in(0,1)$, let $\mu(a) = a^d$
and $\hat\mu(a) =(1/n)\sum_{i=0}^{n-1}1_{{\boldsymbol{x}}_i\in[0,a)^d}$.
Then
\begin{align*}
0&\leqslant \mu\Bigl(\frac23\Bigr)-\mu(a_m)
= \Bigl(\frac23-a_m\Bigr)\sum_{j=0}^{d-1}
\Bigl(\frac23\Bigr)^ja_m^{d-j-1}
\leqslant \Bigl(\frac23\Bigr)^{d-1}\frac{d}n.
\end{align*}
The number of the first $n$ Sobol'
points that belong to $[0,2/3)^d\setminus[0,a_m)^d$
is at most $d$. Therefore
\begin{align*}
0&\leqslant
\hat\mu(2/3)-\hat\mu(a_m)\leqslant d/n
\end{align*}
Now
\begin{align*}
&\quad n(\hat\mu(2/3)-\mu(2/3))
-n(\hat\mu(a_m)-\mu(a_m))\\
&=
n(\hat\mu(2/3)-\hat\mu(a_m)
-n(\mu(2/3)-\mu(a_m))\\
&\in [-(2/3)^{d-1}d,d]\subseteq [-4/3,d]
\end{align*}
so the absolute error in using $n((\hat\mu(a_m)-\mu(a_m))$
instead of $n((\hat\mu(2/3)-\mu(2/3))$
is at most $d$.

Figure~\ref{fig:errorvsmd} shows the results for $d=2,3,4$.
For $d=2$ we see strong linear trend
lines in the scaled error consistent with
needing $r=1$.  For $d=3,4$ the trend is not
obviously linear but it does not have a compelling
and repeating $r>1$ pattern even at $n=2^{50}$ for $d=4$
or $n=2^{100}$ for $d=3$.

\begin{figure}[t]
\centering
\begin{minipage}{0.34\hsize}
\includegraphics[width=\textwidth]{errorvsm,d=2}
\end{minipage}
\hspace*{-.35cm}
\begin{minipage}{0.34\hsize}
\includegraphics[width=\textwidth]{errorvsm,d=3}
\end{minipage}
\hspace*{-.35cm}
\begin{minipage}{0.34\hsize}
\includegraphics[width=\textwidth]{errorvsm,d=4}
\end{minipage}
\caption{\label{fig:errorvsmd}
The panels show the signed scaled error
$n(\hat\mu-\mu)$ for a Sobol' sequence
when $f=\prod_{j=1}^d1_{x_j<2/3}$.
The panels have $d=2,3,4$.
The sample sizes go to $2^{100}$ for
$d=2,3$ and to $2^{50}$ for $d=4$.
The computed values differ from the true
ones by at most $d$ for all $m$.
}
\end{figure}

\section{Discussion}\label{sec:discussion}

For any $\epsilon>0$
and any $d\geqslant1$ and any infinite sequence of points ${\boldsymbol{x}}_i\in[0,1]^d$
and any $c>0$ that there are functions in BVHK$[0,1]^d$ where the QMC error
exceeds $c(\log n)^{r}/n$ infinitely often when $r<(d-1)/2$.
For $d=1$ we can easily construct such functions for $r=1$
using results from discrepancy theory.  We don't know of
any specific examples with $r>1$ and simple multidimensional
generalizations of the functions needing $r=1$ for $d=1$
did not show an apparent need for $r>1$ when $d=2$.
The best candidates we saw for the Sobol' sequence
are $f({\boldsymbol{x}}) =\prod_{j=1}^d1\{x_j<2/3\}$ for $d=3$ or $4$
but we have no proof that they require $r>1$.

One surprise is that comparing
Figures~\ref{fig:vdcempiricals} and~\ref{fig:d2prodempirical} we see an error for $f({\boldsymbol{x}})=(x_1-1/2)(x_2-1/2)$ that appears
to be $O(1/n)$ while the one dimensional
function $f(x)=x$ has error $\Omega(\log(n)/n)$ (theoretically
and also empirically over small $n$).
A second surprise is that for a two dimensional function
of unbounded variation we see very different behavior
for Sobol' and Halton points in
Figure~\ref{fig:d2notbvempirical}.  The error for Sobol'
points appears to grow faster than $\log(n)/n$
while that for Halton points is far lower and might
even grow at a different rate.  Neither of these surprises
contradict known theory but it is odd to see the two
dimensional problem apparently easier to solve than
a corresponding one dimensional one and it is also odd
to see Halton points appear to be so much more robust
to unbounded variation than Sobol' points.

So, where could the logs be?  It is possible that they are only
practically important for enormous $n$, or what is almost
the same thing, that they are present with a very tiny implied constant.
It is also possible that the commonly investigated integrands
have error $O(\log(n)/n)$ even for $d\geqslant2$.

\section*{Acknowledgments}
This work was supported by the U.S.\ NSF under
grant IIS-1837931.  Thanks to Fred Hickernell and
Erich Novak for discussions related to this problem.
We are also grateful to Traub, Wasilkowski and Wo\'zniakowski
for ensuring that Trojan's work was not lost.

\bibliographystyle{plain}

Numerical Analysis (math.NA); Computation (stat.CO)

Where are the logs?


A Geometric Approach for Bounding Average Stopping Time

Abstract: We propose a geometric approach for bounding average stopping times for stopped random walks in discrete and continuous time. We consider stopping times in the hyperspace of time indexes and stochastic processes. Our techniques relies on exploring geometric properties of continuity or stopping regions. Especially, we make use of the concepts of convex sets and supporting hyperplane. Explicit formulae and efficiently computable bounds are obtained for average stopping times. Our techniques can be applied to bound average stopping times involving random vectors, nonlinear stopping boundary, and constraints of time indexes. Moreover, we establish a stochastic characteristic of convex sets and generalize Jensen's inequality, Wald's equations and Lorden's inequality, which are useful for investigating average stopping times.

\section{Stopping Times and Convex Sets}

In this section, we shall propose to investigate stopping times with their geometric representations.  We shall also establish a connection
between stopping times and convex sets.  A stochastic characterization of convex sets is developed.

\subsection{Geometric Representations of Stopping Times}

Existing methods for bounding the average of a stopping time typically focus on exploring the properties of the function defining the stopping
time. Consider, for example, the stopping time mentioned in the introduction of this paper.  To bound the expectation of stopping time \begin{equation}
\label{st8899} \boldsymbol{N} = \inf \{n \in \mathbb{N}: X_1 + \cdots + X_n \geq f(n) \}, \end{equation}
 conventional wisdom is to explore the function $f(n)$ for properties such as
linearity and  monotonicity  which could be useful for bounding the
average stopping time.  We would like to point out that the methods
in this direction usually fail to fully exploit the geometric
information of the underlying continuity or stopping regions.  To
clearly address this point, we shall first provide geometric
representations of stopping times in the sequel.

Throughout the remainder of this paper, we shall use the following notations and definitions.  Let $0 \leq N_0 < N_1 < N_2 < \cdots $ be an
increasing sequence of integers and define $\mathscr{N} = \{ N_1, N_2, \cdots \}$. Let $\mathscr{R}$ be a subset of $\{ (t, s): t \in \mathbb{R}^+, \; s \in
\mathbb{R}^d \}$. Let $X = [\boldsymbol{x}_1, \cdots, \boldsymbol{x}_d ]$ be a $d$-dimensional real-valued random vector with mean $\mu = \mathbb{E} [ X ]$. Let $X_1, X_2,
\cdots$ be i.i.d. samples of $X$. Define $S_0 = 0$ and
\[
S_n = \sum_{i=1}^n X_i, \qquad \overline{X}_n = \frac{S_n}{n}
\]
for $n \in \mathbb{N}$.  Our effort will be devoted to stopping times which are defined in terms of sample sum $S_n$ (or equivalently, sample mean
$\overline{X}_n$), the region $\mathscr{R}$ and the set $\mathscr{N}$. The stopping times defined in this way can be fairy general.

A stopping time can be defined in terms of $S_n$ as $\boldsymbol{N} = \inf \{ n \in \mathscr{N}: (n, S_n) \in \mathscr{R} \}$. Such stopping time is
associated with the stopping rule: Continue observing $S_n$ until $(n, S_n) \in \mathscr{R}$ for some $n \in \mathscr{N}$. For such stopping rule, the
region $\mathscr{R}$ is referred to as a stopping region.

On the other hand, a stopping time can also be defined as $\boldsymbol{N} = \inf \{ n \in \mathscr{N}: (n, S_n) \notin \mathscr{R} \}$.  Such stopping time is
associated with the stopping rule: Continue observing $S_n$ until $(n, S_n) \notin \mathscr{R}$ for some $n \in \mathscr{N}$. For such stopping rule,
the region $\mathscr{R}$ is referred to as a continuity region.

Despite the generality of the above geometric representations,
stopping times are usually expressed in algebraic forms. A familiar
example is the stopping time defined by (\ref{st8899}).  In this
paper, we propose to investigate stopping times based on their
geometric representations. The primary reason is that the bounding
of average stopping times can be much more easier by exploiting the
geometric properties of the underlying continuity or stopping region
$\mathscr{R}$. As will be seen later, this is especially true when the
closure of the region $\mathscr{R}$ is convex. We discovered that, for
a wide variety of stopping times in the context of sequential
hypothesis testing and estimation,  the corresponding continuity or
stopping regions $\mathscr{R}$ in geometric representations are
actually convex.  In the worse case that the continuity or stopping
regions $\mathscr{R}$ are not convex, it is still possible to bound the
average stopping time by replacing $\mathscr{R}$ with its convex hull
$\mathcal{R}$, at the price of extra conservatism.

To illustrate the advantage of geometric representations, consider
stopping time \[ \boldsymbol{N} = \inf \{ n \in \mathbb{N}: f(n, S_n) \geq 0 \},
\]
where $f(t, s)$ is a bivariate function of $t \in \mathbb{R}^+$ and $s
\in \mathbb{R}$.  Clearly, the stopping region is \begin{equation} \label{stopreg}
\mathscr{R} = \{ (t, s): t \in \mathbb{R}^+, \; s \in \mathbb{R}, \; f (t, s)
\geq 0\} \end{equation} and  the stopping time $\boldsymbol{N} = \inf \{ n \in \mathbb{N}:
(n, S_n) \in \mathscr{R} \}$. Similarly, the continuity region is \begin{equation}
\label{contreg} \mathscr{R}^c = \{ (t, s): t \in \mathbb{R}^+, \; s \in
\mathbb{R}, \;  f(t, s) < 0 \} \end{equation} and the stopping time $\boldsymbol{N} = \inf
\{ n \in \mathbb{N}: (n, S_n) \notin \mathscr{R}^c \}$.  It can be shown
that if $f$ is a concave function, then the stopping region
(\ref{stopreg}) is convex. If $f$ is a convex function, then the
continuity region (\ref{contreg}) is convex.  It is important to
note that the convexity of the stopping or continuity region may
also hold in situations when the function $f$ is neither convex nor
concave. Moreover, even if neither the continuity region nor the
stopping region is convex, we can still bound the average stopping
time by using their convex hulls. This example demonstrates that, in
contrast to using algebraic forms of stopping times, it is possible
to exploit the convexity of the continuity or stopping regions in
geometric representations under much weaker conditions.

\subsection{ A Stochastic Characteristic of Convex Sets}

As discussed in the last subsection, there exists a useful connection between stopping times and convex sets.   Since continuity or stopping
regions are convex in many situations, it is natural to consider the question of under what conditions the expectation of a random vector will
be contained by a convex set. Our investigation indicates that if a set in a finite-dimensional Euclidean space is convex, then the set contains
the expectation of any random vector almost surely contained by the set. Conversely, if a set in a finite-dimensional Euclidean space contains
the expectation of any random vector almost surely contained by the set, then the set is convex. More formally, we have established the
following results.

\begin{theorem}

\label{fundamental}

If $\mathscr{D}$ is a convex set in $\mathbb{R}^n$, then $\mathbb{E} [ \boldsymbol{\mathcal{X}} ] \in \mathscr{D}$ holds
for any random vector $\boldsymbol{\mathcal{X}}$ such that $\Pr \{ \boldsymbol{\mathcal{X}}
\in \mathscr{D} \} = 1$ and that $\mathbb{E} [ \boldsymbol{\mathcal{X}} ] $ exists.
Conversely, if $\mathscr{D}$ is a set in $\mathbb{R}^n$ such that $\mathbb{E} [ \boldsymbol{\mathcal{X}} ] \in
\mathscr{D}$ holds for any random vector $\boldsymbol{\mathcal{X}}$
such that $\Pr \{ \boldsymbol{\mathcal{X}}  \in \mathscr{D} \} = 1$ and that $\mathbb{E} [ \boldsymbol{\mathcal{X}} ] $ exists, then
$\mathscr{D}$ is convex.  \end{theorem}

See Appendix \ref{fundamentalapp} for a proof.  This theorem plays a fundamental role in our approach for bounding average stopping times.

We would like to point out that  the first assertion of Theorem
\ref{fundamental} provides a simple proof of Jensen's inequality. To
see this, note that if a function is convex, then its epigraph, the
region above its graph, is a convex set. Hence, if $f$ is a convex
function, then for any random variable $X$, since $(X, \; f(X))$ is
contained by the epigraph of $f$, it follows from Theorem
\ref{fundamental} that $(\mathbb{E}[ X ], \; \mathbb{E} [ f(X) ] )$ is
contained by its epigraph. This implies that $\mathbb{E} [ f(X) ] \geq f
(\mathbb{E} [ X ] )$ by the notion of epigraph.

\section{Generalizations of Jensen's Inequality, Wald's Equations and Lorden's Inequality}

In this section, we shall generalize Jensen's inequality, Wald's equations and Lorden's inequality, which can be useful for evaluating average
stopping times.

\subsection{Generalization of Jensen's Inequality}

For investigating the convexity of continuity and stopping regions, we have the following results.

\begin{theorem}
 \label{BoundaryCon}
Suppose that $g(z)$ is a multivariate convex function of $z \in \mathscr{D}$, where $\mathscr{D}$ is a convex set.
 Define $f (t, s) = t g \left (\frac{s}{t} \right )$ for $t \neq 0$ and $s$ such that $\frac{s}{t} \in \mathscr{D}$. Then, $f(t, s)$
 is a multivariate convex function of $t > 0$ and $s$ such that $\frac{s}{t} \in \mathscr{D}$.  Similarly, $f(t, s)$
 is a multivariate concave function of $t < 0$ and $s$ such that $\frac{s}{t} \in \mathscr{D}$.

\end{theorem}

See Appendix \ref{BoundaryConapp} for a proof.   As an application of Theorem \ref{BoundaryCon}, consider stopping time $\boldsymbol{N} = \inf \{ n \in
\mathscr{N}: f(n, S_n) \geq 0 \}$,  with
\[
f(n, S_n) = \sum_{\ell = 1}^k \left [ n g_\ell \left ( \frac{A_\ell^\top S_n + \beta_\ell}{n} \right ) + h_\ell (n, S_n) \right ],
\]
where $g_\ell, \; h_\ell$ are multivariate convex functions, $A_\ell \in \mathbb{R}^d$, and $\beta_\ell \in \mathbb{R}$.  The convexity of the continuity
region follows immediately from Theorem \ref{BoundaryCon}.

By virtue of Theorem \ref{BoundaryCon}, we have derived the following results.

\begin{theorem}

\label{genJen} Let $\boldsymbol{Z}$ be a random vector and $Y$ be a scalar random variable such that $\frac{ \boldsymbol{Z}  }{Y}$  and $\frac{ \mathbb{E} [ \boldsymbol{Z} ] }{
\mathbb{E} [ Y ] } $ are contained in a convex set $\mathscr{D}$. Assume that $g(z)$ is a multivariate convex function of $z \in \mathscr{D}$. Then,
\[
\mathbb{E} \left [ Y g \left ( \frac{\boldsymbol{Z}}{Y} \right ) \right ] \geq  \mathbb{E} [ Y ] g \left ( \frac{ \mathbb{E} [ \boldsymbol{Z} ]}{\mathbb{E} [ Y ]} \right ) \qquad \text{if $Y$ is a
positive random variable};
\]
\[
\mathbb{E} \left [ Y g \left ( \frac{\boldsymbol{Z}}{Y} \right ) \right ] \leq  \mathbb{E} [ Y ] g \left ( \frac{ \mathbb{E} [ \boldsymbol{Z} ]}{\mathbb{E} [ Y ]} \right ) \qquad \text{if $Y$ is a
negative random variable}.
\]

\end{theorem}

\bigskip

See Appendix \ref{genJenapp} for a proof.   It should be noted that Theorem \ref{genJen} generalizes Jensen's inequality.  In the special case
that $Y = 1$, the first inequality of Theorem \ref{genJen} reduces to Jensen's inequality.

\subsection{Generalization of Wald's Equations}

Making use of Theorem \ref{genJen}, we have generalized Wald's equations \cite{Wald} as follows.

\begin{theorem}

\label{genWald}

Let $X_1, X_2, \cdots$ be i.i.d. samples of random vector $X$  with mean $\mu = \mathbb{E} [ X ]$ and variance $\nu = \mathbb{E} [ | X - \mu |^2 ]$. Assume
that $\boldsymbol{N}$ is an integer-valued  random variable such that $\mathbb{E} [ \boldsymbol{N} ] < \infty$ and that for any possible value $n$ of $\boldsymbol{N}$, the
event $\{ \boldsymbol{N} = n \}$ depends only on $X_1, \cdots, X_n$.  Define
\[
S_{\boldsymbol{N}} = \sum_{i=1}^{\boldsymbol{N}} X_i, \qquad \overline{X}_{\boldsymbol{N}}  = \frac{S_{\boldsymbol{N}} }{\boldsymbol{N}}, \qquad \overline{V}_{ \boldsymbol{N} } = \frac{(S_{\boldsymbol{N}} - \boldsymbol{N} \mu)^2
}{\boldsymbol{N}}.
\]
  The following assertions hold.

(I): If $g$ is a convex function on a convex set $\mathscr{D}$ in $\mathbb{R}^d$ such that $\mathscr{D}$ contains $\mu$ and the range of $\overline{X}_{\boldsymbol{N}}
$, then
\[
\mathbb{ E } \left [ \boldsymbol{N} g ( \overline{X}_{\boldsymbol{N}} ) \right ] \geq  \mathbb{ E } [ \boldsymbol{N} ] g (\mu).
\]

(II): If $g$ is a convex function on a convex set $\mathscr{D}$ in $\mathbb{R}^d$ such that $\mathscr{D}$ contains $\nu$ and the range of $\overline{V}_{
\boldsymbol{N} }$, then \[ \mathbb{ E } \left [ \boldsymbol{N} g \left (  \overline{V}_{ \boldsymbol{N} } \right  ) \right ] \geq  \mathbb{ E } [ \boldsymbol{N} ] g (\nu).
\]

\end{theorem}

See Appendix \ref{genWaldapp} for a proof.   To see why the inequality in the first assertion of Theorem \ref{genWald} is a generalization of
Wald's first equation, consider function $g(x) = x$. By the convexity of $g(x)$, we have $\mathbb{ E } \left [ \boldsymbol{N} \overline{X}_{\boldsymbol{N}} \right ] \geq
\mathbb{ E } [ \boldsymbol{N} ] \mu$.  On the other hand, by the convexity of $- g(x)$, we have $\mathbb{ E } \left [ \boldsymbol{N} (- \overline{X}_{\boldsymbol{N}} ) \right ] \geq \mathbb{
E } [ \boldsymbol{N} ] (- \mu)$ or equivalently, $\mathbb{ E } \left [ \boldsymbol{N} \overline{X}_{\boldsymbol{N}} \right ] \leq \mathbb{ E } [ \boldsymbol{N} ] \mu$.  Hence, it must be true
that $\mathbb{ E } \left [ S_{\boldsymbol{N}} \right ] = \mathbb{ E } \left [ \boldsymbol{N}  \overline{X}_{\boldsymbol{N}}  \right ] =  \mathbb{ E } [ \boldsymbol{N} ] \mu$, which is Wald's first
equation. Similarly, we can demonstrate that the inequality in the second assertion of Theorem \ref{genWald} is a generalization of Wald's
second equation.  As an illustration of the applications of Theorem \ref{genWald}, consider
\[
\boldsymbol{N} = \inf \left \{ n \in \mathscr{N}: n \geq \frac{1}{ g ( \overline{X}_n ) }, \; g ( \overline{X}_n ) > 0 \right \}.
\]
Clearly,
\[
\boldsymbol{N} \geq \frac{1}{ g ( \overline{X}_{\boldsymbol{N}} ) }, \qquad g ( \overline{X}_{\boldsymbol{N}} ) > 0
\]
and thus  $\boldsymbol{N} g ( \overline{X}_{\boldsymbol{N}} )  \geq 1$ almost surely provided that $\mathbb{E} [ \boldsymbol{N} ] < \infty$.  By using the generalization of  Wald's
first equation, we have $\mathbb{E} [ \boldsymbol{N} ] g (\mu) \geq 1$, which implies the following result.

\begin{theorem} \label{UseWald} Assume that $g$ is a concave function on a convex set $\mathscr{D}$ in $\mathbb{R}^d$ such that $\mathscr{D}$ contains $\mu$ and the
range of $\overline{X}_{\boldsymbol{N}}$ and that $g(\mu) > 0$. Then, $\mathbb{E} [ \boldsymbol{N} ] \geq \frac{1}{ g (\mu)}$.

\end{theorem}

As another application example of Theorem \ref{genWald}, consider $L^p$-norm function $g (s) = ||s||_p$, where $s \in \mathbb{R}^d$ and $p \geq 1$.
As a consequence of the absolute homogeneity and subadditivity of the $L^p$-norm,  we have
\[
|| \rho \boldsymbol{x} + (1 - \rho) \boldsymbol{y} ||_p \leq || \rho \boldsymbol{x}  ||_p + || (1 - \rho) \boldsymbol{y} ||_p = \rho || \boldsymbol{x}  ||_p + (1 - \rho) ||  \boldsymbol{y} ||_p
\]
for arbitrary vectors $\boldsymbol{x}, \; \boldsymbol{y} \in \mathbb{R}^d$ and $\rho \in [0, 1]$.  Hence, the $L^p$-norm function $g (s) = ||s||_p$ is a convex
function of $s \in \mathbb{R}^d$.  Applying Theorem \ref{genWald} to the $L^p$-norm function $g (s) = ||s||_p$, we have the following results.

\begin{theorem}

Let $X_1, X_2, \cdots$ be i.i.d. samples of random vector $X$  with mean $\mu = \mathbb{E} [ X ]$ and variance $\nu = \mathbb{E} [ | X - \mu |^2 ]$. Assume
that $\boldsymbol{N}$ is an integer-valued  random variable such that $\mathbb{E} [ \boldsymbol{N} ] < \infty$ and that for any possible value $n$ of $\boldsymbol{N}$, the
event $\{ \boldsymbol{N} = n \}$ depends only on $X_1, \cdots, X_n$.  Define $S_{\boldsymbol{N}}  = \sum_{i=1}^{\boldsymbol{N}} X_i$ and $V_{\boldsymbol{N}} = ( S_{\boldsymbol{N}} -
\boldsymbol{N} \mu )^2$.  Then, \begin{eqnarray*} &  & \mathbb{ E } \left [ || S_{\boldsymbol{N}} ||_p \right ] \geq  \mathbb{ E } [ \boldsymbol{N} ] \; || \mu ||_p, \\
&  & \mathbb{ E } \left [ || V_{\boldsymbol{N}} ||_p \right ] \geq  \mathbb{ E } [ \boldsymbol{N} ] \; || \nu ||_p  \end{eqnarray*}
 for all $p \geq 1$.

 \end{theorem}

\subsection{Generalization of Lorden's Inequality}

In order to obtain tight bounds for stopping times, we need to generalize Lorden's inequality \cite{Lorden}.
In this direction, we have obtained the following
result.

 \begin{theorem} \label{Lordengen}
 Let $Z_1, Z_2, \cdots$ be i.i.d. samples of random variable $Z$ such that $\mathbb{E} [ Z^2 ] < \infty$.
 Assume that $\boldsymbol{\lambda}$ is a random variable independent of $Z_i$ for all $i \in \mathbb{N}$.
 Define {\small $\mathscr{M}_{\boldsymbol{\lambda}} = \inf \left \{ n \in \mathbb{N}: \sum_{i = 1}^n Z_i \geq \boldsymbol{\lambda} \right
\}$} and $R_{\boldsymbol{\lambda}} = \sum_{i =1}^{\mathscr{M}_{\boldsymbol{\lambda}}}  Z_i - \boldsymbol{\lambda}$.  Then, \[ \mathbb{E} [ R_{\boldsymbol{\lambda}} ] \leq \frac{ \mathbb{E} [ (Z^+)^2 ] }{
\mathbb{E} [ Z ] } \Pr \{ Z <  \boldsymbol{\lambda}  \} + \mathbb{E} [ (Z - \boldsymbol{\lambda} )^+ ].
\]

\end{theorem}

See Appendix \ref{Lordengenapp} for a proof.

In the following, we have extended Lorden's inequality to the case that the increment of sample sizes is not a constant.

\begin{theorem}

\label{Lordenspec} Let $Z_1, Z_2, \cdots$ be i.i.d. samples of positive random variable $Z$ such that $\mathbb{E} [ Z^2 ] < \infty$.
 Assume that $\boldsymbol{\lambda}$ is a random variable independent of $Z_i$ for all $i \in \mathbb{N}$.
 Define $\mathscr{M}_{\boldsymbol{\lambda}} = \inf \{ n \in \mathscr{N}: \sum_{i = 1}^n Z_i \geq \boldsymbol{\lambda} \}$ and $R_{\boldsymbol{\lambda}} = \sum_{i =1}^{\mathscr{M}_{\boldsymbol{\lambda}}}  Z_i - \boldsymbol{\lambda}$.
  Define $Y = Z_1 + \cdots + Z_{N_1}$ and $K = \max \{ N_{\ell + 1} - N_\ell: \ell \in \mathbb{N} \}$. Then, \[ \mathbb{E} [ R_{\boldsymbol{\lambda}} ] \leq
\left ( (K - 1) \mathbb{E} [ Z ]  + \frac{ \mathbb{E} [ Z^2 ] }{ \mathbb{E} [ Z ] } \right ) \Pr \{  Y  <  \boldsymbol{\lambda}  \} + \mathbb{E} [ (Y - \boldsymbol{\lambda} )^+ ].
\]

\end{theorem}

See Appendix \ref{Lordenspecapp} for a proof.

\section{Bounding Average Stopping Time via Convex Optimization}

In this section, we shall demonstrate that the general problem of bounding average stopping times can be converted into problems of convex
minimization, which can be readily solved by modern optimization theory and algorithms. In some particular cases, it is possible to obtain
explicit bounds for average stopping times.

\subsection{Lower Bound on Average Stopping Time}

 Consider stopping time $\boldsymbol{N} = \inf \{ n \in \mathscr{N}: (n, S_n) \in \mathscr{R} \}$,
 where the set $\mathscr{R}$ is called a stopping region.  We have the following
results on the average stopping time.

\begin{theorem}

\label{Convexlower}

Suppose that the stopping region $\mathscr{R}$ is a convex set.  Define $\mathscr{A} = \{ t \in \mathbb{R}^+: (t, t \mu) \in \mathscr{R} \}$. Then, $\mathbb{E} [
\boldsymbol{N} ] \geq \min \mathscr{A}$ provided that $\mathscr{A}$ is not empty.  Moreover, $\mathbb{E} [ \boldsymbol{N} ] = \infty$ provided that $\mathscr{A}$ is empty. \end{theorem}

See Appendix \ref{Convexlowerapp} for a proof.

\subsection{Upper Bounds on Average Stopping Time}

 As mentioned before, a general problem is to
bound the stopping time
 \begin{equation}
 \label{defST}
\boldsymbol{N} = \inf \left \{ n \in  \mathscr{N}: (n, S_n)  \notin \mathscr{R} \right \},
 \end{equation}
where $\mathscr{R}$ is called the continuity region.  For the boundedness of $\mathbb{E} [ \boldsymbol{N} ]$, consider the following assumptions:

\begin{description}

\item [(I)] There exist numbers $\lambda > 0$ and $K$ such that
\begin{equation} \label{group88} N_{\ell + 1} \leq \lambda N_\ell + K \end{equation}  for all $\ell \geq 0$.

\item [(II)]  Either $\limsup_{\ell \to \infty} \; ( N_{\ell + 1} - N_\ell ) < \infty$
or $\liminf_{\ell \to \infty} \frac{N_{\ell + 1}}{N_\ell} > 1$.

\item [(III)]  $\overline{\mathscr{R}}$ is a convex set containing $(0, \boldsymbol{0}_d)$.

\item [(IV)] $\{  (N_0, S_{N_0} )
\in \overline{\mathscr{R}} \}$ is a sure event.

\item [(V)] There exists a unique positive number $m$ such that $(m, m \mu) \in \partial \mathscr{R}$.

\item [(VI)] Each element of $\mathbb{E} [ | X |^3 ]$ is finite.

\end{description}

It should be noted sample sizes used in group sequential methods \cite{Bartroff, Jennsion, Mukhopadhyay, Proschan, Whitehead} typically satisfy
the inequality (\ref{group88}) for some numbers $\lambda
> 0$ and $K$.  For the stopping time defined by (\ref{defST}), we have established the following result.

 \begin{theorem} \label{BoundGen}

  If assumptions (I) --(VI) are fulfilled, then $\mathbb{E} [\boldsymbol{N} ] < \infty$.

\end{theorem}

See Appendix \ref{BoundGenapp} for a proof.

For the purpose of bounding $\mathbb{E}[ \boldsymbol{N} ]$, define \begin{equation} \label{defM}
 \boldsymbol{M} = \sup \{ N_\ell: \ell \in \mathbb{Z}^+ , \; \boldsymbol{N} > N_\ell \}.
\end{equation} If $\mathbb{E} [\boldsymbol{N} ] < \infty$,  then it must be true that $\Pr \{ \boldsymbol{N} < \infty \} = 1$.  Let $\boldsymbol{\ell}$ be the index at the termination of
the sampling process. This implies that $\boldsymbol{\ell}$ is a random variable such that $\boldsymbol{N} = N_{\boldsymbol{\ell}}$ and $\boldsymbol{M} = N_{\boldsymbol{\ell} - 1}$. It
should be noted that $\boldsymbol{M}$ is not a stopping time and thus $\mathbb{E} [ S_{\boldsymbol{M}} ]$ is, in general,  not equal to $\mathbb{E} [ \boldsymbol{M} ] \mu$. In
other words, Wald's first equation \cite{Wald} is not applicable to $S_{\boldsymbol{M}}$, although it holds for $S_{\boldsymbol{N}}$.

Clearly, as a consequence of the definition of $\boldsymbol{M}$ and assumption (I), we have $\boldsymbol{N} \leq \lambda \boldsymbol{M} + K$ and \begin{equation} \label{indirect} \mathbb{E} [
\boldsymbol{N} ] \leq \lambda \mathbb{E} [ \boldsymbol{M} ] + K. \end{equation} In view of (\ref{indirect}), to bound $\mathbb{E} [ \boldsymbol{N} ]$, it suffices to bound $\mathbb{E} [ \boldsymbol{M}
]$.  For this purpose, we have the following general result.

\begin{theorem}

\label{GenConvex}  If assumptions (I) -- (VI) are fulfilled,  then  \begin{equation} \label{upper}
 \mathbb{E} [  \boldsymbol{M} ]
\leq \max_{ (t, s) \in \mathscr{D}  }  t, \end{equation} where $\mathscr{D} = \{ (t,
s) \in \overline{\mathscr{R}} :  \; t \alpha + \zeta \leq s \leq t \beta + \eta
\}$, with {\small \[ \alpha = \mu - \lambda  \mathbb{E} [  (X - \mu)^+], \quad
\beta = \mu + \lambda  \mathbb{E} [  (X - \mu)^-], \quad \zeta = ( N_0 - K ) \;
\mathbb{E} [  (X - \mu)^+], \quad \eta = ( K - N_0 ) \; \mathbb{E} [  (X -
\mu)^-].
\]} \end{theorem}

See Appendix \ref{GenConvexapp} for a proof.  It can be checked that $\mathscr{D}$ is a convex set.  Moreover, $\max_{ (t, s) \in \mathscr{D}  }  t =
- \min_{ (t, s) \in \mathscr{D} } f(t,s)$, where $f(t,s) = - t$ is a convex function of $(t, s)$ contained in the convex set $\mathscr{D}$.
Therefore, the upper bound in (\ref{upper}) can be readily evaluated by convex minimization.  With recent improvements in computing and in
optimization theory, convex minimization is nearly as straightforward as linear programming (see, e.g., \cite{Boyd} for a comprehensive
treatment).  Convex minimization problems can be solved by contemporary methods such as subgradient projection methods \cite{Polyak},
interior-point methods \cite{{Nesterov}}, etc.

In the case that $X$ is a bounded random vector, we have the following result.

\begin{theorem}

\label{Convexsecond}  Suppose that $\Pr \{ \boldsymbol{a} \leq X \leq \boldsymbol{b} \} = 1$, where $\boldsymbol{a}, \boldsymbol{b} \in \mathbb{R}^d$, and that assumptions (I) -- (V)
are fulfilled.  Define \[ \boldsymbol{v} = \frac{ (\mu - \boldsymbol{a}) (\boldsymbol{b} - \mu)  }{ \boldsymbol{b} - \boldsymbol{a} }, \quad \alpha = \mu - \lambda \boldsymbol{v}, \quad \beta = \mu + \lambda
\boldsymbol{v}, \quad \zeta = ( N_0 - K ) \; \boldsymbol{v}, \qquad \eta = - \zeta
\]
and \[ \alpha^\prime =  \boldsymbol{b} + \lambda (\mu - \boldsymbol{b}), \qquad \beta^\prime = \boldsymbol{a} + \lambda (\mu - \boldsymbol{a}), \qquad \zeta^\prime = K ( \mu - \boldsymbol{b} ), \qquad
\eta^\prime = K ( \mu - \boldsymbol{a} ). \] Then, \begin{equation} \label{upperbound}
 \mathbb{E} [  \boldsymbol{M} ]
\leq \max_{ (t, s) \in \mathscr{D}  }  t, \end{equation} where $\mathscr{D} = \{ (t,
s) \in \overline{\mathscr{R}} :  \; t \alpha + \zeta \leq s \leq t \beta + \eta,
\; t \alpha^\prime + \zeta^\prime \leq s \leq t \beta^\prime +
\eta^\prime \}$.  \end{theorem}

\bigskip

See Appendix \ref{Convexsecondapp} for a proof.

In many situations, a stopping time is defined in terms of sample mean.  Consider stopping time \begin{equation} \label{STf}
 \boldsymbol{N} = \inf \{ n \in \mathscr{N}: n \geq
g (\overline{X}_n) \}. \end{equation} For such stopping time, we have the following result.

\begin{theorem}

\label{ConvexThird} Assume that $g$ is a concave function on a convex set $D$ in $\mathbb{R}^d$ such that $\mu$ is an interior point of $D$ and that
the range of $\overline{X}_n$ is contained by $D$ for any $n \in \{N_0, N_1, \cdots \}$.  Assume that $K$ and $N_0$ are positive integers such that $\{
N_0 < g(\overline{X}_{N_0} ) \}$ is a sure event and that $N_{\ell + 1} - N_\ell \leq K \leq N_0$ for $\ell = 0, 1, 2, \cdots$.  Assume that each
element of $\mathbb{E} [ | X |^3 ]$ is finite. Then,
\[
\mathbb{E} [ \boldsymbol{N} ] \leq K + \max_{ \theta \in \mathscr{D}  } g (\theta),
\]
where $\mathscr{D} = \{  \theta \in D: \alpha \leq \theta \leq \beta \}$ with $\alpha = \mu - \mathbb{E}[ (X- \mu)^+ ]$ and $\beta =  \mu + \mathbb{E}[ (X - \mu)^- ]$.

\end{theorem}

See Appendix \ref{ConvexThirdapp} for a proof.

If $\mathscr{N} = \mathbb{N}$, the stopping time defined by (\ref{STf}) becomes $\boldsymbol{N} = \inf \{ n \in \mathbb{N}: n \geq  g (\overline{X}_n) \}$. For such
stopping time, we have the following result.

\begin{theorem}

\label{Convexfour} Assume that $g$ is a non-negative concave function on a convex set $D$ in $\mathbb{R}^d$ such that $\mu$ is an interior point of $D$
and that the range of $\overline{X}_n$ is contained by $D$ for any $n \in \mathbb{N}$.  Assume that each element of $\mathbb{E} [ | X |^3 ]$ is finite. Then,
\[
\mathbb{E} [ \boldsymbol{N} ] \leq 2 + \max_{ \theta \in \mathscr{D}  } g (\theta),
\]
where $\mathscr{D} = \{  \theta \in D: \alpha \leq \theta \leq \beta \}$ with $\alpha = \mu - \mathbb{E}[ (X- \mu)^+ ]$ and $\beta =  \mu + \mathbb{E}[ (X - \mu)^- ]$.

\end{theorem}

See Appendix \ref{Convexfourapp} for a proof.  In the case that $X$ is a bounded random vector, we have the following result for the stopping
time defined by (\ref{STf}).

\begin{theorem}

\label{try88} Assume that $\Pr \{ \boldsymbol{a} \leq X \leq \boldsymbol{b} \} = 1$, where $\boldsymbol{a}, \boldsymbol{b} \in \mathbb{R}^d$. Assume that $g$ is a concave function on
a convex set $D$ in $\mathbb{R}^d$ such  that $\mu$ is an interior point of $D$ and that the range of $\overline{X}_n$ is contained by $D$ for any $n \in
\{N_0, N_1, \cdots \}$. Assume that $K$ and $N_0$ are positive integers such that  $\{ N_0 < g (\overline{X}_{N_0} ) \}$ is a sure event and that
$N_{\ell + 1} - N_\ell \leq K \leq N_0$ for $\ell = 0, 1, 2, \cdots$. Then,
\[
\mathbb{E} [ \boldsymbol{N} ] \leq K + \max_{ \theta \in \mathscr{D}  } g (\theta),
\]
where $\mathscr{D} = \{  \theta \in D: \mu - \boldsymbol{v}  \leq \theta \leq \mu + \boldsymbol{v} \}$ with $\boldsymbol{v} = \frac{ (\mu - \boldsymbol{a}) (\boldsymbol{b} - \mu)  }{ \boldsymbol{b} -
\boldsymbol{a} }$.  \end{theorem}

See Appendix \ref{try88app} for a proof.

\section{Bounding Average Stopping Time with Supporting Hyperplane}

In this section, we shall establish explicit bounds for average stopping times.  Consider the stopping time $ \boldsymbol{N} $ defined by (\ref{defST}).
In view of (\ref{indirect}), to bound $\mathbb{E} [ \boldsymbol{N} ]$, it suffices to bound $\mathbb{E} [ \boldsymbol{M} ]$.   For this purpose, we propose to use the
concept of supporting hyperplane to derive explicit bounds for $\mathbb{E} [ \boldsymbol{M} ]$. In the sequel, we shall use equation $As + B t = C$, where
$A^\top \in \mathbb{R}^d, \; B \in \mathbb{R}$ and $C \in \mathbb{R}$ are constants,  to represent a hyperplane which consists of points $(t, s)$ with $t
\in \mathbb{R}$ and $s \in \mathbb{R}^d$ satisfying the equation.  To bound $\mathbb{E} [ \boldsymbol{M} ]$ associated with (\ref{defST}), we have the following
result.

\begin{theorem} \label{lineartheorem}

Suppose that assumptions (I) -- (VI) are fulfilled.  Let $A s + B t = C$, where $C > 0$, be the supporting hyperplane of $\mathscr{R}$ passing
through $(m, m \mu) \in \partial \mathscr{R}$. Let $\mathscr{V} = \{ \boldsymbol{q} \in \mathbb{R}^d: \text{each element of $\boldsymbol{q}$ assumes value} \; 0 \; \text{or} \; 1
\}$.
 Define {\small \[
\alpha = \mu - \lambda  \mathbb{E} [  (X - \mu)^+], \quad \beta = \mu + \lambda  \mathbb{E} [  (X - \mu)^-], \quad \zeta = ( N_0 - K ) \; \mathbb{E} [ (X - \mu)^+], \quad \eta
= ( K - N_0 ) \; \mathbb{E} [  (X - \mu)^-].
\]}
Then, \[ \mathbb{E} [ \boldsymbol{M} ] \leq \max \left \{  \frac{  C - A [ \eta + \boldsymbol{q} (\zeta - \eta) ]  }{ B +  A [ \beta + \boldsymbol{q} (\alpha - \beta) ] }: \boldsymbol{q} \in
\mathscr{V} \right \} \]
provided that the minimum of $\{ B +  A [ \beta + \boldsymbol{q} (\alpha - \beta) ]: \boldsymbol{q} \in \mathscr{V} \}$ is positive. \end{theorem}

See Appendix \ref{lineartheoremapp} for a proof.

From the above theorem, it can be seen that the minimum of $\{ B +  A [ \beta + \boldsymbol{q} (\alpha - \beta) ]: \boldsymbol{q} \in \mathscr{V} \}$ is close to the
positive quantity $A \mu + B$ if $\alpha$ and $\beta$ are close to $\mu$.  Actually, this frequently occurs  in practices.

In situations that $X$ is a bounded random vector, we have obtained explicit bounds for $\mathbb{E} [ \boldsymbol{M} ]$
in connection with the stopping time $\boldsymbol{N} $ defined by (\ref{defST}) as follows.

\begin{theorem} \label{lineartheorembounded}

Suppose that $\Pr \{ \boldsymbol{a} \leq X \leq \boldsymbol{b} \} = 1$, where
$\boldsymbol{a}, \boldsymbol{b} \in \mathbb{R}^d$, and that assumptions (I) -- (V) are
fulfilled.  Let $A s + B t = C$, where $C > 0$, be the supporting
hyperplane of $\mathscr{R}$ passing through $(m, m \mu) \in \partial
\mathscr{R}$. Let $\mathscr{V} = \{ \boldsymbol{q} \in \mathbb{R}^d: \text{each element
of $\boldsymbol{q}$ assumes value} \; 0 \; \text{or} \; 1 \}$.  Define \[
\boldsymbol{v} = \frac{ (\mu - \boldsymbol{a}) (\boldsymbol{b} - \mu)  }{ \boldsymbol{b} - \boldsymbol{a} },
\quad \alpha = \mu - \lambda \boldsymbol{v}, \quad \beta = \mu + \lambda \boldsymbol{v}, \quad \zeta =
( N_0 - K ) \; \boldsymbol{v}, \qquad \eta = - \zeta
\]
and \[ \alpha^\prime =  \boldsymbol{b} + \lambda (\mu - \boldsymbol{b}), \qquad \beta^\prime = \boldsymbol{a} + \lambda (\mu - \boldsymbol{a}), \qquad \zeta^\prime = K ( \mu - \boldsymbol{b} ), \qquad
\eta^\prime = K ( \mu - \boldsymbol{a} ). \]   Then,
\[
\mathbb{E} [ \boldsymbol{M} ] \leq \max \left \{  \frac{  C - A [ \eta + \boldsymbol{q} (\zeta - \eta) ]  }{ B +  A [ \beta + \boldsymbol{q} (\alpha - \beta) ] }: \boldsymbol{q} \in
\mathscr{V} \right \}
\]
provided that the minimum of $\{ B +  A [ \beta + \boldsymbol{q} (\alpha - \beta) ]: \boldsymbol{q} \in \mathscr{V} \}$ is positive.  Similarly, \[ \mathbb{E} [ \boldsymbol{M}
] \leq \max \left \{  \frac{  C - A [ \eta^\prime + \boldsymbol{q} (\zeta^\prime - \eta^\prime) ]  }{ B +  A [ \beta^\prime + \boldsymbol{q} (\alpha^\prime -
\beta^\prime) ] }: \boldsymbol{q} \in \mathscr{V} \right \}
\]
provided that the minimum of $\{ B +  A [ \beta^\prime + \boldsymbol{q} (\alpha^\prime - \beta^\prime) ]: \boldsymbol{q} \in \mathscr{V} \}$ is positive.  \end{theorem}

See Appendix \ref{lineartheoremboundedapp} for a proof.

In the sequel, we shall apply the concept of supporting hyperplane
and Lorden's inequality on overshoot to obtain explicit bounds for
average stopping times. Consider stopping time
\[
\boldsymbol{N} = \left \{ n \in \mathbb{N}: \frac{n}{K} \in \mathbb{N}, \;  (n, S_n) \notin \mathscr{R} \right \},
\]
where  $K$ is a positive integer.  For such stopping time, we have the following results.

\begin{theorem}

\label{ChenLorden}

Assume that $\overline{\mathscr{R}}$ is a convex set containing $(0,
\boldsymbol{0}_d)$.  Assume that each element of $\mathbb{E} [ | X |^2 ]$ is
finite.  Assume that there exists a unique positive number $m$ such
that $(m, m \mu) \in \partial \mathscr{R}$. Assume that there exists a
supporting hyperplane $A s + B t = C$, where $C > 0$,  of $\mathscr{R}$
passing through $(m, m \mu)$. Define $Z = B K + A \sum_{i = 1}^K
X_i$. The following assertions hold true.

(I):  \begin{eqnarray*} \mathbb{E}[ \boldsymbol{N} ]  & \leq & m + \frac{1}{ K  } \left ( \frac{m}{C}
\right )^2 \mathbb{E} [ (Z^+)^2 ]\\
& \leq & m + K +  \left ( \frac{m}{C} \right )^2 \mathbb{E} \left [ \left ( A ( X -
\mu) \right )^2 \right ]\\
& \leq & m + K +  \left ( \frac{m}{C} \right )^2  || A ||_2^2 \times \mathbb{E}
\left [ || X - \mu ||_2^2 \right ]. \end{eqnarray*}

(II): If  the  elements of $X$ are mutually independent, then $\mathbb{E}[ \boldsymbol{N} ]  \leq m + K + \left ( \frac{m}{C} \right )^2 A^2 \; \mathbb{E} [ (X - \mu)^2
]$.

(III): If $\Pr \{ \boldsymbol{a} \leq X \leq \boldsymbol{b} \} = 1$, where $\boldsymbol{a}, \boldsymbol{b} \in \mathbb{R}^d$, then \begin{eqnarray*} \mathbb{E} [ \boldsymbol{N} ] \leq m + \frac{K m  (u + v)
}{ C } - \frac{ K m^2  u v}{C^2},   \end{eqnarray*} where $u = \frac{1}{2} \left [  A (\boldsymbol{a} + \boldsymbol{b}) + |A| (\boldsymbol{a} - \boldsymbol{b}) \right ] + B$ and $v = \frac{1}{2} \left [
A (\boldsymbol{a} + \boldsymbol{b}) + |A| (\boldsymbol{b} - \boldsymbol{a}) \right ] + B$.  In particular, $\mathbb{E} [ \boldsymbol{N} ] \leq  m + \frac{ K v^2 }{ v - u }  \left ( \frac{ m }{C} \right
)^2  \left ( \frac{C}{m} - u \right )$ for $u < 0$.

 \end{theorem}

See Appendix \ref{ChenLordenapp} for a proof.

To illustrate the applications of Theorem \ref{ChenLorden}, consider
stopping time \begin{equation} \label{newST8896}
 \boldsymbol{N} = \inf \{ n \in \mathbb{N}: (n, S_n) \notin
\mathscr{R} \}. \end{equation} We wish to apply assertion (I) of Theorem
\ref{ChenLorden} to derive convenient upper bounds for $\mathbb{E} [
\boldsymbol{N} ]$. For this purpose, we define function \begin{equation} \label{defgv}
 g (v)
= \sup \{ t \in \mathbb{R}^+:  (t, t v) \in \overline{\mathscr{R}} \} \end{equation} for $v
\in \mathbb{R}^d$.  Note that $g (v)$ can be $\infty$.   Let $\nabla(v)$
denote the gradient of $\ln(g(v))$ with respect to $v$, that is, \begin{equation}
\label{defnabla}
 \nabla(v) = \frac{\partial \ln(g(v)) } {\partial v} = \frac{ 1 }{g(v)} \frac{\partial g(v)   }{ \partial v }. \end{equation}
 For the stopping time defined by
(\ref{newST8896}), we have the following results.  \begin{theorem}
\label{discrerebound} Assume that $\overline{\mathscr{R}}$ is a convex set
containing $(0, \boldsymbol{0}_d)$. Assume that $\mathbb{E} [||X||_2^2]$ is
finite. Assume that $g(v)$ is differentiable at a neighborhood of $v
= \mu$. Then,
\[
\mathbb{E} [ \boldsymbol{N} ] \leq g (\mu) + 1 + \mathbb{E} \left [  \left <  \nabla
(\mu), \; X - \mu \right >^2 \right ] \leq g (\mu)  + 1 +   \left | \left |
\nabla (\mu) \right | \right |_2^2 \times
 \mathbb{E} \left [ \left | \left | X - \mu \right | \right |_2^2  \right ].
\]
\end{theorem}

See Appendix \ref{discrereboundap} for a proof.  We can apply
Theorem \ref{discrerebound} to derive a simple bound for the
expectation of the first passage time for a random walk with concave
boundary.  More specifically, consider stopping time \begin{equation}
\label{exam8888}
 \boldsymbol{N} = \inf \{ n \in \mathbb{N}: S_n > f (n) \},
\end{equation} where $S_n = \sum_{i = 1}^n X_i$ is the partial sum of i.i.d
scalar random variables $X_1, X_2, \cdots$, which have the same
distribution as $X$ with mean $\mu = \mathbb{E} [ X ]$ and variance
$\sigma^2 = \mathbb{E} [ | X - \mu |^2 ] < \infty$.  Assume that $f(t)$ is a
concave function of $t \in \mathbb{R}^+$ such that $f(0) > 0$. Assume
that there exists a  positive number $m$ such that
 \begin{equation}
 \label{vipequation}
 m \mu = f (m),
 \end{equation}
that is,  $m = g (\mu)$, where the function $g(.)$ is defined by
 (\ref{defgv}) with the continuity region
\[ \mathscr{R} = \{ (t, s): t \in \mathbb{R}^+, \; s \in \mathbb{R}, \; t \leq
f (s) \}. \]
Assume that $f(t)$ is differentiable in a neighborhood
of $t = m$. Then, $g(v)$ must be differentiable in a neighborhood of
$v = \mu$.  Due to the concavity of the boundary function $f(.)$,
the continuity region $\mathscr{R}$ is a convex set.  To apply Theorem
\ref{discrerebound} to bound the
 stopping time in (\ref{exam8888}), we can calculate  $\nabla(\mu)$ by (\ref{defnabla}) as follows.

At a neighborhood of $v = \mu$, we have
\[
g (v) v = f ( g(v) ).
\]
Differentiating both sides of the above equation with the chain rule
yields
\[
g^\prime (v) v + g (v) = f^\prime (g (v) ) g^\prime (v),
\]
where $f^\prime (.)$ and $g^\prime (.)$ denotes the first
derivatives of $f (.)$ and $g(.)$, respectively. Solving the
equation, we obtain
\[
g^\prime (v) = \frac{ g (v)  }{  f^\prime (g (v) ) - v }.
\]
It follows that the first derivative, $g^\prime (\mu)$, of $g (v)$
at $v = \mu$ can be obtained as
\[
g^\prime (\mu) = \frac{ g (\mu)  }{  f^\prime (g (\mu) ) - \mu } =
\frac{m}{ f^\prime (m) - \mu }.
\]
Therefore, the gradient is
\[
\nabla(\mu) = \frac{ g^\prime (\mu)  } {  g (\mu) } = \frac{m}{ [f^\prime
(m) - \mu ] g (\mu) } = \frac{1}{ f^\prime (m) - \mu }.
\]
It follows from Theorem \ref{discrerebound} that \begin{equation} \label{vipformula}
\mathbb{E} [ \boldsymbol{N} ] \leq m + 1 + \frac{ \sigma^2  }{  | f^\prime (m) - \mu
|^2 }. \end{equation}   To use formula (\ref{vipformula}), we need to obtain
$m$ from equation (\ref{vipequation}). In many cases, it is possible
to derive an explicit expression of $m$ from equation
(\ref{vipequation}).  Even if $m$ cannot be obtained analytically,
it can still be readily computed by numerical methods such as the
bisection search method.   Due to the concavity of $f(.)$ and the
existence of $m$ satisfying (\ref{vipequation}), it must be true
that $t \mu > f(t)$ for large enough $t
> 0$. For example, we can find such value of $t$ as $2^k$ for some
integer $k
> 0$. Then, the number $m$ can be obtained by a bisection search
from interval $(0, 2^k)$.

\section{Bounding Average Stopping Time with Concentration Inequalities}

In this section, we shall propose a method for bounding average stopping times by virtue of concentration inequalities. Consider the stopping
time defined by (\ref{defST}). Define \[ \tau = \min \{ \ell \in \mathbb{N}: N_\ell > m \}, \]
 where $m$ is the unique positive number such that
$(m, m \mu) \in \partial \mathscr{R}$ as defined in assumption (V) of Section 4.2. Define \[ \mathscr{G}_n = \{ z \in \mathbb{R}^d: (n, n z) \in \overline{\mathscr{R}}
\} \]
 and \[
 \rho (n) = \inf_{ z \in \mathscr{G}_n }  || z - \mu ||_2
 \]
  for $n \in \mathbb{N}$. Let $\mathscr{L}$ denote the support of the random index
$\boldsymbol{\ell}$ such that $N_{\boldsymbol{\ell}} = \boldsymbol{N}$. For $\ell = 1, 2, \cdots$ and $k = 1, \cdots, d$, let $\overline{X}_{N_\ell}^k$ denote the $k$-th element
of $\overline{X}_{N_\ell}$. For $k = 1, \cdots, d$, let $\mu_k$ denote the $k$-th element of $\mu$.  We have the following results.

\begin{theorem} \label{BoundConcentrationa88} Suppose that assumptions (I)--(VI) are fulfilled.  Then, \begin{equation} \label{inelinear88a} \mathbb{E} [ \boldsymbol{N} ] \leq N_\tau +
\sum_{\ell \geq \tau \atop{ \ell + 1 \in \mathscr{L} } }  (N_{\ell + 1} - N_\ell )
 \sum_{k = 1}^d \Pr \left \{  \left | \overline{X}_{N_\ell}^k -  \mu_k \right | \geq \frac{ \rho (N_\ell) }{ \sqrt{d} } \right \}.
\end{equation} Moreover, if $X$ is a scalar random variable, then, \begin{equation} \label{citeA}
 \mathbb{E} [ \boldsymbol{N} ] \leq N_\tau + \sum_{\ell \geq \tau \atop{ \ell + 1 \in
\mathscr{L} } }  (N_{\ell + 1} - N_\ell )
 \Pr \left \{  \overline{X}_{N_\ell} \geq \mu + \rho (N_\ell) \right \}
\end{equation} provided that $\mu$ is less than the infimum of $\mathscr{G}_{N_\tau}$;  and similarly, \begin{equation} \label{citeB}
 \mathbb{E} [ \boldsymbol{N} ] \leq N_\tau +
\sum_{\ell \geq \tau \atop{ \ell + 1 \in \mathscr{L} } }  (N_{\ell + 1} - N_\ell )
 \Pr \left \{ \overline{X}_{N_\ell} \leq \mu - \rho (N_\ell) \right \}
\end{equation} provided that $\mu$ is greater than the supremum of $\mathscr{G}_{N_\tau}$. \end{theorem}

See Appendix \ref{BoundConcentrationa88app} for a proof.

 It should be noted that for all $n \in \mathbb{N}$, $\mathscr{G}_n$ is convex due to the convexity of $\overline{\mathscr{R}}$. Consequently,  $\rho (N_\ell)$
 can be readily obtained by convex minimization.  Making use of the concept of supporting hyperplane, we have the following results.

\begin{theorem}  \label{BoundConcentrationb88} Suppose that assumptions (I)--(VI) are fulfilled.  Assume that there exists a supporting hyperplane $A s + B t
= C$ of $\mathscr{R}$ passing through $(m, m \mu) \in \partial \mathscr{R}$.  Then, \begin{equation} \label{inelinear8899}
 \mathbb{E} [ \boldsymbol{N} ] \leq N_\tau + \sum_{\ell \geq
\tau \atop{ \ell + 1 \in \mathscr{L} } }  (N_{\ell + 1} - N_\ell )
 \sum_{k = 1}^d \Pr \left \{  \left | \overline{X}_{N_\ell}^k  -  \mu_k \right | \geq \frac{ \gamma_\ell }{ \sqrt{d} } \right \},
\end{equation} where $\gamma_\ell = \left ( 1 - \frac{m}{N_\ell}  \right ) \frac{|A \mu + B|} { \sqrt{ A A^\top}  }$.  Moreover, if $X$ is a scalar random variable, then,
\begin{equation} \label{citeC}
 \mathbb{E} [ \boldsymbol{N} ] \leq N_\tau + \sum_{\ell \geq \tau \atop{ \ell + 1 \in \mathscr{L} } }  (N_{\ell + 1} - N_\ell )
 \Pr \left \{  \overline{X}_{N_\ell}   \geq \mu + \left ( 1 - \frac{m}{N_\ell} \right ) \left | \mu + \frac{B}{A} \right | \right \}
\end{equation} provided that $\mu$ is less than the infimum of $\mathscr{G}_{N_\tau}$;  and similarly, \begin{equation} \label{citeD}
 \mathbb{E} [ \boldsymbol{N} ] \leq N_\tau + \sum_{\ell \geq \tau \atop{ \ell + 1 \in \mathscr{L} } }  (N_{\ell + 1} - N_\ell )
 \Pr \left \{ \overline{X}_{N_\ell} \leq  \mu - \left ( 1 - \frac{m}{N_\ell} \right ) \left | \mu + \frac{B}{A} \right | \right \}
\end{equation} provided that $\mu$ is greater than the supremum of $\mathscr{G}_{N_\tau}$. \end{theorem}

See Appendix \ref{BoundConcentrationb88app} for a proof.

It should be noted that the probabilistic terms in Theorems \ref{BoundConcentrationa88} and \ref{BoundConcentrationb88} can be bounded by
concentration inequalities such as Chernoff bounds and Hoeffding inequalities \cite{Chernoff, Hoeffding}.

\section{Bounds for Average Stopping Times of Brownian Motion}

In the last few sections, our techniques for bounding stopping times are devoted to discrete-time stochastic processes. Actually, the principle
of such techniques can be extended to continuous-time stochastic processes. To demonstrate this idea, we shall focus on the problem of bounding
stopping times pertaining to Brownian motion \cite{KARATZAS, Morters}.

 Let $W_t \in \mathbb{R}^d$ be a $d$-dimensional Brownian motion with mean drift vector
 $\mu$ such that $W_0 = \boldsymbol{0}_d$ and $\mathbb{E} [ W_t ] = t \mu$ for $t \geq 0$.
 Define
 \[
\nu = \frac{1}{t} \mathbb{E} [ | W_t - t \mu |^2 ], \qquad \overline{W}_t  = \frac{ W_t }{t}, \qquad V_t = ( W_t - t \mu)^2 , \qquad \overline{V}_t = \frac{V_t }{t}
 \]
for $t > 0$.   Making use of Theorem \ref{genJen}, we have obtained the following results.

\begin{theorem}

\label{use66889933}

 Assume that $T$ is random variable such that $\mathbb{E} [ T ] < \infty$ and that for
any possible value $t$ of $T$, the event $\{ T = t \}$ depends only on $\{W_\tau: 0 \leq \tau \leq t \}$.  The following assertions hold.

(I): If $g$ is a convex function on $\mathbb{R}^d$, then
\[
\mathbb{ E } \left [ T g ( \overline{W}_T ) \right ] \geq  \mathbb{ E } [ T ] g (\mu).
\]

(II): If $g$ is a convex function of vectors with non-negative elements, then \[ \mathbb{ E } \left [ T g \left (  \overline{V}_T \right  ) \right ] \geq  \mathbb{ E }
[ T ] g (\nu).
\]

\end{theorem}

The proof of Theorem \ref{use66889933} is similar to that of Theorem  \ref{genWald}, which is given in Appendix \ref{genWaldapp}.

Making use of the convexity of  the $L^p$-norm and Theorem \ref{use66889933},  we have the following results.

\begin{theorem}

Assume that $T$ is random variable such that $\mathbb{E} [ T ] < \infty$ and that for any possible value $t$ of $T$, the event $\{ T = t \}$ depends
only on $\{W_\tau: 0 \leq \tau \leq t \}$.
Then, \begin{eqnarray*} &  & \mathbb{ E } \left [ || W_T ||_p \right ] \geq  \mathbb{ E } [ T ] \; || \mu ||_p, \\
&  & \mathbb{ E } \left [ || V_T ||_p \right ] \geq  \mathbb{ E } [ T ] \; || \nu ||_p  \end{eqnarray*}
 for all $p \geq 1$.

 \end{theorem}

Now,  consider stopping time $T = \inf \{ t > 0: (t, W_t) \notin \mathscr{R} \}$, where $\mathscr{R}$ is called the continuity region. We have the
following result.

\begin{theorem} \label{Brown1}

 Assume that $\overline{\mathscr{R}}$ is a convex set containing $(0, \boldsymbol{0}_d)$ and that there exists a unique positive number $\tau$
such that $(\tau, \tau \mu) \in \partial \mathscr{R}$. Then, $\mathbb{E} [ T ] \leq \tau$.

\end{theorem}

See Appendix \ref{Brown1app} for a proof.

Next,  consider stopping time $T = \inf \{ t > 0: (t, W_t) \in \mathscr{R} \}$,
where $\mathscr{R}$ is called the stopping region.  We have obtained  the following results.

\begin{theorem}  \label{Brown2}  Suppose that the stopping region $\mathscr{R}$ is a convex set.  Define $\mathscr{A} = \{ t \in \mathbb{R}^+: (t,  t \mu ) \in \mathscr{R}
\}$. The following assertions hold.

(I):  $\mathbb{E} [ T ] \geq \inf \mathscr{A}$ provided that the set $\mathscr{A}$ is nonempty.

(II):  $\mathbb{E} [ T ] \leq \sup \mathscr{A}$ provided that $\mathbb{E} [ T ] < \infty$ and the set $\mathscr{A}$ is nonempty.

(III):  $\mathbb{E} [ T ] = \infty$ provided that the set $\mathscr{A}$ is empty.  \end{theorem}

See Appendix \ref{Brown2app} for a proof.

Consider stopping times defined in terms of $\overline{W}_t$. For stopping time $T = \inf \{ t > 0: t \geq g \left ( \overline{W}_t \right ) \}$, we have the
following result.

\begin{theorem}

\label{Brown3}

Assume that $g$ is a concave function on $\mathbb{R}^d$ with $g(\mu) > 0$. Then, $\mathbb{E} [ T ] \leq g (\mu)$.  \end{theorem}

See Appendix \ref{Brown3app} for a proof.

  For stopping time $T = \inf \left \{ t > 0: t \geq \frac{1}{g \left ( \overline{W}_t \right )}, \; g \left ( \overline{W}_t \right ) > 0 \right \}$,
   we have derived the following result.

\begin{theorem} \label{Brown4} Assume that $g$ is a concave function on $\mathbb{R}^d$ with $g(\mu) > 0$. Then, $\mathbb{E} [ T ] \geq \frac{1}{g (\mu)}$. \end{theorem}

See Appendix \ref{Brown4app} for a proof.

\section{Conclusion}

In this paper, we have established a geometric approach for bounding average stopping times.   The central idea of our approach is to explore
the geometric convexity of the continuity or stopping regions.  Our approach are effective for a wide variety of stopping times which involve
random vectors, nonlinear boundary, constraint of sample number, etc.   Tight bounds are obtained for stopping times in a general setting, which
are explicit or readily computable.  A probabilistic characterization is established for convex sets. Extensions are developed for classical
results such as Jensen's inequality, Wald's equations and Lorden's inequality.

Probability (math.PR); Statistics Theory (math.ST)