Concentration of the Langevin Algorithm's Stationary Distribution

Abstract: A canonical algorithm for log-concave sampling is the Langevin Algorithm, aka the Langevin Diffusion run with some discretization stepsize $\eta > 0$. This discretization leads the Langevin Algorithm to have a stationary distribution $\pi_{\eta}$ which differs from the stationary distribution $\pi$ of the Langevin Diffusion, and it is an important challenge to understand whether the well-known properties of $\pi$ extend to $\pi_{\eta}$. In particular, while concentration properties such as isoperimetry and rapidly decaying tails are classically known for $\pi$, the analogous properties for $\pi_{\eta}$ are open questions with direct algorithmic implications. This note provides a first step in this direction by establishing concentration results for $\pi_{\eta}$ that mirror classical results for $\pi$. Specifically, we show that for any nontrivial stepsize $\eta > 0$, $\pi_{\eta}$ is sub-exponential (respectively, sub-Gaussian) when the potential is convex (respectively, strongly convex). Moreover, the concentration bounds we show are essentially tight.Key to our analysis is the use of a rotation-invariant moment generating function (aka Bessel function) to study the stationary dynamics of the Langevin Algorithm. This technique may be of independent interest because it enables directly analyzing the discrete-time stationary distribution $\pi_{\eta}$ without going through the continuous-time stationary distribution $\pi$ as an intermediary.

\section{Introduction}\label{sec:intro}

Sampling from a log-concave distribution $\pi \propto e^{-f}$ over $\mathbb{R}^d$ is a foundational problem with applications throughout statistics, engineering, and the sciences.
A canonical and well-studied approach is the Langevin Algorithm, which is the discrete-time Markov process
\begin{align}
	X_{t+1} = X_t - \eta \nabla f(X_t) + Z_t\,,
	\label{eq:LA}
\end{align}
where $\eta > 0$ is a stepsize parameter, and $Z_t \sim \mathcal{N}(0, 2\eta I_d)$ are independent Gaussians. Briefly, the intuition behind the Langevin Algorithm is that it discretizes the Langevin Diffusion, which is a continuous-time Markov process with stationary distribution $\pi$. More precisely, the Langevin Diffusion is the stochastic differential equation
\begin{align}
	dX_t = - \nabla f(X_t) + \sqrt{2}d B_t\,,
	\label{eq:LD}
\end{align}
where $B_t$ is a standard Brownian motion on $\mathbb{R}^d$, and the Langevin Algorithm is identical except that it updates the gradient $\nabla f(X_t)$ every $\eta$ units of time rather than continuously. We refer to the textbooks~\citep{Chewi22,andrieu2003introduction,robert1999monte,jerrum1996markov,liu2001monte} for a detailed historical account of the Langevin Algorithm and the truly extensive literature surrounding it, which spans multiple decades and communities.

Although the discretization enables simulating the Langevin Diffusion algorithmically, it introduces an asymptotic bias. Specifically, for any discretization stepsize $\eta > 0$, the stationary distribution $\pi_{\eta}$ of the Langevin Algorithm differs from the stationary distribution $\pi$ of the Langevin Diffusion. An important question with direct algorithmic implications is:
\[
\text{\emph{How similar are }} \pi_{\eta} \text{\emph{ and }} \pi\text{?}
\]

\paragraph*{Concentration?}
The motivation of this note is that this question is wide open for concentration properties.
Indeed, while concentration properties for $\pi$ have been classically known for several decades, the analogous properties for $\pi_{\eta}$ have remained unknown---and recent work has even started to prove algorithmic implications of these conjectural properties assuming that they hold.

\par Perhaps the most notable example of this is isoperimetry: while classical results imply that $\pi$ satisfies Poincar\'e/log-Sobolev isoperimetric inequalities when the potential $f$ is convex/strongly-convex~\citep{bakry1985diffusions,bobkov2003spectral,kannan1995isoperimetric}, the analogous properties for $\pi_{\eta}$ are unknown. The influential paper~\citep{VempalaW19} stated this as an open question and proved rapid mixing of the Langevin Algorithm in R\'enyi divergence under this conjectural assumption of isoperimetry of $\pi_{\eta}$. Another example is rapidly decaying tails: while classical results imply that  $\pi$ has sub-exponential/sub-Gaussian tails when the potential $f$ is convex/strongly-convex~\citep{ledoux1999concentration,MilmanBook}, the analogous properties for $\pi_{\eta}$ were previously unknown. Such tail decay results can be exploited to extend fast-mixing results for the Langevin Algorithm from constrained settings to unconstrained settings~\citep{AltTal22mix}.

\paragraph*{Contributions.} The purpose of this note is to provide a first step in this direction. Specifically, in \S\ref{sec:sc} we show that the stationary distribution $\pi_{\eta}$ of the Langevin Algorithm is sub-Gaussian when the potential $f$ is strongly convex; and in \S\ref{sec:convex} we show that $\pi_{\eta}$ is sub-exponential when $f$ is convex. These results hold for any nontrivial\footnote{
    Our results apply for $\eta =O(1/M)$. This is the relevant regime in both theory and practice since the Langevin Algorithm is transient if $\eta > 2/M$. In fact, $\eta$ is typically polynomially smaller than $O(1/M)$ to ensure small bias~\citep{Chewi22}.
} discretization stepsize $\eta > 0$ and mirror the aforementioned classical results for $\pi$. Moreover, the concentration bounds we show are essentially tight.

\paragraph*{Techniques.} A central obstacle for proving this result is that this requires establishing properties of $\pi_{\eta}$ for any discretization stepsize $\eta > 0$, not just $\eta$ arbitrarily small. Since the difference between $\pi$ and $\pi_{\eta}$ grows with $\eta$, we approach this problem by directly analyzing $\pi_{\eta}$---rather than analyzing properties of $\pi$ and then trying to conclude properties of $\pi_{\eta}$ via approximation bounds between $\pi$ and $\pi_{\eta}$. However, directly analyzing $\pi_{\eta}$ comes with the difficulty that many standard analysis techniques for the continuous-time Langevin Diffusion become much more complicated after discretization. To overcome this, our analysis makes use of a rotation-invariant moment generating function (aka Bessel function) to directly study the dynamics of the discrete-time Langevin Algorithm at stationarity. Briefly, the key point is that this Lyapunov function readily tracks the effect of gradient descent updates and Gaussian noise convolutions---the two components of the update equation~\eqref{eq:LA} for the Langevin Algorithm---thereby enabling a simple and direct analysis of this discrete-time process. This technique may be of independent interest and is developed in \S\ref{sec:mgf}.

\paragraph*{Independent work.} After completing an initial draft of this manuscript, we found out that a recent revision of~\citep{VempalaW19} (arXiv v4) has a new result (Theorem 8) which shows that in the strongly convex setting, $\pi$ satisfies LSI and thus is sub-Gaussian with concentration parameters that exactly match our Theorem~\ref{thm:sc}.
The proof techniques are completely different, and the final results are incomparable in their level of generality. On one hand, their result subsumes Theorem~\ref{thm:sc} in that it shows LSI, which implies sub-Gaussianity. On the other hand, our techniques are more general in that they readily extend to the (non-strongly) convex setting, while the techniques of~\citep{VempalaW19} only apply to the strongly convex setting (it is stated as an open question in their paper whether one can obtain the analogous results for this more general setting). We are grateful to Andre Wibisono for making us aware of this recent revision and for discussing the differences with us.

\section{Preliminaries}\label{lem:prelim}

\paragraph*{Notation.}  We write $\mathcal{S}^{d-1}$ to denote the unit sphere in $\mathbb{R}^d$, and we abuse notation slightly by writing $v \sim \mathcal{S}^{d-1}$ to indicate that $v$ is a random vector drawn uniformly from $\mathcal{S}^{d-1}$. Throughout, $\|\cdot\|$ denotes the Euclidean norm. Recall that a differentiable function $f : \mathbb{R}^d \to \mathbb{R}$ is said to be $M$-smooth if $\nabla f$ is $M$-Lipschitz; and is said to be $m$-strongly convex if $f(y) \geq f(x) + \langle \nabla f(x), y-x \rangle + \frac{m}{2}\|x-y\|^2$ for all $x,y$. All logarithms are natural. All other notation is introduced in the main text.

\paragraph*{Integrability assumption.} Throughout, we assume that the convex potential $f : \mathbb{R}^d \to \mathbb{R}$ satisfies $\int e^{-f(x)} dx < \infty$ since this suffices to ensure that the stationary distribution of the Langevin Diffusion exists and is equal to $\pi \propto e^{-f}$~\citep{roberts1996exponential,bhattacharya1978criteria}. This assumption is automatically satisfied in the strongly convex setting, and is a mild assumption in the convex setting (e.g., it holds if there is a minimizer $x^*$ of $f$ and a large enough radius $R$ such that $f(x) > f(x^*)$ at all points $x$ satisfying $\|x-x^*\| = R$.)
\section{Sub-Gaussian concentration for strongly convex potentials}\label{sec:sc}

In this section we show that if the potential $f$ is strongly convex and smooth, then the stationary distribution $\pi_{\eta}$ of the Langevin Algorithm has sub-Gaussian tails. 	This parallels the classically known fact that in this strongly convex setting, the \emph{unbiased} stationary distribution $\pi \propto e^{-f}$ of the continuous Langevin Diffusion is sub-Gaussian~\citep{ledoux1999concentration}.

\begin{theorem}[Sub-Gaussianity of $\pi_{\eta}$ for strongly convex potentials]\label{thm:sc}
	Suppose $f : \mathbb{R}^d \to \mathbb{R}$ is $m$-strongly convex and $M$-smooth for some $0 < m \leq M < \infty$. Consider running the Langevin Algorithm with stepsize $\eta < 2/M$.  Then its stationary distribution $\pi_{\eta}$ satisfies
	\[
		\mathbb{P}_{X \sim \pi_{\eta}} \left[ \|X\|
		\geq
		4\sqrt{\frac{\eta}{1-c}} \left(\sqrt{2d} + \sqrt{\log1/\delta}\right)
		\right] \leq \delta\,, \quad \forall \delta \in (0,1)\,,
	\]
	where $x^*$ denotes the unique minimizer of $f$, and $c := \max_{\rho \in \{m,M\}} |1 - \eta \rho|$.
\end{theorem}

	The operational interpretation of $c$ is that it is the contraction coefficient corresponding to a gradient descent step (see Lemma~\ref{lem:gd-contractive} below). Note that $c < 1$ because $\eta < 2/M$. In the typical setting of $\eta \leq 1/M$, it simplifies to $c = 1 - \eta m$, whereby Theorem~\ref{thm:sc} simplifies to
	\[
	\mathbb{P}_{X \sim \pi_{\eta}} \left[ \|X\|
	\geq
	\frac{4}{\sqrt{m}} \left( \sqrt{2d} + \sqrt{\log1/\delta} \right)
	\right] \leq \delta\,, \quad \forall \delta \in (0,1)\,.
	\]

The rest of the section is organized as follows. In \S\ref{ssec:sc-proof} we prove Theorem~\ref{thm:sc} by showing that the rotationally symmetrized version of $\pi$ is sub-Gaussian with variance proxy $2\eta/(1-c)$, and in \S\ref{ssec:sc-tight} we show that this bound is tight in all parameters (up to a constant factor of at most $2$).

\subsection{Proof}\label{ssec:sc-proof}

The proof makes use of the following helper lemma. This fact is well-known in convex optimization because it immediately implies a tight bound on the convergence rate of gradient descent for strongly-convex, smooth objectives. A short proof can be found in, e.g., Appendix A.1 of~\citep{AltTal22mix}.

\begin{lemma}[Contractivity of gradient descent step]\label{lem:gd-contractive}
		Suppose $f$ is $m$-strongly convex and $M$-smooth over $\mathbb{R}^d$ for some $0 \leq m \leq M < \infty$, and consider any stepsize $\eta \leq 2/M$. Then
		\[
				\|x - \eta \nabla f(x) - x^*\| \leq c \|x - x^*\|\,, \quad \forall x \in \mathbb{R}^d\,,
		\]
		where $x^*$ is any minimizer of $f$, and $c := \max_{\rho \in \{m,M\}} |1 - \eta \rho| \leq 1$.
\end{lemma}

\begin{proof}[Proof of Theorem~\ref{thm:sc}]
	To avoid cumbersome notation, let us assume that $f$ is centered to have minimizer $x^* = 0$ (this is without loss of generality after a translation). We bound the change in the Lyapunov function $\Phi_{d,\lambda}(\cdot)$ from running an iteration of the Langevin Algorithm from some point $x \in \mathbb{R}^d$ to
	\[
		x' = x - \eta \nabla f(x) + Z\,,
	\]
	where $Z \sim \mathcal{N}(0,2\eta I_d)$. To analyze this, we disassociate the two parts of this update: the gradient descent step and the noise convolution.
	\par First, we analyze the gradient descent step:
		\begin{align*}
		\Phi_{d,\lambda}(x - \eta \nabla f(x))
		= \phi_{d}(\lambda \|x - \eta \nabla f(x)\|)
		\leq \phi_{d}(c \lambda \|x\|)
		\leq \big( \phi_{d}(\lambda \|x\|) \big)^c
		=  \big( \Phi_{d,\lambda}(x ) \big)^c\,.
	\end{align*}
	Above, the first and last steps are by definition of $\phi$ in terms of $\Phi$ (see~\eqref{eq:Phi-phi}); the second step is by contractivity of a gradient descent iteration (\cref{lem:gd-contractive}) and monotonicity of $\phi_d$ (item (i) of Lemma~\ref{lem:phi-properties}); and the third step is by an application of Jensen's inequality $\mathbb{E}[Y^c] \leq \mathbb{E}[Y]^c$ where $Y :=e^{\lambda \langle v, x \rangle}$, which applies since the function $y \mapsto y^c$ is concave for $c \leq 1$.

	\par Second, we use Lemma~\ref{lem:convolve} to bound the change in the Lyapunov function from the Gaussian noise convolution:
	\begin{align*}
			\Phi_{d,\lambda}(x') = \Phi_{d,\lambda}(x - \eta \nabla f(x) + Z) = e^{\eta \lambda^2} \Phi_{d,\lambda}(x - \eta \nabla f(x))\,.
	\end{align*}

	\par By combining the above two displays, we conclude that
	\[
	 	\Phi_{d,\lambda}(x') \leq e^{\eta \lambda^2}\big( \Phi_{d,\lambda}(x ) \big)^c\,.
	\]
	Now take expectations on both sides, drawing $X \sim \pi_{\eta}$. Note that by stationarity, $X' \sim \pi_{\eta}$. Thus
	\[
		\mathbb{E}_{X \sim \pi_{\eta}}\big[ \Phi_{d,\lambda}(X) \big] \leq e^{\eta \lambda^2} \mathbb{E}_{X \sim \pi_{\eta}}\big[ \Phi_{d,\lambda}(X) \big]^c\,,
	\]
	where above we have again used Jensen's inequality on the concave function $y \mapsto y^c$. Rearranging and simplifying yields
	\[
		\mathbb{E}_{X \sim \pi_{\eta}}\big[ \Phi_{d,\lambda}(X) \big] \leq e^{\eta \lambda^2/(1-c)} \,.
	\]
	By definition of the Lyapunov function $\Phi_{d,\lambda}$, this implies that
	\[
		\mathbb{E}_{X \sim \pi_{\eta}} \mathbb{E}_{R \sim \mathcal{R}} \big[ e^{\lambda \langle u, RX \rangle} \big] \leq e^{\eta \lambda^2/(1-c)}\,, \qquad \forall u \in \mathcal{S}^{d-1}\,,
	\]
	where $\mathcal{R}$ denotes the Haar measure over rotations of $\mathbb{R}^d$. Define $\tilde{\pi}_{\eta}$ to be the law of $RX$ (in words, $\tilde{\pi}_{\eta}$ is a rotationally symmetrized version of $\pi_{\eta}$). Then by definition of sub-Gaussianity, the above display establishes that $\tilde{\pi}_{\eta}$ is sub-Gaussian with variance proxy $2\eta/(1-c)$. By a standard concentration inequality on the norm of a sub-Gaussian vector (see e.g.,~\citep[Theorem 1.19]{rigollet2015high}),
	\[
			\mathbb{P}_{X \sim \tilde{\pi}_{\eta}} \left[ \|X\|
		\geq
		4\sqrt{\frac{\eta}{1-c}} \left(\sqrt{2d} + \sqrt{\log1/\delta}\right)
		\right] \leq \delta\,, \quad \forall \delta \in (0,1)\,.
	\]
	Since the Euclidean norm is invariant under rotations, this concentration inequality remains true if $\tilde{\pi}_{\eta}$ is replaced by $\pi_{\eta}$ (cf., the discussion in \S\ref{ssec:mgf}). This concludes the proof.
\end{proof}

\subsection{Tightness}\label{ssec:sc-tight}

Here we observe that, modulo a constant factor of at most $2$, the sub-Gaussianity we proved is tight in all parameters. It is simplest to explain this tightness in terms of the sub-Gaussian parameters since the final concentration inequality in Theorem~\ref{thm:sc} was proved as a direct consequence of this. To this end, recall that in the proof of Theorem~\ref{thm:sc} above, we proved that the rotationally symmetric version $\tilde{\pi}_{\eta}$ of $\pi_{\eta}$ is sub-Gaussian with variance proxy $2\eta/(1-c)$. Following is a simple construction which matches this upper bound. This calculation is similar to~\citep[Example 4]{VempalaW19}.

\begin{example}[Tightness of sub-Gaussanity parameters]\label{ex:sc-tight}
	Consider running the Langevin Algorithm on the univariate quadratic potential $f(x) = \frac{\rho}{2}x^2$, where the curvature $\rho$ is chosen so as to maximize the contraction coefficient $c := \max_{\rho \in \{m,M\}} |1 - \eta \rho|$. Then, for any stepsize $\eta > 0$, the Langevin Algorithm has stationary distribution
	\[
			\pi_{\eta} = \mathcal{N}\left( 0, 2\eta \frac{1}{1-c^2} \right)
	\]
	when it is initialized to $X_0 = 0$. (This fact is proved in~\citep[\S4.2]{AltTal22mix} and follows by simply unraveling the definition of a Langevin Algorithm update, composing $t$ times, and taking $t \to \infty$.)
	Note that $\tilde{\pi}_{\eta} = \pi_{\eta}$ in this example since $\pi_{\eta}$ is already rotationally symmetric.
	Moreover, since $1-c^2 = (1-c)(1+c)$ and since also $(1+c) \in (1,2)$, this construction matches our aforementioned upper bound on the sub-Gaussian parameter of $\tilde{\pi}_{\eta}$ up to a constant factor of at most $2$.
\end{example}

\section{Sub-exponential concentration for convex potentials}\label{sec:convex}

In this section we show that if the potential $f$ is convex and smooth, then the stationary distribution $\pi_{\eta}$ of the Langevin Algorithm has sub-exponential tails. This parallels the classically known fact that in this convex setting, the \emph{unbiased} stationary distribution $\pi \propto e^{-f}$ of the continuous Langevin Diffusion is sub-exponential (see Lemma~\ref{lem:pi-convex} below).

\begin{theorem}[Sub-exponentiality of $\pi_{\eta}$ for convex potentials]\label{thm:convex}
    Consider running the Langevin Algorithm on a convex, $M$-smooth potential $f : \mathbb{R}^d \to \mathbb{R}$ with stepsize $\eta \leq 1/M$. Then the stationary distribution $\pi_{\eta}$ of the Langevin Algorithm has sub-exponential concentration; i.e., there exists
    $A,C,R > 0$ such that
    \[
        \mathbb{P}_{X \sim \pi_{\eta}} \left[ \norm{X} \geq R + C \log(A/\delta) \right] \leq \delta\,, \quad \forall \delta \in (0,1)\,.
    \]
\end{theorem}

Note that in this sub-exponential concentration inequality, the parameters $A$, $C$, and $R$ depend on $f$ and $\eta$ (this dependence is made explicit in the proof). A dependence on $f$ is inevitable since even the unbiased distribution $\pi$ has sub-exponential parameters which depend on $f$. As a simple concrete example, consider the potential $f(x) = d(x, B_R)$ where $B_R$ is the Euclidean ball of radius $R$ centered at the origin, and $d(x,B_R)$ is the Euclidean distance to the ball. The distribution $\pi$ then concentrates no better than the uniform distribution on the Euclidean ball of radius $R$, and a randomly drawn point $X$ from $B_R$ has norm $\Theta(R)$ with high probability. See also the tightness discussion in \S\ref{ssec:convex-tight} for a fully worked-out example.

The rest of the section is organized as follows. In \S\ref{ssec:convex-proof} we prove Theorem~\ref{thm:convex}, and in \S\ref{ssec:convex-tight} we discuss tightness of this bound.

\subsection{Proof}\label{ssec:convex-proof}

We begin by recalling the classical fact from functional analysis that every log-concave distribution is sub-exponential. For a proof, see e.g.,~\citep[Lemma 10.6.1]{MilmanBook}.

\begin{lemma}[Log-concavity implies sub-exponentiality]\label{lem:pi-convex}
	If $\pi$ is a log-concave probability distribution, then there exist
	$a,b > 0$ such that $\pi(x) \leq ae^{-b\|x\|}$ for all $x \in \mathbb{R}^d$.
\end{lemma}

To apply this lemma, it is convenient to first assume without loss of generality that $f$ is translated and shifted so that $0$ is a minimizer of $f$ with value $f(0) = 0$. Then $f(x) = \log ( \pi(0) / \pi(x) )$. By Lemma~\ref{lem:pi-convex}, we conclude that
\begin{align}
	f(x) \geq -\alpha + \beta\|x\|\,, \quad \forall x \in \mathbb{R}^d\,,
	\label{eq:convex-superlinear}
\end{align}
for some $\alpha,\beta \in \mathbb{R}$. (Here, $\alpha$ plays the role of $\log a - \log \pi(0)$ and $\beta$ plays the role of $b$, where $a$ and $b$ are the quantities from Lemma~\ref{lem:pi-convex}.)
We exploit this super-linear growth~\eqref{eq:convex-superlinear} of $f$ to show that a gradient descent update on $f$ makes significant progress towards the minimizer $x^* =0$ when the current iterate has sufficiently large norm.

\begin{lemma}[Gradient descent updates make significant progress for super-linear objectives]\label{lem:gd-superlinear}
	Suppose $f : \mathbb{R}^d \to \mathbb{R}$ is convex, $M$-smooth, achieves its minimum value at $f(0) = 0$, and satisfies~\eqref{eq:convex-superlinear}. Then there exists $r_1 := r_1(f)$ such that for any stepsize $\eta \leq 1/M$, it holds that
	\[
	\|x - \eta \nabla f(x)\| \leq \|x \| - \frac{\eta \beta}{4} \,, \quad \forall \|x\| \geq r_1\,.
	\]
\end{lemma}
\begin{proof}
	We bound
	\begin{align*}
		\|x - \eta \nabla f(x)\| & = \sqrt{\|x - \eta \nabla f(x)\| ^2}
		\\ &= \sqrt{\|x\|^2 - 2\eta \langle \nabla f(x), x \rangle + \eta^2\|\nabla f(x)\|^2}\\
		&\leq  \sqrt{\|x\|^2 -  2\eta \langle \nabla f(x), x \rangle + \eta^2 M \langle \nabla f(x), x \rangle} \\
		&\leq \sqrt{\|x\|^2 - \eta \langle \nabla f(x), x \rangle}\\
		&\leq \|x\| - \eta \langle\nabla f(x), x \rangle / (2 \|x\| ) \\
		&\leq \|x\| - \eta f(x)/(2\|x\|) \\
		&\leq \|x\| - \eta \beta/2  + \eta \alpha /(2\|x\|) \,.
	\end{align*}
	Above, the second step expands the square. The third step uses the folklore fact from optimization (see, e.g.,~\citep[Equation (2.1.8)]{nesterov2003introductory}) that $M$-smoothness implies $\langle \nabla f(x) - \nabla f(y), x - y\rangle \geq \frac{1}{M} \|\nabla f(x) - \nabla f(y)\|^2$, which we use here for $y=0$. The fourth step uses the assumption that $\eta \leq 1/M$. The fifth step adds a non-negative quantity to complete the square. The penultimate step uses convexity and the assumption $f(0) = 0$ to bound $\langle \nabla f(x), x \rangle \geq f(x) - f(0) = f(x)$. The final step uses the super-linear growth condition~\eqref{eq:convex-superlinear}.  We conclude that if $\|x\| \geq 2\alpha /\beta$, then $\|x - \eta \nabla f(x)\| \leq \|x\| - \eta \beta/4$.
\end{proof}

\begin{proof}[Proof of Theorem~\ref{thm:convex}]
	As in the proof of Theorem~\ref{thm:sc}, we bound the change in the Lyapunov function $\Phi_{d,\lambda}(\cdot)$ from running an iteration of the Langevin Algorithm from some point $x \in \mathbb{R}^d$ to
	\[
	x' = x - \eta \nabla f(x) + Z\,,
	\]
	where $Z \sim \mathcal{N}(0,2\eta I_d)$. To analyze this, we disassociate the two parts of this update: the gradient descent step and the noise convolution.
	\par First, we analyze the gradient descent step. We consider two cases depending on whether the norm $\|x\|$ of the current iterate is larger than $R := \max(r_0/\lambda + \eta \beta /4, r_1 )$, where $r_0$ is the quantity from Lemma~\ref{lem:phi-properties}, and $r_1$ is the quantity from Lemma~\ref{lem:gd-superlinear}. (We will set $\lambda = \beta/16$.)

	\begin{itemize}
		%
		%
		\item \underline{Case 1: $\|x\| \leq R$.} Then by the fact that a gradient descent step is non-expansive (Lemma~\ref{lem:gd-contractive} with $m=0$), we know that $\|x - \eta \nabla f(x)\| \leq \|x\| \leq R$. Thus
		\begin{align*}
			\Phi_{d,\lambda}(x - \eta \nabla f(x)) = \phi_{d}(\lambda \|x - \eta \nabla f(x)\|) \leq \phi_{d}( \lambda R)\,.
		\end{align*}
		where above the first step is by definition of $\phi$ in terms of $\Phi$ (see~\eqref{eq:Phi-phi}), and the second step is by monotonicity of $\phi_{d}$ (item (i) of Lemma~\ref{lem:phi-properties}).
		%
		%
		\item \underline{Case 2: $\|x\| > R$.} Then $\|x\| \geq r_1$, so $\|x - \eta \nabla f(x)\| \leq \|x\| - \eta \beta /4$ by Lemma~\ref{lem:gd-superlinear}. Thus
		\begin{align*}
			\Phi_{d,\lambda}(x - \eta \nabla f(x)) =  \phi_{d}(\lambda \|x - \eta \nabla f(x)\|) \leq \phi_{d}(\lambda (\|x\| - \eta \beta/4)) \leq e^{-\eta \beta \lambda/8} \phi_{d}(\lambda \|x\|) = e^{-\eta \beta \lambda /8} \Phi_{d,\lambda}(x) \,,
		\end{align*}
		where above the first and last steps are by definition of $\phi$ in terms of $\Phi$ (see~\eqref{eq:Phi-phi}), the second step is by monotonicity of $\phi_{d}$ (item (i) of Lemma~\ref{lem:phi-properties}), and the third step is due to the super-exponential growth of $\phi_{d}$ above $\lambda (R - \eta \beta /4) \geq r_0$ (item (ii) of Lemma~\ref{lem:phi-properties}).
		%
		%
	\end{itemize}

	\par Now, since $\phi$ is non-negative (item (i) of Lemma~\ref{lem:phi-properties}), we can crudely combine these two cases to conclude that
	\begin{align*}
			\Phi_{d,\lambda}(x - \eta \nabla f(x)) \leq \phi_{d}(\lambda R) + e^{-\eta \beta \lambda / 8} \Phi_{d,\lambda}(x)\,.
	\end{align*}

	\par Now use Lemma~\ref{lem:convolve} to bound the change in the Lyapunov function from the Gaussian noise convolution:
	\begin{align*}
		\Phi_{d,\lambda}(x') = \Phi_{d,\lambda}(x - \eta \nabla f(x) + Z) = e^{\eta \lambda^2} \Phi_{d,\lambda}(x -  \eta \nabla f(x))
	\end{align*}

	\par By combining the above two displays, we conclude that
	\[
	\Phi_{d,\lambda}(x') \leq e^{\eta \lambda^2} \left[ \phi_{d}(\lambda R) + e^{-\eta \beta \lambda / 8} \Phi_{d,\lambda}(x) \right] \,.
	\]
	Now take expectations on both sides, drawing $X \sim \pi_{\eta}$. Note that by stationarity, $X' \sim \pi_{\eta}$. Thus
	\[
	\mathbb{E}_{X \sim \pi_{\eta}}\big[ \Phi_{d,\lambda}(X) \big] \leq e^{\eta \lambda^2}\left(  \phi_{d}(\lambda R) + e^{-\eta \beta \lambda/ 8}  \mathbb{E}_{X \sim \pi_{\eta}} \left[ \Phi_{d,\lambda}(X) \right]  \right) \,.
	\]
	Re-arranging and simplifying yields
	\[
		\mathbb{E}_{X \sim \pi_{\eta}}\big[ \Phi_{d,\lambda}(X) \big]  \leq \frac{e^{\eta \lambda^2}}{1 - e^{\eta \lambda (\lambda - \beta/8)}} \phi_d(\lambda R)\,.
	\]
	Setting $\lambda = \beta/16$ yields
	\begin{align}
	\mathbb{E}_{X \sim \pi_{\eta}}\big[ \Phi_{d,\lambda}(X) \big] \leq A \phi_{d}(\lambda R)\,,
	\label{eq:convex-Phi-bound}
	\end{align}
	where for notational shorthand we have defined $A := e^{\eta \beta^2 /256} / (1 - e^{-\eta \beta^2 /256}) > 1$.
	\par From this, we argue a concentration inequality in a way analogous to Chernoff bounds, except using our rotationally-invariant version of the moment generating function. Specifically, bound
	\begin{align*}
			\mathbb{P}_{X \sim \pi_{\eta}} \left[ \|X\| \geq R + 2/\lambda \log (A/\delta) \right]
			&=  \mathbb{P}_{X \sim \pi_{\eta}} \left[ \phi_{d}(\lambda \|X\|) \geq \phi_{d}\left( \lambda R + 2 \log (A/\delta)  \right) \right] \\
			&\leq \frac{\mathbb{E}_{X \sim \pi_{\eta}} \left[ \phi_{d}(\lambda \|X\|) \right] }{ \phi_{d}\left( \lambda R + 2 \log (A/\delta) \right) } \\
			&\leq  \frac{ A \phi_{d}(\lambda R) }{e^{\log (A/\delta)} \phi_{d}\left( \lambda R \right) } \\
			&= \delta\,.
	\end{align*}
	Above, the first step is by monotonicity of $\phi_{d}$ (item (i) of Lemma~\ref{lem:phi-properties}), the second step is by Markov's inequality which is applicable since $\phi_{d}$ is non-negative (item (i) of Lemma~\ref{lem:phi-properties}), and the third step is by~\eqref{eq:convex-Phi-bound} and the super-exponential growth of $\phi_{d}$ above $\lambda R \geq r_0$ (item (ii) of Lemma~\ref{lem:phi-properties}).
\end{proof}

\subsection{Tightness}\label{ssec:convex-tight}

Here we observe that, modulo a constant factor, the sub-exponential concentration bound that we proved for $\pi_{\eta}$ matches the concentration for $\pi$---both in terms of when the sub-exponentiality threshold occurs, and also in terms of the sub-exponential decay rate beyond this threshold. Although no techniques currently exist for extending these lower bounds from $\pi$ to $\pi_{\eta}$ (this is an interesting open question), we conjecture that this tightness extends from $\pi$ to $\pi_{\eta}$ since we suspect that $\pi_{\eta}$ should not have better concentration than $\pi$.

\begin{example}[Tightness of sub-exponentiality parameters]\label{ex:convex-tight}
	Consider running the Langevin Algorithm on the univariate potential
	\[
		f(x) = \begin{cases}
				\beta^2 x^2 /2 & |x| \leq 1 \\
				\beta|x|-\alpha & |x| \geq 1
		\end{cases}
	\]
	where $\beta \in (0,1)$. Above, $\alpha = \beta - \beta^2/2$ is set so that $f$ is continuous at its breakpoints. Intuitively, modulo a translation and a quadratic smoothing around $0$, $f$ is essentially the absolute value function times $\beta$, and therefore $\pi$ is essentially the exponential distribution with parameter $\beta$. It is straightforward to check that $f$ is convex and $M$-smooth with $M=\beta^2$, and thus satisfies the conditions in Theorem~\ref{thm:convex}. In this example, the quantities in the proof of Theorem~\ref{thm:convex} are as follows:
	\begin{itemize}
		\item The quantities $\alpha$ and $\beta$ in the super-linear growth inequality~\eqref{eq:convex-superlinear} exactly match the $\alpha$ and $\beta$ in the construction of this function $f$.
		\item The quantity $r_0$ from Lemma~\ref{lem:phi-properties} can be set to $r_0 = \log(3)/2 \approx 0.5493$ because in dimension $d=1$, the Lyapunov function $\phi_d(r) = \cosh(r)$, and it holds\footnote{Proof: By Jensen's inequality, $\frac{3}{4} \exp(\Delta) + \frac{1}{4} \exp(-\Delta) \geq \exp(\Delta/2)$. Re-arranging gives $e^{2r} \geq 3 \geq \frac{\exp(\Delta/2) - \exp(-\Delta)}{\exp(\Delta) - \exp(\Delta/2)}$ for any $r \geq \log(3)/2$. Re-arranging further establishes the desired inequality $\cosh(r + \Delta) \geq \exp(\Delta/2) \cosh(r)$.} that $\cosh(r + \Delta) \geq \exp(\Delta/2) \cosh(r)$ for all $r \geq \log(3)/2$ and all $\Delta \geq 0$.
		\item The quantity $r_1$ from Lemma~\ref{lem:gd-superlinear} can be set to $r_1 = 2$ because $\frac{2\alpha}{\beta} = 2 - \beta \leq 2$.
		\item The quantity $R = \max(\frac{r_0}{\lambda} + \frac{\eta \beta}{4}, r_1) \asymp \max( \frac{1}{\beta/16} + \frac{\beta}{4\beta^2}, 2) \asymp \frac{1}{\beta}$.
	\end{itemize}
	Therefore, for this potential $f$, Theorem~\ref{thm:convex} establishes sub-exponential tails that decay at rate $2\lambda^{-1}\logdel \asymp\beta^{-1}\logdel$ beyond norms of size $R + 2\lambda^{-1} \log A \asymp \beta^{-1}  - \beta^{-1} \log (\eta \beta^2)$. Similarly, $\pi \propto e^{-f}$ has sub-exponential tails that decay at a rate of order $\beta^{-1} \logdel$ beyond norms of order $\beta^{-1}$, because $\pi$ has tails that are similar to that of an exponential random variable with parameter $\beta$.
	We conclude that for all stepsizes $\eta \leq 1/M$ that are not exponentially small in the problem parameters\footnote{This is a mild assumption, since in all relevant parameter regimes for sampling, $\eta$ is only polynomially small; and moreover, for exponentially small $\eta$, sub-exponential concentration of $\pi_{\eta}$ is immediate from combining any bias bound between $\pi$ and $\pi_{\eta}$ (which depend polynomially on $\eta$) with the classical result that $\pi$ is sub-exponential (Lemma~\ref{lem:pi-convex}).}, the concentration inequality in Theorem~\ref{thm:convex} matches that of $\pi$---both in terms of when the sub-exponentiality threshold occurs, as well as the rate of sub-exponential decay beyond this threshold.
    \par Moreover, this lower bound construction is readily extended to arbitrary parameters $\beta > 0$ (by considering the dilated function $h(x) = f(bx)$ for some $b$ related to $\beta$) and any  $\alpha > 0$ (by adding a flat region near $0$ of size roughly $\alpha/\beta$ before the quadratic and linear pieces as defined in $f$).
\end{example}

\section{Lyapunov function}\label{sec:mgf}

We propose to use the following Lyapunov function in order to analyze $\pi_{\eta}$.

\begin{defin}[Lyapunov function]
	For any dimension $d \in \mathbb{N}$ and weight $\lambda > 0$, let $\Phi_{d,\lambda} : \mathbb{R}^d \to \mathbb{R}$ denote the function
	\begin{align}
	\Phi_{d,\lambda}(x) := \mathbb{E}_{v \sim \mathcal{S}^{d-1}} \left[ e^{\lambda \langle v, x \rangle} \right]\,,
		\label{eq:Phi-def}
	\end{align}
\end{defin}

The rest of this section discusses properties and interpretations of this Lyapunov function.

\subsection{Relation to rotation-invariant MGF}\label{ssec:mgf}

This Lyapunov function has a natural interpretation in terms of a rotationally-invariant version of the standard moment generating function (MGF). Specifically, rather than bounding the standard MGF of a distribution $\mu$---which we recall is
$\mathbb{E}_{X \sim \mu} e^{\lambda \langle v, X \rangle}\,,$
as a function of a fixed vector $v \in \mathcal{S}^{d-1}$---we will bound the average
$	\mathbb{E}_{X \sim \mu} 	\Phi_{d,\lambda}(X) =  \mathbb{E}_{v \sim \mathcal{S}^{d-1}} \mathbb{E}_{X \sim \mu} e^{\lambda \langle v, X \rangle}$. The latter is precisely the average of the standard MGF over all vectors $v \in \mathcal{S}^{d-1}$, hence why we refer to it as the ``rotation-invariant MGF''.

\par How do bounds on this rotation-invariant MGF imply concentration inequalities? Since properties like sub-Gaussianity and sub-exponentiality of a distribution are by definition equivalent to MGF bounds which hold uniformly over all vectors $v \in \mathcal{S}^{d-1}$~\citep{rigollet2015high}, these properties of course imply the same bounds for the rotationally-invariant MGF (simply average over $v \sim \mathcal{S}^{d-1}$). The converse is in general lossy, in that a bound on the rotationally-invariant MGF does not imply the same\footnote{It is possible to extract a weaker bound for the standard MGF, but this loses dimension-dependent factors, which leads to loose bounds. In contrast, our analysis is tight up to a small constant factor.} bound on the standard MGF uniformly over all vectors $v \in \mathcal{S}^{d-1}$. But this conversion is of course lossless for rotationally-invariant distributions $\mu$---and this suffices for the purposes of this note.
\par Let us elaborate: in order to prove concentration inequalities for $\|X\|$ where $X$ is drawn from the (not rotationally-symmetric) distribution $\pi_{\eta}$, it suffices to prove the same concentration inequalities for $\|\tilde{X}\|$, where $\tilde{X}$ is a randomly rotated version of $X$, namely $\tilde{X} = RX$ where $R$ is a random rotation. Since our analysis in the sequel lets us bound the rotationally-invariant MGF for the law of $\tilde{X}$, this immediately gives concentration inequalities for $\|\tilde{X}\|$ and thus also for $\|X\|$, as desired.

\par Briefly, the upshot of using this rotationally-invariant MGF is that, unlike the standard MGF, it is a function of the norm (see \S\ref{ssec:bessel})---and as we describe in the sequel, this enables exploiting contractivity of the gradient descent step in the Langevin Algorithm update. We also mention that since our goal is to prove concentration inequalities on the norm, of course no information is lost by using the rotational-invariant MGF rather than the standard MGF, since the only difference is averaging over directions.

\subsection{Explicit expression via Bessel functions}\label{ssec:bessel}

\par Of course, by rotational invariance, this Lyapunov function $\Phi_{d,\lambda}(x)$ depends on the point $x$ only through its norm $\|x\|$. That is,
\begin{align}
	\Phi_{d,\lambda}(x) := \phi_{d}(\lambda \|x\|)
	\label{eq:Phi-phi}
\end{align}
where $\phi_d : \mathbb{R}_{\geq 0} \to \mathbb{R}_{\geq 0}$ is the univariate function given by
\begin{align}
	\phi_d(z) = \mathbb{E}_{v \sim \mathcal{S}^{d-1}} \left[ e^{z \langle v, e_1 \rangle} \right] \,.
	\label{eq:phi-def}
\end{align}
The following lemma provides an elegant closed-form expression for this univariate function $\phi_d$ in terms of modified Bessel functions $I_n(\cdot)$ of the first kind and the Gamma function $\Gamma(\cdot)$.

\begin{lemma}[Explicit formula for Lyapunov function]\label{lem:bessel}
	For any dimension $d \geq 2$ and argument $z > 0$,
	\[
		\phi_{d}(z) = \Gamma(\alpha+ 1) \cdot \left(\frac{2}{z} \right)^{\alpha} \cdot I_{\alpha}(z)\,,
	\]
	where $\alpha := (d-2)/2$. (In dimension $d=1$, we simply have $\phi_d(z) = \cosh(z)$.)
\end{lemma}
\begin{proof}
	It is a classical fact (see, e.g.,~\citep{Szego39}) that if $v$ is drawn uniformly at random from the unit sphere $\mathcal{S}^{d-1}$, then the induced law $\nu_{d-1}$ of $v_1$ has density
	\begin{align*}
		\frac{d\nu_{d-1}(t)}{dt} = \frac{\Gamma(\alpha + 1)}{\sqrt{\pi} \Gamma(\alpha)} (1-t^2)^{\alpha- 1/2} \mathds{1}_{t \in (-1,1)}\,.
	\end{align*}
	Therefore
	\begin{align*}
			\Phi_{d,\lambda}(x)
			=
			\mathbb{E}_{v \sim \mathcal{S}^{d-1}} \left[ e^{\lambda \langle v, x \rangle} \right]
			=
			\mathbb{E}_{v \sim \mathcal{S}^{d-1}} \left[ e^{z v_1 } \right]
			=
			 \frac{\Gamma(\alpha + 1)}{\sqrt{\pi} \Gamma(\alpha)}
			\int_{-1}^1 e^{zt} (1-t^2)^{\alpha- 1/2} dt\,.
	\end{align*}
	We conclude by using the Poisson integral representation
	\begin{align*}
			I_{\alpha}(z) = \frac{(z/2)^{\alpha}}{\sqrt{\pi}\Gamma(\alpha + 1/2)} \int_{-1}^1 e^{zt} (1-t^2)^{\alpha- 1/2} dt\,.
	\end{align*}
	of the Bessel functions, see, e.g.,~\citep[equation (10.32.2)]{DLMF}.
\end{proof}

\subsection{Properties of the Lyapunov function}

The key reason we use the Lyapunov function $\Phi$ to analyze the dynamics of the Langevin Algorithm is that the Langevin Algorithm's update decomposes into a gradient descent step and a noise convolution step, and $\Phi$ enables tracking the effect of both steps in an elegant manner.
\par The following lemma describes how $\Phi$ is affected by the noise convolution. While a similar property holds for the standard MGF, the advantage of the rotation-invariant MGF over the standard MGF is that it enables exploiting contractivity of the gradient descent step. However, the effect of the gradient descent step is slightly different depending on whether the potential is just convex or also strongly convex, and thus is treated separately in the corresponding sections below.

\begin{lemma}[Behavior of $\Phi$ under Gaussian convolution]\label{lem:convolve}
	For any dimension $d \in \mathbb{N}$, point $x \in \mathbb{R}^d$, weight $\lambda > 0$, and noise variance $\sigma^2$,
	\[
	\mathbb{E}_{Z \sim \mathcal{N}(0, \sigma^2 I_d)} \left[ \Phi_{d,\lambda}\left( x + Z \right) \right] = e^{\lambda^2\sigma^2/2} \cdot \Phi_{d,\lambda}(x)\,.
	\]
\end{lemma}
\begin{proof}
	We compute:
	\begin{align*}
		\mathbb{E}_{Z \sim \mathcal{N}(0, \sigma^2 I_d)}[\Phi_{d,\lambda}(x+Z)]
		&= 	\mathbb{E}_{Z \sim \mathcal{N}(0, \sigma^2 I_d)}\mathbb{E}_{v \sim \mathcal{S}^{d-1}} \left[ e^{\lambda \langle v, x+Z \rangle} \right]\\
		&= \mathbb{E}_{v \sim \mathcal{S}^{d-1}} \left[ e^{\lambda \langle v, x \rangle} \mathbb{E}_{Z \sim \mathcal{N}(0, \sigma^2 I_d)} [e^{\lambda \langle v, Z\rangle}] \right]\\
		&= \mathbb{E}_{v \sim \mathcal{S}^{d-1}} \left[ e^{\lambda \langle v, x \rangle} e^{\lambda^2\sigma^2 \|v\|^2 /2} \right]\\
		&= e^{\lambda^2\sigma^2/2} \cdot \Phi_\lambda(x).
	\end{align*}
	Above, the first and last steps are by definition of $\Phi$; the second step is by Fubini's Theorem; and the third step is by the well-known formula for the moment generating function of the multivariate Gaussian distribution (see, e.g.,~\citep[\S5.8]{grimmett2020probability}).
\end{proof}

The next lemma collects various basic properties of $\Phi$ that we use in our analysis in the sequel. We remark that in the setting of strongly convex potentials, our analysis only makes use of the positivity and monotonicity in item (i) of this lemma; the setting of convex potentials has a more involved analysis and requires the eventual exponential growth in item (ii) of this lemma.

\begin{lemma}[Properties of rotation-invariant MGF]\label{lem:phi-properties}
	For any dimension $d \in \mathbb{N}$:
	\begin{itemize}
		\item[(i)]  The function $\phi_{d}$ is positive and increasing on $\mathbb{R}_{\geq 0}$.
		\item[(ii)] There exists some $r_0 := r_0(d)$ such that
		\[
			\phi_d(r + \Delta) \geq e^{\Delta/2} \phi_d(r)\,
		\]
		for all $r \geq r_0$ and $\Delta > 0$.
	\end{itemize}
\end{lemma}
\begin{proof}
	Item (i) is immediate because $\phi_{d}$ is by definition an expectation over functions which are each positive and increasing. For item (ii), observe that Lemma~\ref{lem:bessel} expresses $\phi_d(z)$ as $\Gamma(\alpha + 1)(2/z)^{\alpha} = \Theta(z^{-\alpha})$, which grows inverse polynomially in $z$, times the Bessel function $I_{\alpha}(z)$, which grows exponentially for large arguments $z \gg \sqrt{\alpha} = \Theta(\sqrt{d})$ due to the Hankel expansion $I_{\alpha}(z) \sim e^z (2\pi z)^{-1/2} (1 - \frac{4\alpha^2 - 1}{8z} + \frac{(4 \alpha^2 - 1)(4\alpha^2 - 9)}{2! (8z)^2} - \dots)$~\citep[equation (10.40.1)]{DLMF}. The desired exponential growth immediately follows\footnote{The ``fudge factor'' of $1/2$ in the exponent is used to simply and crudely bound the lower-order polynomial terms, and is irrelevant for our purposes in the sequel.}.
\end{proof}

Although unnecessary for the purposes of this paper, we remark in passing that non-asymptotic bounds on the exponential growth in item (ii) of the above lemma can be computed in several ways. One way is to use more precise non-asymptotic bounds on Bessel functions since, as mentioned above, $\phi_d(z)$ is equal to $I_{\alpha}(z)$ modulo a lower-order polynomial term. Another way is to show that the logarithmic derivative of $\phi_d$ is eventually lower bounded by a constant; we cannot help but mention that an elegant identity for this purpose is that this logarithmic derivative simplifies to $\frac{d}{dz} \log \phi_d(z) = I_{\alpha+1}(z) / I_{\alpha}(z)$, which is the ratio of Bessel functions of different orders.

Machine Learning (stat.ML); Machine Learning (cs.LG); Probability (math.PR); Statistics Theory (math.ST)


N derivatives are necessary for order N+1 convergence in quadrature: a converse result

Abstract: Results on the error bounds of quadrature methods are well known - most state that if the method has degree N, and the integrand has N derivatives, then the error is order N+1. We prove here a converse: that if the integrand fails to have N derivatives, even only at a finite number of points, no method, regardless of its degree, can guarantee convergence more than order N. Even if the integrand fails to have N derivatives at just 3 (for even N, 2) points, no method can produce order more than N+1 convergence. This is done by an adversarial proof: we explicitly construct the functions that exhibit such error; simple splines turn out to suffice.

\section{Introduction.}
It is well known that a quadrature method of degree $N$ has an error bound of order $N+1$; many proofs rely on Taylor's theorem, which then requires the existence of the $N$th derivative. Of course, it can be asked what happens if the $N$th derivative fails to exist.

The question was first investigated in \cite{stroud}, which proved error bounds using lower derivatives. These bounds do not attain the full order allotted by the degree of the method; instead they are consigned to no more than the number of derivatives plus one.

Recently attention has returned to the issue of integration of insufficiently nice functions, where \cite{engel} proved a few error bounds using lower order derivatives than the degrees of the methods involved. \cite{cruz} looked into the trapezoidal rule and Simpson's rule and found that, for merely once differentiable functions, the trapezoidal rule had a better bound than Simpson's rule.

A different direction is to optimize quadrature rules under the constraint that the integrand is only once differentiable; this work is led by \cite{ujevic} and there are quite a few papers on the subject. The optimal rules indeed differ from the established Gauss-Legendre rules. There is also a focus on Simpson's inequality and its applications, as demonstrated by \cite{huy} who proved a bound using an arbitrary order derivative.

One unifying theme in all these results is the inability to achieve $N+2$th order convergence with only the $N$th derivative. We settle this question once and for all, by proving this impossibility. In other words, $N$ derivatives are not only sufficient (in conjunction with a suitable method), but also a \emph{necessary} condition of achieving $N+1$th order convergence.

We prove the contrapositive by exhibiting functions, exactly $N$ times differentiable, that bound the error of any method from below at $N+1$th order.

\section{Results.}
For space, we use the symbol $k-1$ instead of $N$ in the ensuing discussion.

\begin{theorem}
\label{t1}
\hypertarget{t1}{Let} the region of integration $[a,b]$ be specified, and $k\geq1$ be a fixed natural number. Let $\{I_n\}$ be any sequence of internal quadrature methods, with
\[I_n(f)=\sum_{i=1}^nw_{n,i}f(x_{n,i}),\quad a\leq x_{n,1}<x_{n,2}<\cdots<x_{n,n}\leq b\text{,}\]
with $w_{n,i},x_{n,i}$ fixed. Define the exact integral to be
\[I(f)=\int_a^bf(x)\,\mathrm{d}x\text{.}\]
Then there exists a sequence of functions $\{f_n\}$ along with a sequence of finite sets $\{S_n\}$ satisfying
\begin{itemize}
\item $S_n\subset[a,b]$, and has size $\abs{S_n}\leq4n-3$
\item $f_n\in C^{k-1}[a,b]\setminus C^k[a,b]$
\item $f_n\in C^{\omega}([a,b]\setminus S_n)$
\item $\displaystyle\norm{f_n^{(k-1)}}_{\infty,[a,b]}\leq\frac{(b-a)}{4}k!$
\item $\displaystyle\norm{f_n^{(k)}}_{\infty,[a,b]\setminus S_n}=k!$
\end{itemize}
such that the sequence of integration errors of $I_n$ on $f_n$ has order at most $k$, or in other words
\[\limsup_{n\rightarrow\infty}\frac{\abs{I_n(f_n)-I(f_n)}}{n^k}>0\text{.}\]
\end{theorem}
\begin{proof}
By adversarial construction.

Let $f_n$ be a $k$th order spline such that $f_n(x_{n,i})=0$ for all $i$, and have maximal integral with unit leading coefficient. First, for $1\leq i\leq n-1$ define $\Delta x_i=x_{n,i+1}-x_{n,i}$ and the quarter-interpolated points
\begin{alignat*}{3}
\bar{x}_{n,i+1/4}&=\frac{3x_{n,i}}{4}&&+\hspace{0.049in}\frac{x_{n,i+1}}{4}&&=x_{n,i}+\frac{\Delta x_i}{4}\\
\bar{x}_{n,i+1/2}&=\hspace{0.039in}\frac{x_{n,i}}{2}&&+\hspace{0.049in}\frac{x_{n,i+1}}{2}&&=x_{n,i}+\frac{\Delta x_i}{2}\\
\bar{x}_{n,i+3/4}&=\hspace{0.039in}\frac{x_{n,i}}{4}&&+\frac{3x_{n,i+1}}{4}&&=x_{n,i}+\frac{3\Delta x_i}{4}\text{.}
\end{alignat*}
Then explicitly, if $k$ is odd,
\begin{align*}f_n&=\begin{cases}
-(x-x_{n,1})^k,\;&x\in\intcc{a,x_{n,1}}\\
(x-x_{n,i})^k,\;&x\in\intcc{x_{n,i},\bar{x}_{n,i+1/4}}\\
\left(x-\bar{x}_{n,i+1/2}\right)^k+2\left(\frac{\Delta x_i}{4}\right)^k,\;&x\in\intcc{\bar{x}_{n,i+1/4},\bar{x}_{n,i+1/2}}\\
-\left(x-\bar{x}_{n,i+1/2}\right)^k+2\left(\frac{\Delta x_i}{4}\right)^k,\;&x\in\intcc{\bar{x}_{n,i+1/2},\bar{x}_{n,i+3/4}}\\
-(x-x_{n,i+1})^k,\;&x\in\intcc{\bar{x}_{n,i+3/4},x_{n,i+1}}\\
(x-x_{n,n})^k,\;&x\in\intcc{x_{n,n},b}
\end{cases}
\intertext{for all valid $1\leq i\leq n-1$; and if $k$ is even,}
f_n&=\begin{cases}
(x-x_{n,1})^k,\;&x\in\intcc{a,\bar{x}_{1+1/4}}\\
-\left(x-\bar{x}_{n,i+1/2}\right)^k+2\left(\frac{\Delta x_i}{4}\right)^k,\;&x\in\intcc{\bar{x}_{n,i+1/4},\bar{x}_{n,i+3/4}}\\
(x-x_{n,j})^k,\;&x\in\intcc{\bar{x}_{(j-1)+3/4},\bar{x}_{n,j+1/4}}\\
(x-x_{n,n})^k,\;&x\in\intcc{\bar{x}_{(n-1)+3/4},b}
\end{cases}
\end{align*}
for all valid $1\leq i\leq n-1$ and $2\leq j\leq n-2$.

An easier way to understand this is to consider the ``basic unit'' of structure between two $x_{n,i}$, shown in Figure \ref{base} between $x_{n,i}=0$ and $x_{n,i+1}=4$. The concept is to splice 4 (shifted) copies of the curve $x^k$ together such that the two outer pieces are convex and the inner pieces concave. If $k$ is even, then it is analytic at the endpoints and midpoint.

\begin{figure}[h]
\centering\includegraphics[width=\textwidth,keepaspectratio]{base}
\caption{A basic unit of $f_n$ for various values of $k$, shown for $x_{n,i}=0$ and $x_{n,i+1}=4$}
\label{base}
\end{figure}

Now it is clear that $f_n$ is $C^{k-1}[a,b]$, is analytic except at most at the points
\[S_n=\left\{x_{n,i},\frac{3x_{n,i}+x_{n,i+1}}{4},\frac{x_{n,i}+x_{n,i+1}}{2},\frac{x_{n,i}+3x_{n,i+1}}{4}\right\}_{i=1}^{n-1}\bigcup\;\{x_{n,n}\}\text{,}\]
a set of size $4n-3$ (only $2n-2$ points if $k$ is odd). The $k-1$th derivative of $f_n$ exists and is piecewise linear, with slope $\pm k!$; the length of a piece is bounded above by $(b-a)/4$, possible only if all points $x_{n,i}$ are at endpoints, thus the derivative is bounded by that product. If $k$ is odd the piece can have length $(b-a)/2$, but its midpoint is at 0 and that does not change the bound. The $k$th derivative is the piecewise constant function $\pm k!$ where it exists.\pagebreak

Note that $I_n(f_n)=0$ by construction; it remains only to compute $I(f_n)$, which is easily done by considering the basic unit:
\begin{align*}
\int_{x_{n,i}}^{x_{n,i+1}}\!\!f_n(x)\,\mathrm{d}x&
\begin{cases}\displaystyle=\int_0^{\frac{\Delta x_i}{4}}\!x^k\,\mathrm{d}x+\int_{-\frac{\Delta x_i}{4}}^0\!x^k+2\left(\textstyle\frac{\Delta x_i}{4}\right)^k\!\mathrm{d}x\\
\displaystyle\!\!\quad{}+\int_0^{\frac{\Delta x_i}{4}}\!(-x)^k+2\left(\textstyle\frac{\Delta x_i}{4}\right)^k\!\mathrm{d}x+\int_{-\frac{\Delta x_i}{4}}^0\!(-x)^k\,\mathrm{d}x\!\!&\text{ if $k$ is odd}\\
\displaystyle=\int_0^{\frac{\Delta x_i}{4}}\!x^k\,\mathrm{d}x+\int_{-\frac{\Delta x_i}{4}}^{\frac{\Delta x_i}{4}}\!(-x)^k+2\left(\textstyle\frac{\Delta x_i}{4}\right)^k\!\mathrm{d}x\\
\displaystyle\!\!\quad{}+\int_{-\frac{\Delta x_i}{4}}^0\!x^k\,\mathrm{d}x&\text{ if $k$ is even}
\end{cases}\displaybreak[0]\\
&\hspace{0.12in}=\int_{-\frac{\Delta x_i}{4}}^{\frac{\Delta x_i}{4}}\!2\left(\frac{\Delta x_i}{4}\right)^k\!\mathrm{d}x\quad\bigpar{\forall k}\\
&\hspace{0.12in}=\frac{(\Delta x_i)^{k+1}}{4^k}\text{.}
\end{align*}
The pieces for the endpoints are easy:
\begin{align*}
\int_{a}^{x_{n,1}}f_n(x)\,\mathrm{d}x=\int_0^{x_{n,1}-a}x^k\,\mathrm{d}x&=\frac{(x_{n,1}-a)^{k+1}}{k+1}\\
\int_{x_{n,n}}^{b}f_n(x)\,\mathrm{d}x=\int_0^{b-x_{n,n}}x^k\,\mathrm{d}x&=\frac{(b-x_{n,1})^{k+1}}{k+1}\text{.}
\end{align*}
Thus $I(f_n)$, and the integration error, is simply
\[\frac{(x_{n,1}-a)^{k+1}}{k+1}+\sum_{i=1}^{n-1}\frac{(x_{n,i+1}-x_{n,i})^{k+1}}{4^k}+\frac{(b-x_{n,1})^{k+1}}{k+1}\text{.}\]
This is straightforward to minimize by picking equally spaced points (we ignore the issue of picking points closer to the endpoints, which is unimportant), whereby the error is bounded below by
\[\sum_{i=0}^{n}C\left(\frac{b-a}{n+1}\right)^{k+1}=C(n+1)\left(\frac{b-a}{n+1}\right)^{k+1}\in\Theta(n^{-k})\text{.}\]
Therefore the method has at most order $k$ convergence, and the theorem is proved
\end{proof}

There are two main ways of conceptualizing a ``sequence of quadrature methods''. The first is an arbitrary-order family of methods, for example the Newton-Cotes or Gauss-Legendre, with an increasing number of points. The second is to take a fixed quadrature method, and form a composite rule with $n$ subintervals, for all $n$.

Note that the construction obviates all dependence on the weights chosen by the quadrature method. The restriction to internal quadrature methods is not necessary: a point outside $[a,b]$ can easily be made to evaluate to 0, with the result that the error bound becomes worse.

Nor do methods that rely on the derivatives of the function escape this fate. It is straightforward to check that the $k-1$th and lower derivatives of $f_n$ are all 0 at each $x_{n,i}$, and so unless the method uses the $k$th derivative (which doesn't exist in the first place), the construction stands valid.

The size of $S_n$ grows linearly in $n$, which disallows attempts to sidestep the theorem by using less points for evaluation than the number of points known to be bad.

We may then ask the question: if the number of bad points is finitely bounded, can something more be salvaged? The idea is, say with a composite rule, with a sufficient number of subintervals, to confine all the bad points to specific subinterval(s), whereby all other subintervals may achieve their degree-based bound.

The answer is a qualified yes. The proof of Theorem \ref{t1} can be easily modified to use a very small number of bad points, but this only allows improvement by 1 order, to $k+1$. We shall reuse the setting in the \hyperlink{t1}{preface} of the preceding theorem.

\begin{theorem}
\label{t2}
There exists a sequence of functions $\{f_n\}$ along with a sequence of finite sets $\{S_n\}$ satisfying
\begin{itemize}
\item $S_n\subset[a,b]$, and has size $\abs{S_n}=5$ if $k$ is odd, or $\abs{S_n}=4$ if $k$ is even
\item $f_n\in C^{k-1}[a,b]\setminus C^k[a,b]$
\item $f_n\in C^{\omega}([a,b]\setminus S_n)$
\item $\displaystyle\norm{f_n^{(k-1)}}_{\infty,[a,b]}\leq\frac{(b-a)}{4}k!$
\item $\displaystyle\norm{f_n^{(k)}}_{\infty,[a,b]\setminus S_n}=k!$
\end{itemize}
such that the sequence of integration errors of $I_n$ on $f_n$ has order at most $k+1$, or in other words
\[\limsup_{n\rightarrow\infty}\frac{\abs{I_n(f_n)-I(f_n)}}{n^{k+1}}>0\text{.}\]
\end{theorem}
\begin{proof}
By adversarial construction.

Let $i$ be an index such that $\Delta x=x_{n,i+1}-x_{n,i}$ is maximized. Let $f_n$ be a $k$th order spline such that $f_n=0$ outside of the open interval $(x_{n,i},x_{n,i+1})$, and have maximal integral with unit leading coefficient. Explicitly,
\begin{align*}
f_n&=\begin{cases}
(x-x_{n,i})^k,\;&x\in\intcc{x_{n,i},x_{n,i}+\frac{\Delta x}{4}}\\
\left(x-\frac{\Delta x}{2}\right)^k+2\left(\frac{\Delta x}{4}\right)^k,\;&x\in\intcc{x_{n,i}+\frac{\Delta x}{4},x_{n,i}+\frac{\Delta x}{2}}\\
-\left(x-\frac{\Delta x}{2}\right)^k+2\left(\frac{\Delta x}{4}\right)^k,\;&x\in\intcc{x_{n,i}+\frac{\Delta x}{2},x_{n,i}+\frac{3\Delta x}{4}}\\
-(x-x_{n,i+1})^k,\;&x\in\intcc{x_{n,i}+\frac{3\Delta x}{4},x_{n,i+1}}\\
0\quad&\text{otherwise}
\end{cases}\text{ if $k$ is odd,}\\
f_n&=\begin{cases}
(x-x_{n,i})^k,\;&x\in\intcc{x_{n,i},x_{n,i}+\frac{\Delta x}{4}}\\
-\left(x-\frac{\Delta x}{2}\right)^k+2\left(\frac{\Delta x}{4}\right)^k,\;&x\in\intcc{x_{n,i}+\frac{\Delta x}{4},x_{n,i}+\frac{3\Delta x}{4}}\\
(x-x_{n,i+1})^k,\;&x\in\intcc{x_{n,i}+\frac{3\Delta x}{4},x_{n,i+1}}\\
0\quad&\text{otherwise}
\end{cases}\text{ if $k$ is even.}
\end{align*}
This is the same construction as in Theorem 1, except that only one of the basic units is used.

The discussion about the continuity classification of $f_n$ holds, with the ``bad'' set
\[S_n=\left\{x_{n,i}\,,\,x_{n,i}+\frac{\Delta x}{4}\,,\,x_{n,i}+\frac{\Delta x}{2}\,,\,x_{n,i}+\frac{3\Delta x}{4}\,,\,x_{n,i+1}\right\}\]
of size 5; if $k$ is even, the midpoint is analytic as well. The comments on the bounds on derivatives hold.

$I_n(f_n)$ remains 0 as all evaluation points are 0, and the exact same calculation as before shows that
\[I(f_n)=\int_{a}^{b}f_n(x)\,\mathrm{d}x=\int_{x_{n,i}}^{x_{n,i+1}}f_n(x)\,\mathrm{d}x=\frac{(\Delta x)^{k+1}}{4^k}\text{.}\]
With $n$ points to be chosen in $[a,b]$ it is clear that $\Delta x$ is bounded below by $(b-a)/(n+1)$; an attempt to pick the midpoint $n$ times runs afoul of the fact that the error for the endpoints is even worse. Hence we have that
\[\abs{I_n(f_n)-I(f_n)}\geq\frac{(b-a)^{k+1}}{4^k}(n+1)^{-(k+1)}\in\Theta(n^{-(k+1)})\]
with order at most $k+1$, and the theorem is proved
\end{proof}

We can tighten this result to requiring 3 (or 2, for even $k$) bad points by noticing that the endpoints $x_{n_i},x_{n,i+1}$ do not need to be bad. Instead we can use the analytic continuation of $f_n$ beyond these points (the monomial $(x-x_{i})^{k+1}$), and assume that $I_n$ can exactly integrate the rest of the function; the one piece between those two points alone makes the lower bound.

We may also keep the quadrature result 0 by making the $k+1$th derivative go negative quickly and smoothly connect to the zero function; this however necessitates breaking analyticity.

\section{Discussion and Conclusions.}
It may be asked next, can there be a finite, fixed set of bad points in the function, and how much improvement does that generate? We consider this moot, as when the bad points are fixed, the usual methods may be employed to simply \emph{find} them all, so that we may segment the region of integration at these points. Thus, all problems relating to having less than $N$ derivatives completely vanish.

The main implication of these theorems is that it is asymptotically fruitless to use any method that has 2 degrees or more beyond the differentiability class of the integrand: Theorem \ref{t2} proves the extra degrees have no effect.

An interesting observation is of the non-local effect of a single point of non-niceness destroying convergence everywhere; this has been noted in interpolation theory, under the name of the \emph{principle of contamination} \cite{saff}. Given the strong connection between interpolation polynomials and numerical integration, seeing the principle apply here is hardly a surprise.

Another insight that drops out of the details is a heuristic to minimize the impact of this lower bound: minimize the maximal distance between two evaluation points. That equally spaced points are optimal against this adversary speaks to the difference between optimizing for degrees of exactness and optimizing for absolute error under only $N$ derivatives.

There seems to be a penalty in the error bound, for example found in \cite{cruz}, for achieving a higher degree of exactness, if the extra derivatives for the higher order bound do not exist.

Thus of course the high degree Newton-Cotes rules retain their severe disadvantages, and we still do not recommend them; rather that a highly composite trapezoid or Simpson's rule be considered instead of the theoretically degree-superior Gauss-Legendre or Curtis-Clenshaw families, even at equal degree.

For future work we would like to know if the order $N+1$ in Theorem \ref{t2} is sharp, or if we can find a way to force order $N$ convergence with a fixed finite number of bad points.
\pagebreak
\paragraph{Acknowledgments.}
The author would like to thank Rajesh Pereira, University of Guelph, for help on formatting and literature search. This work was supported by a Natural Sciences and Engineering Research Council of Canada Postgraduate Scholarship.