Simple and optimal high-probability bounds for strongly-convex stochastic gradient descent

Abstract: We consider stochastic gradient descent algorithms for minimizing a non-smooth, strongly-convex function. Several forms of this algorithm, including suffix averaging, are known to achieve the optimal $O(1/T)$ convergence rate in expectation. We consider a simple, non-uniform averaging strategy of Lacoste-Julien et al. (2011) and prove that it achieves the optimal $O(1/T)$ convergence rate with high probability. Our proof uses a recently developed generalization of Freedman's inequality. Finally, we compare several of these algorithms experimentally and show that this non-uniform averaging strategy outperforms many standard techniques, and with smaller variance.

\section{Additional experiments}
\SectionName{additional_experiments}

In each experiment we run SGD for the regularized SVM optimization problem described in \Section{experimental_results}. We use regularization parameter $\lambda = 1/n$ and step size $\eta_t = \frac{2}{\mu (t+1)}$.  For each return strategy, we run many trials of SGD and plot the objective value over time for every trial. At any point in time, the darkness of the plot at a specific objective value indicates the number of trials that achieved that value at that time.

We use the freely available data sets \emph{quantum} ($m=50000$ and $n=78$), \emph{covtype} ($m = 581012$ and $n=54$) and \emph{rcv1} ($m = 20242$ and $n = 47236$). We run 1000 trials of SGD on the \emph{quantum} data set, 80 trials of SGD on the \emph{covtype} data set and 70 trials of SGD on the \emph{rcv1} data set. The \emph{quantum} data set can be found at the \href{http://osmot.cs.cornell.edu/kddcup/datasets.html}{\color{blue}{KDD cup 2004 website}}
and \emph{covtype} and \emph{rcv1} can be found at the \href{https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/}{\color{blue}{LIBSVM website}}.
\begin{landscape}
\begin{figure}[h]
    \centering
    \begin{subfigure}[b]{\paperwidth}
        \centering
        \includegraphics[width=\paperwidth]{figures/quantum_iterates.png}
    \caption{quantum}
    \FigureName{quantum}
    \end{subfigure} \\
    \begin{subfigure}[b]{\paperwidth}
        \centering
        \includegraphics[width=\paperwidth]{figures/covtype2_iterates_80.png}
    \caption{covtype (80 trials)}
    \FigureName{covtype}
    \end{subfigure}
\end{figure}
\begin{figure}
    \ContinuedFloat
    \begin{subfigure}[b]{\paperwidth}
        \centering
        \includegraphics[width=\paperwidth]{figures/rcv1_2_iterates_70.png}
    \caption{rcv1 (70 trials)}
    \FigureName{rcv}
    \end{subfigure}
    \caption{Number of effective passes vs. objective value. \Figure{quantum} plots the results for the \emph{quantum} dataset; \Figure{covtype} plots the results for \emph{covtype} dataset; \Figure{rcv} plots the results for the \emph{rcv1} dataset. From left to right, we plot the objective value over time of the final iterate, uniform average, suffix average and non-uniform average for 1000 trials of SGD.}
    \label{fig:iterates_additional}
\end{figure}
\end{landscape}
\section{Proof of high probability lower bound}
\SectionName{lb_proof}

In this section we show that the error of SGD when returning $\sum_{t=1}^T \frac{t}{T(T+1)/2} x_t$ is $\Omega( \log(1/\delta)/T )$ with probability at least $\delta$. We begin by stating a useful lemma.

\begin{lemma}[{\cite[Lemma~4]{RevChernoff}}]
\LemmaName{ReverseChernoff}
Let $X_1, \ldots, X_n$ be independent random variables taking value $\{-1, +1\}$ uniformly at random and $X = \frac{1}{n} \sum_{i=1}^n X_i$.
Suppose $\sqrt{6} \leq c \leq \sqrt{n}/2$, then
\[
\prob{X \geq  \frac{c}{\sqrt{n}} } \geq \exp(-9c^2/2).
\]
\end{lemma}

\begin{proofof}{\Claim{LB}}
Since the gradient oracle satisfies the assumption in \Claim{lowerBoundIterates} (which we prove below), we may apply \Claim{lowerBoundIterates} to obtain:
\begin{align*}
    \sum_{t=1}^T \gamma_t x_t
        &~=~ \sum_{t=2}^T \gamma_t \left [\frac{1}{t}\sum_{i=1}^{t-1} \hat{z}_i \right ] \qquad\text{(by \Claim{lowerBoundIterates})}\\
        &~=~ \frac{2}{T(T+1)}\sum_{t=2}^T \sum_{i=1}^{t-1} \hat{z}_i \qquad\text{(definition of $\gamma_t$)}\\
        &~=~ \frac{2}{T(T+1)}\sum_{i=1}^{T-1} \hat{z}_i \cdot \left ( T - i \right ) \qquad\text{(swap order of summation)}\\
        &~=~ \frac{2}{T(T+1)}\sum_{i=T/2+1}^{3T/4}\hat{z}_i \cdot \left (T-i\right) \qquad\text{($\hat{z}_i = 0$ for all other $i$)}\\
        &~=~ \frac{2}{4} \left (\frac{1}{T/4}\sum_{i=T/2+1}^{3T/4} X_i \right ) \qquad\text{(definition of $\hat{z}_i$)}.
\end{align*}Now, we may apply \Lemma{ReverseChernoff} with $c = \frac{\sqrt{2\log (1/\delta)}}{3}$, and $n = T/4$ to obtain:
\[  f \left ( \sum_{t=1}^T \gamma_t x_t  \right ) ~=~ \frac{1}{2} \left ( \sum_{t=1}^T \gamma_t x_t \right )^2 ~\geq~ \frac{1}{2}\left ( \frac{1}{2} \frac{\sqrt{2 \log(1/\delta)}}{3 \sqrt{T/4}} \right )^2 ~=~ \frac{\log(1/\delta)}{9 \cdot T},  \]with probability at least $\delta.$
\end{proofof}

The proof of \Claim{LB} required the use of \Claim{lowerBoundIterates}. We now provide a proof of this claim

\begin{proofof}{\Claim{lowerBoundIterates}}
We prove the claim via induction. For the base case consider $x_2 = \Pi_{\mathcal{X}} \left (x_1 -\eta_1 \hat{g}_1 \right)$. Recall that $\hat{g}_1 = g_1 - \hat{z}_1$ where $g_1$ is the gradient of $f$ at $x_1$. Since $x_1 = 0$, we have $g_1 = 0$ and $x_2 = \Pi_{\mathcal{X}} \left (\eta_1 \hat{z}_1 \right) = \frac{1}{2}\hat{z}_1$ because $\abs{\hat{z}_t} \leq 1$ for all $t$ and $\eta_t = \frac{1}{t+1}$.

Next, assume that $x_t = \frac{1}{t}\sum_{i=1}^{t-1}\hat{z}_i$. Then, $x_{t+1} = \Pi_{\mathcal{X}}(y_t)$ where $y_t = x_t - \eta_t \hat{g}_t$ where $\hat{g}_t$ = $\nabla f (x_t) - \hat{z}_t.$ Hence, we have
\[
y_{t} ~=~ \frac{1}{t}\sum_{i=1}^{t-1} \hat{z}_t - \eta_t \left (\frac{1}{t}\sum_{i=1}^{t-1} \hat{z}_i - \hat{z}_t   \right ) ~=~ \frac{1}{t+1} \sum_{i=1}^t\hat{z}_t.
\] Clearly, $y_t \in \mathcal{X}$, and therefore $x_{t+1} = y_t = \frac{1}{t+1} \sum_{i=1}^{t}\hat{z}_t$ as desired.

\end{proofof}
\section{Proof of \Lemma{RakhlinDiameterBound} and \Lemma{RakhlinWBound}}
Both of the proofs in this section are slight modifications of the proofs found in \cite{Rakhlin}.
\begin{proofof}{\Lemma{RakhlinDiameterBound}}
 Due to strong convexity and the fact that $f(x_t) - f(x^*) \geq 0,$ we have
\[  L \norm{x_t-x^*} \geq \norm{g_t}\norm{x_t - x^*} \geq \inner{g_t}{x_t - x^*} \geq \frac{\mu}{2}\norm{x_t - x^*}^2, \]where we used $L$-Lipschitzness of $f$ to bound $\norm{g_t}$ by $L.$

\end{proofof}

\begin{proofof}{\Lemma{RakhlinWBound}}
The definition of strong convexity yields
\[  \inner{g_t}{x_t - x^*} \geq f(x_t) - f(x^*) + \frac{\mu}{2}\norm{x_t - x^*}^2. \]Strong convexity and the fact that $0 \in \partial f(x^*)$ implies
\[  f(x_t) - f(x^*) \geq \frac{\mu}{2}\norm{x_t - x^*}^2. \]Next, recall that for any $x \in \mathcal{X},$ and for any $z$, we have $\norm{\Pi_{\mathcal{X}} (z) - x} \leq \norm{z - x}.$ Lastly, recall $\eta_t = \frac{2}{\mu (t+1)}.$ Using these, we have
\begin{align}
    \norm{x_{t+1}- x^*}^2
        &~=~ \norm{\Pi_{\mathcal{X}} (  x_t - \eta_t \hat{g}_t ) - x^*}^2 \nonumber\\
        &~\leq~ \norm{x_t - \eta_t \hat{g}_t - x^*}^2\nonumber \\
        &~=~ \norm{x_t - x^*}^2 -2\eta_t \inner{\hat{g}_t}{x_t - x^*} + \eta_t^2 \norm{\hat{g}_t}^2\nonumber \\
        &~=~ \norm{x_t - x^*}^2 - 2 \eta_t \inner{g_t}{x_t - x^*} + 2\eta_t \inner{\hat{z}_t}{x_t - x^*} + \eta_t^2 \norm{\hat{g}_t}^2\nonumber\\
        &~\leq~ \norm{x_t -x^*}^2 - 2\eta_t \left (f(x_t) - f(x^*)    \right ) - \eta_t \mu \norm{x_t - x^*}^2 \\& \qquad~+~ 2 \eta_t \inner{\hat{z}_t}{x_t - x^*}  + \eta_t^2 \norm{\hat{g}_t}^2 \nonumber\\
        &~\leq~ \left( 1 - 2\eta_t \mu \right ) \norm{x_t - x^*}^2 + 2 \eta_t \inner{\hat{z}_t}{x_t - x^*} + \eta_t^2 \norm{\hat{g}_t}^2 \nonumber\\
        &~=~ \EquationName{rakhlinEq} \left ( 1 - \frac{4}{t+1}\right )\norm{x_t - x^*}^2 + \frac{4}{\mu (t+1)} \inner{\hat{z}_t}{x_t - x^*} + \frac{4}{ \mu^2 (t+1)^2} \norm{\hat{g}_t}^2.
\end{align}Repeatedly applying \Equation{rakhlinEq} until $t=4$, yields the following
\begin{align}
    \norm{x_{t+1} - x^*}^2 &~\leq~ \frac{4}{\mu} \sum_{i=4}^t \left [ \frac{1}{i+1} \prod_{j=i+1}^t \left (  1 - \frac{4}{j+1}\right ) \right ] \cdot \inner{\hat{z}_i}{x_i - x^*}  \nonumber\\ &\quad+~\frac{4}{\mu^2} \sum_{i=4}^{t} \left [  \frac{1}{(i+1)^2}  \prod_{j=i+1}^t \left ( 1 - \frac{4}{j+1}\right ) \right ] \cdot \norm{\hat{g}_t}.
\end{align}Observing that
\[ \prod_{j=i+1}^t \left ( 1 - \frac{4}{j+1} \right ) ~=~ \prod_{j=i+1}^t\frac{j-3}{j+1} ~=~ \frac{ (i-2) \cdot (i-1)\cdot i \cdot (i+1)}{(t-2) \cdot (t-1)\cdot t \cdot (t+1)},   \]proves the lemma by taking $a_i(t) = \frac{1}{i+1} \cdot \frac{ (i-2)\cdot (i-1)\cdot i \cdot (i+1)}{(t-2) \cdot (t-1)\cdot t \cdot (t+1)}$ and $b_i(t)  = \frac{1}{(i+1)^2} \cdot \frac{(i-2) \cdot (i-1)\cdot i \cdot (i+1)}{(t-2) \cdot (t-1)\cdot t \cdot (t+1)}$
\end{proofof}
\section{Subgaussian noise extension}

The main result in this section is a strengthening of \Theorem{StochasticStrongPGD} by weakening the bounded noise assumption on the stochastic gradient oracle. First, we require a definition.

\begin{definition}
A random variable $X$ is said to be $\kappa$-subgaussian if $\expect{\exp \left ( X^2/\kappa^2 \right ) } \leq 2$.
In additional, we say that $X$ is $\kappa$-subgaussian conditioned on $\mathcal{F}$ if $\expect{\exp \left ( X^2/\kappa^2 \right ) } \leq 2$.
Note that $\kappa^2$ in this setting may itself be a random variable.
\end{definition}
\begin{remark}
Note that the class of subgaussian random variables contains bounded random variables. Furthermore, this class also contains Gaussian random variables (which, of course, are not bounded). Therefore, the following theorem is indeed a strengthening of \Theorem{StochasticStrongPGD}, which only dealt with stochastic gradient oracles that used almost surely bounded noise.
\end{remark}
\begin{theorem}
\TheoremName{StochasticStrongPGDSubGaussian}
Let $\mathcal{X} \subseteq \mathbb{R}^n$ be a convex set.
Suppose that $f : \mathcal{X} \rightarrow \mathbb{R}$ is $\mu$-strongly convex
(with respect to $\norm{\cdot}_2$) and $L$-Lipschitz.
Assume that:
\begin{enumerate}[label=(\emph{\alph*})]
\item $g_t \in \partial f(x_t)$ for all $t$ (with probability $1$).
\item $\norm{\hat{z}_t}$ is $\kappa$-subgaussian conditioned on $\mathcal{F}_{t-1}$ for some $\kappa \in \mathbb{R}$.
\end{enumerate}
Set $\eta_t = \frac{2}{\mu (t+1)}$.
Let $\gamma_t = \frac{t}{T(T+1)/2}$.
Then, for any $\delta \in (0,1)$, with probability at least $1-\delta$ we have,
$$
f\left( \sum_{t=1}^T \gamma_t x_t \right) - f(x^*)
    ~\leq~  O \left (\frac{(L+\kappa)^2}{ \mu}  \cdot \frac{\log \left (1 / \delta \right )}{T}  \right ).
$$
\end{theorem}

\begin{proofof}{\Theorem{StochasticStrongPGDSubGaussian}}

We may follow the proof of \Theorem{StochasticStrongPGD} from \Section{ub} and remove any bound used on $\norm{\hat{g}_t}^2$ to obtain
\begin{equation}
\EquationName{subGaussianNoiseMain}
    f\left (\sum_{t=1}^T \gamma_t x_t    \right ) - f(x^*) ~\leq~ \frac{2}{T(T+1)} \underbrace{\sum_{t=1}^T t \cdot \inner{\hat{z}_t}{x_t - x^*}}_{:= Z_T}  + \frac{2}{\mu T (T+1)} \sum_{t=1}^T\norm{\hat{g}_t}^2.
\end{equation}
\Theorem{StochasticStrongPGDSubGaussian} follows trivially from the following two lemmata:
\begin{lemma}
\LemmaName{subExpSum}
For any $\delta \in (0,1)$, $\sum_{t=1}^T \norm{\hat{g}_t}^2 = O \left ( \left (L + \kappa \right )^2 T \cdot \log (1/\delta)\right )$ with probability at least $1-\delta.$
\end{lemma}
\begin{lemma}
\LemmaName{boundZsubgaussian}
Let $Z_T = \sum_{t=1}^T t \cdot \inner{\hat{z}_t}{x_t - x^*}.$ Then, for any $\delta \in (0,1)$, we have $ Z_T = O \left (\frac{(L + \kappa)^2}{\mu} T\cdot  \log(1/\delta)\right )$ with probability at least $1 -\delta.$
\end{lemma}
\end{proofof}

\subsection{Proof of \Lemma{subExpSum}}

We begin with a fact about subgaussian random variables.

\begin{claim}
\ClaimName{psi2norm}
Let X be a random variable. Define $\norm{X}_{\psi_2}$ as $\inf \setst{ t >0 }{ \expect{ \exp \left ( X^2 / t^2 \right )} \leq 2 }.$ Then, $\norm{\cdot }_{\psi_2}$ is a norm.
\end{claim}

Observe that $X$ is $\kappa$-subgaussian if and only if $\norm{X}_{\psi_2} \leq \kappa$. As a consequence of \Claim{psi2norm}, we have the following claim.

\begin{claim}
\ClaimName{Gpsi2}
There exists $\xi = O(L + \kappa)$, such that  $\norm{\hat{g}_t}$ is $\xi$-subgaussian conditioned on $\mathcal{F}_{t-1}$.
\end{claim}

\begin{proof}
Using the triangle inequality, we have
\[  \norm{\hat{g}_t} ~=~ \norm{g_t -\hat{z}_t } ~\leq~ \norm{g}_t + \norm{\hat{z}_t}.  \]Therefore,
\[ \norm{\norm{\hat{g}_t} \mid  \mathcal{F}_{t-1}}_{\psi_2} ~\leq~ \norm{  \norm{g_t} \mid \mathcal{F}_{t-1}}_{\psi_2}  + \norm{\norm{\hat{z}_t}\mid \mathcal{F}_{t-1}}_{\psi_2} ~\leq~  \norm{  \norm{g}_t \mid \mathcal{F}_{t-1
}}_{\psi_2} + k, \]
because we assumed $\norm{\hat{z}_t}$ is conditionally $\kappa$-subgaussian. Also, note that $\norm{g_t}$ is conditionally $(L/\ln 2)$-subgaussian because $f$ is $L$-Lipschitz and so $\norm{g_t} \leq L$.
\end{proof}
Now, we proceed to prove \Lemma{subExpSum} using an MGF bound:

\begin{claim}There exists $\xi = O\left ( L + \kappa \right )$ such that
\ClaimName{MGFBound}
$\expect{\exp \left ( \sum_{i=1}^T \norm{\hat{g}_i}^2  / \left (T \cdot \xi^2 \right ) \right ) } \leq 2.$
\end{claim}

Using \Claim{MGFBound} we can prove \Lemma{subExpSum}:

\begin{proofof}{\Lemma{subExpSum}}
Using the exponentiated Markov inequality we have for any $\lambda > 0$:
\[ \prob{  \sum_{t=1}^T  \norm{\hat{g}_t}^2 \geq x   } ~\leq~  \frac{\expect{  \exp \left ( \lambda \sum_{t=1}^T  \norm{\hat{g}_t}^2  \right )  }}{\exp \left ( \lambda x\right) }.  \]Plugging in $\lambda  = O \left ( \frac{1}{T \cdot\xi^2} \right )$ and $x = O \left (T \cdot\xi^2 \log (1/\delta) \right)$ completes the proof.
\end{proofof}

It remains to prove \Claim{MGFBound}.

\begin{proofof}{\Claim{MGFBound}}
We will show that for every $1 \leq t \leq T,$
\begin{align}
\EquationName{Intermediate1}
    \expect{ \exp \left (  \sum_{i=1}^t \norm{\hat{g}_i}^2/(T \cdot \xi^2 ) \right )  } ~\leq~ 2^{1/T} \expect{ \exp \left (  \sum_{i=1}^{t-1} \norm{\hat{g}_i}^2/(T \cdot \xi^2 ) \right )  }.
\end{align}
Indeed,
\begin{align*}
    &\expect{ \exp \left (  \sum_{i=1}^t \norm{\hat{g}_i}^2/(T \cdot \xi^2 ) \right )  }  \\
    &=~  \expect{\exp \left (\sum_{i=1}^{t-1} \norm{\hat{g}_i}^2/(T \cdot \xi^2 ) \right ) \expectg{\exp \left ( \norm{\hat{g}_t}^2/(T \cdot \xi^2 ) \right ) }{\mathcal{F}_{t-1}}    }.
\end{align*}Furthermore,
\[  \expectg{\exp \left ( \norm{\hat{g}_t}^2/(T \cdot \xi^2 ) \right ) }{\mathcal{F}_{t-1}} \leq 2^{1/T} \] for all $1 \leq t \leq T$ using \Claim{Gpsi2} and Jensen's inequality. Therefore, \Equation{Intermediate1} is true for all $1\leq t \leq T$.

Hence,
\[\expect{ \exp \left (  \sum_{i=1}^t \norm{\hat{g}_i}^2/(T \cdot \xi^2 ) \right )  } ~\leq~ \left ( 2^{1/T}\right )^T ~=~ 2, \]as desired.
\end{proofof}

\subsection{Proof of \Lemma{boundZsubgaussian}}

We may follow the proof of \Lemma{StronglyConvexSGDBound1} from \Subsection{BoundingZ}. Define $d_t = t \cdot \inner{\hat{z}_t}{x_t -x^*},$ $
\tilde{v}_{t-1} := 2\kappa^2 \cdot t^2 \norm{x_t-x^*}^2,$ and $\tilde{V}_T = \sum_{t=1}^T \tilde{v}_{t-1}.$ Note that $\tilde{v}_{t-1}$ is $\mathcal{F}_{t-1}$-measurable.

\begin{claim}
\ClaimName{subGaussianMGFforFan}
For all $t$ and $\lambda > 0,$ we have
\[  \expectg{\exp \left (\lambda d_t    \right )}{\mathcal{F}_{t-1}} \leq \exp \left ( \frac{\lambda^2}{2} \tilde{v}_{t-1} \right ). \]
\end{claim}

The proof of this requires a lemma from \cite{Ver18}.

\begin{lemma}[{\cite[Proposition 2.5.2]{Ver18}}]
\LemmaName{MeanZeroMGFTrick}
Suppose $X$ is a mean-zero random variable such that  $\expect{\exp \left (  X/\kappa^2 \right )} \leq 2.$ Then, $\expect{\exp(\lambda X )} \leq \exp \left ( \lambda^2 \kappa^2 \right )$ for all $\lambda > 0$.
\end{lemma}

\begin{proofof}{\Claim{subGaussianMGFforFan}}
Because $\norm{\hat{z}_t}$ is $\kappa$-subgaussian conditioned on $\mathcal{F}_{t-1},$ we have by Cauchy-Schwarz
\[  \expectg{ \exp \left ( \frac{t^2 \cdot \inner{ \hat{z}_t}{x_t- x^*}^2}{\kappa^2 \cdot t^2 \norm{x_t - x^*}^2} \right ) }{\mathcal{F}_{t-1}} ~\leq~ \expectg{ \exp \left( \norm{\hat{z}_t}^2 / \kappa^2 \right ) }{\mathcal{F}_{t-1}} ~\leq~ 2.  \]Therefore, by \Lemma{MeanZeroMGFTrick} we have
\[  \expectg{\exp \left ( \lambda \right )}{\mathcal{F}_{t-1}} ~\leq~ \exp \left ( \frac{\lambda^2}{2} (2\kappa^2 \cdot t^2 \norm{x_t - x^*}^2 \right ), \]as desired.
\end{proofof}

To bound $Z_T,$ we will proceed similarly as in \Subsection{BoundingZ}. We will bound the TCV of $Z_T$ by a linear combination of the increments. The only difference is, we will show that this bound holds with high probability, instead of with probability one. This will allow us to use a form of the Generalized Freedman Inequality (\Theorem{FanV2.1}) which the case where we can bound the total conditional variance by a linear transformation of the increments of the martingale with high probability.

\begin{lemma}
\LemmaName{ChickenAndEggSubGaussian}
There exists non-negative constants $\alpha_1, \ldots, \alpha_T = O\left((L +\kappa)^2\frac{T}{\mu}\right)$ and $\beta = O \left( \frac{(L+\kappa)^4}{\mu^2}T^2 \right )$ such that for every $\delta \in (0,1)$,
$\tilde{V}_T \leq \sum_{t=1}^T\alpha_t d_t + \beta \log(1/\delta)$ with probability at least $1 - \delta$.
\end{lemma}

Given \Lemma{ChickenAndEggSubGaussian}, we are ready to prove \Lemma{boundZsubgaussian}. But first, we require a slightly more general version of the Generalized Freedman Inequality where the bound on the TCV by a linear transformation of the increments of the martingale holds only with arbitrarily high probability, rather than with probability 1.

\begin{theorem}[Generalized Freedman, {\cite[Theorem~3.3]{HLPR18}}]
\TheoremName{FanV2.1}
Let $\{ d_t, \mathcal{F}_t \}_{t=1}^T$ be a martingale difference sequence.
Suppose that, for $t \in [T]$, $v_{t-1}$ are non-negative  $\mathcal{F}_{t-1}$-measurable random variables satisfying $\expectg{\exp(\lambda d_t)}{\mathcal{F}_{t-1}} \leq \exp\left(\frac{\lambda^2}{2} v_{t-1}\right)$ for all $\lambda > 0$.
Let $S_T = \sum_{t=1}^T d_t$ and $V_T = \sum_{t=1}^T v_{t-1}$.
Suppose there exists $\alpha_1, \ldots, \alpha_T, \beta \in \mathbb{R}_{\geq 0}$ such that for every $\delta \in (0,1)$, $V_T \leq \sum_{t=1}^T \alpha_t d_t + \beta \log(1/\delta)$.
Let $\alpha \geq \max_{t \in [T]} \alpha_t$.
Then
\[
\prob{S_T \geq x} ~\leq~ \exp\left( -\frac{x^2}{4\alpha\cdot x + 8\beta } \right) + \delta.
\]

\end{theorem}

\begin{proofof}{\Lemma{boundZsubgaussian}}
By \Claim{subGaussianMGFforFan} we have $\expectg{\exp \left ( \lambda d_t \right )}{\mathcal{F}_{t-1}} \leq \exp\left ( \frac{\lambda^2}{2} \tilde{v}_{t-1} \right ).$ By \Lemma{ChickenAndEggSubGaussian} we have that for every $\delta \in (0,1)$ $\tilde{V}_{T}\leq  \sum_{t=1}^T\alpha_t d_t + \beta \log(1/\delta),$ with probability at least $1 -\delta.$ Plugging $\alpha = O \left ((L+\kappa)^2\frac{T}{\mu} \right),$ $\beta = O\left ( \frac{ (L+\kappa)^4}{\mu^2} T^2 \log(1/\delta) \right )$ and $x = O \left ( \frac{(L+\kappa)^2}{\mu} \cdot T \log(1/\delta) \right)$ into \Theorem{FanV2.1}, proves \Lemma{boundZsubgaussian}.
\end{proofof}

It remains to prove \Lemma{ChickenAndEggSubGaussian}.

\begin{proofof}{\Lemma{ChickenAndEggSubGaussian}}
Observe that $\tilde{V}_T = 2\kappa^2 V_T = \sum_{t=1}^T t^2 \cdot \norm{x_t - x^*}^2$ where $V_T$ was defined in \Subsection{BoundingZ}. We focus our attention on bounding $V_T$, and then scale up accordingly at the end.

We may follow the proof of \Lemma{BoundingB} with a key modification: Do not bound $\norm{\hat{g}_i}$ by $L+1$ as this is no longer valid, because we no longer are using the bounded noise assumption.

This yields:

\begin{align}
\EquationName{VTSubgaussian}
    &V_T \\
    &~\leq~ \sum_{i=3}^{T-1}  \underbrace{\frac{4}{\mu}\left( \sum_{t=i+1}^T t^2 \frac{a_i(t-1)}{i}   \right )}_{\coloneqq \alpha_i} \cdot i \cdot \inner{\hat{z}_i}{x_i - x^*} + \frac{4}{\mu^2}\underbrace{\sum_{t=4}^T t^2 \left (  \sum_{i=3}^{t-1}b_i(t-1)  \norm{\hat{g}_i}^2\right )
    }_{\coloneqq G_T} + \frac{56L^2}{\mu^2}.
\end{align}Define $\alpha_1,\alpha_2, \alpha_T = 0.$ We already showed in the proof of \Lemma{BoundingB} that $\alpha_i = O\left( \frac{T}{\mu} \right).$ Therefore, it remains to bound $G_T$ by $O\left ( (L + \kappa)^2T^2 \cdot \log(1/\delta) \right ),$ with probability at least $1-\delta.
$ We rewrite $G_T$ as
\[ G_T ~=~ \sum_{i=3}^{T-1} \underbrace{\left ( \sum_{t=i+1}^T t^2 b_i(t-1) \right )}_{\coloneqq s_i}\cdot \norm{\hat{g}_i}^2 ~=~ \sum_{i=3}^{T-1} s_i \norm{\hat{g}_i}^2 .  \]

We use the following MGF bound on $G_T$, which we prove below.

\begin{claim}
\ClaimName{GTMGFBound}
$\expect{\exp \left ( \lambda G_T \right )} \leq \exp \left ( \lambda O\left (\xi^2\right) \sum_{i=3}^{T-1 }s_i   \right )$ for all $\lambda = O \left( \frac{1}{\xi^2 \max \set{s_i}} \right)$.
\end{claim}
Therefore, via an exponentiated Markov inequality and \Claim{GTMGFBound}, we have
\[  \prob{G_T \geq x} ~\leq~ \frac{\expect{\exp \left ( \lambda G_T \right )}}{\exp\left ( \lambda x \right )} ~\leq~ \exp \left (   \lambda O\left (\xi^2\right )\sum_{i=3}^{T-1} s_i - \lambda x \right ).\]
Setting $\lambda = O \left ( \frac{1}{\xi^2 \sum_{i=3}^{T-1}s_i}  \right )$ and $x = O \left(\xi^2 \sum_{i=3}^{T-1}s_i \cdot \log(1/\delta)\right )$ shows $G_T \leq  O \left(\xi^2 \sum_{i=3}^{T-1}s_i \cdot \log(1/\delta)\right )$ with probability at least $1-\delta.$ Observe that $s_i = O(T):$
\[  s_i ~=~ \sum_{t=i+1}^T t^2 b_i(t-1) ~=~ \sum_{t=i+1}^T t^2 O \left ( \frac{i^2}{t^4} \right ) ~=~ \sum_{t=i+1}^T O\left(1\right) ~=~ O\left (T\right). \]
Therefore $x = O \left ( \xi^2 T^2 \log(1/\delta) \right ) = O\left(  (L + \kappa)^2 T^2 \log(1/\delta)\right ),$ with probability at least $1-\delta.$ That is, with probability at least $1-\delta$
\[  G_T \leq O \left ( \left ( L + \kappa \right )^2 T^2 \log (1/\delta) \right ). \]

Plugging this back in to \Equation{VTSubgaussian}, we obtain
\[  V_T ~\leq~ \sum_{i=1}^{T} \alpha_i d_i + \beta \log(1/\delta)  \]where $\alpha_i = O( \frac{T}{\mu})$ and $\beta = O \left (   \frac{(L + \kappa)^2}{\mu^2} T^2 \cdot \log(1/\delta) \right )$. Multiplying both sides by $2\kappa^2$ yields the desired bound on $\tilde{V}_T.$

\end{proofof}

Now it remains to prove \Claim{GTMGFBound}.
\begin{proofof}{\Claim{GTMGFBound}}
We will show that for every $t$ and for all $\lambda \leq 1/\max \set{s_i}$,
\[  \expect{\exp \left ( \lambda \sum_{i=3}^t s_i \norm{\hat{g}_i}^2  \right) } \leq \exp \left ( \lambda O \left ( \xi^2\right )s_t \right)\expect{\exp\left (\lambda \sum_{i=3}^{t-1} s_i \right ) }. \]The claim then follows by recursively applying the above inequality. Note that $s_i$ is $\mathcal{F}_{i-1}$ measurable. So, we have
\begin{align*}
    \expect{\exp \left ( \lambda \sum_{i=3}^t s_i \norm{\hat{g}_i}^2  \right) }
        &~=~ \expect{\exp \left ( \lambda\sum_{i=3}^{t-1} s_i \norm{\hat{g}_i}^2  \right ) \expectg{\exp\left (\lambda s_t \norm{\hat{g}_t}^2  \right )}{\mathcal{F}_{t-1}}}.
\end{align*}Note that because $\norm{ \norm{\hat{g}_t} \mid \mathcal{F}_{t-1}}_{\psi_2} \leq \xi,$ this implies
\[  \expectg{\exp\left (\norm{\hat{g}_t}^2/\xi^2\right )}{\mathcal{F}_{t-1}} \leq 2. \]Therefore, if $\lambda = O \left ( \frac{1}{\xi^2} \right ).$ Then by Jensens inequality, raising both sides of the above inequality to the power of $\lambda \xi^2$ yields
\[  \expectg{ \exp \left ( \lambda  \norm{\hat{g}_t}^2 \right ) }{\mathcal{F}_{t-1}}  \leq \exp \left ( \lambda O \left ( \xi^2\right ) \right ).\] Hence, if $\lambda = O\left ( \frac{1}{\xi^2 \max\set{s_i}} \right ),$ then for every $t$ we have
\[ \expectg{\exp\left (\lambda s_t \norm{\hat{g}_t}^2  \right )}{\mathcal{F}_{t-1}} \leq \exp \left (\lambda O \left ( \xi^2\right )s_t \right ),   \]which completes the proof.
\end{proofof}
\section{Experimental results}
\SectionName{experimental_results}
The four return strategies discussed in this paper have fairly similar theoretical guarantees. The aim of this section is to compare the strategies on real data sets, focusing on two aspects of their performance: the expectation and the concentration of the objective value. The results are shown in Figure~\ref{fig:iterates_plot_cina0_protein}. Additional experimental results can be found in the supplementary material (\Section{additional_experiments}).

The results of the experiments reveal a clear message. The final iterate and the uniform average return strategies perform noticeably worse than the suffix average and non-uniform average, both in terms of expectation and concentration. This is consistent with the fact that their theoretical guarantees are also worse. The performance of the suffix average and the non-uniform average are nearly indistinguishable, with the suffix average having a slight advantage in expectation.

\paragraph{Methodology. } We consider the regularized SVM optimization problem
\[  f(w) ~\coloneqq~ \frac{\lambda}{2} \norm{w}^2 +  \sum_{i=1}^m \max \set{ 0, 1- y_i w\transpose x_i }, \]where $m$ is the number of data points and we use $n$ to denote the dimension of each data point. We run SGD with step size $\eta_t = \frac{2}{t+1}$ and with regularization parameter $\lambda = 1/m$. This particular step size is required for \Theorem{StochasticStrongPGD}, and the analyses for the other averaging schemes can also accommodate this choice of step size. Furthermore, we found that the relative performance of the different averaging schemes is not particularly sensitive to the choice in step size. We plot the value of $f$ for each return strategy every $m$ iterations (which we refer to as an `effective pass'). Since the output of SGD is random, there is a distribution over the outputs which we would like to capture. We run 1000 trials of SGD. The colored curves are exactly these 1000 trials, which are plotted with low opacity. At any point in time, the darkness of the plot at a specific objective value indicates the number of trials that achieved that value at that time. The dotted dark lines represent the average amongst the trials.

Figure~\ref{fig:iterates_plot_cina0_protein} suggests that practitioners should consider using the suffix average or non-uniform average in lieu of the final iterate or uniform average. It is possible to implement suffix averaging and non-uniform averaging with minimal effort, and the performance boost is significant. Implementing non-uniform averaging (even when the time horizon is not fixed ahead of time) only requires a single additional line of code.

\paragraph{Data sets.} We performed our experiments on a set of freely available binary classification data sets. The experiments from this section use the \emph{cina0} ($n = 16033$ and $d = 132$) and the \emph{protein} ($m= 145751$ and $n = 74$) data sets. We ran the same experiments on the \emph{rcv1} ($m=20242$ and $n = 47236$), \emph{covtype} ($m = 581,012$ and $n=54$) and \emph{quantum} ($m=50000$ and $n=78)$ data sets. The results for these data sets can be found in \Section{additional_experiments}. Sparse features were scaled to $[0,1]$ whereas dense features were scaled to have zero mean and unit variance. Data sets \emph{quantum} and \emph{protein} can be found at the \href{http://osmot.cs.cornell.edu/kddcup/datasets.html}{\color{blue}{KDD cup 2004 website}}, \emph{cina0} can be found at the \href{http://www.causality.inf.ethz.ch/home.php}{\color{blue}{Causality Workbench website}} and \emph{covtype} and \emph{rcv1} can be found at the \href{https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/}{\color{blue}{LIBSVM website}}.

\clearpage
\begin{landscape}
\begin{figure}[h]
    \centering
    \begin{subfigure}[b]{\paperwidth}
        \centering
        \includegraphics[width=\paperwidth]{figures/cina_iterates.png}
    \caption{cina0}
    \end{subfigure} \\
    \begin{subfigure}[b]{\paperwidth}
        \centering
        \includegraphics[width=\paperwidth]{figures/protein_iterates.png}
    \caption{protein}
    \end{subfigure}
    \caption{Number of effective passes vs. objective value. The first row plots the results using the \emph{cina0} data set, whereas the second plots results using the \emph{protein} data set. From left to right, we plot the objective value over time of the final iterate, uniform average, suffix average and non-uniform average for 1000 trials of SGD.}
    \label{fig:iterates_plot_cina0_protein}
\end{figure}
\end{landscape}
\section{Introduction}

Stochastic gradient descent (SGD) is perhaps the single most important algorithm for minimizing strongly convex loss functions. Its popularity is a combined consequence of the simplicity of its statement and its effectiveness in both theory and practice. Gradient descent is an iterative optimization procedure, where the current solution is updated by taking a step in the opposite direction of the current gradient. In the case of using SGD for Empirical Risk Minimization, the gradient of the loss function is often too expensive to compute. So instead, we select a data point uniformly at random and compute the gradient of the loss function using only this single data point. The resulting value is not necessarily a true gradient, but it is in expectation. When it is time to output a solution, the standard textbook choices are either to report the last iterate, or the average of iterates so far.

Surprisingly, there are situations where these textbook output strategies have provably sub-optimal performance, even though the algorithm is so popular. Here, by performance we are referring to the rate at which the loss of the output convergences to the true minimum value of the loss function. Consider the setting where the loss function is strongly convex, but not smooth (for example, the regularized SVM minimization problem). In the absence of smoothness, there is no guarantee that the value of the individual iterates of SGD improve over time. In fact, \citet{HLPR18} construct an example where the value of the iterates \emph{increases} over time.
Moreover, they show that the convergence rate of the individual iterates of even deterministic gradient descent is $\Omega(\log(T)/T),$ whereas the optimal rate is $\Theta(1/T)$ for a first-order algorithm. \citet{Rakhlin} show that returning the average of all of the iterates so far is also sub-optimal by a $\log(T)$ factor (this lower bound holds in expectation).

As a result, researchers have developed several algorithms which achieve the optimal $O(1/T)$ rate in expectation, some of which are simpler than others.
Because the popularity of SGD is largely due to its simplicity, a straightforward variant of the algorithm attaining the optimal rate is significantly more desirable than some other, more complex procedure.
The non-uniform averaging strategy from \cite{NonUniformAvg} and the suffix-averaging strategy from \cite{Rakhlin} are likely the simplest and closest in resemblance to textbook statements of SGD. Each method runs standard SGD until output time. In \cite{NonUniformAvg}, a non-uniform average over all the iterates (with iterate $i$ given a weight proportional to $i$) is returned whereas in \cite{Rakhlin}, a uniform average over the last half of the iterates (referred to as the suffix-average) is returned. There still remains another important issue to resolve though, even provided the existence of simple algorithms which obtain the optimal expected $O(1/T)$ rate.

How many random trials are needed for suffix averaging or the non-uniform averaging strategies to actually achieve the $O(1/T)$ rate? (Recall that the error is random). Usual expositions of SGD provide bounds that hold in \emph{expectation}. This is a weak guarantee because it does not preclude the error of the algorithm from having large variance. Users of SGD want to be confident that the output of a single trial of the algorithm is extremely likely to provide the guaranteed convergence rate. In other words, they would prefer bounds that hold with high probability. Moreover, it is often impossible to run many trials of SGD and select the best one.
For example, considering the Empirical Risk Minimization setting, if we are dealing with many high dimensional data points, it can be prohibitively expensive to even evaluate the loss function.

It was shown recently in \cite{HLPR18}, that suffix-averaging obtains a convergence rate of $O(\log(1/\delta)/T)$ with probability at least $1-\delta.$  However, implementing suffix-averaging when the time horizon is not known ahead of time (for example, stopping SGD when the norm of the gradient is small) requires a modicum of care. Non uniform averaging could be an equally attractive alternative if its convergence rate held with high probability.

\paragraph{Main theoretical results. }We show that running standard SGD and returning the very simple non-uniform average of the iterates from \cite{NonUniformAvg} has error at most $O(\log(1/\delta)/T)$ with probability $1-\delta$. The analysis is simple and exposes a martingale which satisfies a special recursive property which was also observed in \cite{HLPR18}. It is intriguing that this recursive property arises in multiple settings when analyzing SGD. Moreover, we show a matching lower bound of $\Omega(\log(1/\delta)/T)$ with probability at least $\delta.$ The analysis uses the simple univariate function $\frac{1}{2}x^2$.
Thus, we have a tight high probability analysis of a very simple output strategy for SGD which attains the optimal rate.

\paragraph{Experimental results. }In addition, we run detailed experiments for various return schemes of SGD for SVMs on synthetic data and real-world datasets.
Our experimental results strongly suggest that
the suffix averaging and non-uniform averaging schemes
should be preferred over the final iterate and uniform averaging schemes.

\subsection{Related Work}
\SubsectionName{RelatedWork}
There are a number of other algorithms which obtain the optimal $O(1/T)$ convergence rate amongst first-order methods for minimizing a non-smooth, strongly-convex function.
\citet{HK14} proved that a variant of SGD, called Epoch-GD obtains the optimal rate.
Here, they partition the total time $T$ into exponentially growing epochs.
Within each epoch, they run standard SGD (with an appropriate step size) and after the end of each epoch, they reset the current point to the average of the iterates in the previous epoch.

Later, \citet{Rakhlin} and \citet{NonUniformAvg} independently discovered simpler algorithms that also achieve the optimal $O(1/T)$ rate.
In fact, both algorithms run standard SGD with the standard step size proportional to $1/t$; the only difference between the two algorithms is the return value of the algorithm.
In \citet{Rakhlin}, they show that suffix averaging, where one returns the last $\alpha$ fraction of the iterates (for some constant $\alpha > 0$), achieves the optimal rate.
On the other hand, \citet{NonUniformAvg} prove that a certain non-uniform average (see \Algorithm{StrongLipschitzPGD}) of the iterates also achieves the optimal rate.
One advantage of non-uniform averaging is that the iterates can be easily computed on the fly.

Recently, \cite{NesterovShikhman} have devised a modification of gradient descent for which the error of the \emph{last iterate} converges at the optimal rate.
Even more recently, \citet{jain2019making} showed that even for unmodified gradient descent the last iterate can be made to achieve the optimal rate,
if the time horizon is known beforehand, and if the step-size is chosen carefully using the time horizon.
Interestingly, they also show that knowing (or having a bound on) the time horizon is necessary for all the individual iterates to achieve the optimal rate.

\paragraph{High-probability upper bounds.}
All the results stated above hold only in expectation and do not rule out the possibility that the return value has high variance.
Moreover, it can be expensive to compute the objective value of a point.
Hence, it is desirable to have a high-probability upper bound on the return value.

To assist in this task, \citet{HLPR18} recently developed a generalization of Freedman's Inequality.
Using this, they show that if one runs SGD with the standard $1/t$ step sizes, then the last iterate and the suffix average schemes achieve error $O(\log(T)/T)$, and $O(1/T)$, respectively, with high probability.
Using similar methods, \citet{jain2019making} prove that the last iterate of SGD with carefully chosen step sizes achieves an error of $O(1/T)$.
(As mentioned above, this requires advance knowledge of $T$.)
The uniform average was earlier shown by \cite{KT08} to achieve error $O(\log(T)/T)$ with high probability.
Here, we will also employ the generalized Freedman's Inequality to prove a tight high-probability upper bound on the non-uniform averaging scheme.
\section{Description of high probability lower bound}
\SectionName{LB}

\noindent\textbf{Setup of the lower bound. } Consider the one dimensional, $1$-strongly convex function $f(x) = \frac{1}{2}x^2$ with feasible region $\mathcal{X} = [-6,6]$. Suppose, that at any point $x_t$, the gradient oracle returns a value of the form $x_t - \hat{z}_t,$ where $\expect{\hat{z}_t} = 0.$ Clearly, this is a valid subgradient oracle. Suppose we run \Algorithm{StrongLipschitzPGD} with a \emph{slightly modified step size of} $\eta_t = \frac{1}{t+1}$ starting from initial point $x_1 = 0$.

\begin{remark}
Note that we are using a step size of $\frac{1}{t+1}$ instead of the step size $\frac{2}{t+1}$ used in the statement of \Algorithm{StrongLipschitzPGD}. It is possible to modify the analysis to use the step size as stated in \Algorithm{StrongLipschitzPGD}, however the analysis is much cleaner using $\frac{1}{t+1}$ and still captures the main ideas.
\end{remark}

\begin{claim}
\ClaimName{lowerBoundIterates}
Suppose $x_1 = 0$ and assume $\Abs{\hat{z}_t} \leq 6$. Then, $x_t = \frac{1}{t}\sum_{i=1}^{t-1} \hat{z}_i$ for all $2 \leq t \leq T$.
\end{claim}

\noindent\textbf{Definition of gradient oracle.} Let $\hat{z}_t = 0$ if $t \leq \frac{T}{2}$ or $T > \frac{3T}{4}$ and otherwise for $ T/2 +1  \leq t \leq \frac{3T}{4}$, define $\hat{z}_t = \frac{T+1}{T-t} X_t$ where $X_t$ is uniform in $\set{+1,-1}.$ Note that this gradient oracle satisfies the conditions of \Claim{lowerBoundIterates}. That is $\abs{\hat{z}_t} \leq 6$ for all $t,$ as long as $T \geq 2.$

By definition of $\hat{z}_t$ and \Claim{lowerBoundIterates}, one can check that $\sum_{i=1}^T \gamma_t x_t$ is an average of Bernoulli random variables. Applying a reverse Chernoff bound from \cite{RevChernoff} completes the proof.
The complete details can be found in the supplementary materials.

\section{Preliminaries}
Let $\mathcal{X}$ be a closed, convex subset of $\mathbb{R}^n$, $f \colon \mathcal{X} \to \mathbb{R}$ be a convex function, and $\partial f(x)$ be the subdifferential of $f$ at $x$.
Our goal is to solve the convex program $\min_{x \in \mathcal{X}} f(x)$.
We assume that $f$ may not be explicitly represented.
Instead, the algorithm is allowed to query $f$ via a stochastic gradient oracle, i.e.,~if the oracle is queried at $x$ then it returns $\hat{g} = g - \hat{z}$ where $g \in \partial f(x)$ and $\expect{\hat{z}} = 0$ conditioned on all past calls to the oracle.
Furthermore, we assume that $f$ is $L$-Lipschitz, i.e.~$\norm{g} \leq L$ for all $x \in \mathcal{X}$ and $g \in \partial f(x)$ and that $f$ is $\mu$-strongly convex, i.e.
\begin{equation}
\EquationName{strongly_convex_def}
f(y) ~\geq~ f(x) + \inner{g}{y-x} + \frac{\mu}{2}\norm{y - x}^2 \quad \forall y, x \in \mathcal{X}, g \in \partial f(x).
\end{equation}
Throughout this paper, $\norm{\cdot}$ denotes the \emph{Euclidean} norm in $\mathbb{R}^n$, $\Pi_{\mathcal{X}}$ denotes the projection operator onto $\mathcal{X}$ and $[T]$ denotes the set $\set{1, \ldots, T}$.
For the sake of simplicity, we assume that $\norm{\hat{z}} \leq 1$ a.s.

In this paper, we analyze SGD with the averaging scheme proposed by \citet{NonUniformAvg}.
The algorithm is given in \Algorithm{StrongLipschitzPGD}.

\begin{algorithm}
\caption{Stochastic, projected gradient descent for minimizing a $\mu$-strongly convex, $L$-Lipschitz
function with an unknown time horizon.}
\AlgorithmName{StrongLipschitzPGD}
\begin{algorithmic}[1]
\Myprocedure{ProjectedGradientDescent}{$\mathcal{X} \subseteq \mathbb{R}^n$,\, $x_1 \in \mathcal{X}$}
\MyFor{$t \leftarrow 1,\ldots,T$}
    \State Let $\eta_t = \frac{2}{\mu (t+1)}$
    \State $y_{t+1} \leftarrow x_t - \eta_t \hat{g}_t$,
    where $\expectg{\hat{g}_t}{\hat{g}_1, \ldots, \hat{g}_{t-1}} \in \partial f(x_t)$
    \State $x_{t+1} \leftarrow \Pi_\mathcal{X}(y_{t+1})$
\MyEndFor
\State \textbf{return} $\sum_{t=1}^T \frac{t}{T(T+1)/2} x_t $
\EndMyprocedure
\end{algorithmic}
\end{algorithm}

Finally, we will use $\mathcal{F}_t$ to denote the $\sigma$-field generated by
the random vectors $\hat{g}_1, \ldots, \hat{g}_t$.

\begin{remark}
    As noted in \cite{NonUniformAvg}, the return value of \Algorithm{StrongLipschitzPGD} can be computed in an online manner.
    Indeed, we can set $z_1 = x_1$, and we can set $z_t = \rho_t x_t + (1-\rho_t)z_{t-1}$  for $t \geq 2$, where $\rho_t = \frac{2}{t+1}$.
    It is a straightforward calculation to check that $z_T = \sum_{t=1}^T \frac{t}{T(T+1)/2} x_t$.
\end{remark}

\subsection{Probability tools}
Our main probabilistic tool is an extension of Freedman's Inequality \citep{F75} developed recently by \citet{HLPR18}.
Roughly speaking, Freedman's Inequality asserts that a martingale is bounded by the square root of its total conditional variance (TCV).
As we shall see in the sequel, the martingales that arise from analyzing SGD exhibit a ``chicken-and-egg'' phenomenon wherein the TCV of the martingale is bounded by (a linear transformation of) the martingale itself.
Here, we state a specialized form of the Generalized Freedman's Inequality which is a simple corollary from the statement given in \cite{HLPR18}.
\newcommand{\fantwo}{
Let $\{ d_t, \mathcal{F}_t \}_{t=1}^T$ be a martingale difference sequence.
Suppose that, for $t \in [T]$, $v_{t-1}$ are non-negative  $\mathcal{F}_{t-1}$-measurable random variables satisfying $\expectg{\exp(\lambda d_t)}{\mathcal{F}_{t-1}} \leq \exp\left(\frac{\lambda^2}{2} v_{t-1}\right)$ for all $\lambda > 0$.
Let $S_T = \sum_{t=1}^T d_t$ and $V_T = \sum_{t=1}^T v_{t-1}$.
Suppose there exists $\alpha_1, \ldots, \alpha_T, \beta \in \mathbb{R}_{\geq 0}$ such that $V_T \leq \sum_{t=1}^T \alpha_t d_t + \beta$.
Let $\alpha \geq \max_{t \in [T]} \alpha_t$.
Then
\[
\prob{S_T \geq x} ~\leq~ \exp\left( -\frac{x^2}{4\alpha\cdot x + 8\beta } \right).
\]
}

\begin{theorem}[Generalized Freedman, {\cite[Theorem~3.3]{HLPR18}}]
\TheoremName{FanV2}
\fantwo
\end{theorem}
\section{Main results}

Our main result is a high-probability upper bound on the final iterate of \Algorithm{StrongLipschitzPGD}. The proof is given in \Section{ub}.

\begin{theorem}
\TheoremName{StochasticStrongPGD}
Let $\mathcal{X} \subseteq \mathbb{R}^n$ be a convex set.
Suppose that $f : \mathcal{X} \rightarrow \mathbb{R}$ is $\mu$-strongly convex
(with respect to $\norm{\cdot}_2$) and $L$-Lipschitz.
Assume that:
\begin{enumerate}[label=(\emph{\alph*})]
\item $g_t \in \partial f(x_t)$ for all $t$ (with probability $1$).
\item $\norm{\hat{z}_t} \leq 1$ (with probability $1$).
\end{enumerate}
Set $\eta_t = \frac{2}{\mu (t+1)}$ and $\gamma_t = \frac{t}{T(T+1)/2}$.
Then, for any $\delta \in (0,1)$, with probability at least $1-\delta$,
$$
f\left( \sum_{t=1}^T \gamma_t x_t \right) - f(x^*)
    ~\leq~  O  \left (\frac{L \cdot \log (1/\delta) + L^2}{ \mu}  \cdot \frac{1}{T} \right ).
$$
\end{theorem}

\begin{remark}
It is possible to strengthen the statement of \Theorem{StochasticStrongPGD} by replacing assumption (a) with the weaker assumption that $\norm{\hat{z}_t}$ is subgaussian conditioned on $\mathcal{F}_{t-1}$ (for example, $\hat{z}_t \sim N(0, \frac{1}{n} I_n)$). A more detailed discussion can be found in the supplementary material.
\end{remark}

We also show that the bound in \Theorem{StochasticStrongPGD} is tight up to constant factors.
The proof is in \Section{LB}.

\begin{claim}
\ClaimName{LB}
Suppose $ \sqrt{6} \leq  \frac{\sqrt{2\log(1/\delta)}}{3}  \leq \sqrt{T}/4$.
There exists a sub-gradient oracle such that running \Algorithm{StrongLipschitzPGD} on the function $f(x) = \frac{x^2}{2}$ with step sizes $\eta_t = \frac{1}{t+1}$ satisfies the following. With probability at least $\delta$
\[  f\left(   \sum_{t=1}^T \gamma_t x_t \right) - f(x^*) \geq \frac{ \log(1/\delta)}{9 \cdot T},  \]
where $\gamma_t = \frac{t}{T(T+1)/2}.$
\end{claim}

\comment{It is possible to strengthen the statement of \Theorem{StochasticStrongPGD} by weakening assumption (a). Instead of assuming $\norm{\hat{z}_t} \leq 1$ almost surely, we may assume that $\norm{\hat{z}_t}$ is $K$-subgaussian conditioned on $\mathcal{F}_{t-1}$.

First, we define what we mean by a random variable to be $K$-subgaussian.

\begin{definition}
A random variable $X$ is said to be $K$-subgaussian if $\expect{\exp \left ( X^2/K^2 \right ) } \leq 2$.
In additional, we say that $X$ is $K$-subgaussian conditioned on $\mathcal{F}$ if $\expect{\exp \left ( X^2/K^2 \right ) } \leq 2$.
Note that $K^2$ in this setting may itself be a random variable.
\end{definition}
Note that the class of subgaussian random variables contains bounded random variables. Furthermore, this class also contains Gaussian random variables (which, of course, are not bounded). Therefore, the following theorem is indeed a strengthening of \Theorem{StochasticStrongPGD}, which only dealt with stochastic gradient oracles that used almost surely bounded noise.
\begin{theorem}
\TheoremName{StochasticStrongPGDSubGaussian}
Let $\mathcal{X} \subseteq \mathbb{R}^n$ be a convex set.
Suppose that $f : \mathcal{X} \rightarrow \mathbb{R}$ is $\mu$-strongly convex
(with respect to $\norm{\cdot}_2$) and $L$-Lipschitz.
Assume that:
\begin{enumerate}[label=(\emph{\alph*})]
\item $g_t \in \partial f(x_t)$ for all $t$ (with probability $1$).
\item $\norm{\hat{z}_t}$ is $K$-subgaussian conditioned on $\mathcal{F}_{t-1}$ for some $k \in \bN$.
\end{enumerate}
Set $\eta_t = \frac{2}{\mu (t+1)}$.
Let $\gamma_t = \frac{t}{T(T+1)/2}$.
Then, for any $\delta \leq 1/e$, with probability at least $1-\delta$ we have,
$$
f\left( \sum_{t=1}^T \gamma_t x_t \right) - f(x^*)
    ~\leq~  \left (\frac{4.5 \max\set{\frac{4K}{\ln 2}, 1} \max\set{G,1}  + 2 \left ( \frac{L}{\ln 2} + K\right)^2  }{ \mu} \right ) \cdot \frac{\log \left (2 / \delta \right )}{T}.
$$
\end{theorem}
}
\section{Proof of high probability upper bound}
\SectionName{ub}

The proof follows that of \cite{NonUniformAvg} but we must be careful with the noise terms as our goal is obtain a high probability bound.
We will need one technical lemma whose proof we relegate to the next subsection.
\begin{lemma}
\LemmaName{StronglyConvexSGDBound1}
Let $Z_T = \sum_{t=1}^T t \cdot \inner{\hat{z}_t}{x_t - x^*}.$ Then for any $\delta \in (0,1)$, $Z_T \leq O \left ( \frac{L}{\mu} \cdot T \log(1/\delta) \right )$, with probability at least $1-\delta$.
\end{lemma}
\begin{proofof}{\Theorem{StochasticStrongPGD}}
Define $\hat{z}_t = g_t - \hat{g}_t$.
Since $f$ is $\mu$-strongly convex, we have
\begin{align*}
    f(x_t) - f(x^*)
        &~\leq~ \inner{ g_t }{ x_t - x^* }
            - \frac{\mu}{2} \norm{x_t - x^*}_2^2 \\
        &~=~ \inner{ \hat{g}_t }{ x_t - x^* }
            - \frac{\mu}{2} \norm{x_t - x^*}_2^2
            + \inner{ \hat{z}_t }{ x_t - x^* }.
\end{align*}
The first two terms can be bounded as follows.
\begin{align*}
        & \inner{ \hat{g}_t }{ x_t - x^* }
        - \frac{\mu}{2} \norm{x_t - x^*}_2^2 \\
    &~=~ \frac{1}{\eta_t} \inner{ x_t - y_{t+1} }{ x_t - x^* }
        - \frac{\mu}{2} \norm{x_t - x^*}_2^2
        \qquad\text{(by the gradient step)}\\
    &~=~ \frac{1}{2 \eta_t} \Big(
            \norm{x_t - y_{t+1}}_2^2
          + \norm{x_t     - x^*}_2^2
          - \norm{y_{t+1} - x^*}_2^2
        \Big)
        - \frac{\mu}{2} \norm{x_t - x^*}_2^2 \\
    &~\leq~ \frac{1}{2 \eta_t} \Big(
            \norm{x_t - y_{t+1}}_2^2
          + \norm{x_t     - x^*}_2^2
          - \norm{x_{t+1} - x^*}_2^2
        \Big)
        - \frac{\mu}{2} \norm{x_t - x^*}_2^2.
\end{align*}
The last line uses a property of Euclidean projections:
since $x_{t+1}$ is the projected point $\Pi_\mathcal{X}(y_{t+1})$ and $x^* \in \mathcal{X}$,
we have $\norm{x_{t+1} - x^*}_2^2 \leq \norm{y_{t+1} - x^*}_2^2$.

It is convenient to scale by $t$ in order to later obtain a telescoping sum.
Using the definition of the gradient step, i.e.~$x_t - y_{t+1} = \eta_t \hat{g}_t$, we have
\begin{align*}
& t \cdot \Big(f(x_t) - f(x^*)
 - \inner{ \hat{z}_t }{ x_t - x^* } \Big) \\
    &~\leq~ \frac{t \norm{\eta_t \hat{g}_t}_2^2}{2 \eta_t}
          + t \Big(\frac{1}{2 \eta_t} - \frac{\mu}{2}\Big) \norm{x_t-x^*}_2^2
          - \frac{t}{2 \eta_t} \norm{x_{t+1} - x^*}_2^2
          \\
    &~=~ \frac{t \norm{\hat{g}_t}_2^2}{\mu (t+1)}
          + \Big(\frac{\mu t(t+1)}{4} - \frac{2 \mu t}{4}\Big) \norm{x_t-x^*}_2^2
          - \frac{t (t+1) \mu}{4} \norm{x_{t+1} - x^*}_2^2
         \\
    &~\leq~ \frac{(L+1)^2}{\mu}
          + \frac{\mu}{4} \cdot\Big( t(t-1) \norm{x_t-x^*}_2^2
          - t (t+1) \norm{x_{t+1} - x^*}_2^2 \Big).
\end{align*}
Now, summing over $t$, the right-hand side telescopes and we obtain
\begin{align*}
\sum_{t=1}^T t \cdot \big(f(x_t) - f(x^*)\big)
    ~\leq~ \sum_{t=1}^T t \cdot \inner{ \hat{z}_t }{ x_t - x^* }
        + \frac{T \cdot (L+1)^2}{\mu}
\end{align*}
Dividing by $T(T+1)/2$ and applying Jensen's inequality, we obtain
\begin{align*}
f\Big( \sum_{t=1}^T \gamma_t x_t \Big) - f(x^*)
    &~\leq~ \sum_{t=1}^T \gamma_t \cdot \big(f(x_t) - f(x^*)\big) \\
    &~\leq~ \frac{2}{T(T+1)} \underbrace{\sum_{t=1}^T t \cdot \inner{ \hat{z}_t }{ x_t - x^* }}_{=:\, Z_T}
        + \frac{2 \cdot (L+1)^2}{\mu(T+1)}.
\end{align*}
Finally, we can use \Lemma{StronglyConvexSGDBound1} to obtain a high probability bound on $Z_T$, completing the proof of the theorem.
\end{proofof}

\subsection{Bounding $Z_T$}
\SubsectionName{BoundingZ}
Observe that $Z_T$ is a sum of a martingale difference sequence.
Define $d_t = t \cdot \inner{\hat{z}_t}{x_t - x^*}$, $v_{t-1} := t^2 \norm{x_t - x^*}$, and $V_T = \sum_{t=1}^T v_{t-1}$.
Note that $v_{t-1}$ is $\mathcal{F}_{t-1}$-measurable. The next claim shows that $v_{t-1}$ and $d_t$ satisfy the assumptions of Generalized Freedman's inequality (\Theorem{FanV2}).
\begin{claim}
\ClaimName{BIsTCV}
For all $t$ and $\lambda > 0$, we have
$\expectg{\exp \left ( \lambda d_t \right )}{\mathcal{F}_{t-1}} \leq \exp \left ( \frac{\lambda^2}{2} v_{t-1}  \right )$.
\end{claim}
\begin{proof}
    First, we can apply Cauchy-Schwarz to get that $\abs{t \inner{\hat{z}_t}{x_t - x^*}} \leq t \cdot\norm{\hat{z}_t} \cdot \norm{x_t - x^*} \leq t\cdot\norm{x_t - x^*}$ because $\norm{\hat{z}_t} \leq 1$ a.s.
    Next, applying Hoeffding's Lemma (\cite[Lemma~2.6]{massart2007concentration}),
    we have $\expectg{\exp \left( \lambda t \inner{\hat{z}_t}{x_t - x^*} \right )}{\mathcal{F}_{t-1}} \leq \exp \left ( \frac{\lambda^2}{2} t^2 \norm{x_t - x^*}^2  \right )$.
\end{proof}

To bound $Z_T$, we will show that we can bound its TCV by a linear combination of the increments.
This will allow us to use the Generalized Freedman Inequality (\Theorem{FanV2}).
\begin{lemma}
\LemmaName{BoundingB}
There exists non-negative constants $\alpha_1, \ldots, \alpha_T$ such that $\max_{i \in [T]} \set{\alpha_i} = O \left (\frac{T}{\mu} \right)$ and $\beta = O \left (  \frac{L^2}{\mu^2} T^2 \right )$ such that
$V_T \leq \sum_{t=1}^T \alpha_t d_t + \beta$.
\end{lemma}
\begin{proofof}{\Lemma{StronglyConvexSGDBound1}}
By \Claim{BIsTCV}, we have $\expectg{\exp(\lambda d_t)}{\mathcal{F}_{t-1}} \leq \exp\left( \frac{\lambda^2}{2} v_{t-1} \right)$ for all $\lambda > 0$.
By \Lemma{BoundingB}, we have $V_T \leq \sum_{t=1}^T \alpha_t d_t + \beta$.
Plugging $\alpha = O \left (\frac{T}{\mu} \right)$, $\beta = O \left (\frac{L^2}{\mu^2} T^2 \right )$, and $x = O \left  ( \frac{L}{\mu} \cdot T \log(1/\delta) \right )$ into \Theorem{FanV2} proves the lemma.
\end{proofof}

It remains to prove \Lemma{BoundingB}.
To do so, we will need the following two lemmata, which are adapted from \cite{Rakhlin} to use the step sizes $\eta_t = \frac{2}{\mu (t+1)}$. For completeness, we provide a proof in the supplementary material.
\begin{lemma}[{\protect\cite[Lemma 5]{Rakhlin}}]
\LemmaName{RakhlinDiameterBound}
With probability $1$, and for all $t$, $\norm{x_t-x^*} ~\leq~ \frac{2 L}{\mu}.$
\end{lemma}

\begin{lemma}[{\protect\cite[Lemma 6]{Rakhlin}}]
\LemmaName{RakhlinWBound}For all $t \geq 3$, there exists non-negative numbers $a_1(t), \ldots , a_t(t)$ with $a_i(t) = \Theta(i^3/t^4)$ and $b_1(t), \ldots, b_t(t)$ with $b_i(t) = \Theta(i^2/t^4)$, such that with probability $1$
$$
\norm{x_{t+1}-x^*}^2 ~\leq~ \frac{4}{\mu}
    \sum_{i=3}^t a_i(t) \inner{ \hat{z}_i }{ x_i - x^* }
    + \frac{4}{\mu^2} \sum_{i=3}^t b_i(t) \norm{\hat{g}_t}^2.
$$
\end{lemma}
\begin{remark}
\Lemma{RakhlinDiameterBound} and \Lemma{RakhlinWBound} are true regardless of the assumption we place on $\hat{z}_t$.
\end{remark}

\begin{proofof}{\Lemma{BoundingB}}
Recall $\norm{\hat{g}_i} \leq L +1$ because $f$ is $L$-Lipschitz and $\norm{\hat{z}_i}\leq 1$ almost surely. By \Lemma{RakhlinDiameterBound} and \Lemma{RakhlinWBound}, we have:
\begin{align*}
    &V_T \\
        &~=~ \sum_{t=1}^T  t^2 \cdot  \norm{x_t - x^*}^2 \\
        &~\leq~   \frac{56L^2}{\mu^2} + \sum_{t=4}^T t^2  \left (   \frac{4}{\mu} \sum_{i=3}^{t-1} a_i(t-1)\inner{\hat{z}_i}{x_i - x^*} + \frac{4}{\mu^2} \sum_{i=3}^{t-1} b_i(t-1) \norm{\hat{g}_i}^2 \right ) \\
        &~\leq~   \frac{56L^2}{\mu^2} + \sum_{t=4}^T t^2  \left (   \frac{4}{\mu} \sum_{i=3}^{t-1} a_i(t-1)\inner{\hat{z}_i}{x_i - x^*} + \frac{4(L+1)^2}{\mu^2} \sum_{i=3}^{t-1} b_i(t-1)  \right ) \\
        &~=~ \frac{4}{\mu} \sum_{t=4}^T t^2 \left ( \sum_{i=3}^{t-1} a_i(t-1) \inner{\hat{z}_i}{x_i - x^*} \right ) + \frac{4(L+1)^2}{\mu^2} \sum_{t=4}^T t^2 \left (\sum_{i=3}^{t-1}b_i(t-1) \right ) + \frac{56L^2}{\mu^2} \\
        &~=~ \sum_{i=3}^{T-1}  \underbrace{ \frac{4}{\mu} \left ( \sum_{t=i+1}^T t^2 \cdot \frac{a_i(t-1)}{i}\right )}_{\coloneqq \alpha_i} \cdot i  \inner{\hat{z}_i}{x_i - x^*} +  \underbrace{\frac{4(L+1)^2}{\mu^2} \sum_{t=4}^T t^2 \left (\sum_{i=3}^{t-1}b_i(t-1) \right ) + \frac{56L^2}{\mu^2}}_{\coloneqq \beta}\\
\end{align*}
Define $\alpha_1, \alpha_2, \alpha_T = 0.$ We have  $V_T \leq \sum_{i=1}^T \alpha_i \cdot i \cdot \inner{\hat{z}_i}{x_i -x^*} + \beta.$ It remains to show $\max \set{\alpha_i} = O \left ( \frac{T}{\mu}\right )$ and $\beta = O \left ( \frac{L^2}{\mu^2} T^2 \right ).$ To bound $\max \set{\alpha_i}$, observe that for $i \in \set{3, \ldots , T-1},$
\[\sum_{t=i+1}^T t^2 \cdot \frac{a_i(t-1)}{i} ~=~ \sum_{t=i+1}^T t^2 O \left (\frac{i^2}{t^4}\right ) ~=~ \sum_{t=i+1}^T t^2 O \left (\frac{1}{t^2}\right ) ~=~ O \left ( {T -i} \right ).  \]To bound $\beta,$ observe
\[ \sum_{t=4}^T t^2  \left ( \sum_{i=3}^{t-1}b_i(t-1)\right )~=~ \sum_{t=4}^T t^2  \sum_{i=3}^{t-1} O \left ( \frac{i^2}{t^4}  \right ) ~=~   \sum_{t=4}^T t^2  \sum_{i=3}^{t-1} O \left ( \frac{1}{t^2}\right ) ~=~ \sum_{t=4}^T O(t) ~=~ O(T^2).   \]
\end{proofof}

\comment{
Observe that $Z_T$ is a sum of a martingale difference sequence. As such, it is natural to study its total conditional variance (TCV). Indeed, Freedman's inequality \cite{F75} (a standard martingale concentration bound) states that a martingale is roughly bounded by the square root of its TCV. However, this statement only applies if one has a high probability bound on the TCV which we will not be able to obtain.

However, the total conditional variance of $Z_T$ satisfies the rather curious property that its TCV is bounded above by a linear combination of the original martingale (plus a constant).
We call martingales satisfying such a property ``Chicken and Egg'' martingales.

\begin{definition}[``Chicken and Egg'' Martingale]
\DefinitionName{ChickenAndEgg}
Let $\{ d_i, \mathcal{F}_i \}_{i=1}^n$ be a martingale difference sequence. Let $S_t = \sum_{i=1}^t d_i$. We call $S_n$ a ``Chicken and Egg'' martingale if there exists $v_{i-1}$, $i \in [n]$ which are positive and $\mathcal{F}_{i-1}$-measurable random variables such that $\expectg{\exp(\lambda d_i)}{\mathcal{F}_{i-1}} \leq \exp\left(\frac{\lambda^2}{2} v_{i-1}\right)$ for all $i \in [n],\, \lambda > 0$ such that

\[ V_n ~:=~ \sum_{i=1}^n v_{i-1} ~\leq~ \sum_{i=1}^n \alpha_i d_i ~+~ \beta,   \]for some $\alpha_i, \beta \geq 0.$
\end{definition}

The fact that the upper bound on the TCV for Chicken and Egg martingales includes the martingale itself, makes applications of Freedman's inequality quite cumbersome.

Such martingales have made appearances in the stochastic gradient descent literature before. Indeed, in the process of deriving high probability upper bounds on the convergence rate of the final iterate of SGD, \citet{HLPR18} study a martingale representing the accumulated noise, similar to $Z_T$. Their martingale also satisfied the Chicken and Egg phenomenon, and they develop a generalization of Freedman's inequality to control the tail of any such martingale.

\begin{theorem}[Generalized Freedman, {\cite[Theorem~3.3]{HLPR18}}]
\TheoremName{FanV2}
\fantwo
\end{theorem}

\Theorem{FanV2} allows one to obtain a tail bound on any Chicken and Egg martginale.

\begin{corollary}
\CorollaryName{FanCorollary}
Suppose $\{ d_i, \mathcal{F}_i \}_{i=1}^n$ is a martingale difference sequence which satisfies the ``Chicken and Egg'' property from \Definition{ChickenAndEgg} with $(\alpha_i)_{i=1}^n, (v_{i-1})_{i=1}^n, \beta,$ and $\alpha = \max \alpha_i$.   Then,
\[  \prob{  S_n \geq x } ~\leq~ \exp \left ( - \frac{x^2}{4\alpha \cdot x   + 8 \beta}\right ).  \]
\end{corollary}

\begin{proof}
\begin{align*}
    \prob{S_n \geq x}
        &~=~ \prob{ S_n \geq x \text{   and  } V_n \leq \sum_{i=1}^n \alpha_i d_i + \beta  } + \prob{S_n \geq x \text{   and  } V_n > \sum_{i=1}^n \alpha_i d_i + \beta   }\\
        &~\leq~ \prob{ S_n \geq x \text{   and  } V_n \leq \sum_{i=1}^n \alpha_i d_i + \beta  } + \underbrace{\prob{V_n > \sum_{i=1}^n \alpha_i d_i + \beta  }}_{=0} \qquad\text{($S_n$ satisfies \Definition{ChickenAndEgg})}\\
        &~\leq~ \exp \left ( -\frac{x^2}{4\alpha \cdot x   + 8 \beta} \right ). \qquad\text{(by \Theorem{FanV2})}
\end{align*}
\end{proof}

We now make it our main goal to show that $Z_T$ satisfies the Chicken and Egg phenomenon. Let $V_T := \sum_{t=1}^T t^2 \norm{x_t - x^*}^2.$ Observe that this is a valid choice of $V_T$ in \Definition{ChickenAndEgg}:

\begin{claim}
\ClaimName{BIsTCV}
For all $t$, $t^2\norm{x_t - x^*}^2$ is $\mathcal{F}_{t-1}$-measurable and for all $t$ and $\lambda > 0$
\[\expectg{\exp \left ( \lambda \cdot t \cdot\inner{\hat{z}_t}{x_t - x^*} \right )}{\mathcal{F}_{t-1}} \leq \exp \left ( \frac{\lambda^2}{2} t^2 \norm{x_t - x^*}^2  \right ).\]
\end{claim}
The measurability is clear because the $t$-th iterate is determined by the first $t-1$ random sub-gradients. The rest of this claim follows from using the Cauchy-Schwarz inequality and Hoeffding's Lemma (\cite[Lemma~2.6]{massart2007concentration}). In \Subsubsection{BoundingTCV}, we demonstrate that with this definition of $V_T,$ $Z_T$ satisfies the Chicken and Egg phenomenon (\Definition{ChickenAndEgg}).

\begin{lemma}
\LemmaName{BoundingB}
$Z_T$ satisfies \Definition{ChickenAndEgg} with $v_{t-1}  = t^2\norm{x_t - x^*}^2,$ $\alpha_t \leq \frac{8}{\lambda} (T-2)$ for all $t$ and $\beta = \frac{G^2}{\lambda^2} (T(T+1) + 8).$
\end{lemma}
As a result, we may apply \Corollary{FanCorollary} to prove \Theorem{StronglyConvexSGDBound1}:

\begin{proofof}{\Theorem{StronglyConvexSGDBound1}}
\Lemma{BoundingB} allows apply \Corollary{FanCorollary} with $\alpha = \frac{8}{\lambda} (T-2)$ and $\beta = \frac{G^2}{\lambda^2} (T(T+1) + 8)$. That is, \[  \prob{Z_T \geq x} \leq \exp \left ( -\frac{x^2}{4\alpha\cdot x + 8 \beta} \right ). \]One can verify that for every $\delta \leq 1/e,$ if we define $x = \frac{2.25 G}{\lambda}(T+1) \cdot \log(1/\delta),$ then the right hand side above is at most delta.
\end{proofof}

\subsubsection{Bounding the TCV of $Z_T$ by itself}
\SubsubsectionName{BoundingTCV}
We use two results from \cite{Rakhlin} to help us derive the Chicken and Egg phenomenon.
\begin{lemma}[Rakhlin et al.~{\protect\cite[Lemma 5]{Rakhlin}}]
\LemmaName{RakhlinDiameterBound}For all $t$, it holds that with probability 1
$$
\norm{x_t-x^*} ~\leq~ \frac{2 G}{\lambda}.
$$
\end{lemma}

\begin{lemma}[Rakhlin et al.~{\protect\cite[Lemma 6]{Rakhlin}}]
\LemmaName{RakhlinWBound}For all $t \geq 2$, it holds that
$$
\norm{x_{t+1}-x^*}^2 ~\leq~ \frac{2}{\lambda (t-1)t}
    \sum_{i=2}^t (i-1) \inner{ \hat{z}_i }{ x_i - x^* }
    + \frac{G^2}{\lambda^2 t}.
$$
\end{lemma}

\begin{proofof}{\Lemma{BoundingB}}
By \Lemma{RakhlinWBound}, we have:
\begin{align*}
    V_T
        &~=~ \sum_{t=1}^T  t^2 \cdot  \norm{x_t - x^*}^2 \\
        &~\leq~   \frac{8G^2}{\lambda^2} + \sum_{t=3}^T t^2  \left (   \frac{2}{\lambda (t-2)(t-1)} \sum_{i=2}^{t-1} (i-1)\inner{\hat{z}_i}{x_i - x^*} + \frac{G^2 }{\lambda^2 (t-1)} \right ) \qquad\text{(By \Lemma{RakhlinDiameterBound} and \Lemma{RakhlinWBound})}\\
        &~=~ \sum_{t=2}^T \left (\frac{2t^2}{\lambda (t-2)(t-1)} \sum_{i=2}^{t-1} (i-1) \inner{\hat{z}_i}{x_i - x^*} \right ) + \frac{G^2}{\lambda^2} \sum_{t=3}^T \frac{t^2}{(t-1) } + \frac{8G^2}{\lambda^2} \\
        &~=~ \sum_{i=2}^{T-1} \left [\left ( \frac{2}{ \lambda}  \sum_{t=i+1}^T \frac{t^2}{(t-2)(t-1)}   \right ) (i-1)\inner{\hat{z}_t}{x_t - x^*} \right ] + \frac{G^2}{\lambda^2} \sum_{t=3}^T t \cdot \underbrace{(1 + \frac{1}{t-1})}_{\leq 2}  + \frac{8G^2}{\lambda^2} \qquad\text{(Swap order of sum)}\\
        &~\leq~ \sum_{i=2}^{T-1} \left [\left ( \frac{2}{ \lambda}  \sum_{t=i+1}^T \frac{t^2}{(t-2)(t-1)}   \right ) (i-1)\inner{\hat{z}_t}{x_t - x^*} \right ] + \frac{G^2}{\lambda^2} \cdot T (T+1)  + \frac{8G^2}{\lambda^2}\\
        &~=~ \sum_{i=2}^{T-1} \underbrace{\left ( \frac{2}{ \lambda}  \sum_{t=i}^T \frac{t^2 \cdot (i-1)}{(t-2)(t-1) \cdot i}   \right )}_{=:\, \alpha_i, \ i \geq 2} i \cdot\inner{\hat{z}_t}{x_t - x^*}  ~+~ \underbrace{\frac{G^2}{\lambda^2} \left (T(T+1)  + 8 \right) }_{=:\, \beta}
\end{align*}
Define $\alpha_1, \alpha_2, \alpha_T = 0$. Observe \[   a_i ~\leq~ \alpha_3 ~\leq~    \frac{2}{\lambda} \sum_{t=3}^{T-1} \frac{t^2}{(t-2)(t-1)} ~\leq~ \frac{2}{\lambda}\sum_{t=3}^{T-1} \underbrace{\left( 1  + \frac{3t -2}{(t-2)(t-1)} \right )}_{\leq 4}~\leq~ \frac{8}{\lambda} (T-2). \]
\end{proofof}
}

Machine Learning (cs.LG); Optimization and Control (math.OC); Machine Learning (stat.ML)


Application of the Asymptotic Iteration Method to a Perturbed Coulomb Model

Abstract: We show that the asymptotic iteration method converges and yields accurate energies for a perturbed Coulomb model. We also discuss alternative perturbation approaches to that model.

\section{Introduction \label{sec:intro}}

The asymptotic iteration method (AIM) is an iterative algorithm for the
solution of Sturm--Liouville equations~\cite{CHS03,F04}. Although this
method does not seem to be better than other existing approaches, it has
been applied to quantum--mechanical~\cite{B05,B06,CHS05} as well as
mathematical problems~\cite{B05b}. For example, the AIM has proved suitable
for obtaining both accurate approximate and exact eigenvalues~\cite
{CHS03,F04,B05,B05b,B06,CHS05} and it has also been applied to the
calculation of Rayleigh--Schr\"{o}dinger perturbation coefficients~\cite
{B06,CHS05b}.

Recently, Barakat applied the AIM to a Coulomb potential with a radial
polynomial perturbation~\cite{B06}. By means of a well--known transformation
he converted the perturbed Coulomb problem into an anharmonic oscillator.
Since straightforward application of the AIM exhibited considerable
oscillations and did not appear to converge Barakat resorted to perturbation
theory in order to obtain acceptable results \cite{B06}.

It is most surprising that the straightforward application of the AIM failed
for the anharmonic oscillator studied by Barakat~\cite{B06} since it had
been found earlier that the approach should be accurate in such cases~\cite
{F04}.

The main purpose of this paper is to verify whether the AIM gives accurate
eigenvalues of the perturbed Coulomb model or if its sequences are
oscillatory divergent as mentioned above. We also discuss the application of
perturbation theory to that model.

In Sec.~\ref{sec:model} we present the model and discuss useful scaling
relations for the potential parameters. In Sec.~\ref{sec:AIM} we apply the
AIM to the perturbed Coulomb model directly; that is to say we do not
convert it into an anharmonic oscillator. In Sec.~\ref{sec:PT} we outline
alternative perturbation approaches, and in Sec.~\ref{sec:concl} we
interpret our results and draw conclusions.

\section{The model \label{sec:model}}

The problem studied by Barakat~\cite{B06} is given by the following radial
Schr\"{o}dinger equation
\begin{eqnarray}
\hat{H}\Psi &=&E\Psi ,  \nonumber \\
\hat{H} &=&-\frac{1}{2}\frac{d^{2}}{dr^{2}}+\frac{l(l+1)}{2r^{2}}+V(r)
\nonumber \\
V(r) &=&-\frac{Z}{r}+gr+\lambda r^{2},  \label{eq:Schro}
\end{eqnarray}
where $l=0,1,2,\ldots $ is the angular--momentum quantum number, and the
boundary conditions are $\Psi (0)=\Psi (\infty )=0$. We restrict to the case
$\lambda >0$ in order to have only bound states; on the other hand, $Z$ and $%
g$ can take any finite real value.

It is most useful to take into account the scaling relations
\begin{eqnarray}
E(Z,g,\lambda ) &=&Z^{2}E(1,gZ^{-3},\lambda
Z^{-4})=|g|^{2/3}E(Z|g|^{-1/3},g|g|^{-1},\lambda |g|^{-4/3})  \nonumber \\
&=&\lambda ^{1/2}E(Z\lambda ^{-1/4},g\lambda ^{-3/4},1).  \label{eq:scaling}
\end{eqnarray}
Notice that we can set either $Z$ or $\lambda $ equal to unity without loss
of generality, and that, for example, $E(1,-g,\lambda )=E(-1,g,\lambda )$.
Following Barakat~\cite{B06} we choose $n=0,1,\ldots $ to be the radial
quantum number, and we may define a ``principal '' quantum number $\nu
=n+l+1=1,2,\ldots $.

\section{Direct application of the AIM\label{sec:AIM}}

Barakat mentions that straightforward application of the AIM does not give
reasonable results because the sequences oscillate when the number of
iteration is greater than $30$ approximately~\cite{B06}. This conclusion is
surprising because it has been shown that the AIM yields accurate results
for anharmonic oscillators~\cite{F04}, and Barakat converted the perturbed
Coulomb model into one of them\cite{B06}. In this section we apply the AIM
directly to the original radial Schr\"{o}dinger equation (\ref{eq:Schro}).

By means of the transformation $\psi (r)=\phi (r)y(r)$ we convert the
perturbed Coulomb model (\ref{eq:Schro}) into a Sturm--Liouville equation
for $y(r)$:
\begin{eqnarray}
y^{\prime \prime }(r) &=&Q(r)y^{\prime }(r)+R(r)y(r)  \nonumber \\
Q(r) &=&-\frac{2\phi ^{\prime }(r)}{\phi (r)}  \nonumber \\
R(r) &=&\left\{ 2[V(r)-E]-\frac{\phi ^{\prime \prime }(r)}{\phi (r)}\right\}
,  \label{eq:St_Li_gen}
\end{eqnarray}
where $\phi (r)$ is arbitrary. It seems reasonable to choose
\begin{equation}
\phi (r)=r^{l+1}e^{-\beta r-\alpha r^{2}}  \label{eq:phi(r)}
\end{equation}
that resembles the asymptotic behaviour of the eigenfunction for a harmonic
oscillator when $\beta =0$ or for a Coulomb interaction when $\alpha =0$. It
leads to
\begin{eqnarray}
Q(r) &=&4\alpha r-\frac{2(l+1)}{r}+2\beta  \nonumber \\
R(r) &=&\left( 2\lambda -4\alpha ^{2}\right) r^{2}+(2g-4\alpha \beta )r+%
\frac{2\beta (l+1)-2Z}{r}  \nonumber \\
&&+2\alpha (2l+3)-2E-\beta ^{2}.  \label{eq:P(r)_Q(r)}
\end{eqnarray}
We can set the values of the two free parameters $\alpha $ and $\beta $ to
obtain the greatest rate of convergence of the AIM sequences. From now on we
call asymptotic values of $\alpha $ and $\beta $ to such values of those
parameters that remove the terms of $R(r)$ that dominate at large $r$; that
is to say: $\beta =g/(2\alpha )$ and $\alpha =\sqrt{\lambda /2}$. Since the
asymptotic values of the free parameters do not necessarily lead to the
greatest convergence rate\cite{F04}, in what follows we will also look for
optimal values of $\alpha $.

The Sturm--Liouville equation (\ref{eq:St_Li_gen}) with the functions $Q(r)$
and $R(r)$ (\ref{eq:P(r)_Q(r)}) is suitable for the application of the AIM.
We do not show the AIM equations here because they have been developed and
discussed elsewhere\cite{CHS03,F04}. Since the AIM quantization condition
depends not only on the energy but also on the variable $r$ for non--exactly
solvable problems, we have to choose a convenient value for the latter\cite
{CHS03,F04}. Later on we will discuss the effect of the value of $r$ on the
convergence of the method; for the time being we follow Barakat~\cite{B06}
and select the positive root of $\phi ^{\prime }(r)=0$:
\begin{equation}
r_{0}=\frac{\sqrt{8\alpha (l+1)+\beta ^{2}}-\beta }{4\alpha }  \label{eq:r_0}
\end{equation}

For concreteness we restrict to $Z=\lambda =1$ and $n=l=0$, and select $%
g=-2,-1,1,2$ from Barakat's paper\cite{B06}. As expected from earlier
calculations on anharmonic oscillators~\cite{F04}, the rate of convergence
of the AIM depends on the value of $\alpha $. In order to investigate this
point we choose $g=-2$ because it is the most difficult of all the cases
considered here. More precisely, we focus on the behaviour of the
logarithmic error $L_{N}=\log |E^{(N)}-E^{exact}|$, where $E^{(N)}$ is the
AIM energy at iteration $N$ and $E^{exact}=-1.1716735847196510437987056$ was
obtained by means of the rapidly converging Riccati--Pad\'{e} Method (RPM)~%
\cite{F95,F96} from sequences of determinants of dimension $D=2$ through $%
D=22$.

We first consider the asymptotic value $\alpha =1/\sqrt{2}$.
Fig.~\ref
{fig:asympt} shows that $L_{N}$ decreases rapidly with $N$ when $%
N<\approx 20$ and then more slowly but more smoothly for $N>20$.
In the transition region about $N\approx 20$ we observe
oscillations that can mislead one into believing that the AIM
starts to diverge.

Fig.~\ref{fig:optimal} shows that the behaviour of $L_{N}$ for a
nearly optimal value $\alpha =1/2$ is similar to the previous
case, except that the transition takes place at a larger value of
$N$ and the convergence rate is
greater. More precisely, $L_{N}$ decreases rapidly with $N$ when $%
N<\approx 50$ approximately as $L_{N}\approx
0.22-0.064N-0.0029N^{2}$ and more slowly and smoothly for $N>50$
as $L_{N}\approx -6.5-0.068N$. Again, the transition region
exhibits oscillations.

Table~\ref{tab:Table2} shows the ground--state energies for
$g=-2,-1,1,2$ and the corresponding nearly optimal values of
$\alpha $. We estimated those eigenvalues from the sequences of
AIM roots for $N=10$ through $N=80$. Notice that the optimal
values of $\alpha $ in Table~\ref{tab:Table2} depend on $g$ and do
not agree with the asymptotic value $\alpha =\sqrt{1/2}$.
Table~\ref{tab:Table2} also shows that the AIM eigenvalues agree
with those calculated by means of the RPM~\cite{F95,F96} from
sequences of determinants of dimension $D=2$ through $D=15$.

The rate of convergence also depends on the chosen value of $r$. The
calculation of $L_{N}$ as a function of $\xi =r/r_{0}$ shows that $L_{N}(\xi
)$ exhibits a minimum at $\xi _{N}$ and that $\xi _{N}$ increases with $N$
approximately as $\xi _{N}=0.435+0.005N$ (for $g=-2$). However, in order to
keep the application of the AIM as simple as possible we just choose $%
r=r_{0} $ for all the calculations.

\section{Alternative perturbation approaches\label{sec:PT}}

Barakat~\cite{B06} first converted the radial Schr\"{o}dinger equation (\ref
{eq:Schro}) into another one for an anharmonic oscillator by means of the
standard transformations $r=u^{2}$ and $\Phi (u)=u^{-1/2}\Psi (u^{2})$.
Finally, he derived the Sturm--Liouville problem
\begin{equation}
f^{^{\prime \prime }}(u)+2\left( \frac{L+1}{u}-\alpha u^{3}\right) f^{\prime
}(u)+\left( \epsilon u^{2}-8gu^{4}+8Z\right) f(u)=0,  \label{eq:SL_u}
\end{equation}
where $L=2l+1/2$ and $\epsilon =8E-(2L+5)\alpha $, through factorization of
the asymptotic behaviour of the solution:
\begin{equation}
\Phi (u)=u^{L+1}e^{-\alpha u^{2}/4}f(u),\;\alpha =\sqrt{8\lambda }.
\label{eq:factor1}
\end{equation}
Notice that present $\alpha $ and Barakat's $\alpha $ are not exactly the
same but they have a close meaning and are related by $\alpha
_{asymptotic}^{present}=\alpha _{Barakat}/4$. Since Barakat's application of
the AIM to Eq. (\ref{eq:SL_u}) did not appear to converge~\cite{B06} he
opted for a perturbation approach that consists of rewriting Eq. (\ref
{eq:SL_u}) as
\begin{equation}
f^{^{\prime \prime }}(u)+2\left( \frac{L+1}{u}-\alpha u^{3}\right) f^{\prime
}(u)+\left[ \epsilon u^{2}+\gamma \left( -8gu^{4}+8Z\right) \right] f(u)=0
\label{eq:SL_u_PT}
\end{equation}
and expanding the solutions in powers of $\gamma $:
\begin{equation}
f(u)=\sum_{j=0}^{\infty }f^{(j)}(u)\gamma ^{j},\;\epsilon
=\sum_{j=0}^{\infty }\epsilon ^{(j)}\gamma ^{j}  \label{eq:AIM_PT_series_u}
\end{equation}
The perturbation parameter $\gamma $ is set equal to unity at the end of the
calculation. The series for the energy exhibits considerable convergence
rate and consequently Barakat obtained quite accurate results with just two
to five perturbation corrections~\cite{B06}. Barakat calculated the
coefficient $\epsilon ^{(0)}$ exactly and all the others approximately~\cite
{B06}.

The model (\ref{eq:Schro}) is suitable for several alternative
implementations of perturbation theory in which we simply write $%
V(r)=V_{0}(r)+\gamma V_{1}(r)$ and expand the solutions in powers of $\gamma
$.

If we choose $V_{0}(r)=-Z/r$ (when $Z>0$) and $V_{1}(r)=gr+\lambda r^{2}$
then we can calculate all the perturbation coefficients exactly by means of
well known algorithms~\cite{F00}. One easily realizes that the perturbation
series can be rearranged as
\begin{equation}
E=Z^{2}\sum_{i=0}^{\infty }\sum_{j=0}^{\infty }d_{ij}g^{i}\lambda
^{j}Z^{-(3i+4j)}.  \label{eq:E_series_Z}
\end{equation}
It is well known that this series is asymptotic divergent for all values of
the potential parameters.

The other reasonable perturbation split of the potential energy is $%
V_{0}(r)=\lambda r^{2}$, $V_{1}(r)=-Z/r+gr$. In this case we can rearrange
the series as
\begin{equation}
E=\lambda ^{1/2}\sum_{i=0}^{\infty }\sum_{j=0}^{\infty
}c_{ij}Z^{i}g^{j}\lambda ^{-(i+3j)/4}.  \label{eq:E_series_lambda}
\end{equation}
One expects that this series has a finite radius of convergence. This is
exactly the series obtained by Barakat~\cite{B06} by means of the AIM and,
consequently, it is not surprising that he derived accurate results from it.
In this case one can obtain exact perturbation corrections at least for the
first two energy coefficients.

For simplicity we concentrate on the states with $n=0$. The eigenfunctions
and eigenvalues of order zero are
\begin{eqnarray}
\Psi _{0l}^{(0)}(r) &=&\frac{\sqrt{2}(2\lambda )^{(2l+3)/8}}{\Gamma (l+3/2)}%
r^{l+1}e^{-\sqrt{2\lambda }r^{2}/2}  \nonumber \\
E_{0l}^{(0)} &=&\frac{\sqrt{2\lambda }}{2}(2l+3)  \label{eq:E^(0)}
\end{eqnarray}
respectively. With the unperturbed eigenfunctions one easily obtains the
perturbation correction of first order to the energy
\begin{equation}
E_{0l}^{(1)}=\frac{(l+1)!g}{(2\lambda )^{1/4}\Gamma (l+3/2)}-\frac{%
l!Z(2\lambda )^{1/4}}{\Gamma (l+3/2)}  \label{eq:E^(1)}
\end{equation}
that is the term of the series (\ref{eq:E_series_lambda}) with $i+j=1$. One
can easily carry out the same calculation for the states with $n>0$ using
the appropriate eigenfunctions of the harmonic oscillator.

Equation (\ref{eq:E^(1)}) yields all the numerical results for $\epsilon
_{0l}^{(1)}$ in Tables 1-3 of Barakat's paper~\cite{B06}. In particular, $%
E_{0l}^{(1)}=0$ when $g=\sqrt{2\lambda }Z/(l+1)$ as in Table 1 of Barakat's
paper~\cite{B06}. This particular relationship between the potential
parameters also leads to exact solutions of the eigenvalue equation (\ref
{eq:Schro}). Some of them are given by
\begin{eqnarray}
\Psi _{0l}^{exact}(r) &=&N_{l}r^{l+1}e^{-\alpha r^{2}-\beta r},\;\alpha =%
\sqrt{\frac{\lambda }{2}},\;\beta =\frac{Z}{l+1},  \nonumber \\
E_{0l}^{exact} &=&\alpha (2l+3)-\frac{Z^{2}}{2(l+1)^{2}},\;g=\frac{\sqrt{%
2\lambda }Z}{l+1},  \label{eq:QES}
\end{eqnarray}
where $N_{l}$ is a normalization constant.

\section{Conclusions \label{sec:concl}}

We have shown that the AIM converges for the perturbed Coulomb
model if the values of the free parameters in the factor function
that converts the Schr\"{o}dinger equation into a Sturm--Liouville
one are not too far from optimal. It is clear that it is not
necessary to transform the perturbed Coulomb model into an
anharmonic oscillator for a successful application of the AIM. Our
results do not exhibit the oscillatory divergence reported by
Barakat\cite{B06} even when choosing the asymptotic value of
$\alpha$.

The perturbation approach proposed by Barakat~\cite{B06} is equivalent to
choosing the harmonic oscillator as unperturbed or reference Hamiltonian,
and if we apply perturbation theory to the original radial Schr\"{o}dinger
equation we easily obtain two energy coefficients exactly instead of just
only one. It is worth mentioning that the coefficients calculated by Barakat~%
\cite{B06} are quite accurate and, consequently, the resulting series
provide a suitable approach for the eigenvalues of the perturbed Coulomb
potential. This application of the AIM to perturbation theory is certainly
much more practical than the calculation of exact perturbation corrections
proposed earlier~\cite{CHS05b} that can certainly be carried out more
efficiently by other approaches~\cite{F00}.

\noindent \textbf{Acknowledgements}

\noindent P.A. acknowledges support from Conacyt grant
C01-40633/A-1


Monomial to ultraspherical basis transformation and the zeros of polynomials

Abstract: We examine a result of A. Iserles and E. B. Saff, use it to prove a conjecture of S. Fisk that a linear operator which maps monomials to Legendre polynomials also preserves zeros in the open interval $|x|<1$, and state a more general version of the conjecture for the Jacobi polynomials.

\section{Introduction}

Let $P_n^{(\alpha,\beta)}(x)$ represent the $n$-th Jacobi polynomial with parameters $\alpha$ and $\beta$, defined by the generating function \cite[p. 271]{R},

\begin{equation}\label{jacobi-gen}
F(x,t) := \frac{2^{\alpha+\beta}}{\rho(1+t+\rho)^{\beta}(1-t+\rho)^{\alpha}} = \sum_{k=0}^\infty P_{n}^{(\alpha,\beta)}(x) t^n ,
\end{equation}
\\
where $\rho = (1-2xt+t^2)^{1/2}$.
The set $\{P_n^{(\alpha,\alpha)}(x)\}_{n=0}^\infty$, known as the ultraspherical polynomials \cite[p. 276]{R}, is also obtained from the simpler generating function
\\
\begin{equation}\label{Pgen}
G(x,t):=(1-2xt+t^2)^{-\alpha-\frac{1}{2}} = \sum_{n=0}^\infty \frac{(1+2\alpha)_n}{(1+\alpha)_n}P^{(\alpha,\alpha)}_n(x)t^n,
\end{equation}
\\
where $(x)_n = x(x+1)(x+2)\dots(x+n-1)$ is the commonly used \emph{Pochhammer symbol}.  When $\alpha=0$, the ultraspherical polynomials are the classical Legendre polynomials, which we denote by $P_k(x)$ ($P_k(x)=P^{(0,0)}_k(x)$).
We prove the following conjecture from S. Fisk's electronic book \cite{fisk}, which is left as an open problem in the book's appendices.
\begin{conj}{\rm (S. Fisk \cite[p. 724, Question 39]{fisk})}\label{Pconj}
If $f(x)=\sum_{k=0}^n a_k x^k\in\rl[x]$ has all of its zeros in the interval $(-1,1)$ then $T[f](x)=\sum_{k=0}^n a_k P_k(x)$ has all of its zeros in $(-1,1)$.
\end{conj}
The proof of Conjecture \ref{Pconj} is accomplished through a technique of A.~Iserles and E.~B.~Saff.  Many similar transformations involving orthogonal bases have already been established by Iserles, N{\o}rsett, and Saff using the same method in conjunction with other identities \cite{IN1, IN2, IS}.  Our main result is the following.

\begin{theorem}\label{main}
If $f(x)=\sum_{k=0}^n a_k x^k\in\rl[x]$ has all of its zeros in the interval $(-1,1)$ then
\[
T[f](x)=\sum_{k=0}^n a_k \frac{k!}{\Gamma(k+1+\alpha)} P^{(\alpha,\alpha)}_k(x)
\]
 has all of its zeros in $(-1,1)$ for $\alpha>-1$.
\end{theorem}

Conjecture \ref{Pconj} follows from Theorem \ref{main} with $\alpha=0$.  Iserles and Saff already proved that $x^n\to  (n!/\Gamma(n+1+\alpha)) (2n+2\alpha+1)P^{(\alpha,\alpha)}_n(x)$ maps polynomials whose zeros lie in the interval $(-1,1)$ to polynomials whose zeros lie in the interval $[-1,1]$ \cite[Proposition 3]{IS}.  We modify the generating function \eqref{Pgen} with a differential operator and apply the observation in Lemma \ref{open-sets} (below) to change the Iserles--Saff proof into the one we require.

We provide a short proof of the Iserles--Saff theorem in Appendix \ref{biortho-polys}, both for reference and to aid in the discussion, as we state it in a more general form.  The characterization of all linear transformations which preserve the set of polynomials whose zeros lie on an interval is an open problem as mentioned in \cite{IN2}, and stated more formally in \cite{BB1, CC3}.  A successful characterization would hopefully provide simple proofs of Theorem \ref{main} and Theorem \ref{key}.

Total positivity and sign regularity are introduced in Section \ref{section-tp}, along with Theorem \ref{key} of Iserles and Saff.  We also discuss problems which appear difficult with the currently available techniques for establishing sign regularity.
In Section \ref{section-proof}, we prove Theorem \ref{main} and pose a conjecture which extends it (Conjecture \ref{jacobi-trans}).

{\bf Acknowledgements.}  The author thanks Professor P.~Br\"and\'en for his comments and corrections, and Professor G.~Csordas for suggesting the application of the Iserles--Saff theorem to some of Fisk's conjectures.

\section{Total positivity and the Iserles-N{\o}rsett-Saff theorem}\label{section-tp}

In the proof of Conjecture \ref{Pconj}, we rely on the strict sign regularity (Definition \ref{def:ssr}) of a generating function related to the transformation.

\begin{defn}\label{def:ssr}
The kernel function $G:X\times Y\to\rl$ is said to be \emph{strictly sign regular} (SSR) if, given any two increasing m-tuples  $x_1 < x_2 < \ldots < x_m$,  $y_1 < y_2 < \ldots < y_m$, in $X$ and $Y$ respectively,

\beq
\varepsilon(m)\left| \begin{array}{cccc}
G(x_1,y_1) &  G(x_1,y_2) & \ldots & G(x_1,y_m) \\
G(x_2,y_1) & G(x_2,y_2)  & \ldots & G(x_2,y_m)\\
\vdots & \vdots & \ddots & \vdots \\
G(x_m,y_1)  & G(x_m,y_2)  & \ldots & G(x_m,y_m) \end{array} \right|>0.
\label{detcond}
\eeq
for all $m\in\nn$, where $\varepsilon(m)$ is a function of $m$ only and has the range $\{-1,1\}$.  If $\varepsilon(m)=1$, the kernel $G(x,t)$ is said to be \emph{strictly totally positive} (STP), and if equality is allowed \emph{totally positive} (TP)  (see \cite{karlin}).
\end{defn}

A sequence of functions $\phi_0(x), \phi_1(x),\dots, \phi_n(x)$ is said to constitute a \emph{Tchebyshev system} on $a<x<b$ if for any set of real constants $\{c_k\}$, not all zero, $\sum_{k=0}^n c_k\phi_k(x)$ does not vanish more than $n$ times on $a<x<b$ \cite[p.~24]{karlin}.  If every subset of a Tchebyshev system is also a Tchebyshev system, then $\phi_y(x)$ is SSR.

\begin{theorem}[{\cite[p. 18]{karlin}}]\label{factortheorem}
Let $K(x,y)$ be $SSR$ \textup{(}$x\in X$ and $y\in Y$\textup{)}, and suppose $\varphi(x)$ and $\psi(x)$ maintain the same constant sign on $X$ and $Y$, respectively.  Then
\bnum
\item $L(x,y)=\varphi(x)\psi(y)K(x,y)$ is $SSR$.

\item Now suppose $u=\varphi^{-1}(x)$ and $v=\psi^{-1}(x)$ be strictly increasing functions mapping $X$ and $Y$ onto $U$ and $V$, respectively, where $\varphi^{-1}$ and $\psi^{-1}$ are the inverse functions of $\varphi$ and $\psi$, respectively.

Consider
$$ L(u,v) = K[\varphi(u),\psi(x)] \text{ \hspace{ 5mm} } u\in U, v\in V.$$
Then $L(u,v)$ is $SSR$ and with signs $\varepsilon_m(K) = \varepsilon_m(L)$.  If $\phi(u)$ is strictly increasing while $\psi(v)$ is strictly decreasing, then $L(u,v)$ is $SSR$ and $\varepsilon_m(K) = (-1)^{\frac{m(m-1)}{2}}\varepsilon_m(L)$.
\enum
\end{theorem}

\begin{theorem}[Composition Rule]\label{cbtheorem}
Let $K$, $L$ and $M$ be Borel-measurable functions of two variables satisfying
\[
M(x,y) = \int_Z K(x,\eta)L(\eta,y) d\sigma(\eta),
\]
where the integral is assumed to converge absolutely.  Suppose the variables $x$, $y$, and $\eta$ are in subsets of the real line $X$, $Y$, and $Z$ respectively, and $d\sigma(\eta)$ is a sigma finite measure on $Z$.  If both $K$ and $L$ are SSR (STP), then so is $M$ \cite[p. 16]{karlin}.
\end{theorem}

\begin{theorem}[{\cite[p. 15]{karlin}}]\label{exy}
$$K(x, y)=e^{xy}$$
is STP for $x,y\in\rl$.
\end{theorem}
It is worth noting that a generalization of Theorem \ref{exy} exists for power series in the form $\sum_{k=0}^\infty a_kx^ky^k$ \cite[Part V, \#86]{PSz2}.

Let $\RR^n/S_n$ be $n$-dimensional Euclidean space modulo permutations of the coordinates, e.g. $\RR^2/S_2=\{ \{(x_1,x_2),(x_2,x_1)\} : x_1, x_2\in\RR \}$.  Then, $\RR^n/S_n$ is a metric space when endowed with a Hausdorff distance between equivalence classes,
\[
d(y_1,y_2)= \inf_{a\in y_1, b\in y_2} ||a-b||_{\infty},
\]
for $y_1,y_2\in\RR^2/S_2$, where $||(x_1,\dots,x_n)||_{\infty}=\max\{|x_1|,\dots,|x_n|\}$.  This provides a natural topology for the zeros of polynomials of a fixed degree.  Let $I$ be an open or closed interval, and let $I^n/S_n$ denote the subset of $\RR^n/S_n$ where every component lies in the interval $I$.  The topology on $\RR^n/S_n$ induced by the metric $d(\cdot,\cdot)$ then has open sets which are generated by unions of the ``open intervals'' $(a,b)^n/S_n$, $a,b\in\RR$, $a\le b$.

\begin{lemma}\label{open-sets}
Suppose $T:\RR[x]\to\RR[x]$ is degree preserving, linear, invertible, and maps polynomials whose zeros lie in the interval $(c,d)$ to polynomials whose zeros lie in the interval $[a,b]$. Then $T$ furthermore maps polynomials whose zeros lie in $(c,d)$ to polynomials whose zeros lie in $(a,b)$.
\end{lemma}

\begin{proof}
Let $Q(x)=\sum_{k=0}^nq_kx^k$ be an arbitrary polynomial with zeros in $(c,d)$.
Let the map $\phi:(c,d)^n/S_n\to[a,b]^n/S_n$ be defined by the mapping of the zeros of $Q$ to the zeros of $T[Q]$.  As $T$ is degree preserving and invertible, both $\phi$ and $\phi^{-1}$ are well-defined, and both are continuous by linearity and Hurwitz's theorem.  Therefore, open sets in $(c,d)^n/S_n$ are mapped by $\phi$ to open sets in $[a,b]^n/S_n$, whence the zeros of $T[Q(x)]$ must lie in $(a,b)$, which covers all open sets contained in $[a,b]^n/S_n$.
\end{proof}

Theorem \ref{key} of A.~Iserles and E.~B.~Saff is required for our proof of Conjecture \ref{Pconj}, and a short proof is provided for the reader's convenience in Appendix \ref{biortho-polys}.  The original version of Theorem \ref{key} is sharpened here by using the observation in Lemma \ref{open-sets}, and relaxing the domain basis to a complete Tchebyshev system.

\begin{theorem}[A. Iserles and E. B. Saff \cite{IS}]\label{key}
Let $\{Q_n(x)\}_{n=0}^\infty$ be an orthogonal sequence of polynomials satisfying
\[
\int_{a}^b Q_n(x)Q_m(x) dx = \begin{cases}h_m , & \text{ for } m=n\cr
0, & \text{ for } m\neq n.
\end{cases}
\]
Let $\{R_k\}_{k=0}^\infty$ be a set of polynomials, with $\deg(R_k)=k$ (thus any subset  $\{R_k\}_{k=0}^n$, $n\in\NN$ is a Tchebyshev system on $(c,d)$).
If the generating function $G(x,t)=\sum_{t=0}^\infty \delta_kQ_k(x)R_k(t)$ is SSR for all $x\in(a,b)$ and $t\in(c,d)$, and all the zeros of the polynomial $\sum_{k=0}^nq_kR_k(x)$ are in the interval $(c,d)$, then all the zeros of the polynomial
\[
\sum_{k=0}^n\frac{q_k}{\delta_kh_k}Q_k(x)
\]
are in the interval $(a,b)$.
\end{theorem}

All of the transformations which have been obtained so far with Theorem \ref{key} have $R_k(t)=t^k$.  For other starting bases, showing the kernel
\[
G(x,t)=\sum_{t=0}^\infty \delta_kQ_k(x)R_k(t)
\]
is SSR on some region appears quite difficult. Two papers by J.~Burbea \cite{burbea2,burbea} establish the strict total positivity of several replicating kernels by extending the notion of total positivity and sign regularity to curves in the complex plane, but this approach is very restrictive.  For natural extensions of total positivity to the complex plane, compositions of the type in Theorem \ref{cbtheorem} are generally not possible without unusually convenient phase behavior of the kernels involved.
The lack of tools for determining totally positivity and sign regularity of kernels prevents application of Theorem \ref{key} in some desirable situations.
For example, in \cite{IS}, a comment is made that although the kernel
\[
\frac{1}{(1-2xy+y^2)^{\beta}}
\]
is not strictly totally positive on $(-1,1)\times (-1,1)$ for $\beta<0$, it appears to be strictly sign regular there.
Similarly, if the range of parameters for which \eqref{jacobi-gen} is STP or SSR could be determined, a large number of transformations involving the Jacobi polynomials would follow.

\section{Proof for the ultraspherical polynomial transform}\label{section-proof}

Recall that for $\Re(\alpha)>-1$, $\Re(\beta)>-1$, the Jacobi polynomials satisfy the orthogonality relation \cite[p. 260]{R}

\begin{align}
&\int_{-1}^1 P^{(\alpha,\beta)}_n(x)P^{(\alpha,\beta)}_m(x) (1-x)^\alpha(1+x)^\beta dx = \label{ortho} \\
&\qquad \begin{cases} \displaystyle \frac{2^{1+\alpha+\beta}\Gamma(1+\alpha+n)\Gamma(1+\beta+n)}{(2n+1+\alpha+\beta)n!\Gamma(1+\alpha+\beta+n)}, & \text{ for } m=n\\
\\
 \displaystyle 0, & \text{ for } m\neq n.\nonumber
\end{cases}
\end{align}
From Theorems \ref{cbtheorem} and \ref{exy} one obtains the next lemma.

\begin{lemma}[\cite{IS}]\label{handy}
The function
\[
K(x,y) = \frac{1}{(x+y)^{\beta}},
\]
where $\beta>0$, is STP for $x,y\in(0,\infty)$
\end{lemma}

By simple substitutions with Theorem \ref{factortheorem}, see \cite{IS}, Lemma \ref{handy} is transformed into what we require.

\begin{lemma}\label{legendre-gen-stp}
The function
\[
K(x,y) = \frac{1}{(1-2xy+y^2)^{\beta}},
\]
where $\beta>0$, is STP for $x,y\in(-1,1)$.
\end{lemma}

\begin{proof}[Proof of Theorem \ref{main}]
The series on the right-hand side of  \eqref{Pgen} is known to converge for all $x,t\in(-1,1)$.
Operating on \eqref{Pgen} with $\left(2t\frac{\partial}{\partial t} + 2\alpha + 1 \right)$, we obtain a second generating function which converges for the same range of parameters:
\begin{align*}
G_2(x, t) =& \left(2t\frac{\partial}{\partial t} + 2\alpha + 1 \right)G(x,t)\\
=& \frac{(2\alpha+1)(1-t^2)}{(1-2xt+t^2)^{\alpha+3/2}} \\
=& \sum_{k=0}^\infty (2k+\alpha+1)\frac{(1+2\alpha)_k}{(1+\alpha)_k}P^{(\alpha,\alpha)}_k(x)t^k.
\end{align*}
\noindent
By Theorem \ref{key}, with $\delta_k=2k+2\alpha+1$, and (from \eqref{ortho} with $\alpha=\beta> -1$)
\[
h_k=\frac{2^{1+\alpha}(\Gamma(1+\alpha+n))^2}{k!(2k+2\alpha+1)\Gamma(1+2\alpha+2k)},
\]
it is sufficient to show that $G_2(x,t)$ is SSR for $x,t\in(-1,1)$.  By Theorem \ref{factortheorem}, the factor $(2\alpha+1)(1-t^2)$ can be dropped, and it is sufficient to check that
\[
 \frac{1}{(1-2xt+t^2)^{\alpha+3/2}}
\]
is SSR for $\alpha>-1$, which follows by Lemma \ref{legendre-gen-stp}.
\end{proof}

It is natural to question whether Theorem \ref{main} can be extended to the Jacobi Polynomials. Fisk has posed the following related question.

\begin{question}[{\cite[p. 724, Question 40]{fisk}}]\label{jacobi-fact}
For which $\alpha$ and $\beta$ does the transformation $x^n\to P^{(\alpha,\beta)}_n(x)/n!$ preserve the set of polynomials with only real zeros?
\end{question}

Numerical experiments using polynomials of the form $(x-1)^n(x+1)^m$, $n+m\le14$, suggest the restriction $\alpha,\beta\ge0$ in Question \ref{jacobi-fact}.  As an extension of Conjecture \ref{Pconj} (settled by Theorem \ref{main}) we state the following conjecture, which has been verified using Mathematica for $\alpha,\beta<5$, with $(x-1)^n(x+1)^m$, $n+m\le14$, and a large number of random polynomials of degree less than ten with zeros in the interval $(-1,1)$.

\begin{conjecture}\label{jacobi-trans}
If $\alpha$ and $\beta$ are non-negative integers, then $x^n\to P^{(\alpha,\beta)}_n(x)$ preserves the set of polynomials whose zeros lie only in the interval $(-1,1)$.
\end{conjecture}

\renewcommand\refname{References}