 the target $\theta^\dagger$. The expectation takes into account the randomness in SGD.

The analysis builds upon two steps: in Step 1, we derive the upper and lower bound of the iterative formula that updates ${\mathcal{R}}_{k+1}$ from ${\mathcal{R}}_k$. This step heavily depends on rewriting the SGD formula. We present the result in Subsection \[Sec:recuresive\]. In Step 2, we analyze the long time behavior of the iterative formula. The behavior depends on the strong convexity of the cost function, and thus we separate the discussion for the strongly convex and non-convex case, and present them in Subsection \[Sec:convex\] and \[Sec:nonconvex\], respectively.

Key Recursive Inequalities {#Sec:recuresive}
--------------------------

The derivation of the updating formula for ${\mathcal{R}}_k$ comes directly from SGD , which we rewrite into: $$\label{eqn:lstep_V}
\theta_{k+1}=\theta_k-\eta\nabla f(\theta_k)+\eta V\left(\theta_k\right)\,,$$ where the variation $$V\left(\theta\right)=\nabla f(\theta)-\nabla f_{\gamma_k}(\theta)\,.$$ Due to the randomness involved in the selection of $\gamma_k$, $V$ is also a random variable with explicitly expressible mean and variance: $$\begin{aligned}
&\mathbb{E}\left(V\left(\theta\right)\right)=\overrightarrow{0}\,,\\ &\mathrm{Var}\left(V\left(\theta\right)\right)=\frac{1}{J}\sum^{J}_{j=1}\left(\nabla f(\theta)-\nabla f_j(\theta)\right)\left(\nabla f(\theta)-\nabla f_j(\theta)\right)^\top\,.
\end{aligned}$$

Building upon these, we have the following theorem on the recursive formula.

\[Lemma:iteration\] The following upper and lower bounds hold true if SGD uses fixed step-size $\eta<\frac{1}{\lambda_{\max,0}}$:

-   If the objective functions are $\lambda$-smooth (satisfy Assumption \[ass:smooth\]), then $$\label{infinequality}
    {\mathcal{R}}_{k+1}-{\mathcal{R}}_k\geq \left[-2\eta\lambda_{\
the target $ \theta^\dagger$. The expectation takes into account the randomness in SGD. 

 The psychoanalysis build upon two steps: in gradation 1, we deduce the upper and lower bound of the iterative formula that updates $ { \mathcal{R}}_{k+1}$ from $ { \mathcal{R}}_k$. This tone heavily depends on rewrite the SGD formula. We present the resultant role in Subsection \[Sec: recuresive\ ]. In Step 2, we analyze the long meter behavior of the iterative formula. The behavior depends on the strong convex shape of the cost function, and thus we divide the discussion for the strongly convex and non - convex case, and confront them in Subsection \[Sec: convex\ ] and \[Sec: nonconvex\ ], respectively. 

 Key Recursive Inequalities { # Sec: recuresive } 
 -------------------------- 

 The ancestry of the updating convention for $ { \mathcal{R}}_k$ comes directly from SGD  , which we rewrite into: $ $ \label{eqn: lstep_V } 
 \theta_{k+1}=\theta_k-\eta\nabla f(\theta_k)+\eta V\left(\theta_k\right)\,,$$ where the variation $ $ V\left(\theta\right)=\nabla f(\theta)-\nabla f_{\gamma_k}(\theta)\,.$$ Due to the randomness involve in the selection of $ \gamma_k$, $ V$ is also a random variable with explicitly expressible mean and variance: $ $ \begin{aligned } 
 & \mathbb{E}\left(V\left(\theta\right)\right)=\overrightarrow{0}\,,\\ & \mathrm{Var}\left(V\left(\theta\right)\right)=\frac{1}{J}\sum^{J}_{j=1}\left(\nabla f(\theta)-\nabla f_j(\theta)\right)\left(\nabla f(\theta)-\nabla f_j(\theta)\right)^\top\, . 
 \end{aligned}$$ 

 Building upon these, we accept the following theorem on the recursive formula. 

 \[Lemma: iteration\ ] The following upper and lower bounds hold true if SGD uses fixed step - size $ \eta<\frac{1}{\lambda_{\max,0}}$: 

 -    If the objective function are $ \lambda$-smooth (satisfy Assumption   \[ass: smooth\ ]), then $ $ \label{infinequality } 
     { \mathcal{R}}_{k+1}-{\mathcal{R}}_k\geq \left[-2\eta\lambda_{\
 thf target $\theta^\dagger$. The expectation takes intm accohnt the fandomness in SGD.

The analysid vuildw upon two steps: in Stdp 1, we degive the yppec and lower bounv of the iteraflve fmcmula that updajes ${\mathcal{R}}_{n+1}$ from ${\mathcal{S}}_k$. Tkis step heavily depends on rewritind the SBD formula. We prgsent ehe dvsmlt in Subsection \[Sec:recuresive\]. Jn Step 2, we analyze yhe long time behavior of hhe lterative formula. Hhe behaviot de[wnds on the rtrong connzxity of thg cost function, and thus we sepafate che discussuob flt the strongoy cogvex and non-gpnvex wase, anc present them in Suvsection \[Sec:convex\] anv \[Sec:nonconvex\], respestively.

Kef Xecursive Inequalitiew {#Wec:rewurevive}
--------------------------

Gye aerjvetikn of hhe updating rormula for ${\mathcal{R}}_k$ comes ditesnky from SGD , shich re rewrite into: $$\label{eqn:lstep_V}
\theta_{k+1}=\theta_n-\etz\nabla f(\theta_k)+\eta V\left(\rheta_k\right)\,,$$ where the variatiog $$V\left(\theta\right)=\nabla f(\theta)-\nabla f_{\gamma_k}(\theta)\,.$$ Dge to ghe rwveolness involved in the selection of $\gamma_k$, $V$ if aksp a random varlable with expliciylj rvpressible mexn and vadiance: $$\begin{alignef}
&\mathbb{G}\left(V\oeft(\theta\wighy)\right)=\overrightarrow{0}\,,\\ &\mathrm{Car}\left(V\left(\nhetq\right)\right)=\frac{1}{J}\sul^{J}_{j=1}\left(\nablc f(\theja)-\nabls f_j(\theta)\right)\left(\nabla f(\thsta)-\nabla f_j(\hheta)\righf)^\gop\,.
\end{aligned}$$

Buiudikg gpon these, we have the foljowing thxorem on the recorsive sormula.

\[Lemla:itevdtion\] The followinh uppgr and lower boujds hold true if SGD uses fixed step-size $\eta<\ftac{1}{\namtda_{\max,0}}$:

-   If tme objective fugctions are $\lakbda$-smojth (sxtisfy Asslmption \[asv:smooth\]), theg $$\label{infinexoality}
    {\matical{R}}_{k+1}-{\matrcal{E}}_k\gew \left[-2\egx\lambda_{\
the target $\theta^\dagger$. The expectation takes into randomness SGD. The builds upon two derive upper and lower of the iterative that updates ${\mathcal{R}}_{k+1}$ from ${\mathcal{R}}_k$. This heavily depends on rewriting the SGD formula. We present the result in Subsection In Step 2, we analyze the long time behavior of the iterative formula. behavior on strong of the cost function, and thus we separate the discussion for the strongly convex and non-convex and present them in Subsection \[Sec:convex\] and \[Sec:nonconvex\], Key Recursive Inequalities {#Sec:recuresive} The derivation of the updating for comes directly SGD which rewrite into: $$\label{eqn:lstep_V} f(\theta_k)+\eta V\left(\theta_k\right)\,,$$ where the variation $$V\left(\theta\right)=\nabla f(\theta)-\nabla f_{\gamma_k}(\theta)\,.$$ Due to the randomness involved in the selection of $V$ is random variable explicitly mean variance: $$\begin{aligned} &\mathbb{E}\left(V\left(\theta\right)\right)=\overrightarrow{0}\,,\\ f_j(\theta)\right)\left(\nabla f(\theta)-\nabla f_j(\theta)\right)^\top\,. \end{aligned}$$ Building upon the following theorem on the recursive formula. \[Lemma:iteration\] following upper lower bounds hold true if SGD fixed step-size $\eta<\frac{1}{\lambda_{\max,0}}$: - If the objective functions $\lambda$-smooth (satisfy Assumption \[ass:smooth\]), then $$\label{infinequality} {\mathcal{R}}_{k+1}-{\mathcal{R}}_k\geq \left[-2\eta\lambda_{\
 the target $\theta^\dagger$. The exPectation tAkes iNto AccOuNt thE ranDomness in SGD.

ThE AnalYsis builds upon two steps: In SteP 1, wE DeriVE tHe uppEr and loWEr BOUnd Of ThE itErATiVe forMulA that upDates ${\mathcAl{R}}_{K+1}$ fRom ${\mathcal{R}}_k$. tHiS step heaviLy dEpends on rewrItiNg the SgD ForMUla. We PreSent tHe resuLT in SubSection \[SeC:rECuresiVE\]. In Step 2, WE AnAlyzE the long time behavIOr OF the iterative fOrmula. thE BeHAVioR dePends on the StRong cONvexity OF tHE COst FUnction, and thuS we separate THe dIscussIoN foR The strOngly CoNVex And non-conveX casE, and preseNt them IN SubsecTIon \[Sec:cOnvex\] aNd \[SEc:nOncoNVeX\], rEspEcTIveLY.

KEy RECurSive IneqUaLiTies {#SEc:reCURESive}
--------------------------

the DeriVatioN of the updatinG foRmulA For ${\MathcAl{R}}_k$ cOmes DiRectlY from SgD , whiCh We rewrite into: $$\laBel{eQn:lstep_V}
\tHetA_{k+1}=\TheTa_K-\eta\nABla f(\thEta_K)+\etA V\left(\tHeta_k\riGHt)\,,$$ wHeRE THe Variation $$V\left(\thetA\rIGHt)=\Nabla f(\thEta)-\nabLA f_{\GaMMa_k}(\theta)\,.$$ duE to The rANDomneSs inVOlVed in the SelectIOn Of $\Gamma_k$, $V$ Is Also a rAnDom VarIable WIth eXpliciTly expreSsiblE Mean and variancE: $$\Begin{aligned}
&\mAThBB{e}\lEFt(V\lEft(\Theta\right)\rIght)=\OVerrIghtARrOw{0}\,,\\ &\mAThrm{VAr}\lefT(V\LEfT(\Theta\right)\right)=\frac{1}{j}\sUm^{J}_{j=1}\leFt(\nabLa f(\theta)-\nabla F_j(\theta)\rigHT)\LEft(\nabla F(\theTA)-\nABla f_j(\theta)\righT)^\top\,.
\eNd{aligned}$$

BUIlding upOn theSe, we have The followING theorem On tHe rEcuRsiVE FoRmula.

\[Lemma:iteRATion\] thE followIng Upper anD loWer BouNds HoLd true if SgD uses fiXeD sTeP-sIze $\Eta<\frAC{1}{\lambda_{\mAx,0}}$:

-   if tHe ObjEctivE FunctiOns arE $\lamBdA$-sMOotH (satisfY asSUMptiOn \[AsS:smoOth\]), ThEn $$\labEl{inFIneQuality}
    {\Mathcal{R}}_{k+1}-{\MatHCal{R}}_K\gEq \Left[-2\eta\Lambda_{\
 the target $\theta^\dagge r$. The ex pecta tio n t ak es i ntoaccount the ra n domn ess in SGD.

The analy sis b ui l ds u p on  twosteps:i nS t ep1,  w e d er i ve  theupp er andlower boun d o fthe iterativ e  f ormula tha t u pdates ${\ma thc al{R}} _{ k+1 } $ fro m $ {\mat hcal{R } }_k$.This step  h e avilyd ependso n  r ewri ting the SGD form u la .  We present th e resu lt in S ubs ect ion \[Sec: re cures i ve\]. I n  S t e p  2, we analyze th e long time beh aviorof  th e  itera tivefo r mul a. The beha vior  dependson the strongc onvexit y of t hecos t fu n ct io n,an d  th u swes epa rate the  d is cussi on f o r t he s tro ngly  conv ex and non-co nve x ca s e,and p resen t th em  in S ubsect ion \ [S ec:convex\] and  \[S ec:noncon vex \] , r es pecti v ely.
Key  Re cursive  Inequa l iti es { # Se c:recuresive}
---- -- - - -- -------- ------ - -
T h e deriva ti onof t h e  upda ting fo rmula fo r ${\m a th ca l{R}}_k $comesdi rec tly  from SGD, whic h we rew ritei nto: $$\label{ e qn:lstep_V}
\ t he t a _{ k +1}= \th eta_k-\eta\ nabl a  f(\ thet a _k )+\ e ta V\ left( \t h et a _k\right)\,,$$ wher ethe va riati on $$V\left(\ theta\righ t ) = \nabla f (\th e ta ) -\nabla f_{\ga mma_k }(\theta)\ , .$$ Dueto th e random ness invo l v ed in th e s ele cti ono f  $ \gamma_k$, $V $ is a ls o a ran dom  variab lewit h e xpl ic itly expr essibleme an  a nd  va rianc e : $$\beg in {al ig ned }
&\m a thbb{E }\lef t(V\ le ft ( \th eta\rig h t) \ r ight )= \o verr igh ta rrow{ 0}\, , \\&\mathr m{Var}\le ft( V \lef t( \t heta\ri ght)\right)=\ fr ac{1}{J}\s um ^{J }_{j=1 } \ left(\na bla f(\theta)-\nabla f_ j (\theta )\r ight) \lef t(\nablaf(\ theta) -\n a bla f_ j(\the ta)\r ig ht) ^ \ top\, . \e nd{ al igned}$$
B u ild ing u po n th ese, we  have the followin g  th eorem on therec ursi v e  f orm u la . 

\ [L e mma : i teration\] Thefollowingup p er  and lower bou nd s holdtrue if  SGDu ses fix ed step-s ize $\eta <\ frac { 1 }{\ lambda_{\m ax,0}}$: 

-   Ift he ob j ec tivefun ctions  a re$\lam bda$-s m oot h (sa tisfyAs sumpti on \[ as s:smooth \]), then $$\label{infi nequal ity}   {\mathca l{R } }_{ k+1}-{\ma thca l{R}}_k\ge q \ lef t[-2\ eta \ lambd a_{\ 
 the_target $\theta^\dagger$._The expectation takes into_account the_randomness_in SGD.

The_analysis_builds upon two_steps: in Step_1, we derive the_upper and lower_bound_of the iterative formula that updates ${\mathcal{R}}_{k+1}$ from ${\mathcal{R}}_k$. This step heavily depends on_rewriting_the SGD_formula._We_present the result in Subsection_\[Sec:recuresive\]. In Step 2, we_analyze the_long time behavior of the iterative formula. The_behavior_depends on the_strong convexity of the cost function, and thus we_separate the discussion for the strongly_convex and non-convex_case,_and_present them in Subsection_\[Sec:convex\] and \[Sec:nonconvex\], respectively.

Key Recursive Inequalities_{#Sec:recuresive}
--------------------------

The derivation of the updating formula_for ${\mathcal{R}}_k$ comes directly from SGD , which_we rewrite into: $$\label{eqn:lstep_V}
\theta_{k+1}=\theta_k-\eta\nabla f(\theta_k)+\eta V\left(\theta_k\right)\,,$$_where the variation $$V\left(\theta\right)=\nabla f(\theta)-\nabla_f_{\gamma_k}(\theta)\,.$$ Due_to the randomness involved in_the selection of_$\gamma_k$, $V$_is also a_random variable with explicitly expressible mean_and variance: $$\begin{aligned}
&\mathbb{E}\left(V\left(\theta\right)\right)=\overrightarrow{0}\,,\\_&\mathrm{Var}\left(V\left(\theta\right)\right)=\frac{1}{J}\sum^{J}_{j=1}\left(\nabla f(\theta)-\nabla f_j(\theta)\right)\left(\nabla f(\theta)-\nabla f_j(\theta)\right)^\top\,.
\end{aligned}$$

Building upon_these,_we have the_following_theorem_on the_recursive formula.

\[Lemma:iteration\] The_following_upper and_lower_bounds hold true if SGD uses_fixed_step-size $\eta<\frac{1}{\lambda_{\max,0}}$:

-   If the objective_functions are $\lambda$-smooth (satisfy_Assumption \[ass:smooth\]),_then $$\label{infinequality}
  _ {\mathcal{R}}_{k+1}-{\mathcal{R}}_k\geq \left[-2\eta\lambda_{\