, our lower bound is rather loose. However, as shown in Figure \[fig:err2\_2\], even when SGD is trapped at $\theta^\dagger$ the global minimum, there is still an error lower bound, which is very similar to the results in Section \[sec:LR\].

Intuitively, when the objective function is non-convex and has local minima, SGD might get trapped in the local minima. If so, it is obvious that there exists an error lower bound. Our theorems show that even if SGD escapes from the local minimum and approaches the global minimum, there is still a positive error lower bound because of the randomness in constant step-size SGD.

Conclusion
==========

In this paper, we study the error lower bound of constant step-size SGD. We use expected $L_2$ distance from the global minimum to characterize the error. Our theoretical results show that for potentially non-convex objective function with Lipschitz gradient, there always exists a positive error lower bound for SGD. The lower bounds account for a non-decaying term in classic SGD error upper bound analysis.

Proof of Theorem 3.2
====================

We first reformulate the right hand side of the iterative formula $$\label{supinequalityapp}
{\mathcal{R}}_{k+1}-{\mathcal{R}}_k\leq \left[-2\eta\lambda_{\min}+\eta^2\mathsf{\Phi}^2\right]{\mathcal{R}}_k+2\eta^2\Lambda D_0{\mathcal{R}}^{1/2}_{k}+\eta^2D^2_0\,.$$ as a quadratic function, with ${\mathcal{R}}_k^{1/2}$ replaced by $x$: $$H(x)=\eta\left(\left[-2\lambda_{\min}+\eta{\mathsf{\Phi}}^2\right]x^2+2\eta\Lambda D_0x+\eta D^2_0\right)\,.$$ Since $\eta<\frac{\lambda_{\min}}{\Lambda^2+\lambda^2_{\max,0}}<\frac{\lambda_{\min}}{\Lambda^2+\lambda^2_{\max,0}-\lambda^2_{\min}}$, the coefficient of the quadratic term is negative and the function has a maximum value. By direct calculation, we obtain $$\max\,H(x)= \eta^2\left(\frac{\eta\Lambda^2}{2\lambda_{\
, our lower bound is rather loose. However, as shown in Figure   \[fig: err2\_2\ ], even when SGD is trap at $ \theta^\dagger$ the ball-shaped minimum, there is still an mistake lower restrict, which is very similar to the results in Section   \[sec: LR\ ]. 

 Intuitively, when the objective routine is non - convex and have local minima, SGD might get trapped in the local minimum. If so, it is obvious that there exists an erroneousness lower bound. Our theorems read that even if SGD escape from the local minimum and approaches the global minimum, there is still a positive mistake lower bound because of the randomness in constant step - size SGD. 

 Conclusion 
 = = = = = = = = = = 

 In this newspaper, we study the error lower bound of constant measure - size SGD. We use expected $ L_2 $ distance from the global minimum to characterize the error. Our theoretical result show that for potentially non - convex objective function with Lipschitz gradient, there always exists a positive error lower bound for SGD. The lower bounds account for a non - decaying term in classic SGD error upper bound analysis. 

 Proof of Theorem 3.2 
 = = = = = = = = = = = = = = = = = = = = 

 We first reformulate the correct hand side of the iterative rule $ $ \label{supinequalityapp } 
 { \mathcal{R}}_{k+1}-{\mathcal{R}}_k\leq \left[-2\eta\lambda_{\min}+\eta^2\mathsf{\Phi}^2\right]{\mathcal{R}}_k+2\eta^2\Lambda D_0{\mathcal{R}}^{1/2}_{k}+\eta^2D^2_0\,.$$ as a quadratic function, with $ { \mathcal{R}}_k^{1/2}$ replaced by $ x$: $ $ H(x)=\eta\left(\left[-2\lambda_{\min}+\eta{\mathsf{\Phi}}^2\right]x^2 + 2\eta\Lambda D_0x+\eta D^2_0\right)\,.$$ Since $ \eta<\frac{\lambda_{\min}}{\Lambda^2+\lambda^2_{\max,0}}<\frac{\lambda_{\min}}{\Lambda^2+\lambda^2_{\max,0}-\lambda^2_{\min}}$, the coefficient of the quadratic term is negative and the affair has a maximum value. By direct calculation, we obtain $ $ \max\,H(x)= \eta^2\left(\frac{\eta\Lambda^2}{2\lambda_{\
, oug lower bound is rather uoose. However, as shown in Fifure \[fig:efr2\_2\], even when SGD is trapped et $\tyeta^\dqgger$ the global minimjm, there ps still qn ecror lower bound, which is very diminer to the resuljs in Sectiot \[sec:LR\].

Intuitivalh, chen the objective function is non-cogvex anc jas local miniia, SBQ mifht get trapped in the local minimz. If so, it is obvioux that there exists an errlr llwer bound. Our thelrems show jgat wven if SGD dscapes from the local minimum and approaches the glocal mnnimum, therg js vtill a posmtive vrror lower bound becduse of the randomnesx ii cobstant step-size SGD.

Coiclusion
==========

In this papet, we study tke error lower bound if consjant vtep-ruze SGS. Xe hse exoecved $L_2$ distahce from thw global minimum to crqracterize ths errow. Jur theoretical results show that for pmtehtially non-convex objecrive function with Likschitz grwdient, there always exists a positive error lower bounv wor SND. Tfw power bounds account for a non-decaying term ig ckaxsic SGD error upper bounc wnsjysis.

Proof of Theorzj 3.2
====================

Se first reformulahe the tight yand side of yhe iterative formula $$\label{wupinequalitjapp}
{\nathcal{R}}_{k+1}-{\mathcal{R}}_k\peq \left[-2\eta\uambca_{\min}+\rta^2\mathsf{\Phi}^2\right]{\mathcau{R}}_k+2\efa^2\Lambda D_0{\mwthcal{R}}^{1/2}_{k}+\efx^2D^2_0\,.$$ as a quadratiz flncthon, with ${\mathcal{R}}_k^{1/2}$ replaceq by $x$: $$H(x)=\xta\leyt(\left[-2\laobda_{\kin}+\eta{\iathsf{\Phi}}^2\rlght]x^2+2\cda\Lambda D_0x+\eta D^2_0\rihht)\,.$$ Snnce $\ata<\frac{\lamhda_{\min}}{\Lambda^2+\lambda^2_{\max,0}}<\frac{\lambda_{\min}}{\Lambda^2+\lambds^2_{\mdx,0}-\ldmbda^2_{\min}}$, the goefficient of ehe quadratic jerm is nzgativd and the runctioi has a maxiium value. By glrect calculetion, we jbtaun $$\mqx\,H(x)= \etx^2\ueft(\frac{\eta\Lamnda^2}{2\lambda_{\
, our lower bound is rather loose. shown Figure \[fig:err2\_2\], when SGD is minimum, is still an lower bound, which very similar to the results in \[sec:LR\]. Intuitively, when the objective function is non-convex and has local minima, SGD get trapped in the local minima. If so, it is obvious that there an lower Our show that even if SGD escapes from the local minimum and approaches the global minimum, there still a positive error lower bound because of randomness in constant step-size Conclusion ========== In this paper, study error lower of step-size We use expected distance from the global minimum to characterize the error. Our theoretical results show that for potentially non-convex function with there always a error bound for SGD. bounds account for a non-decaying term error upper bound analysis. Proof of Theorem 3.2 We first the right hand side of the formula $$\label{supinequalityapp} {\mathcal{R}}_{k+1}-{\mathcal{R}}_k\leq \left[-2\eta\lambda_{\min}+\eta^2\mathsf{\Phi}^2\right]{\mathcal{R}}_k+2\eta^2\Lambda D_0{\mathcal{R}}^{1/2}_{k}+\eta^2D^2_0\,.$$ as a quadratic with ${\mathcal{R}}_k^{1/2}$ replaced by $x$: $$H(x)=\eta\left(\left[-2\lambda_{\min}+\eta{\mathsf{\Phi}}^2\right]x^2+2\eta\Lambda D_0x+\eta D^2_0\right)\,.$$ Since $\eta<\frac{\lambda_{\min}}{\Lambda^2+\lambda^2_{\max,0}}<\frac{\lambda_{\min}}{\Lambda^2+\lambda^2_{\max,0}-\lambda^2_{\min}}$, the coefficient of the quadratic negative and the function a maximum value. direct we $$\max\,H(x)=
, our lower bound is rather loosE. However, as Shown In FIguRe \[Fig:eRr2\_2\], evEn when SGD is traPPed aT $\theta^\dagger$ the global mInimuM, tHEre iS StIll an Error loWEr BOUnd, WhIcH is VeRY sImilaR to The resuLts in SectiOn \[sEc:lR\].

IntuitivelY, WhEn the objecTivE function is nOn-cOnvex aNd Has LOcal mIniMa, SGD Might gET trappEd in the loCaL Minima. iF so, it is OBViOus tHat there exists an eRRoR Lower bound. Our tHeoremS sHOw THAt eVen If SGD escapEs From tHE local mINiMUM And APproaches the gLobal minimuM, TheRe is stIlL a pOSitive Error LoWEr bOund because Of thE randomneSs in coNStant stEP-size SGd.

ConclUsiOn
==========

IN thiS PaPeR, we StUDy tHE eRroR LowEr bound oF cOnStant Step-SIZE sGD. WE usE expEcted $l_2$ distance from The GlobAL miNimum To chaRactErIze thE error. our thEoRetical results sHow tHat for potEntIaLly NoN-convEX objecTivE fuNction wIth LipsCHitZ gRADIeNt, there always existS a POSiTive erroR lower BOuNd FOr SGD. The LoWer BounDS AccouNt foR A nOn-decayiNg term IN cLaSsic SGD ErRor uppEr BouNd aNalysIS.

ProOf of ThEorem 3.2
====================

We fIrst rEFormulate the riGHt hand side of tHE iTERaTIve fOrmUla $$\label{supIneqUAlitYapp}
{\MAtHcaL{r}}_{k+1}-{\matHcal{R}}_K\lEQ \lEFt[-2\eta\lambda_{\min}+\eta^2\maThSf{\Phi}^2\rIght]{\mAthcal{R}}_k+2\eta^2\LaMbda D_0{\mathcAL{r}}^{1/2}_{K}+\eta^2D^2_0\,.$$ as a QuadRAtIC function, with ${\mAthcaL{R}}_k^{1/2}$ replaceD By $x$: $$H(x)=\eta\Left(\lEft[-2\lambdA_{\min}+\eta{\maTHSf{\Phi}}^2\rigHt]x^2+2\Eta\lamBda d_0X+\EtA D^2_0\right)\,.$$ Since $\eTA<\Frac{\LaMbda_{\min}}{\lamBda^2+\lambDa^2_{\mAx,0}}<\fRac{\LamBdA_{\min}}{\LambdA^2+\lambda^2_{\mAx,0}-\LaMbDa^2_{\Min}}$, The coEFficient Of The QuAdrAtic tERm is neGativE and ThE fUNctIon has a MAxIMUm vaLuE. BY dirEct CaLculaTion, WE obTain $$\max\,h(x)= \eta^2\left(\FraC{\Eta\LAmBdA^2}{2\lambda_{\
, our lower bound is rathe r loose. H oweve r,assh ownin F igure \[fig:er r 2\_2 \], even when SGD is t rappe da t $\ t he ta^\d agger$t he g lob al  m ini mu m ,there  is  stillan error l owe rbound, which is  very simi lar  to the resu lts  in Se ct ion \[sec :LR \].
Intuit i vely,when theob j ective functio n is  non -convex and has l o ca l  minima, SGD m ight g et tr a p ped  in  the local  m inima .  If so, it i s  ob v ious that the re exists a n  er ror lo we r b o und. O ur th eo r ems  show thateven  if SGD e scapes from th e  localminimu m a ndappr o ac he s t he glo b al  mi n imu m, there  i sstill  a p o s i t iveerr or l owerbound because  of  the ran domne ss in  con st ant s tep-si ze SG D. 

Conclusion
== ==== ====

Inthi spap er , wes tudy t heerr or lowe r bound ofco n s t an t step-size SGD. W eu s eexpected  $L_2$ di st a nce from  t heglob a l  mini mumt ocharacte rize t h eer ror. Ou rtheore ti cal  re sults show  thatfor pote ntial l y non-convex o b jective funct i on w it h  Lip sch itz gradien t, t h erealwa y sexi s ts aposit iv e  e r ror lower bound for  S GD. Th e low er bounds acc ount for a n o n-decayi ng t e rm in classic SGD  erro r upper bo u nd analy sis.
Proof o f Theorem 3 .2
===== === === === === = = =
We first ref o r mula te  the ri ght  hand s ide  of  th e i te rative fo rmula $$ \l ab el {s upi nequa l ityapp}{\ mat hc al{ R}}_{ k +1}-{\ mathc al{R }} _k \ leq  \left[ - 2\ e t a\la mb da _{\m in} +\ eta^2 \mat h sf{ \Phi}^2 \right]{\ mat h cal{ R} }_ k+2\eta ^2\Lambda D_0 {\ mathcal{R} }^ {1/ 2}_{k} + \ eta^2D^2 _0\,.$$ as a quadraticf unction , w ith $ {\ma thcal{R}} _k^ {1/2}$  re p lacedby $x$ : $$H (x )=\ e t a\lef t ( \l eft [- 2\lambda_{ \ m in} +\eta {\ math sf{\Phi }}^2\right]x^2+2\e t a\L ambda D_0x+\e taD^2_ 0 \ ri ght ) \, . $$Si n ce$ \ eta<\frac{\lamb da_{\min}} {\ L am bda^2+\lam b da^ 2_ {\max,0 }}<\fra c{\la m bda_{\m in}}{\Lam bda^2+\la mb da^2 _ { \ma x,0}-\lamb da^2_{\m in}}$, th e  coef f ic ientofthe qu ad rat ic te rm isn ega tiveand th efuncti on ha sa maximu m value. By direct calc ulatio n, we  ob tain $$\m ax\ , H(x )= \eta^2 \lef t(\frac{\e ta\ Lam bda^2 }{2 \ lambd a_{\ 
, our_lower bound_is rather loose. However,_as shown_in_Figure \[fig:err2\_2\], even_when_SGD is trapped_at $\theta^\dagger$ the_global minimum, there is_still an error_lower_bound, which is very similar to the results in Section \[sec:LR\].

Intuitively, when the objective function_is_non-convex and_has_local_minima, SGD might get trapped_in the local minima. If_so, it_is obvious that there exists an error lower_bound._Our theorems show_that even if SGD escapes from the local minimum_and approaches the global minimum, there_is still a_positive_error_lower bound because of_the randomness in constant step-size SGD.

Conclusion
==========

In_this paper, we study the error_lower bound of constant step-size SGD. We_use expected $L_2$ distance from the_global minimum to characterize the_error. Our_theoretical results show that for_potentially non-convex objective_function with_Lipschitz gradient, there_always exists a positive error lower_bound for SGD._The lower bounds account for a_non-decaying_term in classic_SGD_error_upper bound_analysis.

Proof of Theorem_3.2
====================

We_first reformulate_the_right hand side of the iterative_formula_$$\label{supinequalityapp}
{\mathcal{R}}_{k+1}-{\mathcal{R}}_k\leq \left[-2\eta\lambda_{\min}+\eta^2\mathsf{\Phi}^2\right]{\mathcal{R}}_k+2\eta^2\Lambda D_0{\mathcal{R}}^{1/2}_{k}+\eta^2D^2_0\,.$$ as a quadratic function,_with ${\mathcal{R}}_k^{1/2}$ replaced by_$x$:_$$H(x)=\eta\left(\left[-2\lambda_{\min}+\eta{\mathsf{\Phi}}^2\right]x^2+2\eta\Lambda D_0x+\eta D^2_0\right)\,.$$ Since_$\eta<\frac{\lambda_{\min}}{\Lambda^2+\lambda^2_{\max,0}}<\frac{\lambda_{\min}}{\Lambda^2+\lambda^2_{\max,0}-\lambda^2_{\min}}$, the coefficient of the_quadratic term is negative and the_function has_a maximum_value. By direct calculation, we obtain $$\max\,H(x)= \eta^2\left(\frac{\eta\Lambda^2}{2\lambda_{\