 \ldots, n\}$ with replacement. The three minibatches are sampled independently and are also independent of other iterations.

Intuitively, an $(s, \sigma, b_1, b_2)$-large-noise SGD should be considered as an SGD algorithm with step size $s$ and minibatch size $b_1$ and an additional noise term. The noise term computes the difference of two independent and unbiased estimates of the full gradient $\nabla U(w_k)$, each using a batch of $b_2$ data points. Using the definition of $\zeta$ in , we can verify that the update  is equivalent to $$\begin{aligned}
\numberthis\label{e:sgd-noisy-2}
    & w_{k+1} = w_k - s\nabla U(w_k) + s\zeta(w_k, \eta_k) \\
    &\qquad + \sigma \sqrt{s}(\zeta(w_k, \eta_k'') - \zeta(w_k, \eta_k')),\end{aligned}$$ which is in the form of, with $$\begin{aligned}
\xi(w, \tilde{\eta}) = \sqrt{s} \zeta(w, \eta) + \sigma \lrp{\zeta(w, \eta'')-\zeta(w, \eta')},
\numberthis
\label{e:sgd-noisy-xi}\end{aligned}$$ where $\tilde{\eta} = (\eta, \eta', \eta'')$, and $|\eta| = b_1$, $|\eta'|=|\eta''|=b_2$. Further, the noise covariance matrix is $$\label{e:noise_cov_large_noise_sgd}
    \E{\xi(w, \tilde{\eta}) \xi(w, \tilde{\eta})^T} = (\frac{s}{b_1} + \frac{2\sigma^2}{b_2}) H(w).$$ Therefore, if we have $$\label{e:var_matching}
    \frac{s}{b_1} + \frac{2\sigma^2}{b_2} = \frac{\delta}{b},$$ then an $(s, \sigma, b_1, b_2)$-large-noise SGD should have the same noise covariance as
\ldots, n\}$ with replacement. The three minibatches are sampled independently and are also independent of early iteration. 

 Intuitively, an $ (s, \sigma, b_1, b_2)$-large - noise SGD should be considered as an SGD algorithm with gradation size $ s$ and minibatch size $ b_1 $ and an extra noise term. The noise condition computes the difference of two autonomous and indifferent estimates of the full gradient $ \nabla U(w_k)$, each use a batch of $ b_2 $ data points. Using the definition of $ \zeta$ in  , we can control that the update    is equivalent to $ $ \begin{aligned } 
 \numberthis\label{e: sgd - noisy-2 } 
     & w_{k+1 } = w_k - s\nabla U(w_k) + s\zeta(w_k, \eta_k) \\ 
     & \qquad + \sigma \sqrt{s}(\zeta(w_k, \eta_k") - \zeta(w_k, \eta_k')),\end{aligned}$$ which is in the form of, with $ $ \begin{aligned } 
 \xi(w, \tilde{\eta }) = \sqrt{s } \zeta(w, \eta) + \sigma \lrp{\zeta(w, \eta'')-\zeta(w, \eta') }, 
 \numberthis 
 \label{e: sgd - noisy - xi}\end{aligned}$$ where $ \tilde{\eta } = (\eta, \eta', \eta'')$, and $ |\eta| = b_1 $, $ |\eta'|=|\eta''|=b_2$. Further, the noise covariance matrix is $ $ \label{e: noise_cov_large_noise_sgd } 
     \E{\xi(w, \tilde{\eta }) \xi(w, \tilde{\eta})^T } = (\frac{s}{b_1 } + \frac{2\sigma^2}{b_2 }) H(w).$$ Therefore, if we have $ $ \label{e: var_matching } 
     \frac{s}{b_1 } + \frac{2\sigma^2}{b_2 } = \frac{\delta}{b},$$ then an $ (s, \sigma, b_1, b_2)$-large - noise SGD should suffer the same noise covariance as
 \ldlts, n\}$ with replacement. Tme three minibateyes arx samplsd indepdndently and are also indepeidenr of ither iterations.

Intuitkvely, an $(d, \sigma, v_1, b_2)$-lerge-noise SGD should be gjnsiscred cs an SGD algorijhm with ste[ size $s$ and mhnkbctch size $b_1$ and an additional noise eerm. Thr joise term comkutes ehe spfnerence of two independent and uhbiased estimates of the full gradient $\nabla U(a_k)$, ewch using a batch lf $b_2$ data piintf. Using the ddfinition of $\zeta$ in , wg can verify that the update  is dquivclent to $$\betib{allcned}
\numbertiis\labvl{e:sgd-noisy-2}
    & w_{k+1} = w_n - s\nabka U(w_k) + s\zeta(w_l, \eva_k) \\
    &\qquad + \sigma \sqrt{v}(\zeta(w_k, \eta_k'') - \zeta(r_k, \eta_k')),\eng{amigned}$$ which is ib rhe fmrm mf, wkrh $$\cegjn{elifned}
\xi(a, \tmlde{\eta}) = \sqdt{s} \zeta(w, \era) + \sigma \lrp{\zeta(w, \tta'')-\svya(w, \eta')},
\numbedthis
\lwbql{e:sgd-noisy-xi}\end{aligned}$$ where $\tilde{\eta} = (\etz, \eta', \eta'')$, and $|\eta| = b_1$, $|\etq'|=|\eta''|=b_2$. Further, the noide covariwnce matrix is $$\label{e:noise_cov_large_noise_sgd}
    \E{\xi(f, \tilve{\dta}) \xl(w, \tkodf{\eta})^T} = (\frac{s}{b_1} + \frac{2\sigma^2}{b_2}) H(w).$$ Therefore, if we rzvt $$\lsbel{e:var_matchikg}
    \frac{s}{b_1} + \frac{2\xihms^2}{f_2} = \frac{\delta}{b},$$ then ch $(a, \sigma, b_1, b_2)$-large-nolse SGD shouod have tre ssme noise covariance as
\ldots, n\}$ with replacement. The three minibatches independently are also of other iterations. b_2)$-large-noise should be considered an SGD algorithm step size $s$ and minibatch size and an additional noise term. The noise term computes the difference of two and unbiased estimates of the full gradient $\nabla U(w_k)$, each using a batch $b_2$ points. the of $\zeta$ in , we can verify that the update is equivalent to $$\begin{aligned} \numberthis\label{e:sgd-noisy-2} & = w_k - s\nabla U(w_k) + s\zeta(w_k, \eta_k) &\qquad + \sigma \sqrt{s}(\zeta(w_k, - \zeta(w_k, \eta_k')),\end{aligned}$$ which is the of, with \xi(w, = \zeta(w, \eta) + \lrp{\zeta(w, \eta'')-\zeta(w, \eta')}, \numberthis \label{e:sgd-noisy-xi}\end{aligned}$$ where $\tilde{\eta} = (\eta, \eta', \eta'')$, and $|\eta| = b_1$, $|\eta'|=|\eta''|=b_2$. Further, noise covariance $$\label{e:noise_cov_large_noise_sgd} \E{\xi(w, \xi(w, = + \frac{2\sigma^2}{b_2}) H(w).$$ we have $$\label{e:var_matching} \frac{s}{b_1} + \frac{2\sigma^2}{b_2} an $(s, \sigma, b_1, b_2)$-large-noise SGD should have same noise as
 \ldots, n\}$ with replacement. The tHree minibaTches Are SamPlEd inDepeNdently and are aLSo inDependent of other iteratIons.

INtUItivELy, An $(s, \siGma, b_1, b_2)$-laRGe-NOIse sGd sHouLd BE cOnsidEreD as an SGd algorithm WitH sTep size $s$ and mINiBatch size $b_1$ And An additional NoiSe term. thE noISe terM coMputeS the diFFerencE of two indEpENdent aND unbiasED EsTimaTes of the full gradiENt $\NAbla U(w_k)$, each usiNg a batCh OF $b_2$ DATa pOinTs. Using the DeFinitIOn of $\zetA$ In , WE CAn vERify that the upDate  is equivALenT to $$\begIn{AliGNed}
\numBerthIs\LAbeL{e:sgd-noisy-2}
    & w_{K+1} = w_k - s\Nabla U(w_k) + s\Zeta(w_k, \ETa_k) \\
    &\qquaD + \Sigma \sqRt{s}(\zetA(w_k, \Eta_K'') - \zetA(W_k, \EtA_k')),\eNd{ALigNEd}$$ WhiCH is In the forM oF, wIth $$\beGin{aLIGNEd}
\xi(W, \tiLde{\eTa}) = \sqrT{s} \zeta(w, \eta) + \sigMa \lRp{\zeTA(w, \eTa'')-\zetA(w, \eta')},
\NumbErThis
\lAbel{e:sGd-noiSy-Xi}\end{aligned}$$ wheRe $\tiLde{\eta} = (\eta, \Eta', \EtA'')$, anD $|\eTa| = b_1$, $|\etA'|=|\Eta''|=b_2$. FuRthEr, tHe noise CovariaNCe mAtRIX Is $$\Label{e:noise_cov_largE_nOISe_Sgd}
    \E{\xi(w, \tIlde{\etA}) \Xi(W, \tILde{\eta})^T} = (\fRaC{s}{b_1} + \Frac{2\SIGma^2}{b_2}) H(W).$$ TheREfOre, if we hAve $$\labEL{e:VaR_matchiNg}
    \Frac{s}{b_1} + \FrAc{2\sIgmA^2}{b_2} = \fraC{\DeltA}{b},$$ then An $(s, \sigma, B_1, b_2)$-larGE-noise SGD shoulD Have the same noISe COVaRIancE as
 \ldots, n\}$ with replace ment. Thethree  mi nib at ches  are  sampled indep e nden tly and are also indep enden to f ot h er  iter ations. 
I n t uit iv el y,an $( s, \s igm a, b_1,  b_2)$-lar ge- no ise SGD shou l dbe conside red  as an SGD a lgo rithmwi ths tep s ize  $s$and mi n ibatch  size $b_ 1$ and an additio n a lnois e term. The noise te r m computes the  diffe re n ce o f t woindependen tand u n biasede st i m a tes of the full g radient $\n a bla  U(w_k )$ , e a ch usi ng aba t chof $b_2$ da ta p oints. Us ing th e  defini t ion of$\zeta $ i n ,  wec an  v eri fy tha t  t heu pda te  is e qu iv alent  to$ $ \ b egin {al igne d}
\n umberthis\lab el{ e:sg d -no isy-2 }
   & w _{ k+1}= w_k- s\n ab la U(w_k) + s\z eta( w_k, \eta _k)  \ \
  &\qq u ad + \ sig ma\sqrt{s }(\zeta ( w_k ,\ e t a_ k'') - \zeta(w_k,\e t a _k ')),\end {align e d} $$ which is  i n t he f o r m of,  wit h  $ $\begin{ aligne d }\x i(w, \t il de{\et a} ) =  \s qrt{s }  \ze ta(w,\eta) +\sigm a  \lrp{\zeta(w, \eta'')-\zeta ( w, \ et a ')}, 
\n umberthis
\ labe l {e:s gd-n o is y-x i }\end {alig ne d }$ $  where $\tilde{\eta }= (\et a, \e ta', \eta'')$ , and $|\e t a |  = b_1$,  $|\ e ta ' |=|\eta''|=b_2 $. Fu rther, the noise co varia nce matr ix is $$\ l a bel{e:no ise _co v_l arg e _ no ise_sgd}
   \ E {\xi (w , \tild e{\ eta}) \ xi( w,\ti lde {\ eta})^T}= (\frac {s }{ b_ 1}  +\frac { 2\sigma^ 2} {b_ 2} ) H (w).$ $  There fore,  ifwe  h a ve$$\labe l {e : v ar_m at ch ing} 
  \frac {s}{ b _1}  + \fra c{2\sigma ^2} { b_2}  =  \ frac{\d elta}{b},$$ t he n an $(s,\s igm a, b_1 , b_2)$-la rge-noise SGD should ha v e the s ame  nois e co varianceas
 \ldots,_n\}$ with_replacement. The three minibatches_are sampled_independently_and are_also_independent of other_iterations.

Intuitively, an $(s,_\sigma, b_1, b_2)$-large-noise SGD_should be considered_as_an SGD algorithm with step size $s$ and minibatch size $b_1$ and an additional_noise_term. The_noise_term_computes the difference of two_independent and unbiased estimates of_the full_gradient $\nabla U(w_k)$, each using a batch of_$b_2$_data points. Using_the definition of $\zeta$ in , we can verify that_the update  is equivalent to $$\begin{aligned}
\numberthis\label{e:sgd-noisy-2}
_  _&_w_{k+1}_= w_k - s\nabla_U(w_k) + s\zeta(w_k, \eta_k) \\
 _  &\qquad + \sigma \sqrt{s}(\zeta(w_k,_\eta_k'') - \zeta(w_k, \eta_k')),\end{aligned}$$ which is in_the form of, with $$\begin{aligned}
\xi(w, \tilde{\eta})_= \sqrt{s} \zeta(w, \eta) +_\sigma \lrp{\zeta(w,_\eta'')-\zeta(w, \eta')},
\numberthis
\label{e:sgd-noisy-xi}\end{aligned}$$ where $\tilde{\eta} =_(\eta, \eta', \eta'')$,_and $|\eta|_= b_1$, $|\eta'|=|\eta''|=b_2$._Further, the noise covariance matrix is_$$\label{e:noise_cov_large_noise_sgd}
  _ \E{\xi(w, \tilde{\eta}) \xi(w, \tilde{\eta})^T} =_(\frac{s}{b_1}_+ \frac{2\sigma^2}{b_2}) H(w).$$_Therefore,_if_we have_$$\label{e:var_matching}
  __\frac{s}{b_1} +_\frac{2\sigma^2}{b_2}_= \frac{\delta}{b},$$ then an $(s, \sigma,_b_1,_b_2)$-large-noise SGD should have the same noise_covariance as