▁access ▁to ▁no isy ▁samples ▁of ▁the ▁( gradient ▁of ▁the ) ▁function . ▁The ▁goal ▁is ▁to ▁have ▁an ▁estimate ▁of ▁the ▁optimal ▁parameter ▁( min im izer ) ▁per ▁new ▁sample . ▁AD MM ▁is ▁originally ▁a ▁batch ▁method . ▁However , ▁with ▁some ▁modifications ▁it ▁can ▁also ▁be ▁used ▁for ▁st ochastic ▁optimization . ▁Since ▁in ▁st ochastic ▁setting ▁we ▁only ▁have ▁access ▁to ▁no isy ▁samples ▁of ▁gradient , ▁we ▁use ▁an ▁in ex act ▁approximation ▁of ▁the ▁L agr ang ian ▁as ▁ ▁where ▁ ▁is ▁a ▁time - v ary ing ▁step ▁size . ▁ ▁The ▁altern ating ▁direction ▁method ▁of ▁multip liers ▁( AD MM ) ▁ ▁is ▁a ▁popular ▁method ▁for ▁online ▁and ▁distributed ▁optimization ▁on ▁a ▁large ▁scale , ▁and ▁is ▁employed ▁in ▁many ▁applications , ▁e . g . ▁AD MM ▁is ▁often ▁applied ▁to ▁solve ▁regular ized ▁problems , ▁where ▁the ▁function ▁optimization ▁and ▁regular ization ▁can ▁be ▁carried ▁out ▁locally , ▁and ▁then ▁coordin ated ▁glob ally ▁via ▁constraints . ▁Reg ular ized ▁optimization ▁problems ▁are ▁especially ▁relevant ▁in ▁the ▁high ▁dimensional ▁regime ▁since ▁regular ization ▁is ▁a ▁natural ▁mechanism ▁to ▁overcome ▁ill - posed ness ▁and ▁to ▁encou rage ▁par sim ony ▁in ▁the ▁optimal ▁solution , ▁e . g ., ▁sp ars ity ▁and ▁low ▁rank . ▁Due ▁to ▁the ▁efficiency ▁of ▁AD MM ▁in ▁solving ▁regular ized ▁problems , ▁it ▁has ▁a ▁good ▁potential ▁for ▁st ochastic ▁optimization ▁in ▁high ▁dimensions . ▁However , ▁conventional ▁st ochastic ▁AD