Randomized Iterative Methods for Linear Systems: Momentum, Inexactness and Gossip
Nicolas Loizou




 

=1










𝐱
ŁL



    




Input:
Initialize:




















































































plain
thmTheorem[]
factFact[]

lem[thm]Lemma
con[thm]Conjecture
que[thm]Question
prop[thm]Proposition
defn[thm]Definition
remRemark[]
cor[thm]Corollary

remark

exaExample[]
*noteNote


 
















 
 

*assumption*

assumption[2]
 
  
  
  @edefcurrentlabel#1#2
 
 
  
 
























P[1]>p#1
M[1]>m#1

 
 
 
 
















 Randomized Iterative Methods for Linear Systems: Momentum, Inexactness and Gossip
    Nicolas Loizou
    2019
==================================================================================





To my parents, 
 Christaki and Agathi.


In the era of big data, one of the key challenges is the development of novel optimization algorithms that can accommodate vast amounts of data while at the same time satisfying constraints and limitations of the problem under study. The need to solve optimization problems is ubiquitous in essentially all quantitative areas of human endeavor, including industry and science. In the last decade there has been a surge in the demand from practitioners, in fields such as machine learning, computer vision, artificial intelligence, signal processing and data science, for new methods able to cope with these new large scale problems. 

In this thesis we are focusing on the design, complexity analysis and efficient implementations of such algorithms. In particular, we are interested in the development of randomized first order iterative methods for solving large scale linear systems, stochastic quadratic optimization problems and the distributed average consensus problem. 

In Chapter <ref>, we study several classes of stochastic optimization algorithms enriched with  heavy ball momentum. Among the methods studied are: stochastic gradient descent, stochastic Newton, stochastic proximal point  and stochastic dual subspace ascent. This is the first time momentum variants of several of these methods are studied. We choose to perform our analysis in a setting in which all of the above methods are equivalent: convex quadratic problems. We prove global non-asymptotic linear convergence rates for all methods and various measures of success, including primal function values, primal iterates, and dual function values. We also show that the primal iterates converge at an accelerated linear rate in a somewhat weaker sense. This is the first time a linear rate is shown for the stochastic heavy ball method (i.e., stochastic gradient descent method with momentum). Under somewhat weaker conditions,  we establish a sublinear convergence rate for Cesàro averages of primal iterates.  Moreover, we propose a novel concept, which we call stochastic momentum, aimed at decreasing the cost of performing the momentum step. We prove linear convergence of several stochastic methods with stochastic momentum, and show that in some sparse data regimes and for sufficiently small momentum parameters, these methods enjoy better overall complexity than methods with deterministic momentum. Finally, we perform extensive numerical testing on artificial and real datasets. 

In Chapter <ref>, we present a convergence rate analysis of inexact variants of stochastic gradient descent, stochastic Newton, stochastic proximal point and stochastic subspace ascent. A common feature of these methods is that in their update rule a certain sub-problem needs to be solved exactly. We relax this requirement by allowing for the sub-problem to be solved inexactly. In particular, we propose and analyze inexact randomized iterative methods for solving three closely related problems: a convex stochastic quadratic optimization problem, a best approximation problem and its dual – a concave quadratic maximization problem. We provide iteration complexity results under several assumptions on the inexactness error. Inexact variants of many popular and some more exotic methods, including randomized block Kaczmarz, randomized Gaussian Kaczmarz and randomized block coordinate descent, can be cast as special cases. Finally, we present numerical experiments which demonstrate the benefits of allowing inexactness.

When the data describing a given optimization problem is big enough, it becomes impossible to store it on a single machine. In such situations, it is usually preferable to distribute the data among the nodes of a cluster or a supercomputer.  In one such setting the nodes cooperate to minimize  the sum (or average) of private functions (convex or non-convex) stored at the nodes. Among the most popular protocols for solving this problem in a decentralized fashion (communication is allowed only between neighbors) are randomized gossip algorithms.

In Chapter <ref> we propose a new approach for the design and analysis of randomized gossip algorithms which can be used to solve the distributed average consensus problem, a fundamental problem in distributed computing, where each node of a network initially holds a number or vector, and the aim is to calculate the average of these objects by communicating only with its neighbors (connected nodes). The new approach consists in establishing new connections to recent literature on randomized iterative methods for solving large-scale linear systems. Our general framework recovers a comprehensive array of well-known gossip protocols as special cases and allow for the development of block and arbitrary sampling variants of all of these methods. In addition, we present novel and provably accelerated randomized gossip protocols where in each step all nodes of the network update their values using their own information but only a subset of them exchange messages. The accelerated protocols are the first randomized gossip algorithms that converge to consensus with a provably accelerated linear rate. The theoretical results are validated via computational testing on typical wireless sensor network topologies.

Finally, in Chapter <ref>, we move towards a different direction and present the first randomized gossip algorithms for solving the average consensus problem while at the same time protecting the private values stored at the nodes as these may be sensitive.
In particular, we develop and analyze three privacy preserving variants of the randomized pairwise gossip algorithm (“randomly pick an edge of the network and then replace the values stored at vertices of this edge by their average”) first proposed by Boyd et al. <cit.> for solving the average consensus problem. The randomized methods we propose are all dual in nature.  That is, they are designed to solve the dual of the best approximation optimization formulation of the average consensus problem.
We call our three privacy preservation techniques “Binary Oracle”, “ϵ-Gap Oracle” and “Controlled Noise Insertion”. We give iteration complexity bounds for the proposed privacy preserving randomized gossip protocols and perform extensive numerical experiments. 







CHAPTER: ACKNOWLEDGMENTS

I would like to express my sincere gratitude to my supervisor, Prof. Peter Richtárik, for his guidance through every facet of the research world. Thank you for being such a great advisor, mentor and teacher, and for being an inspiring role model that I live up to both in my academic and personal life. Thanks for your continuous feedback, your constant encouragement, your great suggestions for writing and presentation, for offering career advice and for all the nice moments. I couldn't ask for a better advisor, mentor and a friend during my PhD studies.

I would also like to thank my second supervisor and mentor Dr. Lukasz Szpruch for various discussions about research and differences between related fields. I am also grateful to my examination committee, Prof. Coralia Cartis and Prof. Miguel F. Anjos for their suggestions for improvement and for their time.

For stimulating discussions and fun we had together during last four years, I thank other past and current members of our research group: Robert Mansel Gower, Jakub Konečný, Dominik Csiba, Filip Hanzely, Konstantin Mishchenko, Samuel Horváth, Aritra Dutta, El Houcine Bergou, Xun Qian, Adil Salim, Dmitry Kovalev, Elnur Gasanov, Alibek Sailanbayev. I would also like to thank my office mates Matt Booth, Juliet Cooke and Rodrigo Garcia Nava and the rest of my friends from Operational Research and Optimization group, Chrystalla Pavlou, Spyros Pougkakiotis, Minerva Martin del Campo, Xavier Cabezas Garcia, Ivet Galabova, Saranthorn Phusingha, Marion Lemery and Wenyi Qin for all the nice moments.

I am indebted to the University of Edinburgh and to the Principal's Career Development PhD Scholarship  for funding my PhD studies. I would like to thank 
the school of Mathematics of the University of Edinburgh, for providing me with a wonderful work environment and for funding several trips during my PhD studies.
I am also extremely grateful to Gill Law, Iain Dornan and Tatiana Chepelina who were always very helpful in overcoming every formal or administrative problems I encountered.
I am most grateful to the Dr. Laura Wisewell Travel scholarship for funding my conference travels expenses in 2016 and 2017 and to Prof. Peter Richtárik for funding my conference travels expenses in 2018 and 2019. The chance to participate in international conferences, with all the experts in my area in one place, was invaluable.
I would also like to thank the A.G. Leventis Foundation for the financial support during both my MSc and PhD studies.

I appreciate the advice, discussions and fun I had with many amazing people during my internship at Facebook AI Research in Montreal. More specifically, I would like to thank Dr. Mike Rabbat, Dr. Nicolas Ballas and Mahnoud Assran for the excellent collaboration and Prof. Joelle Pineau, Prof. Pascal Vincent, Dr. Adriana Romero, Dr. Michal Drozdzal for the great research environment. A special thanks should go to Dr. Mike Rabbat for the immerse trust and support I received during my internship. Mike has been an inspiration to me, a great enthusiastic collaborator, and warm person in general. 

For willingness to help and provide recommendation, reference, or connection at various stages of my study, I would like to thank Dr. Martin Takáč, Dr. Martin Jaggi, Dr. Panos Parpas, Dr. Wolfram Wiesemann, Dr. Lin Xiao, Dr. Kimon Fountoulakis and Dr. Anastasios Kyrillidis. In addition, I am extremely thankful to Prof. Apostolos Burnetas and Prof. Apostolos Giannopoulos for introducing me to the areas of operational research and convex analysis, respectively.
 
This journey would not have been possible without the support of my parents, Christaki and Agathi. Thank you for encouraging me in all of my pursuits and inspiring me to follow my dreams. I dedicate this thesis to you.

Finally, I would like to thank the two people that were the most important to me during my time in Edinburgh:

My brother George, thank you for all the funny moments and the joyful experiences. You have been a constant support throughout my PhD studies and throughout my life.  You have made my time in Edinburgh one of a kind. :) 

My girlfriend, Katerina, thank you for your endless patience and for your continuous encouragement over the last four years. Thank you for reminding me what is important in life, for traveling around the world with me, for supporting all of my decisions. Thank you for being so amazing!!! 



 




CHAPTER: INTRODUCTION


In this thesis we study the design and analysis of new efficient randomized iterative methods for solving large scale linear systems, stochastic quadratic optimization problems, the best approximation problem and quadratic optimization problems. A large part of the thesis is also devoted to the development of efficient methods for obtaining average consensus on large scale networks. As we will explain later in more detail, some of our proposed algorithms for solving the average consensus problem are carefully constructed special cases of methods for solving linear systems. All methods presented in the thesis (except two algorithms in the last chapter that converge with a sublinear rate) converge with global linear convergence rates, which means that they achieve an approximate solution of the problem fast.

In this introductory chapter we present the setting shared throughout the thesis and explain the relationships between the four problems mentioned above. We describe some baseline methods for solving these problems and present their convergence rates. Finally, we give a summary of the main contributions of each chapter. 



  
Organization of thesis. The thesis is divided into two main parts. 
In the first part (Chapters <ref> and <ref>) we present and analyze novel momentum (Chapter <ref>) and inexact (Chapter <ref>) variants of several randomized iterative methods for solving three closely related problems: 


  (i) stochastic convex quadratic minimization, 

  (ii) best approximation, and 

  (iii) (bounded) concave quadratic maximization.
 

In the second part (Chapters <ref> and <ref>), we focus on the design and analysis of novel randomized gossip algorithms for solving the average consensus problem. This is a fundamental problem in distributed computing with the following goal: each node of a network initially holds a number or a vector, and the aim is for every node to calculate the average of these objects in a decentralized fashion (communicating with neighbors only). The proposed decentralized algorithms are inspired by recent advances in the area of randomized numerical linear algebra and optimization.  In particular, in Chapter <ref> we propose a new framework for the design and analysis of efficient randomized gossip protocols. We show how randomized iterative methods for solving linear systems can be interpreted as gossip algorithms when applied to special systems encoding the underlying network. Using the already developed framework of Chapter <ref>,  we move towards a different direction and in Chapter <ref> we present the first randomized gossip algorithms for solving the average consensus problem while at the same time protecting the information about the initial private values stored at the nodes. 

Excluding some introductory results presented in this section, each chapter of the thesis is self-contained, including the objective, contributions, definitions and notation. However, to the extend that this was possible and meaningful, a unified notation has been adopted throughout the thesis.



§ THESIS' PHILOSOPHY: PLACE IN THE LITERATURE

Here we start by providing a bird's-eye view of the main concepts and a simple first explanation of the context and of the problems under study.  We present how this thesis is related to different areas of research and provide important connections that allow readers with different backgrounds to easily navigate through the main contributions of this work.



 §.§ Bridge across several communities


This thesis is related to three different areas of research: linear algebra, stochastic optimization and machine learning. 



Linear Algebra

Linear systems form the backbone of most numerical codes used in industry and academia.
Solving large linear systems is a central problem in numerical linear algebra and plays an important role in computer science, mathematical computing, optimization, signal processing, engineering and many other fields.

In this thesis we are concerned with the problem of solving a consistent linear system.
In particular,  given a matrix ∈^m× n and a vector b ∈^m, we are interested to solve the problem:

    x = b.


Main Assumption: Consistency. Throughout the thesis we assume that the linear system (<ref>) has a solution x^* ∈^n (not necessarily unique) that satisfies  x^* =b. That is, the linear system is consistent, i.e., {x :  x=b }≠∅. We make no extra assumption on the form, positive definiteness, rank or any other property of matrix . Thus, all methods proposed in this thesis converge under virtually no additional assumptions on the system beyond consistency. However, our methods are particularly well suited for the case of large over-determined linear systems. That is, to the case when the number of linear equations (rows) of the matrix is much larger than number of columns (variables) (m ≫ n).


Stochastic Optimization

This thesis is also related to the stochastic optimization literature, through a recently proposed  stochastic optimization reformulation of linear systems <cit.>. 

It is well known that the linear system (<ref>) can be expressed as an optimization problem as follows <cit.>:

    min_x∈^n1/2 x-b^2 = 1/2∑_i=1^m (_i:^⊤ x- b_i )^2,

where _i: denotes the i^th row of matrix . 
Note that if we denote with  the solution set of problem (<ref>), then =, where  is the solution set of the consistent linear system (<ref>).

The above approach of reformulating a linear system to an optimization problem is without doubt one of the most popular. However as we will later explain in more detail, it is not the only one. For example, one may instead consider the more general formulation

    min_x∈^n1/2 x-b_^2=1/2(  x-b)^⊤(  x-b ) ,

where ∈^m × m is a symmetric and positive definite matrix.
In <cit.>, Richtárik and Takáč  proposed a stochastic optimization reformulation of linear systems similar to (<ref>). In particular, they consider (<ref>) with =_∼[] and allow   to be positive semi-definite. The expectation is over random matrices  ( is matrix expression involving random matrix ) that depend on an arbitrary user-defined distribution  and the matrix  of the linear system (<ref>).  
Under a certain assumption on , for which the term exactness was coined in <cit.>, the solution set of the stochastic optimization reformulation is identical to the solution set of the linear system.  In <cit.>, the authors provide necessary and sufficient conditions for exactness. Later in Sections <ref> and <ref> we describe exactness and comment on its importance in more detail.

In this thesis we design and analyze randomized iterative methods (stochastic optimization algorithms) for solving the stochastic convex quadratic minimization reformulation proposed in <cit.>.


Machine Learning

Stochastic optimization problems are at the heart of many machine learning and statistical techniques used in data science. Machine learning practitioners, as part of their data analysis,  are often interested in minimizing function f which in full generality takes the form:

    min_x∈^n[f(x) = _i ∼f_i(x)],

where _i ∼ denotes the expectation over an arbitrary distribution . 

If we further assume that the distribution  is uniform over m functions f_i, the stochastic optimization problem is simplified to the finite-sum structure problem:

    min_x∈^n f(x)= 1/m∑_i=1^m f_i(x).

Problem (<ref>) is referred to as Empirical Risk Minimization (ERM), and is one of the key optimization problems arising in large variety of models, ranging from simple linear regressions to deep learning.  For example, note that problem (<ref>) is also a special case of ERM (<ref>) when functions f_i: ^n → are chosen to be f_i(x) m/2(_i:^⊤ x- b_i )^2. 

A trivial benchmark for solving problem (<ref>) in the case of differentiable function is Gradient Descent (GD). That is,

    x^k+1 = x^k - ω^k ∇ f(x^k)=x^k - ω^k1/m∑_i=1^m ∇f_i(x),

where ω^k is the stepsize parameter (learning rate). 
However in modern machine learning applications the number m of component functions f_i can be very large (m ≫ n). As a result, computing the full gradient in each iteration is prohibitively expensive and GD becomes impractical for most state-of-the-art applications.

To avoid such issues, machine learning practitioners use Stochastic Gradient Descent (SGD), a randomized variant of GD, first proposed by Robbins and Monro <cit.> in 1951 and which has enjoyed a lot of success ever since. For solving (<ref>),  SGD first uniformly at random samples function f_i (where i ∈{1,…,m}) and then performs the iteration:

    x^k+1 = x^k - ω^k ∇ f_i(x^k),

where ω^k is the stepsize parameter (learning rate).  
SGD has become the workhorse for training supervised machine learning problems which have the generic form  (<ref>) and many papers devoted to the understanding of its convergence behavior in different applications and under different assumptions on the functions f_i <cit.>.

This thesis is closely related to machine learning literature and the papers devoted to the analysis of SGD and its variants. In particular, besides other methods, we focus on analyzing SGD and two of its most popular variants: SGD with momentum (Chapter <ref>) and Inexact SGD (Chapter <ref>) for solving the stochastic optimization reformulation of linear systems proposed in <cit.>.



 §.§ Roadmap



In this subsection,  by following the flowchart of Figure <ref> we present the hierarchy of the main problems under study, explain the relationships between them and provide a brief summary of the chapters of the thesis. More details will be provided in the remaining sections of the Introduction.

In this thesis, we are studying the problem of solving large-dimensional consistent linear systems of the form x=b. In particular, we are adopting the stochastic optimization reformulation of linear systems first proposed in <cit.>. As we have already briefly mentioned, under a certain assumption (exactness) on the randomness of the stochastic reformulation, the solution set of the stochastic convex quadratic minimization problem is equal to the solution set of the original linear system. Hence, solving the stochastic optimization problem _∼f_(x) is equivalent to solving the original linear system.

For solving the stochastic convex quadratic minimization problem one can use stochastic gradient descent (SGD), a popular stochastic optimization algorithm, particularly useful in machine learning applications (large scale setting). In Section <ref> we explain how other stochastic optimization methods, like stochastic Newton (SN) method and stochastic proximal point (SPP) method, are identical to SGD for solving this particular problem and provide a simple analysis for their linear (exponential) convergence. 

As it turns out, SGD and its equivalent methods converge to one particular minimizer of the stochastic optimization problem: the projection of their starting point x^0 onto the solution set of the linear system (<ref>). This leads to the best approximation problem, which is the problem of projecting a given vector onto the solution space of the linear system. The best approximation problem is popular in numerical linear algebra and is normally solved using sketching techniques. We show how the sketch-and-project method proposed in <cit.> for solving the best approximation problem has also identical updates to SGD. 

The dual of the best approximation problem is a bounded unconstrained concave quadratic maximization problem. In this thesis, we are also interested in the development and convergence analysis of efficient, dual in nature, algorithms for directly solving the dual of the best approximation problem. The baseline method for solving the dual of the best approximation problem is stochastic dual subspace accent (SDSA) <cit.>. As we will explain later, the random iterates of SGD, SN and SPP arise as affine images of the random iterates produced by SDSA.

In Chapters <ref> and <ref> we study novel momentum and inexact variants of several randomized iterative methods for solving the above problems.  Among the methods studied are: stochastic gradient descent, stochastic Newton, stochastic proximal point  and stochastic dual subspace ascent. 

As we can also see in Figure <ref>, a large part of the thesis will be devoted to the development of efficient methods for solving the average consensus (AC) problem.  In particular, we will explain how the AC problem can be expressed as a best approximation problem once we choose special linear systems encoding the underlying network (average consensus systems). In Chapter <ref> we show how classical randomized iterative methods for solving the best approximation problem can be interpreted as gossip algorithms and explain in detail their decentralized nature. In Chapter <ref>, we present the first privacy-preserving randomized gossip algorithms that solve the AC problem while at the same time protect the private values stored at the nodes as these may be sensitive.




§ STOCHASTIC OPTIMIZATION REFORMULATION OF LINEAR SYSTEMS
 

The starting point of this thesis is a general framework for studying consistent linear systems via carefully designed stochastic reformulations recently proposed by Richtárik and Takáč <cit.>. In particular, given the consistent linear system (<ref>),
the authors provide four reformulations in the form of a stochastic optimization problem,  stochastic linear system,  stochastic fixed point problem and a stochastic feasibility problem.  These reformulations are equivalent in the sense that their solution sets are identical. That is, the set of minimizers of  the stochastic optimization problem is equal to the set of solutions of the stochastic linear system and so on. Under a certain assumption on the randomness defining these reformulations, for which the term exactness was coined in <cit.>, the solution sets  of these reformulations are equal to the solution set of the linear system. 

For the sake of a simplified narrative, in this thesis we choose to focus mostly on one of the above reformulations: the stochastic convex quadratic optimization problem, which can be expressed as follows:

    min_x∈^n f(x) _∼f_(x).

Here the expectation is over random matrices  drawn from an arbitrary, user defined, distribution  and f_ is a stochastic convex quadratic function of a  least-squares type, defined as 
 
    f_(x) 1/2 x - b_^2 = 1/2( x - b)^⊤ ( x - b).
 
Function  f_ depends on the matrix ∈^m× n and vector b∈^m of the linear system (<ref>) and on a random symmetric positive semidefinite matrix  

    (^⊤^-1^⊤)^†^⊤.

The n× n positive definite matrix , in the expression of matrix , defines the geometry of the space and throughout the thesis, gives rise to an inner product 

    ⟨ x,z ⟩_⟨ x, z⟩

and the induced norm x_ (x^⊤ x)^1/2 on ^n. By † we denote the Moore-Penrose pseudoinverse.  



  
On Moore-Pernose Pseudoinverse: The Moore-Pernose pseudoinverse matrix (or simply pseudoinverse) ^† of a matrix  was first intoduced by Moore <cit.> and Penrose <cit.> in their pioneering work. 

A computationally simple and accurate way to compute the matrix ^† is by using the singular value decomposition <cit.>. That is, if = ^⊤  is the singular value decomposition of matrix , then ^†= ^†^⊤, where the diagonal matrix ^†  is computed by taking the reciprocal of each non-zero element on the diagonal of matrix , leaving the zeros in place. That is ^†_ii=1/_ii for all _ii>0.   Note that by its definition Moore-Penrose pseudoinverse is uniquely defined for all matrices (not necessarily square) whose entries are real or complex numbers.

It is worth to highlight that in this thesis, we applied the Moore-Pernose pseudoinverse only on symmetric positive semidefinite matrices. In particular, if ≽ 0 is a symmetric m × m matrix then its pseudoinverse will appear as a part of the expression ^† b, where b ∈^m.  Using properties of pseudoinverse this is equivalent to the least-norm solution of the least-squares problem min_x  x -b^2 <cit.>. Hence, if the system x = b has a solution the following holds:

    ^† b= argminx^2 
      subject to   x = b.

Let us know present some basic properties of the pseudoinverse:


  * If matrix  is invertible, its pseudoinverse is its inverse. That is, ^†=^-1.

  * The pseudoinverse of the pseudoinverse is the original matrix. That is , (^†)^†=

  * If  is symmetric positive semidefinite matrix , then ^† is also symmetric positive semidefinite matrix.

  * There are several identities that can be used to cancel certain subexpressions or expand expressions involving pseudoinverses: ^† =, ^†^†=^†, (^⊤)^†^⊤=^†, (^⊤)^† = (^†)^⊤  and (^⊤)^† =(^†)^⊤^†. For more useful identities see <cit.>.


As we have already mentioned, problem (<ref>) is constructed in such a way that the set of minimizers of f is identical to the set of solutions of the given (consistent) linear system (<ref>).  In this sense, (<ref>) can  be seen as the reformulation of the linear system (<ref>) into a stochastic optimization problem. As argued in <cit.>, such reformulations  provide an explicit connection between the fields of linear algebra and stochastic optimization, and allow the transfer of knowledge, techniques, and algorithms from one field to another. For instance, the randomized Kaczmarz method of Strohmer and Vershynin <cit.> for solving (<ref>) is equivalent to the stochastic gradient descent method applied to (<ref>), with  corresponding to a discrete distribution over unit coordinate vectors in ^m <cit.>. However, the flexibility of being able to choose  arbitrarily allows for numerous generalizations of the randomized Kaczmarz method <cit.>. Likewise, provably faster variants of the randomized Kaczmarz method (for instance, by utilizing importance sampling) can be designed using the connection.

Since their introduction in <cit.>, stochastic reformulations of otherwise deterministic problems have found surprising applications in various areas, and are hence an
important object of study in its own right.  For instance, using a different stochastic reformulation Gower et al. <cit.> performed a tight convergence analysis of stochastic gradient descent in a more general convex setting, while <cit.> utilized “controlled" stochastic reformulations to develop a new approach to variance reduction for finite-sum problems appearing in machine learning. Further, this approach led to the development of the first accelerated quasi-Newton matrix update rules in the literature <cit.> and to the design of efficient randomized projection methods for convex feasibility problems <cit.>; all solving open problems in the literature.



  
Closed form expressions.

We shall often refer to matrix expressions involving the random matrix  and the matrices  and . In order to keep these expressions brief throughout the thesis, it will be useful to define the n × n matrix:

    ^⊤(<ref>)=^⊤(^⊤^-1^⊤)^†^⊤.


Using matrix  we can easily express important quantities related to the problems under study. For example, the stochastic functions f_ defined in (<ref>) can be also expressed as 

    f_(x) (<ref>)=1/2( x - b)^⊤ ( x - b)= 1/2 (x-x^*)^⊤ (x-x^*),

where x^* ∈. In addition, the gradient and the Hessian of f_ with respect to the  inner product (<ref>) are equal to

    ∇ f_(x) (<ref>)=^-1^⊤ ( x - b)  = ^-1^⊤ (x -x^*)  (<ref>)=^-1 (x -x^*),

where x^* ∈ and ∇^2 f_(x)=^-1 <cit.>. 

Using the above expressions, the gradient and the Hessian of the objective function f of problem (<ref>) are given by 

    ∇ f(x)= _∼[∇ f_(x)] =^-1(x-x^*),
 and

    ∇^2 f = _∼[∇^2 f_(x)]= ^-1,
 respectively.



  
Projections. Let Υ⊆^n be a closed convex set. 
Throughout the thesis, with Π_Υ, we denote the projection operator onto Υ, in the -norm. That is, Π_Υ, (x) min_x' ∈Υx'-x_.
In particular, we are interested in the projection onto . An explicit formula for the projection onto  is given by

    Π_,(x)min_x' ∈x'-x_ =x-^-1^⊤ (^-1 ^ ⊤ )^† ( x-b).

A formula for the projection onto the sketched system _{x : ^⊤ x=^⊤ b } is obtained by simply replacing matrix  and vector b in (<ref>) with the matrix ^⊤ and vector ^⊤ b, respectively. In this case we write Π__,(x). 



  
On complexity results.
The complexity of the linearly convergent methods presented in this thesis is described by the spectrum of the following key matrix: 

    ^-1/2^-1/2.
 
Matrix  has the same spectrum as the Hessian matrix 
∇^2 f = ^-1 and at the same time is symmetric and positive semidefinite (with respect to the standard inner product). Note that ∇^2 f is a not symmetric matrix (although it is self-adjoint with respect to the -inner product).

Let = ^⊤ = ∑_i=1^n λ_i u_i u_i^⊤ be the eigenvalue decomposition of , where  = [u_1,…,u_n] is an orthonormal matrix composed of eigenvectors, and =Diag(λ_1, λ_2, … ,λ_n) is the diagonal matrix of eigenvalues with λ_1 ≥λ_2 ≥⋯≥λ_n. In this thesis, by λ_min^+ we will denote the smallest nonzero eigenvalue, and by λ_max  = λ_1 the largest eigenvalue of matrix . It was shown in <cit.> that 0≤λ_i ≤ 1 for all i∈ [n].



  
Main Assumption: Exactness. Note that in view of (<ref>), f_(x) = 0 whenever x∈. However, f_ can be zero also for points x outside of . Clearly, f is nonnegative, and f(x)=0 for x∈. However, without further assumptions, the set of minimizers of f can be larger than . The exactness assumption mentioned above ensures that this does not happen. For necessary and sufficient conditions for exactness, we refer the reader to <cit.>. One of a number of equivalent characterizations of exactness is the condition: 

    Null(_∼ [])= Null().

For this thesis it suffices to remark that a sufficient condition for exactness is to require  to be positive definite. This is easy to see by observing that f(x) = f_(x) =1/2 x - b^2_. In other words,  if =argmin f(x) is the solution set of the stochastic optimization problem (<ref>) and  the solution set of the linear system (<ref>), then the notion of exactness is captured by: 
    =




§ STOCHASTIC GRADIENT DESCENT (SGD) AND EQUIVALENT ITERATIVE METHODS


Problem  (<ref>)  has several peculiar characteristics which are of key importance to this thesis. For instance, the Hessian of f_ is a (random) projection matrix, which can be used to show that f_(x) = 1/2∇ f_(x)_^2=1/2∇ f_(x)^⊤∇ f_(x) (see equation (<ref>) in Lemma <ref>). Moreover, as we have already mentioned the Hessian of f has all eigenvalues bounded by 1, and so on. These characteristics can be used to show that several otherwise distinct  stochastic algorithms for solving the stochastic optimization problem (<ref>) are identical.

In particular, the following optimization methods for solving (<ref>) are identical 


  * Stochastic Gradient Descent (SGD):

    x^k+1 = x^k - ω∇ f__k(x^k),


  * Stochastic Newton Method (SN)[In this method we take the -pseudoinverse of the Hessian of f__k instead of the classical inverse, as the inverse does not exist. When =, the  pseudoinverse specializes to the standard Moore-Penrose pseudoinverse.]:

    x^k+1 = x^k - ω (∇^2 f__k(x^k))^†_∇ f__k(x^k),


  * Stochastic Proximal Point Method (SPP)[In this case, the equivalence only works for 0<ω≤ 1.]:

    x^k+1 = min_x∈^n{ f__k(x) + 1-ω/2ωx-x^k_^2}.


In all methods ω>0, is a fixed stepsize and  _k is sampled afresh in each iteration from distribution . 

Note that the equivalence of these methods for solving problem (<ref>) is useful for the purposes of the thesis as it allows us to study their variants  with momentum (Chapter <ref>) and their inexact variants (Chapter <ref>) by studying a single algorithm only. 

Using the closed form expression (<ref>) of the gradient of functions f_, the update rules of the equivalent algorithms (<ref>),(<ref>) and (<ref>) can be also written as:

    x^k+1=x^k-ω^-1^⊤_k (_k^⊤^-1^⊤_k)^†_k^⊤( x^k-b)


Following <cit.>, we name the algorithmic update of equation (<ref>), basic method and we use this in several parts of this thesis to simultaneously refer to the above equivalent update rules.

By choosing appropriately the two main parameters of the basic method,  the matrix  and distribution  of the random matrices , we can recover a comprehensive array of well known algorithms for solving linear systems as special cases, such as the randomized Kaczmarz method, randomized Gauss Seidel (randomized coordinate descent) and their block variants. In addition, it is worth to notice that the basic method allows for a much wider selection of these two parameters, which means that it is possible to obtain a number of new specific and possibly more exotic algorithms as special cases. Hence, by having a convergence analysis for the general method (<ref>) we can easily obtain the convergence rates of all these special cases by choosing carefully the combinations of the two main parameters.



As a special case of the general framework,  let us choose = and _k=e_i, where i ∈ [m] is chosen in each iteration independently, with probability p_i>0. Here with e_i ∈^m we denote the i^th unit coordinate vector in ^m.
 In this setup the update rule (<ref>) simplifies to:

    x^k+1=x^k - ω_i : x^k -b_i/_i :^2_i :^ ⊤  .

where _i: indicates the i^th row of matrix . This is a relaxed variant (stepsize not necessarily ω=1) of the  randomized Kaczmarz method <cit.>.
 



  
On Exactness.
An important assumption that is required for the convergence analysis of the randomized iterative methods presented in this thesis is exactness. The exactness property is of key importance for the setting under study, and should be seen as an assumption on the distribution  and not on matrix . 

Clearly, an assumption on the distribution  of the random matrices  should be required for the convergence of (<ref>). For an instance, if  in the randomized Kaczmarz method (<ref>),  is such that, =e_1 with probability 1, where e_1 ∈^m  be the 1^st unit coordinate vector in ^m, then the algorithm will select the same row of matrix  in each step. For this choice of distribution it is clear that the algorithm will not converge to a solution of the linear system. The exactness assumption guarantees that this will not happen.

For necessary and sufficient conditions for exactness, we refer the reader to <cit.>. For this thesis, it suffices to remark that the exactness condition is very weak, allowing  to be virtually any reasonable distribution of random matrices. For instance, as we have already mentioned, a sufficient condition for exactness is for the matrix  to be positive definite <cit.>. 

A much stronger condition than exactness is ≻ 0 which has been used for the convergence analysis of (<ref>) in <cit.>. In this case, the matrix ∈^m × n of the linear system requires to have full column rank and as a result the consistent linear system has a unique solution. 



§ BEST APPROXIMATION AND ITS DUAL PROBLEM
 




  
Best approximation problem.
It was shown in <cit.> that SGD, SN and SPP converge to a very particular minimizer of f: the projection in the -norm, of the starting point x^0 onto the solution set of the linear system (<ref>). That is, x^*=Π_,(x^0). This naturally leads  to the best approximation problem, which is the problem of projecting a given vector onto the solution space of the linear system (<ref>):

    min_x∈^n P(x) 1/2x-x^0_^2   subject to   x = b.

Note that, unlike the linear system (<ref>), which is allowed to have multiple solutions, the best approximation problem has always (from its construction) a unique solution. 

For solving problem (<ref>), the Sketch and Project Method (SPM): 


    x^k+1      argmin_x ∈^nx-x^k_^2
       subject to      _k^⊤ x=_k^⊤ b  .

was analyzed in <cit.>. The name “sketch-and-project" method is justified by the iteration structure which consists of two steps: (i) draw a random matrix _k from distribution   and formulate the sketched system __k, (ii) project the last iterate x^k onto __k.  Analysis in <cit.> was done under the assumption that  has full column rank. This assumption was lifted in <cit.>, and a duality theory for the method developed. 

Using the closed form expression of projection (<ref>), the iterative process of (<ref>) can be equivalently written as <cit.>:

    x^k+1 =  Π___k, (x^k) (<ref>)= x^k-^-1^⊤_k (_k^⊤^-1 ^ ⊤_k)^† (_k^⊤ x^k- _k^⊤ b),

and for the more general case of ω≠1 the update takes the form:

    x^k+1 =  ωΠ___k, (x^k) + (1-ω) x^k.


By combining the definition of projection (<ref>) and the update rule of equation (<ref>) it can be easily observed that the sketch and project method is identical to the basic method (<ref>). As a result, it is also identical to the previously mentioned algorithms, SGD (<ref>), SN (<ref>) and SPP (<ref>).  Thus, these methods can be also interpreted as randomized projection algorithms.



  
On Sketching. In numerical linear algebra, sketching is one of the most popular techniques used for the evaluation of an approximate solution of large dimensional linear systems x= b where ∈^m × n and b ∈^m <cit.>. 

Let ∈^m × q be a random matrix with the same number of rows as  but far fewer columns (n ≫ q). The goal of sketching is to design the distribution of random matrix  such that the solutions set of the much smaller (and potential much easier to solve) sketched system ^⊤ x=^⊤ b to be close to the solution set of the original large dimensional system, with high probability. That is, ^⊤ x-^⊤ b^2 ≤ (1+ϵ) x-b^2 with high probability. Determining the random matrix  that satisfy the above constraint can be challenging and often depends on the properties and form of matrix . For recent advances in the area of sketching we suggest <cit.>.

In our setting, as we have already described above, sketching is part of our iterative process. In each iteration sketching is used in combination with a projection step in order to evaluate an exact solution of the sketched system. 



  
On Sketch and Project Methods. Variants of the sketch-and-project method  have been recently proposed for solving several other problems.
 <cit.> use sketch-and-project ideas for the development of linearly convergent randomized iterative methods for computing/estimating the inverse and pseudoinverse of a large matrix, respectively. A limited memory variant of the stochastic block BFGS method  for solving the empirical risk minimization problem arising in machine learning  was proposed by  <cit.>. Tu et al. <cit.> utilize the sketch-and-project framework to show that breaking block locality can accelerate block Gauss-Seidel methods. In addition, they develop an accelerated variant of the method for a specific distribution .  An accelerated (in the sense of Nesterov) variant of the sketch and prokect method proposed in <cit.> for the more general Euclidean setting and applied to matrix inversion and quasi-Newton updates. As we have already mentioned, in <cit.>,  through the development of stochastic reformulations, a stochastic gradient descent interpretation of the sketch and project method have been proposed. Similar stochastic reformulations that have the sketch and project method as special case were also proposed in the more general setting of convex optimization <cit.> and in the context of variance-reduced methods <cit.>.



  
The dual problem.
Duality is an important tool in optimization literature and plays a major role in the development and understanding of many popular randomized optimization algorithms.
In this thesis, we are also interested in the development of efficient, dual in nature, algorithms for directly solving the dual of the best approximation problem.

In particular, the Lagrangian dual of (<ref>) is the (bounded) unconstrained concave quadratic maximization problem[Technically problem (<ref>) is both the Lagrangian and the Fenchel dual of (<ref>) <cit.>.]

    max_y∈^m D(y)  (b- x^0)^⊤ y - 1/2^⊤ y^2_^-1.

Boundedness follows from consistency. It turns out that by varying , and b (but keeping consistency of the linear system), the dual problem in fact captures all bounded unconstrained concave quadratic maximization problems.

Let us define an affine mapping from ^m to ^n as follows:

    ϕ(y)  x^0 + ^-1^⊤ y.

It turns out, from Fenchel duality[For more details on Fenchel duality, see <cit.>.], that for any dual optimal y^*, the vector ϕ(y^*) must be primal optimal <cit.>. That is:

    x^* = ϕ(y^*) = x^0 + ^-1^⊤ y^*.


A dual variant of the basic method for solving problem (<ref>) was first proposed in <cit.>. The dual method—Stochastic Dual Subspace Ascent (SDSA)—updates the dual vectors y^k as follows:

    y^k+1 = y^k + ω_k λ^k,

where the random matrix _k is sampled afresh in each iteration from distribution , and λ^k is chosen to maximize the dual objective D: λ^k ∈max_λ D(y^k + _k λ).  More specifically, SDSA is defined by picking the maximizer with the smallest (standard Euclidean) norm. This leads to the formula:

    λ^k =  (_k^⊤^-1^⊤_k )^†_k^⊤(b-(x^0 + ^-1^⊤ y^k) ).


Note that, SDSA proceeds by moving in random subspaces spanned by the random columns of _k. In the special case when ω=1 and  y^0=0,  Gower and Richtárik <cit.> established the following relationship (affine mapping ϕ:^m ↦^n) between the iterates {x^k}_k≥0 produced by the primal methods (<ref>), (<ref>),  (<ref>),  (<ref>) (which are equivalent), and the iterates {y^k}_k≥0 produced by the dual method (<ref>):

    x^k = ϕ(y^k) (<ref>)= x^0 + ^-1^⊤ y^k.


In Section <ref> we show with a simple proof how this equivalence extends beyond the ω=1 case, specifically for 0<ω<2 (see Proposition <ref>). Later, a similar approach will be used in the derivation of the momentum and inexact variant of SDSA in Chapters <ref> and <ref> respectively.

An interesting property that holds between the suboptimalities of the primal methods and SDSA is that the dual suboptimality of y in terms of the dual function values is equal to the primal suboptimality of ϕ(y) in terms of distance <cit.>. That is,

    D(y^*)-D(y)=1/2ϕ(y^*)- ϕ(y)^2_.

This simple-to-derive result (by combining the expression of the dual function D(y) (<ref>) and the equation (<ref>)) gives for free the convergence analysis of SDSA, in terms of dual function suboptimality once the analysis of the primal methods is available (see Proposition <ref> in Section <ref>). 

Note that SDSA update (<ref>)+(<ref>) depends on the same two parameters, matrix  and distribution , of the basic method (SGD, SN and SPP). Therefore, similar to the previous subsection, by choosing appropriately the two parameters we can recover many known algorithms as special cases of SDSA. 

 
Let = and _k=e_i, where i ∈ [m] is chosen in each iteration independently, with probability p_i>0. Here with e_i ∈^m we denote the i^th unit coordinate vector in ^m.
 In this setup the update rule (<ref>) simplifies to:

    y^k+1=y^k + ωb_i-_i : x^0 -_i :^⊤ y^k/_i :^2 e_i.

where _i: indicates the i^th row of matrix . 
This is the randomized coordinate ascent method <cit.> applied to the dual problem. Having said that, the analysis provided in <cit.> does not apply because the objective of the dual problem is not strongly concave function.




§ SIMPLE ANALYSIS OF BASELINE METHODS


Having presented the problems that we are interested in this thesis and explained the relationships between them, let us know present some interesting properties of our setting and a simple convergence analysis of the baseline methods for solving them.  

In Sections <ref> and <ref> we have introduced these baseline methods. As a reminder, these are the stochastic gradient descent (SGD) (<ref>), stochastic Newton method (SN) (<ref>), stochastic proximal point method (SPP) (<ref>), sketch and project method (SPM) (<ref>), and  stochastic dual subspace ascent (SDSA)(<ref>).

To simplify the presentation, in the remaining sections of the introduction we focus on two of these algorithms: SGD and SDSA. Recall at this point that SGD, SN, SPP and SPM have identical updates for the problems under study. This means that the analysis presented here for SGD holds for all of these methods.

We start by presenting some interesting properties of the stochastic quadratic optimization problem (<ref>) and discussing connections with existing literature. Then we focus on the convergence analysis results.



 §.§ Technical preliminaries

Recently, linear convergence of opimization methods has been established under several conditions that are satisfied in many realistic scenarios. We refer the interested reader to <cit.> and <cit.> for more details on these conditions and how they are related to each other.

In this thesis, we are particularly interested in the Quadratic Growth condition (QG).  We say that a function f satisfies the QG inequality if the following holds for some μ>0:

    μ/2x-x^*^2 ≤ f(x) -f(x^*).

Here x^* is the projection of vector x onto the solution set  of the optimization problem min_x ∈^n f(x) and f(x^*) denotes the optimal function value.

Under this condition it can be shown that SGD with constant stepsize ω converges with a linear rate up to a neighborhoud around the optimal point that is proportional to the value of ω <cit.>.  For general convex function the convergence of SGD is sublinear <cit.>.

In <cit.> it was shown the the function of the stochastic quadratic optimization problem (<ref>) satisfies the QG condition as well. In particular the following lemma was proved.



For all x ∈^n and x^* ∈ the objective function of the stochastic optimization problem (<ref>) satisfies:

    λ_min^+ f(x) ≤1/2∇ f(x) ^2_≤λ_max f(x)

and 

    f(x) ≤λ_max/2x-x^*^2_.

Moreover, if exactness is satisfied, and we let x^* =Π_, (x), we have 

    λ_min^+/2x-x^*^2_≤  f(x)  .



Note that inequality (<ref>) is precisely the quadratic growth condition (<ref>) with μ=λ_min^+ and f(x^*)=0. 

The following identities were also established in <cit.>. For completeness, we include different (and somewhat simpler) proofs here.


For all x ∈^n and any ∼ we have

    f_(x) = 1/2∇ f_(x)^2_.

Moreover, if x^*∈ (i.e., if x^* satisfies x^* =b), then for all x∈^n we have

    f_(x) = 1/2⟨∇ f_(x),x-x^* ⟩_,

and

    f(x) = 1/2⟨∇ f(x),x-x^* ⟩_.



 In view of (<ref>), and  since ^-1 = (see <cit.>), we have

    ∇ f_(x)^2_   (<ref>)=   ^-1 (x-x^*)^2_ = (x-x^*)^⊤^-1 (x-x^*) =  (x-x^*)^⊤ (x-x^*) 
       (<ref>)=   (x-x^*)^⊤^⊤ (x-x^*)  = ( x-b)^⊤ (  x- b )  (<ref>)= 2f_(x).

Moreover,

    ⟨∇ f_(x),x-x^* ⟩_   (<ref>)=   ⟨^-1 (x-x^*),x-x^* ⟩_
        =    (x-x^*)^⊤^-1 (x-x^*)   (<ref>)=   2f_(x).

By taking expectations in the last identity with respect to the random matrix , we get
⟨∇ f(x),x-x^* ⟩_=2f(x).




  
No need for variance reduction
SGD  is arguably  one of the most popular algorithms in machine learning. Unfortunately, SGD suffers from slow convergence, which is due to the fact that the variance of the stochastic gradient as an estimator of the gradient does not naturally diminish. For this reason, SGD is typically used with a decreasing stepsize rule, which ensures that the variance converges to zero. However, this has an adverse effect on the convergence rate. For instance, SGD has a sublinear rate even if the function to be minimized is strongly convex (conergence to the optimum point, not to a neighborhoud around it). To overcome this problem, a new class of so-called  variance-reduced methods was developed over the last 8 years, including SAG <cit.>, SDCA <cit.>, SVRG/S2GD  <cit.>, minibatch SVRG/S2GD <cit.>, and SAGA <cit.>.
 
In our setting, we assume that the linear system (<ref>) is feasible. Thus, it follows that the stochastic gradient vanishes at the optimal point (i.e., ∇ f_(x^*)=0 for any ). This suggests that additional variance reduction techniques are not necessary since the variance of the stochastic gradient drops to zero as we approach the optimal point x^*. In particular, in our context, SGD with fixed stepsize enjoys linear rate without any variance reduction strategy (see Theorem <ref>). Hence, in this thesis we can bypass the development of variance reduction techniques, which allows us to focus on the momentum term in Chapter <ref> on the inexact computations in Chapter <ref> and on gossip protocols that converge to consensus in Chapters <ref> and <ref>.




 §.§ Theoretical guarantees

The following convergence rates of SGD and SDSA are easy to establish, having the bounds and identities of Lemmas <ref> and <ref>. Nevertheless we present the statements of the theorems and proofs for completeness and because we use similar ideas and approaches in the rest of the thesis. For the benefit of the reader, we also include the derivations of the two equations (<ref>) and (<ref>) presented in the previous section which connect the primal and the dual iterates.



Let assume exactness and let {x^k}_k=0^∞ be the iterates produced by SGD with constant stepsize ω∈(0,2).  Set x^*=Π_,(x^0). Then,

    [x^k-x^*^2_] ≤[1-ω(2-ω) λ_min^+ ]^k x^0-x^*^2_.




    x^k+1-x^*^2_   (<ref>)=   x^k-ω∇ f__k(x^k)-x^*^2_
       =   x^k-x^*^2_-2 ω⟨ x^k-x^*,∇ f__k(x^k) ⟩_ +ω^2 ∇ f__k(x^k)^2_
       (<ref>),(<ref>)=   x^k-x^*^2_-4ω f__k(x^k)+2ω^2 f__k(x^k) 
       =   x^k-x^*^2_-2ω(2-ω)f__k(x^k).

By taking expectation with respect to _k and using quadratic growth inequality (<ref>):

    __k[x^k+1-x^*^2_]    =   x^k-x^*^2_-2ω(2-ω)f(x^k)
       ω∈ (0,2),  (<ref>)≤   x^k-x^*^2_-ω(2-ω) λ_min^+  x^k-x^*^2_
       =   [1-ω(2-ω) λ_min^+ ] x^k-x^*^2_.

Taking expectation again and by unrolling the recurrence we obtain (<ref>).


 

Let {x^k}_k=0^∞ be the iterates produced by SGD(<ref>) with ω∈(0,2). Let y^0=0, and let {y^k}_k=0^∞ be the iterates of SDSA (<ref>).   Assume that the  methods use the same stepsize ω∈(0,2) and the same sequence of random matrices _k.  Then x^k = ϕ(y^k) = x^0 + ^-1^⊤ y^k holds
for all k. That is, the primal iterates arise as affine images of the dual iterates.


First note that

    ∇ f__k(ϕ(y^k))(<ref>)=^-1^⊤_k ( _k^⊤^-1^⊤_k)^†_k^⊤ (ϕ(y^k) - b) (<ref>)=  - ^-1^⊤_k λ^k.

We now use this to show that

    ϕ(y^k+1)    (<ref>)=    x^0  + ^-1^⊤ y^k+1(<ref>)= x^0+^-1^⊤[y^k+ω_k λ^k ]
       =    x^0+^-1^⊤ y^k +ω^-1^⊤_k λ^k 
       (<ref>),(<ref>)=   ϕ(y^k) -ω∇ f__k(ϕ(y^k)).

So, the sequence of vectors {ϕ(y^k)} satisfies the same recursion to the sequence {x^k} defined by SGD.  It remains to check that the starting vectors of both recursions coincide. Indeed, since y^0=0, we have x^0 = ϕ(0) =  ϕ(y^0).




Let y^* be a solution of the dual problem (<ref>). Then,

    D(y^*)-D(y^k)=1/2x^k-x^*^2_.




    D(y^*)-D(y^k)    (<ref>)=    (b- x^0)^⊤ y^* - 1/2^⊤ y^*^2_^-1 -(b- x^0)^⊤ y^k + 1/2^⊤ y^k^2_^-1
        =    (b- x^0)^⊤ (y^*-y^k) - 1/2 (^⊤ y^*)^⊤^-1^⊤ y^* 
           +   1/2 (^⊤ y^k)^⊤^-1^⊤ y^k
       (*)=    (^-1^⊤ y^*)^⊤ (y^*-y^k) - 1/2 (^⊤ y^*)^⊤^-1^⊤ y^* 
           +  1/2 (^⊤ y^k)^⊤^-1^⊤ y^k
        =   1/2 (y^*-y^k) ^-1^⊤ (y^*-y^k)
        =   1/2^-1^⊤ (y^*-y^k)^2_
       (<ref>)+(<ref>)=   1/2x^k-x^*^2_.

In the (*) equality above we use (<ref>) for the optimal primal and dual values. In particular, x^0=x^*-^-1^⊤ y^* ⇒ x^0= x^*-^-1^⊤ y^*=b-^-1^⊤ y^*.


The next theorem has been proved in <cit.> for the case of ω=1.  Here we extend this convergence to the more general case of 0<ω<2.
 


Let us assume exactness. Choose y^0= 0 ∈^m. Let {y^k}_k=0^∞ be the sequence of random iterates produced by SDSA with stepsize 0≤ω≤ 2. Then,

    [D(y^*)-D(y^k)] ≤[1-ω(2-ω) λ_min^+ ]^k ( D(y^*)-D(y^0) ).



 This follows by applying Theorem <ref> together with Proposition <ref>.




 §.§ Iteration Complexity

In several parts of this thesis we compare the performance of linearly convergent algorithms using their iteration complexity bounds. That is, we derive a lower bound on the number of iterations that are sufficient to achieve a prescribed accuracy. The following lemma shows the derivation of this bound for the sequence {A^k}_k=0^∞.



Consider a non-negative sequence {A^k}_k=0^∞ satisfying 

    A^k ≤ρ^k A^0,

where ρ∈ (0,1). Then, for a given ϵ∈ (0,1) and for:

    k ≥1/1-ρlog(1/ϵ)

it holds that:

    A^k ≤ϵ A^0




Note that since ρ∈ (0,1), we have:

    1/1-ρlog(1/ρ) >1.

Therefore,

    log(A^0/A^k)(<ref>)≥ k log(1/ρ)(<ref>)≥1/1-ρlog(1/ϵ) log(1/ρ)(<ref>)≥log(1/ϵ).

Applying exponentials to the above inequality completes the proof.


As an instance, of how the above lemma can be used, recall the convergence result of Theorem <ref>, where we have proved that SGD with constant stepsize ω∈ (0,2) converges as follows:

    [x^k-x^*^2_] ≤[1-ω(2-ω) λ_min^+ ]^k x^0-x^*^2_.

In this setting, Lemma <ref> can be utilized with A^k=[x^k-x^*^2_] and ρ=1-ω(2-ω) λ_min^+ to obtain:

    k ≥1/ω(2-ω) λ_min^+log(1/ϵ)   ⇒  [x^k-x^*_^2]≤ϵx^0-x^*_^2.


In this case we say that SGD converges with iteration complexity

    (1/ω(2-ω) λ_min^+log(1/ϵ)).




§ STRUCTURE OF THE THESIS


In the remainder of this section we give a summary of each chapter of this thesis. The detailed proofs and careful deductions of any claims made here are left to the chapters.



 §.§ Chapter 2: Randomized Iterative Methods with Momentum and Stochastic Momentum

The baseline first-order method for minimizing a differentiable function f is the gradient descent (GD) method, 
    x^k+1 = x^k - ω^k ∇ f(x^k),
 where ω^k>0 is a stepsize <cit.>. For convex functions with L-Lipschitz gradient,  GD converges at at the rate of (L/ϵ). When, in addition, f is μ-strongly convex, the rate is linear: ((L /μ) log(1/ϵ)) <cit.>. To improve the convergence behavior of the method, Polyak proposed to modify GD by the introduction of a (heavy ball) momentum term[
A more popular, and certainly theoretically much better understood alternative to Polyak's momentum is the momentum introduced by Nesterov <cit.>, leading to the famous accelerated gradient descent (AGD) method. This method converges non-asymptotically and globally; with optimal sublinear rate (√(L/ϵ))  <cit.> when applied to minimizing a smooth convex objective function (class ^1,1_0,L), and with the optimal linear  rate (√(L/μ)log(1/ϵ)) when minimizing smooth strongly convex functions (class ^1,1_μ,L).   Recently, variants of Nesterov's momentum have also been introduced for the acceleration of stochastic gradient descent. We refer the interested reader to <cit.> and the references therein. Both Nesterov's and Polyak's update rules are known in the literature as “momentum” methods. In Chapter <ref>, however, we focus exclusively on Polyak's heavy ball momentum.], β(x^k-x^k-1) <cit.>. This leads to the gradient descent method with momentum (mGD), popularly  known as the heavy ball method: 

    x^k+1 = x^k - ω^k ∇ f(x^k) + β (x^k-x^k-1).
  More specifically, Polyak proved that with the correct choice of the stepsize parameters ω^k and momentum parameter β, a local accelerated linear convergence rate of  (√(L/μ)log(1/ϵ))  can be achieved in the case of twice continuously differentiable, μ-strongly convex objective functions with L-Lipschitz gradient <cit.>.

The theoretical behavior of the above deterministic heavy ball method is now well understood in different settings. In contrast to this, there has been less progress in understanding the convergence behavior of  stochastic variants of the heavy ball method. The key method in this category is stochastic gradient descent with momentum (mSGD; stochastic heavy ball method): x^k+1 = x^k -ω^k g (x^k)  + β (x^k-x^k-1), where g is an unbiased estimator of the true gradient ∇ f(x^k).
 In our setting, where our goal is to solve the stochastic optimization problem (<ref>),  mSGD takes the following form:

    x^k+1 = x^k - ω∇ f__k(x^k)+ β (x^k-x^k-1),

where ω > 0 denotes a fixed stepsize and matrix _k is sampled afresh in each iteration from distribution .

In Chapter <ref>, we study several classes of stochastic optimization algorithms enriched with the heavy ball momentum for solving the three closely related problems already described in the introduction. These are the stochastic quadratic optimization problem (<ref>), the best approximation problem (<ref>) and the dual quadratic optimization problem (<ref>). Among the methods studied are: stochastic gradient descent (<ref>), stochastic Newton (<ref>), stochastic proximal point (<ref>) and stochastic dual subspace ascent (<ref>). This is the first time momentum variants of several of these methods are studied.  
We prove global non-asymptotic linear convergence rates for all methods and various measures of success, including primal function values, primal iterates, and dual function values. We also show that the primal iterates converge at an accelerated linear rate in a somewhat weaker sense. This is the first time a linear rate is shown for the stochastic heavy ball method (i.e., stochastic gradient descent method with momentum). Under somewhat weaker conditions,  we establish a sublinear convergence rate for Cesàro averages of primal iterates.  Moreover, we propose a novel concept, which we call stochastic momentum, aimed at decreasing the cost of performing the momentum step. We prove linear convergence of several stochastic methods with stochastic momentum, and show that in some sparse data regimes and for sufficiently small momentum parameters, these methods enjoy better overall complexity than methods with deterministic momentum. Finally, we perform extensive numerical testing on artificial and real datasets. 



 §.§ Chapter 3: Inexact Randomized Iterative Methods

A common feature of existing randomized iterative methods is that in their update rule a particular subproblem needs to be solved exactly. In the large scale setting, often this step can be computationally very expensive. The purpose of the work in Chapter <ref> is to reduce the cost of this step by incorporating inexact updates in the stochastic methods under study.

From a stochastic optimization viewpoint, we analyze the performance of inexact SGD (iSGD):

    x^k+1=x^k-ω∇ f__k(x^k)  + ϵ^k,

where ω >0 denotes a fixed stepsize, matrix _k is sampled afresh in each iteration from distribution  and ϵ^k represents a (possibly random) error coming from inexact computations.

In Chapter <ref>, we propose and analyze inexact variants of the exact algorithms presented in previous sections for solving the stochastic optimization problem (<ref>), the best approximation problem (<ref>) and the dual problem (<ref>). Among the methods studied are: stochastic gradient descent (SGD), stochastic Newton (SN), stochastic proximal point (SPP), sketch and project method (SPM) and stochastic subspace ascent (SDSA). 
In all of these methods, a certain potentially expensive calculation/operation needs to be performed in each step; it is this operation that we propose to be performed inexactly.  For instance, in the case of SGD, it is the computation of the stochastic gradient ∇ f__k(x^k), in the case of  SPM is the computation of the projection Π__, (x^k), and in the case of SDSA it is the computation of the dual update _k λ^k.

We perform an iteration complexity analysis under an abstract notion of inexactness and also under a more structured form of inexactness appearing in practical scenarios. Typically, an inexact solution of these subproblems can be obtained much more quickly than the exact solution. Since in practical applications the savings thus obtained are larger than the increase in the number of iterations needed for convergence, our inexact methods can be dramatically faster.

Inexact variants of many popular and some more exotic methods, including  randomized block Kaczmarz, randomized Gaussian Kaczmarz and randomized block coordinate descent, can be cast as special cases of our analysis. Finally, we present numerical experiments which demonstrate the benefits of allowing inexactness.



 §.§ Chapter 4: Revisiting Randomized Gossip Algorithms

In Chapter <ref> we present a new framework for the analysis and design of randomized gossip algorithms for solving the average consensus (AC) problem,  a fundamental problem in distributed computing and multi-agent systems.

In the AC problem we are given an undirected connected network =(,) with node set ={1,2,…,n} and edges . Each node i ∈ “knows” a private value c_i ∈. The goal of AC is for every node to compute the average of these private values, c̅1/n∑_i c_i, in a decentralized fashion. That is, the exchange of information can only occur between connected nodes (neighbors). 

In an attempt to connect the AC problem to optimization, consider the simple optimization problem:

    min_x = (x_1,…, x_n) ∈^n1/2x-c^2 
      subject to   x_1=x_2=…=x_n,

where c=(c_1,…,c_n)^⊤ is the vector of the initial private values c_i. Observe that its optimal solution x^* must necessarily satisfy x^*_i=c̅ for all i ∈ [n], where c̅ is the value that each node needs to compute in the AC problem. Note also that, if we represent the constraint x_1=x_2=…=x_n of (<ref>) as a linear system then the optimization problem (<ref>) is an instance of the best approximation problem (<ref>) with = (identity matrix). Perhaps, there is a deeper link here? Indeed, it turns out that properly chosen randomized algorithms for solving (<ref>) can be interpreted as decentralized protocols for solving the AC problem.

A simple way to express the constraint of problem (<ref>) as linear system x= b is by selecting  to be the incidence matrix of the network and the right hand side to be the zero vector (b=0 ∈^m). By using this system the most basic randomized gossip algorithm (“randomly pick an edge e=(i,j) ∈ and then replace the values stored at vertices i and j by their average") is an instance of the randomized Kaczmarz (RK) method (<ref>) for solving consistent linear systems, applied to this system.

Using this observation as a starting point, in Chapter <ref> we show how classical randomized iterative methods for solving linear systems can be interpreted as gossip algorithms when applied to special systems encoding the underlying network and explain in detail their decentralized nature. 
Our general framework recovers a comprehensive array of well-known gossip algorithms as special cases, including the pairwise randomized gossip algorithm and path averaging gossip, and allows for the development of provably faster variants. The flexibility of the new approach enables the design of a number of new specific gossip methods. 
For instance, we propose and analyze novel block and the first provably  accelerated randomized gossip protocols, and dual randomized gossip algorithms.

From a numerical analysis viewpoint, our work is the first that explores in depth the decentralized nature of randomized iterative methods for linear systems and proposes them as methods for solving the average consensus problem. 

We evaluate the performance of the proposed gossip protocols by performing extensive experimental testing on typical wireless network topologies.



 §.§ Chapter 5: Privacy Preserving Randomized Gossip Algorithms

In Chapter <ref>, we present three different approaches to solving the Average Consensus problem while at the same time protecting the information about the initial values of the nodes. To the best of our knowledge, this work is the first which combines the gossip framework with the privacy concept of protection of the initial values. 

The randomized methods we propose are all dual in nature. That is, they solve directly the dual problem (<ref>). In particular, the three different techniques that we use for preserving the privacy are “Binary Oracle”, “ϵ-Gap Oracle” and “Controlled Noise Insertion”. 

Binary Oracle:
We propose to reduce the amount of information transmitted in each iteration to a single bit. More precisely, when an edge is selected, each corresponding node will only receive information whether the value on the other node is smaller or larger. Instead of setting the value on the selected nodes to their average, each node increases or decreases its value by a pre-specified amount.

ϵ-Gap Oracle:
In this case, we have an oracle that returns one of three options and is parametrized by ϵ. If the difference in values of sampled nodes is larger than ϵ, an update similar to the one in Binary Oracle is taken. Otherwise, the values remain unchanged. An advantage compared to the Binary Oracle is that this approach will converge to a certain accuracy and stop there, determined by ϵ (Binary Oracle will oscillate around optimum for a fixed stepsize). However, in general, it will disclose more information about the initial values.

Controlled Noise Insertion:
This approach protects the initial values by inserting noise in the process. Broadly speaking, in each iteration, each of the sampled nodes first adds a noise to its current value, and an average is computed afterwards. Convergence is guaranteed due to the correlation in the noise across iterations. Each node remembers the noise it added last time it was sampled, and in the following iteration, the previously added noise is first subtracted, and a fresh noise of smaller magnitude is added. Empirically, the protection of initial values is provided by first injecting noise into the system, which propagates across the network, but is gradually withdrawn to ensure convergence to the true average.

We give iteration complexity bounds for all proposed privacy preserving randomized gossip algorithms and perform extensive numerical experiments.



§ SUMMARY

The content of this thesis is based on the following publications and preprints:



  
Chapter 2


  * Nicolas Loizou and Peter Richtárik.“Momentum and Stochastic Momentum for Stochastic Gradient, Newton, Proximal Point and Subspace Descent Methods", arXiv preprint arXiv:1712.09677 (2017). <cit.>

  * Nicolas Loizou and Peter Richtárik. “Linearly Convergent Stochastic Heavy Ball Method for Minimizing Generalization Error", Workshop on Optimization for Machine Learning, NIPS 2017. <cit.>




  
Chapter 3


  * Nicolas Loizou and Peter Richtárik. “Convergence Analysis of Inexact Randomized Iterative Methods", arXiv preprint arXiv:1903.07971 (2019). <cit.>




  
Chapter 4


  * Nicolas Loizou and Peter Richtárik. “A New Perspective on Randomized Gossip Algorithms", IEEE Global Conference on Signal and Information Processing (GlobalSIP), pp.440-444, 2016  <cit.>

  * Nicolas Loizou and Peter Richtárik. “Accelerated Gossip via Stochastic Heavy Ball Method." 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton) (pp. 927-934), 2018. <cit.>

  * Nicolas Loizou, Mike Rabbat and Peter Richtárik. “Provably Accelerated Randomized Gossip Algorithms" 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7505-7509 <cit.>

  * Nicolas Loizou and Peter Richtárik. “Revisiting Randomized Gossip Algorithms: General Framework, Convergence Rates and Novel Block and Accelerated Protocols", arXiv preprint arXiv:1905.08645, <cit.>.




  
Chapter 5


  * Filip Hanzely, Jakub Konečný, Nicolas Loizou,  Peter Richtárik and  Dmitry Grishchenko. “A Privacy Preserving Randomized Gossip Algorithm via Controlled Noise Insertion", NeurIPS 2018 - Privacy Preserving Machine Learning Workshop,  <cit.>

  * Filip Hanzely, Jakub Konečný, Nicolas Loizou,  Peter Richtárik and Dmitry Grishchenko.  “Privacy Preserving Randomized Gossip Algorithms",  arXiv preprint arXiv:1706.07636, 2017 <cit.>


During the course of my study, I also co-authored the following works which were not used in the formation of this thesis:


  * Mahmoud Assran, Nicolas Loizou, Nicolas Ballas and Mike Rabbat. “Stochastic Gradient Push for Distributed Deep Learning", Proceedings of the 36th International Conference on Machine Learning (ICML), 2019 <cit.>

  * Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin and Peter Richtárik. “SGD: General Analysis and Improved Rates" Proceedings of the 36th International Conference on Machine Learning (ICML), 2019 <cit.>

  * Nicolas Loizou. “Distributionally Robust Games with Risk-Averse Players", In Proceedings of 5th International Conference on Operations Research and Enterprise Systems (ICORES), 186-196, 2016 <cit.>


In <cit.>, we propose a general theory describing the convergence of Stochastic Gradient Descent (SGD) under the “arbitrary sampling paradigm". Our theory describes the convergence of an infinite array of variants of SGD, each of which is associated with a specific probability law governing the data selection rule used to form minibatches. This is the first time such an analysis is performed, and most of our variants of SGD were never explicitly considered in the literature before. 

In <cit.>, we study Stochastic Gradient Push (SGP), an algorithm which combines PushSum gossip protocol with stochastic gradient updates for distributed deep learning. We prove that SGP converges to a stationary point of smooth, non-convex objectives at the same sub-linear rate as SGD, that all nodes achieve consensus, and that SGP achieves a linear speedup with respect to the number of compute nodes. Furthermore, we empirically validate the performance of SGP on image classification (ResNet-50, ImageNet) and machine translation (Transformer, WMT’16 En- De) workloads.

In <cit.> we present a new model of incomplete information games without private information in which the players use a distributionally robust optimization approach to cope with the payoff uncertainty. 



CHAPTER: RANDOMIZED ITERATIVE METHODS WITH MOMENTUM AND STOCHASTIC MOMENTUM





§ INTRODUCTION
 

Two of the most popular algorithmic ideas  for solving optimization problems involving big volumes of data are stochastic approximation  and momentum. By stochastic approximation we refer to the practice pioneered by Robins and Monro <cit.> of replacement of costly-to-compute quantities (e.g.,  gradient of the objective function) by cheaply-to-compute stochastic approximations thereof (e.g., unbiased estimate of the gradient).  By momentum we refer to the heavy ball technique originally developed by Polyak <cit.> to accelerate the convergence rate of gradient-type methods.  

While much is known about the effects of stochastic approximation and momentum in isolation, surprisingly little is known about the combined effect of these two popular algorithmic techniques. For instance, to the best of our knowledge, there is no context in which a method combining stochastic approximation with momentum  is known to have a linear convergence rate. One of the  contributions of this work is to show that there are important problem classes for which a linear rate can indeed be established for a range of stepsize and momentum parameters.



 §.§ The setting

In this chapter we study the three closely related problems described in the introduction of this thesis. These are: 


  (i) stochastic quadratic optimization (<ref>), 

  (ii) best approximation (<ref>), and 

  (iii) (bounded) concave quadratic maximization (<ref>).
 

In particular we are interested in the complexity analysis and efficient implementation of momentum variants of the baseline algorithms presented in the introduction. As a reminder, these methods are the following: stochastic gradient descent (SGD), stochastic Newton method (SN), stochastic proximal point methods (SPP), sketch and project method (SPM) and stochastic dual subspace ascent (SDSA).

We are not aware of any successful attempts to analyze momentum variants of SN and SPP, SPM and SDSA and to the best of our knowledge there are no linearly convergent variants of SGD with momentum in any setting.

In addition, we propose and analyze a novel momentum strategy for SGD, SN, SPP and SPM, which we call stochastic momentum. It is a stochastic approximation of the popular deterministic heavy ball momentum which in some situations could be particularly beneficial in terms of overall complexity. Similar to the classical momentum, we prove linear convergence rates for this momentum strategy. 



 §.§ Structure of the chapter


This chapter is organized as follows. In Section <ref> we summarize our contributions in the context of existing literature. In  Section <ref> we describe and analyze  primal  methods  with momentum (mSGD, mSN and mSPP), and in Section <ref> we describe and analyze the dual method with momentum (mSDSA). In Section <ref> we describe and analyze primal methods with stochastic momentum (smSGD, smSN and smSPP). Numerical experiments are presented in Section <ref>.  Proofs of all key results can be found in Section <ref>.

 

 §.§ Notation


The following notational conventions are used in this chapter. Boldface upper-case letters denote matrices;  is the identity matrix. By  we denote the solution set of the linear system x=b. By _, where  is a random matrix, we denote the solution set of the sketched linear system ^⊤ x= ^⊤ b. By _i: and _:j we indicate the i_th row and the j_th column of matrix , respectively. Unless stated otherwise, throughout the chapter, x^* is the projection of x^0 onto  in the -norm: x^*=Π_, (x^0). We also write [n]{1,2, … ,n}. Finally, we say that a function f : ^n→ belongs to the class ^1,1_0,L if it is convex, continuously differentiable, and its gradient is Lipschitz continuous with constant L. If in addition the function f is μ-strongly convex with strong convexity constant μ>0, then we say that it belongs to the class ^1,1_μ,L. When it is also twice continuously differentiable, it belongs to the function class ^2,1_μ,L.



§ MOMENTUM METHODS AND MAIN CONTRIBUTIONS
 
In this section we give a brief review of the relevant literature, and provide a  summary of our contributions.



 §.§ Heavy ball method


As we have already mentioned in Section <ref>, Polyak's seminal work <cit.> showed that deterministic heavy ball method:

    x^k+1 = x^k - ω^k ∇ f(x^k) + β (x^k-x^k-1),

converges with a local accelerated linear convergence rate of  (√(L/μ)log(1/ϵ))  in the case of twice continuously differentiable, μ-strongly convex objective functions with L-Lipschitz gradient (function class ℱ_μ, L^2,1). 

Recently, Ghadimi et al. <cit.> performed a global convergence analysis for the heavy ball method. In particular, the authors showed that for a certain combination of the stepsize and momentum parameter, the method  converges sublinearly to the optimum when the objective function is convex  and has Lipschitz gradient (f∈ℱ_0, L^1,1), and  linearly when the function is also strongly convex (f∈ℱ_μ, L^1,1). A particular selection of the parameters ω and β that gives the desired accelerated linear rate was not provided.
 
To the best of our knowledge, despite considerable amount of work on the heavy ball method, there is still no  global convergence analysis which would guarantee an accelerated linear rate for f∈ℱ_μ, L^1,1. However, in the special case of a strongly convex quadratic, an elegant proof was recently proposed in <cit.>. Using the notion of integral quadratic constraints from robust control theory, the authors proved that by choosing ω^k = ω=4 / (√(L)+√(μ))^2 and β=(√(L/μ)-1)^2/(√(L/μ)+1)^2, the heavy ball method enjoys a global asymptotic accelerated convergence rate of (√(L/μ)log(1/ϵ)). The aforementioned results are summarized in the first part of Table <ref>.

Extensions of the heavy ball method have been recently proposed in the proximal setting <cit.>, non-convex setting <cit.> and for distributed optimization <cit.>. For more recent analysis under several combinations of assumptions we suggest <cit.>.



 §.§ Stochastic heavy ball method


In contrast to the recent advances in our theoretical understanding of the (classical) heavy ball method, there has been less progress in understanding the convergence behavior of  stochastic variants of the heavy ball method. The key method in this category is stochastic gradient descent with momentum (mSGD; aka: stochastic heavy ball method):

    x^k+1 = x^k -ω^k g (x^k)  + β (x^k-x^k-1),

where g is an unbiased estimator of the true gradient ∇ f(x^k). While mSGD is used extensively in practice, especially in deep learning <cit.>, its convergence behavior is not very well understood.

In fact, we are aware of only two  papers, both recent, which set out to study the complexity of mSGD: the work of Yang et al. <cit.>, and the work of Gadat et al. <cit.>. In the former paper, a unified convergence analysis for stochastic gradient methods with momentum  (heavy ball and Nesterov's momentum) was proposed; and an analysis for  both convex and non convex functions was performed. For a general Lipschitz continuous convex objective function with bounded variance, a rate of (1/√(k)) was proved. For this, the authors employed a decreasing stepsize strategy: ω^k=ω^0/√(k+1), where ω^0 is a positive constant.  In <cit.>, the authors first describe several almost sure convergence results in the case of general non-convex coercive functions, and then provide a complexity analysis for the case of quadratic strongly convex function. However, the established rate is slow. More precisely, for strongly convex quadratic  and coercive functions, mSGD  with diminishing stepsizes ω^k=ω^0 / k^β was shown to convergence as (1/k^β) when the momentum parameter is β <1, and with the rate (1/log k) when β=1. The convergence rates established in  both of these papers  are sublinear. In particular, no insight is provided into whether the inclusion of the momentum term provides what it was aimed to provide: acceleration. 

The above results are summarized in the second part of Table <ref>. From this perspective, our contribution lies in providing an in-depth analysis of mSGD (and, additionally, of SGD with stochastic momentum).  

Many recent papers have built upon our analysis <cit.> and have already extended our results in several settings. For more details see  <cit.>. 



  
On definitions of convergence presented in Table <ref>.
In Table <ref> we present two main notions to characterize the convergence guarantees presented in the literature for the analysis of deterministic and stochastic heavy ball methods. These are, (i) Local/ Global convergence and (ii) Asymptotic/ Non-asymptotic convergence. For clarity, in this paragraph, we present these definitions of convergence.

Local convergence we have only if the convergence guarantees depend on the starting point of the method. That is, if we can guarantee convergence only if x^0 is in a neighborhood of the optimal point x^*. If the method convergence for any starting point then we have global convergence. 

We have asymptotic convergence when the provided rate can be shown to hold only after specific number of iterations.  For example, we say that a deterministic method converges with asymptotic linear rate if there is K>0 such that for k>K we have x^k-x^*^2 <ρ ^k x^0-x^*, where ρ∈ (0,1). We have a non-asymptotic convergence if the rate satisfy the above definition for K=0.





 


 §.§ Connection to incremental gradient methods


Assuming  is discrete distribution (i.e., we sample from M matrices, ^1,…,^M, where ^i is chosen with probability p_i>0. Here, 0<M ∈ R is fixed.), we can write the stochastic optimization problem (<ref>) in the finite-sum form

    min_x∈^n f(x) = ∑_i=1^M p_i f_^i(x).

Choosing x^0=x^1, mSGD with fixed stepsize ω^k=ω applied to (<ref>) can be written in the form

    x^k+1 =  x^k-ω∑_t=1^kβ^k-t∇ f__t (x^t)+β^k(x^1-x^0)
    x^0=x^1= x^k-ω∑_t=1^kβ^k-t∇ f__t (x^t),
 
where _t = ^i with probability p_i. Problem (<ref>) can be also solved using incremental  average/aggregate gradient methods, such as the IAG method of Blatt et al. <cit.>. These methods have a similar form to (<ref>); however the past gradients are aggregated somewhat differently. While (<ref>) uses a geometric weighting of the gradients, the incremental average gradient methods use a uniform/arithmetic weighting. The  stochastic average gradient (SAG) method of Schmidt et al. <cit.> can be also written in a similar form. Note that mSGD  uses a geometric weighting of previous gradients, while the the incremental and stochastic average gradient methods use an arithmetic weighting. Incremental and incremental average gradient methods are widely studied algorithms for minimizing objective functions which can expressed as a sum of finite convex functions. For a review of key works on incremental methods and a detailed presentation of the connections with stochastic gradient descent, we refer the interested reader to the excellent survey of Bertsekas  <cit.>; see also the work of Tseng  <cit.>. 

In <cit.>, an incremental average gradient method with momentum was proposed for minimizing strongly convex functions. It was proved that the method converges to the optimum with linear rate. The rate is always worse than that of the no-momentum variant. However, it was  shown experimentally that in practice the method is faster, especially in problems with high condition number.  In our setting, the objective function has a very specifc structure (<ref>). It is not a finite sum problem as the distribution  could be continous; and we also do not assume strong convexity. Thus, the convergence analysis of <cit.> can not be directly applied to our problem.  




 §.§ Summary of contributions


We now summarize the contributions of this chapter.

New momentum methods.  We study several  classes of stochastic optimization algorithms (SGD, SN, SPP and SDSA) with momentum, which we call mSGD, mSN, mSPP and mSDSA, respectively (see the first and second columns of Table <ref>). We do this in a simplified  setting with quadratic objectives where all of these algorithms are  equivalent. These methods can be seen as solving three related optimization problems: the stochastic optimization problem (<ref>), the best approximation problem (<ref>) and its dual.  To the best of our knowledge, momentum variants of SN, SPP and SDSA were not analyzed before. 




Linear rate. We prove several (global and non-asymptotic) linear convergence results for our primal momentum methods mSGD/mSN/mSPP. First, we establish a linear rate for the decay of x^k-x^*_^2 to zero, for a range of stepsizes ω> 0 and momentum parameters β≥ 0. We show that the same rate holds for the decay of the expected function values f(x^k)-f(x^*) of (<ref>) to zero. Further, the same rate holds for  mSDSA, in particular, this is for the convergence of the dual objective to the optimum. For a summary of these results, and pointers to the relevant theorems, refer to lines 1, 2 and 6 of Table <ref>. Unfortunately, the theoretical rate for all our momentum methods is optimized for β = 0, and  gets worse as the momentum parameter increases. However, no prior linear rate for any of these methods with momentum are known. We give the first linear convergence rate for SGD with momentum (i.e., for the stochastic heavy ball method). 



Accelerated linear rate. We then study the decay of the larger quantity x^k-x^*_^2  to zero.  In this case, we establish  an accelerated linear rate, which depends on the square root of the condition number (of the Hessian of f). This is a quadratic speedup when compared to  the no-momentum methods as these depend on the condition number. See lines 4 and 5 of Table <ref>. To the best of our knowledge, this is the first time an  accelerated rate is obtained for the stochastic heavy ball method (mSGD).  Note that there are no global non-asymptotic accelerated linear rates proved even in the non-stochastic setting (i.e., for the heavy ball method). Moreover, we are not aware of any accelerated linear convergence results for  the stochastic proximal point method.



Sublinear rate for Cesàro averages. We show that the Cesàro averages, x̂^k = 1/k∑_t=0^k-1x^t, of all primal momentum methods enjoy a sublinear (1/k) rate (see line 3 of Table <ref>). This holds under weaker assumptions than those which lead to the linear convergence rate. 

Primal-dual correspondence. We show that SGD, SN and SPP with momentum arise as affine images of SDSA with momentum (see Theorem <ref>). This extends the result of <cit.> where this was shown for the no-momentum methods (β=0) and in the special case of the unit stepsize (ω=1).

Stochastic momentum. We propose a new momentum strategy, which we call stochastic momentum.   Stochastic momentum is a stochastic (coordinate-wise) approximation of the deterministic momentum, and hence is much less costly, which in some situations leads to computational savings in each iteration. On the other hand, the additional noise introduced this way increases the number of iterations needed for convergence. We analyze the SGD, SN and SPP methods with stochastic momentum, and prove linear convergence rates. We prove that in some settings  the overall complexity of SGD with stochastic momentum is better than the overall complexity of SGD with momentum. For instance, this is the case if we consider the randomized Kaczmarz (RK) method as a special case of SGD, and if  is sparse.

Space for generalizations. We hope that the present work can serve as a starting point for the development of SN, SPP and SDSA methods with momentum for more general classes (beyond special quadratics) of convex and perhaps also nonconvex optimization problems. In such more general settings, however, the symmetry which implies equivalence of these algorithms will break, and hence a different analysis will be needed for each method.



§ PRIMAL METHODS WITH MOMENTUM
 
 Applied to problem (<ref>), i.e., 
min_x∈^n f(x) = f_(x),
  the gradient descent method with momentum (also known as the  heavy ball method) of Polyak <cit.> takes the form

    x^k+1 = x^k - ω∇ f(x^k) + β(x^k - x^k-1),

where ω>0 is a stepsize and β≥ 0 is a momentum parameter. Instead of marrying the momentum term with gradient descent, we can marry it with SGD. This leads to SGD with momentum (mSGD), also known as the stochastic heavy ball method:

    x^k+1 = x^k - ω∇ f__k(x^k) + β(x^k-x^k-1).


Since SGD is equivalent to SN and SPP, this way we obtain momentum variants of the stochastic Newton  (mSN) and  stochastic proximal point (mSPP) methods.  The method is formally described below:



To the best of our knowledge, momentum variants of SN and SPP were not considered  in the literature before. Moreover, as far as we know, there are no momentum variants of even deterministic variants of (<ref>), (<ref>) and (<ref>), such as incremental or batch Newton method,  incremental or batch proximal point method and  incremental or batch projection method; not even for a problem formulated differently.



In the rest of this section we state our convergence results for mSGD/mSN/mSPP. 



 §.§ Convergence of iterates and function values: linear rate

In this section we study the convergence rate of the quantity [x^k-x^*_^2] to zero for mSGD/mSN/mSPP. We show that for a range  of stepsize parameters ω > 0 and  momentum terms β≥ 0, the method enjoys global  linear convergence rate; see (<ref>). To the best of our knowledge, these results are the first of their kind for the stochastic heavy ball method.
As a corollary of this result, we obtain convergence of the expected function values; see (<ref>).




Choose x^0= x^1∈^n.  Assume exactness. Let {x^k}_k=0^∞ be the sequence of random iterates produced by mSGD/mSN/mSPP.  Assume 0< ω < 2 and β≥ 0 and that the expressions

    a_1  1+3β+2β^2 - (ω(2-ω) +ωβ)λ_min^+,     and    
    a_2 β +2β^2 + ωβλ_max

satisfy a_1+a_2<1. Let x^* = Π_ℒ,(x^0). Then 

    [x^k-x^*^2_] ≤ q^k (1+δ)  x^0-x^*^2_

and 

    [f(x^k)] ≤ q^k  λ_max/2 (1+δ) x^0-x^*^2_,

where  q=a_1+√(a_1^2+4a_2)/2 and δ=q-a_1. Moreover, a_1+a_2 ≤ q <1.

 See Section <ref>.




In the above theorem we obtain a global linear rate. To the best of our knowledge, this is the first time that linear rate is established for a stochastic variant of the heavy ball method (mSGD) in any setting. All existing results are sublinear. These seem to be  the first momentum variants of SN and SPP methods.



If we choose ω∈ (0,2), then the condition a_1+a_2<1 is satisfied for all   
    0≤β< 1/8( -4+ωλ_min^+-ωλ_max +√((4-ωλ_min^++ωλ_max)^2+16ω (2-ω) λ_min^+ )).




If β=0, mSGD reduces to  SGD analyzed in <cit.>. In this special case, q = 1-ω(2-ω)λ_min^+, which is the rate established in <cit.>. Hence, our result is more general. 


Let q(β) be the rate as a function of β. Note that since β≥ 0, we have

    q(β)    ≥    a_1 + a_2 
       =    1 + 4β + 4β^2 + ωβ(λ_max-λ_min^+) - ω(2-ω)λ_min^+ 
       ≥    1-ω(2-ω)λ_min^+ = q(0).

Clearly, the lower bound on q is an increasing function of β. 
Also, for any β the rate is always inferior to that of SGD (β=0). It is an open problem whether one can prove a strictly better rate for mSGD than for SGD.








Our next proposition states that Π_,(x^k) = x^* (recall that x^*Π_, (x^0)) for all iterations k of mSGD. This invariance property plays an important role in our convergence analysis,  and “explains” why the algorithm to converges to x^*. 


Let x^0=x^1∈^n be the starting points of the mSGD method and let {x^k} be the random iterates generated by mSGD. Then Π_,(x^k)=Π_,(x^0) for all k≥ 0.

 Note that in view of (<ref>), ∇ f_(x) = ^-1^⊤ ( x - b) ∈ Range(^-1^⊤). Since 

    x^k+1 = x^k -ω∇ f__k(x^k) + β(x^k - x^k-1),

and since x^0=x^1, it can shown by induction that x^k ∈ x^0 +  Range(^-1^⊤) for all k. However, Range(^-1^⊤) is the orthogonal complement to Null() in the -inner product. Since  is parallel to Null(), vectors x^k must have the same -projection onto  for all k: Π_, (x^0) = x^*.


This property also intuitively explains  why mSGD converges to the projection of the starting point onto . Indeed, one may ask: why is the starting point special? After all, each iterate depends on the previous two iterates only, and all older iterates, including the starting point x^0, seem to be eventually “forgotten”. Still, the iterative process has the property that all iterates live in the affine space passing through x^0 and orthogonal to , which means that the projection of all iterates onto  is identical.






 §.§ Cesàro average: sublinear rate without exactness assumption


In this section we present the convergence analysis of the function values computed on the Cesàro average. Again our results are global in nature. To the best of our knowledge these are the first results that show (1/k) convergence of the stochastic heavy ball method. Existing results apply in more general settings at the expense of slower rates. In particular, <cit.> and <cit.> get (1/√(k)) and (1/k^β) convergence when β∈ (0,1), respectively. When β=1, <cit.> gets (1/log(k)) rate.



Choose x^0=x^1 and let {x^k}_k=0^∞ be the random iterates produced by mSGD/mSN/ mSPP, where the momentum parameter 0≤β <1 and relaxation parameter (stepsize) ω > 0 satisfy ω + 2β <2. Let x^* be any vector satisfying f(x^*)=0. If we let x̂^k=1/k∑_t=1^kx^t, then

    [f(x̂^k)] ≤(1-β)^2x^0-x^*_^2 + 2ωβ f(x^0)/2ω(2-2β-ω) k.



 See Section <ref>.


In the special case of β=0, the above theorem gives the rate

    [f(x̂^k)] ≤x^0-x^*^2_/2ω (2 -ω) k .

This is the convergence rate for Cesàro averages of the “basic method” (i.e., SGD) established in <cit.>. 

Our proof strategy  is similar to  <cit.> in which the first global convergence analysis of the (deterministic) heavy ball method was presented. There it was shown that when the objective function has a Lipschitz continuous gradient, the Cesàro averages of the iterates converge to the optimum at a rate of (1/k). To the best of our knowledge, there are no results in the literature that prove the same rate of convergence in the stochastic case for any class of objective functions.  

In <cit.> the authors analyzed  mSGD for general Lipshitz continuous convex objective functions (with bounded variance) and proved the sublinear rate (1/√(k)). In <cit.>, a complexity analysis is provided for the case of quadratic strongly convex smooth coercive functions. A  sublinear convergence  rate of (1/k^β), where β∈ (0,1), was proved. In contrast to our results, where we assume fixed stepsize ω, both papers analyze mSGD with diminishing stepsizes. 




 §.§ Accelerated linear rate for expected iterates


In this section we show that by a proper combination of the relaxation (stepsize) parameter  ω and the momentum parameter β, mSGD/mSN/mSPP enjoy an accelerated linear convergence rate in mean.  That is, while SGD  needs (θlog (1/ϵ)) iterations to find x^k such that x^k-x^*_^2 ≤ϵ <cit.>,  mSGD only needs (√(θ)log (1/ϵ)) iterations (see Theorem <ref>(ii)), where θ = λ_max/λ_min^+.  The word acceleration typically refers to improvement from a leading factor of θ to √(θ), which is significant in the ill-conditioned case, i.e., when θ is very large. In other words, the linear rate  (1-√(0.99/θ))^k of mSGD  is much better than linear rate (1-1/θ)^k of SGD, which in view of (<ref>) is better  than the linear rate of mSGD  established in (<ref>) (for a different quantity converging to zero) . 









Assume exactness. Let {x^k}_k=0^∞ be the sequence of random iterates produced by mSGD / mSN / mSPP, started with x^0, x^1 ∈^n satisfying the relation x^0-x^1 ∈ Range(^-1^ ⊤), with relaxation parameter (stepsize)  0<ω≤1/λ_max and momentum parameter  (1-√(ωλ_min^+))^2 < β <1. Let x^* = Π_,(x^0). Then there exists constant C >0 such that for all k≥0 we have 

    [x^k -x^*] _^2  ≤β^k C.




  (i) If we choose ω= 1 and β= (1- √(0.99 λ_min^+)) ^2 then
 [x^k -x^*] _^2  ≤β^k C
and the iteration complexity becomes
(√(1/ λ_min^+)log(1/ϵ)).

  (ii) If we choose ω= 1/λ_max and β= (1- √(0.99λ_min^+/λ_max))^2 then
[x^k -x^*] _^2  ≤β^k C
and the iteration complexity becomes
(√(λ_max/ λ_min^+)log(1/ϵ)).



 See Section <ref>.


Note that the convergence factor is precisely equal to the value of the momentum parameter β. Let x be any random vector in ^n with finite mean [x], and x^*∈^n is any reference vector (for instance, any solution of x = b). Then we have the identity (see, for instance <cit.>)

    x-x^*_^2 = x-x^*_^2 + x-[x]^2_.

This means that the quantity x-x^*_^2 appearing in the convergence result of Theorem <ref> is larger than 
x-x^*_^2 appearing in the the convergence result of Theorem <ref>, and hence harder to push to zero. As a corollary, the convergence  rate of x-x^*_^2 to zero established in Theorem <ref>)   implies the same rate for the convergence of  x-x^*_^2 to zero. However, note that in Theorem <ref> we have established an accelerated rate for x-x^*_^2.  A similar theorem, also obtaining an accelerated rate for x-x^*_^2, was established in <cit.> for an accelerated variant of SGD in the sense of Nesterov.




§ DUAL METHODS WITH MOMENTUM
 

In the previous sections we focused on methods for solving the stochastic optimization problem (<ref>) and the best approximation problem (<ref>). In this section we focus on the dual of the best approximation problem, and propose a momentum variant of SDSA, which we call mSDSA. 





 §.§ Correspondence between primal and dual methods


In our first result we show that the random iterates of the mSGD/mSN/mSPP methods arise as an affine image of mSDSA under the mapping ϕ defined in (<ref>).


   Let x^0=x^1 and let {x^k} be the iterates of mSGD/mSN/mSPP. Let y^0=y^1=0, and let {y^k} be the iterates of mSDSA.   Assume that the  methods use the same stepsize ω > 0, momentum parameter β≥ 0, and the same sequence of random matrices _k.  Then  

    x^k = ϕ(y^k) = x^0 + ^-1^⊤ y^k

for all k. That is, the primal iterates arise as affine images of the dual iterates.


First note that

    ∇ f__k(ϕ(y^k))    (<ref>)=   ^-1^⊤_k ( _k^⊤^-1^⊤_k)^†_k^⊤ (ϕ(y^k) - b)   =   - ^-1^⊤_k λ^k.

We now use this to show that

    ϕ(y^k+1)    (<ref>)=    x^0  + ^-1^⊤ y^k+1
       =    x^0+^-1^⊤[y^k+ω_k λ^k +β(y^k-y^k-1)]
       =   x^0+^-1^⊤ y^k_ϕ(y^k)+ω^-1^⊤_k λ^k_-∇ f__k(ϕ(y^k)) +β^-1^⊤ (y^k-y^k-1)
       =   ϕ(y^k)- ω∇ f__k(ϕ(y^k)) +β  (^-1^⊤ y^k-^-1^⊤ y^k-1)
       (<ref>)=   ϕ(y^k) -ω∇ f__k(ϕ(y^k)) +β  (ϕ(y^k) - ϕ(y^k-1)).

So, the sequence of vectors {ϕ(y^k)} mSDSA satisfies the same recursion of degree as the sequence {x^k} defined by mSGD.  It remains to check that the first two elements of both recursions coincide. Indeed, since y^0=y^1=0 and x^0=x^1, we have x^0 = ϕ(0) =  ϕ(y^0), and x^1 = x^0= ϕ(0) =  ϕ(y^1).





 §.§ Convergence


We are now ready to state a linear convergence convergence result describing the behavior of mSDSA in terms of the dual function values D(y^k).

 

Choose y^0= y^1∈^n.  Assume exactness. Let {y^k}_k=0^∞ be the sequence of random iterates produced by mSDSA.  Assume 0≤ω≤ 2 and β≥ 0 and that the expressions

    a_1  1+3β+2β^2 - (ω(2-ω) +ωβ)λ_min^+,     and    
    a_2 β +2β^2 + ωβλ_max

satisfy a_1+a_2<1. Let x^* = Π_ℒ,(x^0) and let y^* be any dual optimal solution. Then 

    [D(y^*)-D(y^k)] ≤ q^k (1+δ) [D(y^*)-D(y^0)]

where  q=a_1+√(a_1^2+4a_2)/2 and δ=q-a_1. Moreover, a_1+a_2 ≤ q <1.

 This follows by applying Theorem <ref> together with Theorem <ref> and the identity 1/2x^k-x^0^2_ = D(y^*) - D(y^k).



Note that for β=0, mSDSA simplifies to SDSA. Also recall that for unit stepsize (ω=1), SDSA was analyzed in  <cit.>. In the ω=1 and β=0 case, our result specializes to that established in  <cit.>. Following similar arguments to those in  <cit.>,  the same rate of convergence can be proved for the duality gap [P(x^k)-D(y^k)].





§ METHODS  WITH STOCHASTIC MOMENTUM
 

To motivate stochastic momentum, for simplicity fix =, and assume that _k is chosen as the jth random unit coordinate vector  of ^m with probability p_j>0. In this case, SGD (<ref>) reduces to the randomized Kaczmarz method for solving the linear system x = b, first analyzed for p_j∼_j:^2 by Strohmer and Vershynin <cit.>. 

In this case, mSGD becomes the randomized Kaczmarz method with momentum (mRK), and the iteration (<ref>)  takes the explicit form

    x^k+1 = x^k - ω_j: x^k -b_j/_j:^2_j:^ ⊤ + β(x^k-x^k-1).

Note that the cost of one iteration of this method is (_j:_0 + n), where the cardinality term _j:_0 comes from the stochastic gradient part, and n comes from the momentum part.
When  is sparse, the second term will dominate. Similar considerations apply for many other (but clearly not all) distributions . 

In such circumstances, we propose to replace the expensive-to-compute momentum term by a cheap-to-compute stochastic approximation term. In particular, we let i_k be chosen from [n] uniformly at random,  and replace x^k-x^k-1 with v_i_k e_i_k^⊤ (x^k-x^k-1)e_i_k,  where e_i_k∈^n is the i_k-th unit basis vector in ^n, and β with γ nβ. Note that v_i_k can be computed in (1) time. Moreover, 
    _i_k[γ v_i_k] =β( x^k-x^k-1).
 Hence,  we replace the momentum term by  an unbiased estimator, which allows us to cut the cost to (_j:_0). 




 §.§ Primal methods with stochastic momentum


We now propose a  variant of the SGD/SN/SPP methods employing stochastic momentum  (smSGD/smSN/smSPP). Since SGD, SN and SPP are equivalent, we will describe the development from the perspective of SGD.
In particular, we propose the following method:

    x^k+1 = x^k - ω∇ f__k(x^k) + γ e_i_k^⊤ (x^k-x^k-1)e_i_k.


The method is formalized below:






 §.§ Convergence


In the next result we establish linear convergence of smSGD/smSN/smSPP. For this we will require the matrix  to be equal to the identity matrix.

 Choose x^0= x^1∈^n.  Assume exactness. Let =.  Let {x^k}_k=0^∞ be the sequence of random iterates produced by smSGD/smSN/smSPP.  Assume 0< ω < 2 and γ≥ 0 and that the expressions

    a_1  1+3γ/n+2γ^2/n - (ω(2-ω) +ωγ/n)λ_min^+,     and    
    a_2 1/n(γ +2γ^2 + ωγλ_max)

satisfy a_1+a_2<1. Let x^* = Π_ℒ,(x^0). Then 

    [x^k-x^*^2] ≤ q^k (1+δ)  x^0-x^*^2

and 
[f(x^k)] ≤ q^k  λ_max/2 (1+δ) x^0-x^*^2,
where  qa_1+√(a_1^2+4a_2)/2 and δ q-a_1. Moreover, a_1+a_2 ≤ q < 1.


 See Section <ref>.



It is straightforward to see that if we choose ω∈ (0,2), then the condition a_1+a_2<1 is satisfied for all γ belonging to the interval   
    0≤γ < 1/8 (-4+ωλ_min^+-ωλ_max+√((4-ωλ_min^++ωλ_max)^2+16nω (2-ω)  λ_min^+)).

The  upper bound  is similar to that for mSGD/mSN/mSPP; the only difference is an extra factor of n next to the constant 16.



 §.§ Momentum versus stochastic momentum



As indicated above, if we wish to compare mSGD with momentum parameter β to smSGD with momentum parameter γ, it makes sense to set γ = β n. Indeed, this is because in view of (<ref>), the momentum term in smSGD will then be an unbiased estimator of the deterministic momentum term used in mSGD.

Let q(β) be the convergence constant for mSGD with stepsize ω=1 and an admissible momentum parameter β≥ 0. Further, let a̅_1(γ),a̅_2(γ),q̅(γ) be the convergence constants for smSGD with stepsize ω=1 and momentum parameter γ.  We have

    q̅(β n) ≥a̅_1(β n) + a̅_2(β n)    (<ref>)=    1 + 4β + 4β^2n + β(λ_max-λ_min^+) - λ_min^+
       (<ref>)=    a_1(β) + a_2(β) + 4β^2 (n-1)
       ≥    a_1(β) + a_2(β).

 Hence, the lower bound on the rate for smSGD is worse than the lower bound for mSGD. 
 
The same conclusion holds for the convergence rates themselves. Indeed, note that since a̅_1(β n) - a_1(β) = 2β^2(n-1) ≥ 0 and a̅_2(β n) - a_2(β) = 2β^2 (n-1) ≥ 0, we have

    q̅(β n) = a̅_1(β n) + √(a̅^2_1(β n) +4 a̅_2(β n))/2≥a_1(β ) + √(a_1^2(β ) +4 a_2(β ))/2 = q(β),

and hence the rate of mSGD is always better than that of smSGD.

However, the expected cost of a single iteration of mSGD may be significantly larger than that of smSGD. Indeed, let g be the expected cost of evaluating a stochastic gradient. Then we need to compare (g+n) (mSGD) against (g) (smSGD). If g≪ n, then one iteration of smSGD is significantly cheaper than one iteration of mSGD. Let us now compare the total complexity to investigate the trade-off between the rate and cost of stochastic gradient evaluation. Ignoring constants, the total cost of the two methods (cost of a single iteration multiplied by the number of iterations) is:

    C_mSGD(β) g+n/1-q(β) = g+n/1-a_1(β) + √(a_1^2(β) + 4 a_2(β))/2,

and

    C_smSGD(β n) g/1-q̅(β n) = g/1-a̅_1(β n) + √(a̅_1^2(β n) + 4 a̅_2(β n))/2.


Since  
    q(0) = q̅(0 n),
 and since q(β) and q̅(β n) are continuous functions of β, then because g+n > g, for small enough β we will have 
C_mSGD(β) > C_smSGD(β n). In particular, the speedup of smSGD compared to mSGD for β≈ 0 will be close to

    C_mSGD(β)/C_smSGD(β n)≈lim_β' →_+ 0C_mSGD(β')/C_smSGD(β' n)(<ref>)+(<ref>)+(<ref>)=g+n/g = 1 + n/g.


Thus, we have shown the following statement.

  For small momentum parameters satisfying γ=β n, the total complexity of smSGD is approximately 1+ n/g times smaller than the total complexity of mSGD, where n is the number of columns of , and g is the expected cost of evaluating a stochastic gradient ∇ f_(x).  




§ SPECIAL CASES: RANDOMIZED KACZMARZ WITH MOMENTUM AND RANDOMIZED COORDINATE DESCENT WITH MOMENTUM
 

In Table <ref> we specify several special instances of mSGD  by choosing distinct combinations of the parameters  and .  We use e_i to denote the ith unit coordinate vector in ^m, and _:C for the column submatrix of the m× m identity matrix indexed by (a random) set C. 




The updates for smSGD can be derived by substituting the momentum term β(x^k-x^k-1) with its stochastic variant nβ e_i_k^⊤ (x^k-x^k-1)e_i_k. We do not aim to be comprehensive. For more details on the possible combinations of the parameters  and   we refer the interested reader to Section 3 of <cit.>. 

In the rest of this section we present in detail two special cases: the randomized Kaczmarz method with momentum (mRK) and the randomized coordinate descent method with momentum (mRCD). Further, we compare the convergence rates obtained in Theorem <ref> (i.e., bounds on x^k-x^*_^2) with rates that can be inferred from known results for their no-momentum variants.



 §.§ mRK: randomized Kaczmarz with momentum
 We now provide a discussion on mRK (the method in the first row of Table <ref>).  Let = and let pick in each iteration the random matrix =e_i with probability p_i=_i:^2 / _F^2. In this setup the update rule of the mSGD simplifies to 
    x^k+1=x^k -ω_i: x^k -b_i/_i:^2_i:^ ⊤ + β(x^k - x^k-1)

and 

    (<ref>)=   ^-1/2^⊤_∼[] ^-1/2  =  [ ^⊤] 
       =   ∑_i=1^mp_i_i:^⊤_i:/_i:^2  =  1/^2_F∑_i=1^m_i:^⊤_i:  = ^⊤/^2_F.


The objective function takes the following form:

    f(x)=_∼𝒟 [f_(x)]= ∑_i=1^m p_i f__i(x)= x -b^2/2^2_F.


For β=0, this method reduces to the randomized Kaczmarz method with relaxation, first analyzed in <cit.>. If we also have ω=1, this is equivalent with the randomized Kaczmarz method of Strohmer and Vershynin <cit.>.  RK without momentum (β=0)  and without relaxation (ω=1) 
converges  with iteration complexity <cit.> of 
    (1/λ_min^+()log(1/ϵ))=(^2_F/λ_min^+(^⊤)log(1/ϵ)).


In contrast, based on Theorem <ref> we have


  * For ω= 1 and β= (1- √(0.99 λ_min^+))^2=(1- √(0.99/^2_Fλ_min^+(^⊤)))^2, the iteration complexity of the mRK is: 
    (√(^2_F/λ_min^+(^⊤))log(1/ϵ)).


  * For ω= ^2_F/λ_max(^⊤) and β= (1- √(0.99 λ_min^+(^⊤)/λ_max(^⊤)))^2
the iteration complexity becomes:

    (√(λ_max(^⊤)/λ_min^+(^⊤))log(1/ϵ)).



 This is quadratic improvement on the  previous best result (<ref>).




  
Related Work. 
The Kaczmarz method for solving consistent linear systems was originally introduced by Kaczmarz in 1937 <cit.>. This classical method selects the rows to project onto in a cyclic manner.  In practice, many different selection rules can be adopted. For non-random selection rules (cyclic, greedy, etc) we refer the interested reader to <cit.>. In this work we are interested in randomized variants of the Kaczmarz method, first analyzed by Strohmer and Vershynin <cit.>. In <cit.> it was shown that RK converges with a linear convergence rate to the unique solution of a full-rank consistent linear system. This result sparked renewed interest in design of randomized methods for solving linear systems <cit.>.  All existing results on accelerated variants of RK use the Nesterov's approach of acceleration <cit.>. To the best of our knowledge, no convergence analysis of mRK exists in the literature (Polyak's momentum). Our work fills this gap.




 §.§ mRCD: randomized coordinate descent with momentum
 

We now provide a discussion on the mRCD method (the method in the second row of Table <ref>). If the matrix  is positive definite, then we can choose =  and =e_i with probability p_i=_ii/ Trace(). It is easy to see that =/ Trace(). In this case,   is positive definite and as a result, λ_min^+()=λ_min(). Moreover, we have

    f(x)=_∼𝒟 [f_(x)]= ∑_i=1^m p_i f__i(x)= x -b^2/2 Trace().


For β=0 and ω=1 the method is equivalent with  randomized coordinate descent of Leventhal and Lewis <cit.>, which was shown to converge with iteration complexity  
    Previous  best result:    ( Trace()/λ_min()log(1/ϵ)).


In contrast, following  Theorem <ref>, we can obtain the following iteration complexity results for mRCD:
 
  
  * For ω= 1 and β= (1- √(0.99/ Trace()λ_min()))^2, the iteration complexity is 
    (√( Trace()/λ_min())log(1/ϵ)).


  * For  ω=  Trace()/λ_max() and β= (1- √(0.99 λ_min()/λ_max()))^2  the iteration complexity becomes 

    (√(λ_max()/λ_min())log(1/ϵ)).

 
 
 This is quadratic improvement on the previous best result (<ref>).
 
 


  
Related Work.  It is known that if  is positive definite, the popular randomized Gauss-Seidel method can be interpreted as randomized coordinate descent (RCD). RCD methods were first analyzed by Lewis and Leventhal in the context of linear systems and least-squares problems <cit.>, and later extended by several authors to more general settings, including smooth convex optimization <cit.>, composite convex optimization <cit.>, and parallel/subspace descent variants  <cit.>. These results were later further extended to handle  arbitrary sampling distributions  <cit.> . Accelerated variants of RCD were studied in <cit.>. For other non-randomized coordinate descent variants and their convergence analysis, we refer the reader to <cit.>. To the best of our knowledge, mRCD and smRCD have never  been analyzed before in any setting.



 §.§ Visualizing the acceleration mechanism



We devote this section to the graphical illustration of the acceleration mechanism behind momentum. Our goal is to shed more light on how the proposed algorithm works in practice. For simplicity, we illustrate this by comparing RK and mRK.




 
 
 


 
 
 







In Figure <ref> we present in a simple ^2 illustration of the difference between the workings of RK and mRK. Our goal is to show graphically how the addition of momentum leads to acceleration.  Given iterate x^k, one can think of the update rule of the mRK (<ref>) in two steps:


  * The Projection: The projection step corresponds to the first part x^k - ω∇ f__k(x^k) of the  mRK update (<ref>) and it means that the current iterate x^k is projected onto a randomly chosen hyperplane H_i[In the plots of Figure  <ref>, the hyperplane of each update is chosen in an alternating fashion for illustration purposes]. The value of the stepsize ω∈ (0,2) defines whether the projection is exact or not. When ω=1 (no relaxation) the projection is exact, that is the point  Π_H_i(x^k) belongs in the hyperplane H_i.  In Figure <ref> all projections are exact.

  * Addition of the momentum term: The momentum term (right part of the update rule) β (x^k-x^k-1) forces the next iterate x^k+1 to be closer to the solution x^* than the corresponding point Π_H_i(x^k). Note also that the vector x^k+1-Π_H_i(x^k) is always parallel to x^k-x^k-1 for all k≥0.



In the example of Figure <ref>, the performance of mRK is similar to the performance of RK until iterate x^3. After this point, the momentum parameter becomes more effective and the mRK method accelerates. This behavior appears also in our experiments in the next section where we work with  matrices with many rows. There we can notice that the momentum parameter seems to become more effective after the first m+1 iterations.




§ NUMERICAL EXPERIMENTS
 


In this section we study the computational behavior of the two proposed algorithms, mSGD and smSGD. In particular, we focus mostly on the evaluation of the performance of mSGD. To highlight the usefulness of smSGD, an empirical verification of Theorem <ref> is presented in subsection <ref>. As we have already mentioned, both mSGD and smSGD can be interpreted as sketch-and-project methods (with relaxation), and as a result a comprehensive array of well-known algorithms can be recovered as special cases by varying the main parameters of the methods (check Section <ref>). In our experiments we focus on the popular special cases of randomized Kaczmarz method (RK) and the randomized coordinate descent method (RCD) without relaxation (ω=1), and  show the practical benefits of adding the momentum term[The experiments were repeated with various values of the main parameters and initializations, and similar results were obtained in all cases.]. The choice of the stepsize ω=1 is not arbitrary. Recently, in <cit.> both relaxed RK and relaxed RCD were analyzed, and it was proved that the quantity x^k-x^*^2_ converges linearly to zero for ω∈ (0,2), and that the best convergence rate is obtained precisely for ω=1. Thus the comparison is with the best-in-theory  no-momentum variants. 


Note that, convergence analysis of the error x^k-x^*^2_ and of the expected function values f(x^k) in Theorem <ref> shows that mSGD enjoys global non-asymptotic linear convergence rate but not faster than the no-momentum method. 
The accelerated linear convergence rate has been obtained only in the weak sense (Theorem <ref>). Nevertheless, in practice as indicated from our experiments, mSGD is faster than its no momentum variant. Note also that in all of the presented experiments the momentum parameters β of the methods are chosen to be positive constants that do not depend on parameters that are not known to the users such as λ_min^+ and λ_max.

In comparing the methods with their momentum variants we use both the relative error measure x^k-x^*^2_ / x^0-x^*^2_ and the function values f(x^k)[Remember that in our setting we have f(x^*)=0 for the optimal solution x^* of the best approximation problem; thus f(x)-f(x^*)=f(x). The function values f(x^k) refer to function (<ref>) in the case of RK and to function (<ref>) for the RCD.  For block variants the objective function of problem (<ref>) has also closed form expression but it can be very difficult to compute. In these cases one can instead evaluate the quantity x-b^2_.]. In all implementations, except for the experiments on average consensus (Section <ref>), the starting point is chosen to be x^0=0. In the case of average consensus the starting point must be the vector with the initial private values of the nodes of the network. 
All the code for the experiments is written in the Julia programming language.
For the horizontal axis we use either the number of iterations or the wall-clock time measured using the tic-toc Julia function.

This section is divided in three main experiments. In the first one we evaluate the performance of the mSGD method in the special cases of mRK and mRCD for solving both synthetic consistent Gaussian systems and consistent linear systems with real matrices. In the second experiment we computationally verify Theorem <ref> (comparison between the mSGD and smSGD methods). In the last experiment building upon the recent results of <cit.> we show how the addition of the momentum accelerates the pairwise randomized gossip (PRG) algorithm for solving the average consensus problem.





 §.§ Evaluation of mSGD


In this subsection we study the computational behavior of mRK and mRCD when they compared with their no momentum variants for both synthetic and real data.



  §.§.§ Synthetic Data


The synthetic data for this comparison is generated as follows[Note that in the first experiment we use Gaussian matrices which by construction are full rank matrices with probability 1 and as a result the consistent linear systems have unique solution. Thus, for any starting point x^0, the vector z that is used to create the linear system is the solution mSGD converges to.  This is not true for general consistent linear systems, with no full-rank matrix. In this case, the solution   x^*=Π_,(x^0) that mSGD converges to is not necessarily equal to z. For this reason, in the evaluation of the relative error measure x^k-x^*^2_ / x^0-x^*^2_, one should be careful and use the value x^*=x^0+^† (b-  x^0)x^0=0=^† b.].

For mRK:
All elements of matrix ∈^m × n and vector z ∈^n are chosen to be i.i.d 𝒩(0,1). Then the right hand side of the linear system is set to b= z. With this way the consistency of the linear system with matrix  and right hand side b is ensured. 

For mRCD:  A Gaussian matrix ∈^m × n is generated and then matrix = ^⊤∈^n × n is used in the linear system. The vector z∈^n is chosen to be i.i.d 𝒩(0,1) and again to ensure consistency of the linear system, the right hand side is set to b= z.

In particular for the evaluation of mRK we generate Gaussian matrices with m=300 rows and several columns while for the case of mRCD the matrix  is chosen to be Gaussian with m=500 rows and several columns[RCD converge to the optimal solution only in the case of positive definite matrices. For this reason = ^⊤∈^n × n is used which with probability 1 is a full rank matrix]. 
Linear systems of these forms were extensively studied <cit.> and it was shown that the quantity 1/λ_min^+(condition number) can be easily controlled. 

For each linear system we run mRK (Figures <ref> and <ref>)  and mRCD (Figures <ref> and <ref>) for several values of momentum parameters β and fixed stepsize ω=1 and we plot the performance of the methods (average after 10 trials) for both the relative error measure and the function values. Note that for β=0 the methods are equivalent with their no-momentum variants RK and RCD respectively.

From Figures <ref>, <ref>, <ref> and <ref> it is clear that the addition of momentum term leads to an improvement in the performance of both, RK and RCD. More specifically, from the four figures we observe the following:



  * For the well conditioned linear systems (1/λ_min^+ small) it is known that even the no-momentum variant converges rapidly to the optimal solution. In these cases the benefits of the addition of momentum are not obvious. The momentum term is beneficial for the case where the no-momentum variant (β=0) converges slowly, that is when  1/λ_min^+ is large (ill-conditioned linear systems). 

  * For the case of fixed stepsize ω=1, the problems with small condition number require smaller momentum parameter β to have faster convergence. Note the first two rows of Figures <ref> and <ref>, where β=0.3 or β =0.4, are good options. 

  * For large values of 1/λ_min^+, it seems that the choice of β=0.5 is the best. As an example for matrix ∈^300 × 280 in Figure <ref>, (where 1/λ_min^+=208,730), note that to reach relative error 10^-10, RK needs around 2 million iterations, while mRK with momentum parameter β=0.5 requires only half that many iterations. The acceleration is obvious also in terms of time where in 12 seconds the mRK with momentum parameter β=0.5 achieves relative error of the order 10^-9 and RK requires more than 25 seconds to obtain the same accuracy. 

  * We observe that both mRK and mRCD, with appropriately chosen momentum parameters 0< β≤ 0.5, always converge  faster than their no-momentum variants, RK and RCD, respectively. This is a smaller momentum parameter than β≈ 0.9 which is being used extensively with mSGD for training deep neural networks <cit.>. 

  * In <cit.> a stochastic power iteration with momentum is proposed for principal component analysis (PCA). There it was demonstrated empirically that a naive application of momentum to the stochastic power iteration does not result in a faster method. To achieve faster convergence, the authors proposed mini-batch and variance-reduction techniques on top of the addition of momentum. In our setting, mere addition of the momentum term  to SGD (same is true for  special cases such as RK and RCD) leads to empirically faster methods.











  §.§.§ Real Data

In the following experiments we test the performance of mRK using real matrices (datasets) from the library of support vector machine problems LIBSVM <cit.>. Each dataset consists of a matrix ∈^m × n  (m features and n characteristics) and a vector of labels b ∈^m.  In our experiments we choose to use only the matrices of the datasets and ignore the label vector. As before, to ensure consistency of the linear system,  we choose a Gaussian vector z∈^n and the right hand side of the linear system is set to b= z. 
Similarly as in  the case of synthetic data, mRK is tested for several values of momentum parameters β and fixed stepsize ω=1. 

In Figure <ref> the performance of all methods for both relative error measure x^k-x^*^2 / x^*^2_ and function values f(x^k) is presented. Note again that β=0 represents the baseline RK method. The addition of momentum parameter is again often beneficial and leads to faster convergence. As an example, inspect the plots for the mushrooms dataset in Figure <ref>, where mRK with β=0.5 is much faster than the simple RK method in all presented plots, both in terms of iterations and time. In particular, the addition of a momentum parameter leads to visible speedup for the datasets mushrooms, splice, a9a and ionosphere. For these datasets the acceleration is obvious in all plots both in terms of relative error and function values. For the datasets  australian, gisette and madelon the speedup is less obvious in the plots of the relative error, while for the plots of function values it is not present at all.







 §.§ Comparison of momentum & stochastic momentum


In Theorem <ref>, the total complexities (number of operations needed to achieve a given accuracy) of mSGD and smSGD have been compared and it has been shown that for small momentum parameter β,

    C_β=C_mSGD(β)/C_smSGD(β n)≈  1 + n/g,
  where C_mSGD and C_smSGD represent the total costs of the two methods. The goal of this experiment is to show that this relationship holds also in practice. 

For this experiment we assume that the non-zeros of matrix  are not concentrated in certain rows but instead that each row has the same number of non-zero coordinates. We denote by g the number the non-zero elements per row. Having this assumption it can be shown that for the RK method the cost of one projection is equal to 4g operations while 
the cost per iteration of the mRK and of the smRK are 4g+3n and 4g+1 respectively. For more details about the cost per iteration of the general mSGD and smSGD check Table <ref>.

As a first step a Gaussian matrix ∈^m × n is generated.
Then using this matrix several consistent linear systems are obtained as follows. Several values for g ∈ [1,n] are chosen and for each one of these a matrix _g ∈^m × n with the same elements as  but with n-g zero coordinates per row is produced. 
For every matrix _g, a Gaussian vector z_g ∈^n is drawn and to ensure consistency of the linear system, the right hand side is set to b_g=_g z.

We run both mSGD and smSGD with small momentum parameter β=0.0001 for solving the linear systems _g x=b_g for all selected values of g ∈ [1,n]. The starting point for each run is taken to be x^0=0 ∈^n.  The methods run until ϵ=x^k-x^*<0.001, where x^*=Π__g(x^0)[To pre-compute the solution x^* for each linear system _g x=b_g we use the closed form expression of the projection (<ref>). ] and _g is the solution set of the linear system _g x=b_g. In each run the number of operations needed to achieve the accuracy ϵ have been counted. 
For each linear system the average after 10 trials of the value C_mSGD(β)/C_smSGD(β n) is computed. 



In Figure <ref> the actual ratio C_mSGD(β)/C_smSGD(β n) and the theoretical approximation 1 + n/g are plot and it is shown that they have similar behavior. Thus the theoretical prediction of Theorem <ref> is numerically confirmed.
In particular in the implementations we use the Gaussian matrices ∈^200 × 100 and ∈^1000 × 300.




 §.§ Faster method for average consensus




  §.§.§ Background

Average consensus (AC) is a fundamental problem in distributed computing and multi-agent systems <cit.>. Consider a connected undirected network =(,) with node set ={1,2,…,n} and edges , (||=m), where each node i ∈ owns a private value c_i ∈. The goal of the AC problem is for each node of the network to compute the average of these private values, c̅1/n∑_i c_i, via a protocol which allows communication between neighbours only. The problem comes up in many real world applications such as coordination of autonomous agents, estimation, rumour spreading in social networks, PageRank and distributed data fusion on ad-hoc networks and decentralized optimization. 

It was shown recently that several randomized methods for solving linear systems  can be interpreted as randomized gossip algorithms for solving the AC problem when applied to a special system encoding the underlying network <cit.>. As we have already explained both basic method <cit.> and basic method with momentum (this chapter) find the solution of the linear system that is closer to the starting point of the algorithms. That is, both methods converge linearly to x^*=Π_, (x^0); the projection of the initial iterate onto the solution set of the linear system. As a result (check the Introduction), they can be interpreted as methods for solving the best approximation problem (<ref>).
In the special case that


  * the linear system in the constraints of (<ref>) is the homogeneous linear system (x=0) with matrix ∈^ m × n being the incidence matrix of the undirected graph =(,), and 

  * the starting point of the method are the initial values of the nodes x^0=c,
 
it  is straightforward to see that the solution of the best approximation problem is a vector with all components equal to the consensus value c̅1/n∑_i c_i. Under this setting, the famous randomized pairwise gossip algorithm (randomly pick an edge e∈ E and replace the private values of its two nodes to their average) that was first proposed and analyzed in <cit.>, is equivalent to the RK method without relaxation (ω=1) <cit.>.


In the gossip framework, the condition number of the linear system when RK is used has a simple structure and it depends on the characteristics of the network under study. More specifically, it depends on the number of the edges m and on the Laplacian matrix of the network[Matrix  of the linear system is the incidence matrix of the graph and it is known that the Laplacian matrix is equal to =^⊤, where ^2_F=2m.]:

    1/λ_min^+()(<ref>)=1/λ_min^+(^⊤/ ^2_F)^2_F=2m=2m/λ_min^+(^⊤)=2m/λ_min^+(),

where =^⊤ is the Laplacian matrix of the network and the quantity λ_min^+() is the very well studied algebraic connectivity of the graph <cit.>. 



 The convergence analysis in this chapter holds for any consistent linear system x= b without any assumption on the rank of the matrix . The lack of any assumption on the form of matrix  allows us to solve the homogeneous linear system x=0 where  is the incidence matrix of the network which by construction is rank deficient. More specifically, it can be shown that rank()=n-1 <cit.>. Note that many existing methods for solving linear systems make the assumption that the matrix  of the linear systems is full rank <cit.> and as a result can not be used to solve the AC problem.




  §.§.§ Numerical Setup

Our goal in this experiment is to show that the addition of the momentum term to the randomized pairwise gossip algorithm (RK in the gossip setting) can lead to faster gossip algorithms and as a result the nodes of the network will converge to the average consensus faster both in number of iterations and in time. We do not intend to analyze the distributed behavior of the method (this is an ongoing research work). In our implementations we use three of the most popular graph topologies in the literature of wireless sensor networks. These are the line graph, cycle graph and the random geometric graph G(n,r). In practice, G(n,r) consider ideal for modeling wireless sensor networks, because of their particular formulation. In the experiments the 2-dimensional G(n,r) is used which is formed by placing n nodes uniformly at random in a unit square with edges only between nodes that have euclidean distance less than the given radius r. To preserve the connectivity of G(n, r) a radius r = r(n) =  log(n)/n is used <cit.>. 
The AC problem is solved for the three aforementioned networks for both n=100 and n=200 number of nodes. We run mRK with several momentum parameters β for 10 trials and we plot their average. Our results are available in Figures <ref> and <ref>.

Note that the vector of the initial values of the nodes can be chosen arbitrarily, and the proposed algorithms will  find the average  of these values. In Figures <ref> and <ref> the initial value of each node is chosen independently at random from the uniform distribution in the interval (0,1).




  §.§.§ Experimental Results


By observing Figures <ref> and <ref>, it is clear that the addition of the momentum term improves the performance of the popular pairwise randomized gossip (PRG) method <cit.>.  The choice β=0.4 as the momentum parameter improves the performance of the vanilla PRG for all networks under study and β=0.5 is a good choice for the cases of the cycle and line graph. Note that for networks such as the cycle and line graphs there are known closed form expressions for the algebraic connectivity <cit.>. Thus, using equation (<ref>), we can compute the exact values of the condition number 1/λ_min^+ for these networks. Interestingly, as we can see in Table <ref> for n=100 and n=200 (number of nodes), the condition number 1/λ_min^+ appearing in the iteration complexity of our methods is not very large. This is in contrast with experimental  observations from Section <ref> where it was shown that the choice β=0.5 is good  for very ill conditioned problems only (1/λ_min^+ very large). 


 









§ CONCLUSION



In this chapter, we studied the convergence analysis of several stochastic optimization algorithms enriched with heavy ball momentum for solving  stochastic optimization problems of special structure.  We proved global, non-asymptotic linear convergence rates of all of these methods as well as accelerated linear rate for the case of the norm of expected iterates. We also introduced a new momentum strategy called stochastic momentum which is beneficial for the case of sparse data and proved linear convergence in this setting. We corroborated our theoretical results with extensive experimental testing.

Our work is amenable to further extensions. A natural extension of our results is the analysis of heavy ball momentum variants of our proposed methods (SGD,SN, SPP, etc) in the case of general convex or strongly convex functions. While we have shown that the expected iterates converge in an accelerated manner, it is an open problem whether an accelerated rate can be established for the expected distance, i.e., for x^k-x^*_^2.    In our analysis we also focus on the case of fixed constant step-size and momentum parameters. A study of the effect of decreasing or adaptive choice of the parameters might provide novel insights. 

The obtained results hold under the exactness condition which as we explain is very weak, allowing for virtually arbitrary distributions  from which the random matrices are drawn. One may wish to design optimized distributions in terms of the  convergence rates or overall complexity. 

Finally, we show how the addition of momentum on top of gossip algorithms can lead to faster convergence. An interesting question is to interpret the distributed nature of these algorithms and try to understand how we can improve the analysis using the properties of the underlying network. This is precisely what we are doing later in Chapter <ref>. 



§ PROOFS OF MAIN RESULTS





 §.§ Technical lemmas




Fix F^1=F^0≥ 0 and let {F^k}_k≥ 0 be a  sequence of nonnegative  real numbers satisfying the relation 

    F^k+1≤ a_1F^k +a_2 F^k-1,     ∀ k≥ 1,
 where  a_2 ≥ 0,  a_1 + a_2 <1 and at least one of the coefficients a_1,a_2 is positive. Then the sequence satisfies the relation
F^k+1≤ q^k (1+ δ)  F^0 for all k≥ 1,
where q=a_1+√(a_1^2+4a_2)/2  and δ=q-a_1≥ 0. Moreover,

    q ≥ a_1 + a_2,

with equality if and only if a_2=0 (in which case q=a_1 and δ=0).



Choose   δ = -a_1+√(a_1^2+4a_2)/2. We claim δ≥ 0 and a_2 ≤ (a_1+δ)δ.
Indeed, non-negativity of δ  follows from a_2≥ 0, while the second   relation follows from the fact that δ satisfies 
    (a_1+δ)δ - a_2 = 0.
 

In view of these two relations,  adding δ F_k  to both sides of (<ref>),  we get 

    F^k+1 + δ F^k  ≤   (a_1+δ)F^k + a_2 F^k-1
    ≤ (a_1+δ)(F^k + δ F^k-1) = q(F^k+δ F^k-1).


Let us now argue that  0<q<1. Non-negativity of q follows from non-negativity of a_2.  Clearly, as long as a_2>0, q is positive. If a_2=0, then a_1>0 by assumption, which implies that q is positive. The inequality q<1 follows directly from the assumption a_1+a_2<1.
By unrolling the recurrence (<ref>), we obtain
F^k+1≤ F^k+1 + δ F^k ≤ q^k (F^1+ δ F^0) = q^k(1+δ) F^0.

Finally, let us establish (<ref>). Noting that a_1 = q-δ, and since  in view of (<ref>) we have a_2=qδ, we conclude that a_1+a_2 = q + δ(q-1) ≤ q, where the inequality follows from q <1.


Finally, let us present a simple lemma of an identity that we use in our main proofs. This preliminary result is known to hold for the case of Euclidean norms (=). We provide the proof for the more general -norm for completeness.



Let a,b,c be arbitrary vectors in ^n and let  be a positive definite matrix. Then the following identity holds: 
2 ⟨ a-c,c-b ⟩_=a-b^2_-c-b^2_-a-c^2_,



    LHS    =    2 ⟨ a-c,c-b ⟩_ = 2(a-c)^⊤ (c-b)
        =    2a^⊤ c-2a^⊤ b-2c^⊤ c+2c^⊤ b

and 

    RHS    =   a-b^2_-c-b^2_-a-c^2_
        =    (a-b)^⊤ (a-b)- (c-b)^⊤ (c-b)-(a-c)^⊤ (a-c)
       =    a^⊤ a-a^⊤ b-b^⊤ a+b^⊤ b-c^⊤ c+c^⊤ b+b^⊤ c-b^⊤ b
           - a^⊤ a+a^⊤ c+c^⊤ a-c^⊤ c 
        =    2a^⊤ c-2a^⊤ b-2c^⊤ c+2c^⊤ b

LHS (left-hand side)=RHS (right-hand side) and this completes the proof.





 §.§ Proof of Theorem <ref>
 
First, we decompose

    x^k+1-x^*^2_    =   x^k-ω∇ f__k(x^k)+β(x^k-x^k-1)-x^*^2_
        =   x^k-ω∇ f__k(x^k)-x^*^2__1
             +2⟨ x^k-ω∇ f__k(x^k)-x^*, β(x^k-x^k-1)  ⟩__2
             +β^2x^k-x^k-1^2__3.

We will now analyze the three expressions 1, 2, 3 separately. The first expression can be written as

    1   =   x^k-x^*^2_-2 ω⟨ x^k-x^*,∇ f__k(x^k) ⟩_ +ω^2 ∇ f__k(x^k)^2_
       (<ref>),(<ref>)=   x^k-x^*^2_-4ω f__k(x^k)+2ω^2 f__k(x^k) 
       =   x^k-x^*^2_-2ω(2-ω)f__k(x^k).

We will now bound the second expression. First, we have

    2   = 2β⟨ x^k-x^*, x^k-x^k-1⟩_ +2ωβ⟨∇ f__k(x^k), x^k-1 - x^k  ⟩_
       = 2β⟨ x^k-x^*, x^k-x^*⟩_ + 2β⟨ x^k-x^*, x^*-x^k-1⟩_ +2ωβ⟨∇ f__k(x^k),x^k-1- x^k  ⟩_
       = 2βx^k-x^*^2_  + 2β⟨ x^k-x^*, x^*-x^k-1⟩_ +2ωβ⟨∇ f__k(x^k),x^k-1- x^k  ⟩_.

Using the identity from Lemma <ref> for the vectors x^k, x^* and x^k-1 we obtain:

    2 ⟨ x^k-x^*, x^*-x^k-1⟩_=  x^k-x^k-1^2_- x^k-1-x^*^2_-x^k-x^*^2_.

Substituting this into (<ref>) gives

    2    = βx^k-x^*^2_+βx^k-x^k-1^2_-βx^k-1-x^*^2_ + 2ωβ⟨∇ f__k(x^k),x^k-1- x^k  ⟩_.


The third expression can be bounded as

    3 =β^2(x^k-x^*)+(x^*-x^k-1)^2_≤  2β^2x^k-x^*^2_+2
    β^2x^k-1-x^*^2_.


By substituting the  bounds (<ref>), (<ref>), (<ref>) into  (<ref>) we obtain

    x^k+1-x^*^2_   ≤   x^k-x^*^2_-2ω(2-ω)f__k(x^k)
             + βx^k-x^*^2_+βx^k-x^k-1^2_-βx^k-1-x^*^2_
             + 2ωβ⟨∇ f__k(x^k),x^k-1- x^k  ⟩_  +  2β^2x^k-x^*^2_+2β^2x^k-1-x^*^2_
       ≤    (1+3β + 2β^2)x^k-x^*^2_+ (β + 2β^2 )x^k-1-x^*^2_-2ω(2-ω)f__k(x^k)
             + 2ωβ⟨∇ f__k(x^k),x^k-1- x^k  ⟩_.

Now by first taking expectation with respect to _k, we obtain:

    __k[x^k+1-x^*^2_]    ≤    (1+3β+2β^2)x^k-x^*^2_+ (β +2β^2)x^k-1-x^*^2_
             -2ω(2-ω)f(x^k) + 2ωβ⟨∇ f(x^k),x^k-1- x^k  ⟩_
       ≤    (1+3β+2β^2)x^k-x^*^2_+ (β +2β^2)x^k-1-x^*^2_
             -2ω(2-ω)f(x^k) + 2ωβ(f(x^k-1)-f(x^k))
        =     (1+3β+2β^2)x^k-x^*^2_+ (β +2β^2)x^k-1-x^*^2_
             - (2ω(2-ω) +2ωβ)f(x^k) + 2ωβ f(x^k-1).

where in the second step we used the inequality
 ⟨∇ f(x^k),x^k-1- x^k ⟩≤ f(x^k-1)-f(x^k) and the fact that ωβ≥ 0, which follows from the assumptions. We now apply  inequalities  (<ref>) and (<ref>), obtaining


    __k[x^k+1-x^*^2_]    ≤   (1+3β+2β^2 - (ω(2-ω) +ωβ)λ_min^+)_a_1x^k-x^*^2_
             + (β +2β^2 + ωβλ_max)_a_2x^k-1-x^*^2_.


By taking expectation again, and letting F^k[x^k-x^*^2_], we get  the relation

    F^k+1≤ a_1 F^k  + a_2 F^k-1 .


It suffices to apply  Lemma <ref> to the relation  (<ref>). The conditions of the lemma are satisfied. Indeed, a_2≥ 0, and if a_2=0, then β=0 and hence a_1=1-ω(2-ω)λ_min^+>0. The condition a_1+a_2<1 holds by assumption.


The convergence result in function values, [f(x^k)], follows as a corollary by applying inequality (<ref>) to  (<ref>).




 §.§ Proof of Theorem <ref>



Let p^t=β/1-β(x^t-x^t-1) 
and d^t = x^t + p^t -x^*_^2. In view of (<ref>),  we can write 


    x^t+1+p^t+1    =    x^t+1+β/1-β(x^t+1-x^t) 
       (<ref>)=    x^t-ω∇ f__t(x^t)+β(x^t-x^t-1)+β/1-β(-ω∇ f__t(x^t)+β(x^t-x^t-1)) 
        =    x^t-[ω +β/1-βω] ∇ f__t(x^t)+[β+β^2/1-β](x^t-x^t-1)
        =    x^t-ω/1-β∇ f__t(x^t)+β/1-β(x^t-x^t-1)
        =     x^t+p^t-ω/1-β∇ f__t(x^t)


and therefore

    d^t+1   (<ref>)=   x^t+p^t-ω/1-β∇ f__t(x^t) -x^* ^2_ 
        =    d^t -2 ω/1-β⟨ x^t+p^t-x^*,  ∇ f__t(x^t) ⟩_ + ω^2/(1-β)^2∇ f__t(x^t)^2_
        =    d^t -2ω/1-β⟨ x^t-x^*,  ∇ f__t(x^t) ⟩_ - 2 ωβ/(1-β)^2⟨ x^t-x^t-1,  ∇ f__t(x^t) ⟩_
             + ω^2/(1-β)^2∇ f__t(x^t)^2_.


Taking expectation with respect to the random matrix _t we obtain:

    __t[d^t+1]     =    d^t -2ω/1-β⟨ x^t-x^*,  ∇ f(x^t) ⟩_ - 2 ωβ/(1-β)^2⟨ x^t-x^t-1,  ∇ f(x^t) ⟩_
             + ω^2/(1-β)^2 2 f(x^t) 
       (<ref>)=    d^t -4ω/1-β  f(x^t) - 2 ωβ/(1-β)^2⟨ x^t-x^t-1,  ∇ f(x^t) ⟩_ + ω^2/(1-β)^2 2 f(x^t)
       ≤    d^t -4ω/1-β  f(x^t) - 2 ωβ/(1-β)^2   [f(x^t)-f(x^t-1)] + ω^2/(1-β)^2 2 f(x^t)
        =     d^t + [ -4ω/1-β - 2 ωβ/(1-β)^2 +2 ω^2/(1-β)^2] f(x^t)  +  2 ωβ/(1-β)^2 f(x^t-1),

where the inequality follows from convexity of f.  After rearranging the terms we get

    __t[d^t+1] +   2 ωβ/(1-β)^2 f(x^t) + α f(x^t)  ≤ d^t + 2 ωβ/(1-β)^2 f(x^t-1),

where α =   4ω/1-β -2 ω^2/(1-β)^2 > 0. Taking expectations again and using the tower property, we get

    θ^t+1 + α[f(x^t)]  ≤θ^t,      t=1,2,…,

where θ^t = [d^t] + 2 ωβ/(1-β)^2[ f(x^t-1)]. By summing up (<ref>) for t=1,…, k we get

    ∑_t=1^k [f(x^t)] ≤θ^1-θ^k-1/α≤θ^1/α.

Finally, using Jensen's inequality, we get

    [f(x̂^k)] = [f(1/k∑_t=1^k x^t)] ≤[1/k∑_t=1^k f(x^t)] =  1/k∑_t=1^k [f(x^t)] (<ref>)≤θ^1/α k.


It remains to note that θ^1 = x^0-x^*_^2 + 2ωβ/(1-β)^2 f(x^0).





 §.§ Proof of Theorem <ref>
 

In the proof of Theorem <ref> the following two lemmas are used.



Assume exactness. Let x∈^n and x^* = Π_ℒ, (x). If λ_i=0, then u_i^⊤^1/2 (x-x^*)=0.





Consider the second degree linear homogeneous recurrence relation:

    r^k+1= a_1r^k+a_2 r^k-1

with initial conditions r^0,r^1 ∈.
Assume that the constant coefficients a_1 and a_2 satisfy the inequality a_1^2 +4a_2<0 (the roots of the characteristic equation t^2-a_1t-a_2=0 are imaginary). Then there are complex constants C_0 and C_1 (depending on the initial conditions r^0 and r^1) such that:

    r^k=2 M^k (C_0 cos( θ k) + C_1 sin(θ k))

where M= (√(a_1^2/4+(-a_1^2-4a_2)/4))=√(-a_2) and θ is such that a_1=2 M cos(θ) and √(-a_1^2-4a_2)=2 M sin(θ).



We can now turn to the proof of Theorem <ref>.
Plugging in the expression for the stochastic gradient, mSGD can be written in the form

    x^k+1    =    x^k -ω∇ f__k(x^k) + β(x^k - x^k-1) 
       (<ref>)=    x^k- ω^-1_k(x^k-x^*) + β(x^k - x^k-1).


Subtracting  x^* from both sides of (<ref>), we get

    x^k+1-x^* 
        =     ( - ω^-1_k)(x^k-x^*) + β(x^k -x^* +x^* - x^k-1)
        =   ((1+β) - ω^-1_k)(x^k-x^*) - β(x^k-1-x^*).


Multiplying the last identity from the left by ^1/2, we get

    ^1/2 (x^k+1-x^*)
       =   ((1+β) - ω^-1/2_k ^-1/2) ^1/2(x^k -x^*) - β^1/2(x^k-1-x^*).


Taking expectations, conditioned on x^k (that is, the expectation is with respect to _k):

    ^1/2[x^k+1 -x^*  |  x^k]=((1+β) - ω^-1/2[]  ^-1/2) ^1/2(x^k -x^*) - β^1/2(x^k-1-x^*) .


Taking expectations again, and using the tower property, we get

    ^1/2[x^k+1 -x^*]     =    ^1/2[[x^k+1 -x^*  |  x^k]]
       (<ref>)=   ((1+β) - ω^-1/2[] ^-1/2) ^1/2[x^k -x^*] - β^1/2[x^k-1-x^*].



Plugging the eigenvalue decomposition Λ^⊤ of the matrix =^-1/2[] ^-1/2 into the above, and multiplying both sides from the left by ^⊤, we obtain

    ^⊤^1/2[x^k+1 -x^*]  = ^⊤((1+β) - ωΛ^⊤)^1/2[x^k -x^*] - β^⊤^1/2[x^k-1-x^*].


Let us define s^k^⊤^1/2[x^k -x^*]  ∈^n. Then relation (<ref>) takes the form of the recursion

    s^k+1= [(1+β) - ωΛ ] s^k - β s^k-1,

which can be written in a coordinate-by-coordinate form as follows:

    s^k+1_i= [(1+β) - ωλ_i ] s^k_i - β s^k-1_i     for all   i= 1,2,3,...,n,

where s^k_i indicates the ith coordinate of s^k.

We will now fix i and analyze recursion (<ref>) using Lemma <ref>. Note that (<ref>) is a second degree linear homogeneous recurrence relation of the form (<ref>) with a_1=1+β - ωλ_i and a_2=- β. Recall  that 0≤λ_i ≤1 for all i. Since we assume that 0< ω≤ 1/λ_max, we know that 0≤ωλ_i ≤ 1 for all i. We now consider two  cases:



  * λ_i =0.   

In this case, (<ref>) takes the form: 
    s^k+1_i=(1+β)s^k_i-β s^k-1_i .
 Applying Proposition <ref>, we know that x^*=Π_ℒ,(x^0)=Π_ℒ,(x^1). Using Lemma <ref> twice, once for x=x^0 and then for x=x^1, we observe that s^0_i=u_i^⊤^1/2 (x^0-x^*)=0 and s^1_i=u_i^⊤^1/2 (x^1-x^*)=0. Finally, in view of (<ref>) we conclude that 
    s^k_i=0   for all   k≥ 0 .





  * λ_i >0. 

Since 0<ωλ_i ≤ 1 and β≥ 0, we have 1+β - ωλ_i ≥ 0 and hence

    a_1^2+4a_2=(1+β -ωλ_i)^2-4β≤  (1+β -ωλ_min^+)^2-4β  < 0,

where the last inequality can be shown to hold[The lower bound on β is tight. However,  the upper bound is not. However, we do not care much about the regime of large β as β is the convergence rate, and hence is only interesting if smaller than 1.] for  (1-√(ωλ_min^+))^2 < β < 1. Applying Lemma <ref> the following bound can be deduced

    s^k_i     =    2(-a_2)^k/2 (C_0  cos(θ k) +C_1 sin(θ k))  ≤  2 β^k/2 P_i,

where P_i is a constant depending  on the initial conditions (we can simply choose P_i = |C_0| + |C_1|).





Now putting the two cases together, for all k≥0 we have

    [x^k -x^*]_^2   =   [x^k -x^*]^⊤[x^k -x^*]   =  [x^k -x^*] ^1/2^⊤^1/2[x^k -x^*]  
       =   ^⊤^1/2[x^k -x^*] ^2   =  s^k^2  =  ∑_i=1^n (s^k_i)^2 
       =    ∑_i: λ_i=0 (s^k_i)^2  +  ∑_i:  λ_i >0 (s^k_i)^2  (<ref>)= ∑_i:  λ_i >0 (s^k_i)^2
       (<ref>)≤   ∑_i:  λ_i >0 4 β ^k P_i^2 
       =   β^k C,

where C=4∑_i:  λ_i >0  P_i^2.



 §.§ Proof of Theorem <ref> 
 

The proof follows a similar pattern to that of Theorem <ref>. However, stochasticity in the momentum term introduces an additional layer of complexity, which we shall tackle by utilizing   a more involved version of the tower property.

For simplicity, let i=i_k and r^k_i   e_i^⊤(x^k-x^k-1)e_i. First, we decompose

    x^k+1-x^*^2 
        =   x^k-ω∇ f__k(x^k)+γ r^k_i -x^*^2 
        =   x^k-ω∇ f__k(x^k)-x^*^2+2⟨ x^k-ω∇ f__k(x^k)-x^*, γ r^k_i  ⟩ + γ^2 r^k_i^2.

 We shall use the tower property in the form

    [[[ X   |  x^k,  _k]  |  x^k]] = [X],

where X is some random variable. We shall perform the three expectations in order, from the innermost to the outermost.  Applying the inner expectation to the identity (<ref>), we get

    [x^k+1-x^*^2  |  x^k, _k ] 
       =   [x^k-ω∇ f__k(x^k)-x^*^2 |  x^k, _k]_1
             +[2⟨ x^k-ω∇ f__k(x^k)-x^*, γ r^k_i  ⟩ |  x^k, _k]_2
             +[γ^2 r^k_i^2 |  x^k, _k]_3.


We will now analyze the three expressions 1, 2, 3 separately. The first expression is constant under the expectation, and hence we can write

    1   =   x^k-ω∇ f__k(x^k)-x^*^2
       =   x^k-x^*^2-2 ω⟨ x^k-x^*,∇ f__k(x^k) ⟩ +ω^2 ∇ f__k(x^k)^2
       (<ref>) + (<ref>)=   x^k-x^*^2-4ω f__k(x^k)+2ω^2 f__k(x^k)
       =   x^k-x^*^2-2ω(2-ω)f__k(x^k).


We will now bound the second expression. Using the identity

    [r^k_i   |  x^k, _k] = _i [r^k_i] = ∑_i=1^n 1/nr^k_i = 1/n(x^k-x^k-1),

we can write

    2   =   [2⟨ x^k-ω∇ f__k(x^k)-x^*, γ r^k_i  ⟩ |  x^k, _k]
       =    2⟨ x^k-ω∇ f__k(x^k)-x^*, γ[r^k_i   |  x^k, _k] ⟩
       (<ref>)=    2⟨ x^k-ω∇ f__k(x^k)-x^*, γ/n (x^k - x^k-1)  ⟩
        =   
    2γ/n⟨ x^k-x^*, x^k-x^k-1⟩ +2ωγ/n⟨∇ f__k(x^k), x^k-1 - x^k  ⟩
       =    2γ/n⟨ x^k-x^*, x^k-x^*⟩ + 2γ/n⟨ x^k-x^*, x^*-x^k-1⟩ +2ωγ/n⟨∇ f__k(x^k),x^k-1- x^k  ⟩
       =    2γ/nx^k-x^*^2 + 2γ/n⟨ x^k-x^*, x^*-x^k-1⟩ +2ωγ/n⟨∇ f__k(x^k),x^k-1- x^k  ⟩.

Using the fact that for arbitrary vectors a,b,c ∈^n we have the identity
2 ⟨ a-c,c-b ⟩ =a-b^2-c-b^2-a-c^2,
we obtain

    2 ⟨ x^k-x^*, x^*-x^k-1⟩=  x^k-x^k-1^2- x^k-1-x^*^2-x^k-x^*^2.

Substituting this into (<ref>) gives

    2    = γ/nx^k-x^*^2+γ/nx^k-x^k-1^2-γ/nx^k-1-x^*^2 + 2ωγ/n⟨∇ f__k(x^k),x^k-1- x^k  ⟩.


The third expression can be bound as

    3   =   [γ^2 r^k_i^2 |  x^k, _k] 
       =   γ^2 _i[ r^k_i^2] 
       =   γ^2 ∑_i=1^n 1/n (x^k_i -x^k-1_i)^2 
       =   γ^2/nx^k-x^k-1^2 
       =   γ^2/n(x^k-x^*)+(x^*-x^k-1)^2 
       ≤   2γ^2/nx^k-x^*^2+ 2γ^2/nx^k-1-x^*^2.


By substituting the  bounds (<ref>), (<ref>), (<ref>) into  (<ref>) we obtain

    [x^k+1-x^*^2  |  x^k, _k ] 
       ≤   x^k-x^*^2-2ω(2-ω) f__k(x^k)
             + γ/nx^k-x^*^2+ γ/nx^k-x^k-1^2 -γ/nx^k-1-x^*^2 
             + 2ωγ/n⟨∇ f__k(x^k), x^k-1- x^k  ⟩  +  2γ^2/nx^k-x^*^2 
             + 2γ^2/nx^k-1-x^*^2 
       (<ref>)≤   (1+3γ/n + 2γ^2/n)x^k-x^*^2+ (γ/n + 2γ^2/n) x^k-1-x^*^2 
             - 2ω(2-ω)f__k(x^k) + 2ωγ/n⟨∇ f__k(x^k),x^k-1- x^k  ⟩.

We now take the middle expectation (see (<ref>)) and apply it to inequality (<ref>):

    [[x^k+1-x^*^2  |  x^k, _k ]  |  x^k]     ≤   (1+3γ/n + 2γ^2/n)x^k-x^*^2+ (γ/n + 2γ^2/n) x^k-1-x^*^2 
             -2ω(2-ω)f(x^k) + 2ωγ/n⟨∇ f(x^k),x^k-1- x^k  ⟩
       ≤   (1+3γ/n + 2γ^2/n)x^k-x^*^2+ (γ/n + 2γ^2/n) x^k-1-x^*^2 
             -2ω(2-ω)f(x^k) + 2ωγ/n(f(x^k-1)-f(x^k))
       =   (1+3γ/n + 2γ^2/n)x^k-x^*^2+ (γ/n + 2γ^2/n) x^k-1-x^*^2 
             - (2ω(2-ω) +2ωγ/n)f(x^k) + 2ωγ/n f(x^k-1).

where in the second step we used the inequality
 ⟨∇ f(x^k),x^k-1- x^k ⟩≤ f(x^k-1)-f(x^k) and the fact that ωγ≥ 0, which follows from the assumptions. We now apply  inequalities  (<ref>) and (<ref>), obtaining

    [[x^k+1-x^*^2  |  x^k, _k ]  |  x^k]   ≤   (1+3γ/n+2γ^2/n - (ω(2-ω) +ωγ/n)λ_min^+ )_a_1x^k-x^*^2 
             + 1/n(γ +2γ^2 + ωγλ_max)_a_2x^k-1-x^*^2.


By taking expectation again (outermost expectation in the tower rule (<ref>)), and letting F^k[x^k-x^*^2_], we get  the relation

    F^k+1≤ a_1 F^k  + a_2 F^k-1 .


It suffices to apply  Lemma <ref> to the relation  (<ref>). The conditions of the lemma are satisfied. Indeed, a_2≥ 0, and if a_2=0, then γ=0 and hence a_1=1-ω(1-ω)λ_min^+>0. The condition a_1+a_2<1 holds by assumption.

The convergence result in function values follows as a corollary by applying inequality (<ref>) to  (<ref>).



CHAPTER: INEXACT RANDOMIZED ITERATIVE METHODS





§ INTRODUCTION

In the era of big data where data sets become continuously larger, randomized iterative methods become very popular and are increasingly playing a major role in areas such as numerical linear algebra, scientific computing and optimization. They are preferred mainly because of their cheap per-iteration cost which leads to improvements in terms of complexity upon classical results by orders of magnitude. In addition they can easily scale to extreme dimensions.
However, a common feature of these methods is that in their update rule a particular subproblem needs to be solved exactly. In a large scale setting, often this step is computationally very expensive. The purpose of this work is to reduce the cost of this step by allowing for inexact updates in the stochastic methods under study.



 §.§ The setting

In this chapter we are interested to solve the three closely related problems described in the previous chapters. As a reminder, these are:


  (i) stochastic quadratic optimization (<ref>), 

  (ii) best approximation (<ref>), and 

  (iii) (bounded) concave quadratic maximization (<ref>).
 

In particular we propose and analyze inexact variants of the exact algorithms presented in the introduction of this thesis for solving the above problems. Among the methods studied are: stochastic gradient descent (SGD), stochastic Newton (SN), stochastic proximal point (SPP),  sketch and project method (SPM) and stochastic subspace ascent (SDSA). 
In all of these methods, a certain potentially expensive calculation/operation needs to be performed in each step; it is this operation that we propose to be performed inexactly.  For instance, in the case of SGD, it is the computation of the stochastic gradient ∇ f__k(x^k), in the case of  SPM is the computation of the projection Π__, (x^k), and in the case of SDSA it is the computation of the dual update _k λ^k.

We perform an iteration complexity analysis under an abstract notion of inexactness and also under a more structured form of inexactness appearing in practical scenarios. An inexact solution of these subproblems can be obtained much more quickly than the exact solution. Since in practical applications the savings thus obtained are larger than the increase in the number of iterations needed for convergence, our inexact methods can be dramatically faster.



 §.§ Structure of the chapter and main contributions


Let us now outline the main contribution and the structure of this chapter.

In Section <ref> we describe the subproblems and introduce two notions of inexactness (abstract and structured) that will be used in the rest of this chapter. The Inexact Basic Method (iBasic) is also presented. iBasic is a method that simultaneously captures inexact variants of the algorithms (<ref>), (<ref>), (<ref>) for solving the stochastic optimization problem (<ref>) and algorithm (<ref>) for solving the best approximation problem (<ref>). It is an inexact variant of the Basic Method, first presented in <cit.>, where the inexactness is introduced by the addition of an inexactness error ϵ^k in the original update rule. We illustrate the generality of  iBasic  by presenting popular algorithms that can be cast as special cases. 

In Section <ref> we establish convergence results of iBasic under general assumptions on the inexactness error ϵ^k of its update rule (see Algorithm <ref>). In this part we do not focus on any specific mechanisms which lead to inexactness; we treat the problem abstractly. However, such errors appear often in practical scenarios and can be associated with inaccurate numerical solvers, quantization,  sparsification and compression mechanisms. In particular, we introduce several abstract assumptions on the inexactness level  and describe our generic convergence results. For all assumptions we establish linear rate of decay of the quantity [x^k-x^*_^2] (i.e. L2 convergence)[As we explain later, a convergence of the expected function values of problem <ref> can be easily obtained as a corollary of L2 convergence.]. 

Subsequently, in Section <ref> we apply our general convergence results to a more structured notion of inexactness error and  propose a concrete mechanisms leading to such errors.  We provide theoretical guarantees for this method  in situations when  a linearly convergent iterative method (e.g., Conjugate Gradient)  is used to solve the subproblem inexactly. We also highlight the importance of the dual viewpoint through a sketch-and-project interpretation.

In Section <ref> we study an inexact variant of SDSA, which we called iSDSA, for directly solving the dual problem (<ref>).  We provide a correspondence between  iBasic and iSDSA and we show that the random iterates of iBasic arise as affine images of iSDSA. We consider both abstract and structured inexactness errors and provide linearly convergent rates in terms of the dual function suboptimality  D(y^*) - D(y^0).

Finally, in Section <ref> we evaluate the performance of the proposed inexact methods through numerical experiments and show the benefits of our approach on both synthetic and real datasets. Concluding remarks are given in Section <ref>. 

A summary of the convergence results of iBasic under several assumptions on the inexactness error  with pointers to the relevant theorems is available in Table <ref>. We highlight that similar convergence results can be also obtained for  iSDSA in terms of the dual function suboptimality  D(y^*) - D(y^0) (check Section <ref> for more details on iSDSA).





 §.§ Notation

Following the rest of this thesis, with boldface upper-case letters we denote matrices and  is the identity matrix. By  we denote the solution set of the linear system x=b. By _, where  is a random matrix, we denote the solution set of the sketched linear system ^⊤ x= ^⊤ b.  In general, we use * to express the exact solution of a sub-problem and ≈ to indicate its inexact variant. 
Unless stated otherwise, throughout the chapter, x^* is the projection of x^0 onto  in the -norm: x^*=Π_, (x^0). 



§ INEXACT UPDATE RULES


In this section we start by explaining the key sub-problems that need to be solved exactly in the update rules of the previously described methods. We present iBasic, a method that solves problems (<ref>) and (<ref>) and we show how by varying the main parameters of the method we recover inexact variants of popular algorithms as special cases. Finally closely related work on inexact algorithms for solving different problems is also presented.


 §.§ Expensive sub-problems in update rules


Let us devote this subsection on explaining how the inexactness can be introduced in the current exact update rules of SGD[Note that SGD has identical updates to the Stochastic Newton and Stochastic proximal point method. Thus the inexactness can be added to these updates in similar way.] (<ref>), Sketch and Project (<ref>) and SDSA (<ref>) for solving the stochastic optimization, best approximation  and the dual problem respectively. As we have shown these methods solve closely related problems and the key subproblems in their update rule are similar. However the introduction of inexactness in the update rule of each one of them can have different interpretation. 

For example for the case of SGD for solving the stochastic optimization problem (<ref>) (see also Section <ref> and <ref> for more details), if we define λ^k_*=(_k^⊤^-1^⊤_k)^†_k^⊤(b- x^k)  then the stochastic gradient of function f becomes  ∇ f__k(x^k)(<ref>)=-^-1 ^⊤_k λ^k_* and the update rule of SGD takes the form: x^k+1 = x^k+ω^-1^⊤_kλ^k_*.  Clearly in this update the expensive part is the computation of the quantity λ^k_* that can be equivalently computed to be the least norm solution of the smaller (in comparison to x=b) linear system _k^⊤^-1^⊤_kλ =_k^⊤ (b- x^k). In our work we are suggesting to use an approximation λ^k_≈ of the exact solution and with this way avoid executing the possibly expensive step of the update rule.  Thus the inexact update is taking the following form:

    x^k+1 =x^k+ω^-1^⊤_kλ^k_≈= x^k - ω∇ f__k(x^k)+ω^-1^⊤_k (λ^k_≈-λ^k_*)_ϵ^k.

Here ϵ^k denotes a more abstract notion of inexactness and it is not necessary to be always equivalent to the quantity ω^-1^⊤_k(λ^k_≈-λ^k_*). It can be interpreted as an expression that acts as an perturbation of the exact update. In the case that ϵ^k has the above form we say that the notion of inexactness is structured.
In our work we are interested in both the abstract and more structured notions of inexactness. We first present general convergence results where we require the error ϵ^k to satisfy general assumptions (without caring how this error is generated) and later we analyze the concept of structured inexactness by presenting algorithms where ϵ^k= ω^-1^⊤_k(λ^k_≈-λ^k_*).

In similar way, the expensive operation of SPM (<ref>) is the exact computation of the projection Π___k,^*(x^k). Thus we are suggesting to replace this step with an inexact variant and compute an approximation of this projection. The inexactness here can be also interpreted using both, the abstract ϵ^k error and its more structured version ϵ^k=ω( Π___k,^≈(x^k)- Π___k,^*(x^k) ). At this point, observe that, by using the expression (<ref>) the structure of the ϵ^k in SPM and SGD has the same form.

In the SDSA the expensive subproblem in the update rule is the computation of the λ^k_* that satisfy λ^k_* ∈max_λ D(y^k + _k λ). Using the definition of the dual function (<ref>) this value can be also computed by evaluating the least norm solution of the linear system _k^⊤^-1^⊤_kλ  =_k^⊤(b-(x^0 + ^-1^⊤ y^k )). Later in Section <ref> we analyze both notions of inexactness (abstract and more structured) for inexact variants of SDSA.

Table <ref> presents the key sub-problem that needs to be solved in each algorithm as well as the part where the inexact error is appeared in the update rule.  





 §.§ The inexact basic method

In each iteration of the all aforementioned exact methods a sketch matrix ∼ D is drawn from a given distribution and then a certain subproblem is solved exactly to obtain the next iterate. The sketch matrix ∈^m × q requires to have m rows but no assumption on the number of columns is made which means that the number of columns q allows to vary through the iterations and it can be very large.  The setting that we are interested in is precisely that of having such large random matrices . In these cases we expect that having approximate solutions of the subproblems will be beneficial.

Recently randomized iterative algorithms that requires to solve large subproblems in each iteration have been extensively studied and it was shown that are really beneficial when they compared to their single coordinates variants (∈^m × 1) <cit.>. However, in theses cases the evaluation of an exact solution for the suproblem in the update rule can be computationally very expensive.
In this work we propose and analyze inexact variants by allowing to solve the  subproblem that appear in the update rules of the stochastic methods, inexactly. In particular, following the convention established in <cit.> of naming the main algorithm of the paper Basic method we propose the inexact Basic method (iBasic) (Algorithm <ref>). 



The ϵ^k in the update rule of the method represents the abstract inexactness error described in Subsection <ref>. Note that, iBasic can have several equivalent interpretations. This allow as to study the methods (<ref>),(<ref>),(<ref>) for solving the stochastic optimization problem and the sketch and project method (<ref>) for the best approximation problem in a single algorithm only. 
In particular iBasic can be seen as inexact stochastic gradient descent (iSGD) with fixed stepsize applied to (<ref>). From (<ref>), ∇ f__k(x^k) = ^-1^⊤_k ( x^k - b) and as a result the update rule of iBasic can be equivalently written as: x^k+1=x^k-ω∇ f__k(x^k)  + ϵ^k. In the case of the best approximation problem (<ref>), iBasic can be interpreted as inexact Sketch and Project method (iSPM) as follows:

    x^k+1    =     x^k-ω^-1^⊤_k (_k^⊤^-1^⊤_k)^†_k^⊤( x^k-b)+ϵ^k  
        =    ω[ x^k- ^-1(_k^⊤)^⊤ (_k^⊤ B^-1(_k^⊤) ^ ⊤ )^† (_k^⊤ x^k-_k^⊤ b)] + (1-ω) x^k   +ϵ^k 
       (<ref>) =   ωΠ___k,(x^k) + (1-ω) x^k +ϵ^k

For the dual problem (<ref>) we devote Section <ref> for presenting an inexact variant of the SDSA (iSDSA) and analyze its convergence using the rates obtained for the iBasic in Sections <ref> and <ref>.



 §.§ General framework and further special cases



The proposed inexact methods, iBasic  (Algorithm <ref>) and iSDSA (Section <ref>),  belong in the general sketch and project framework, first proposed from Gower and Richtarik in <cit.> for solving consistent linear systems and where a unified analysis of several randomized methods was studied. This interpretation of the algorithms allow us to recover a comprehensive array of well-known methods as special cases by choosing carefully the combination of the main parameters of the algorithms.

In particular, the iBasic has two main parameters (besides the stepsize ω>0 of the update rule). These are the distribution  from which we draw random matrices  and the positive definite matrix ∈^n× n. By choosing carefully combinations of the parameters  and  we can recover several existing popular algorithms as special cases of the general method.  For example, special cases of the exact Basic method are the Randomized Kaczmarz, Randomized Gaussian Kaczmarz[Special case of the iBasic, when the random matrix  is chosen to be a Gaussian vector with mean 0 ∈ R^m and a positive definite covariance matrix Σ∈^m× m. That is ∼ N(0,Σ) <cit.>.], Randomized Coordinate Descent and their block variants. For more details about the generality of the sketch and project framework and further algorithms that can be cast as special cases of the analysis we refer the interested reader to Section 3 of <cit.>. Here we present only the inexact update rules of two special cases that we will later use in the numerical evaluation. 

Special Cases: 
Let us define with _:C the column concatenation of the m × m identity matrix indexed by a random subset C of [m]. 


  * Inexact Randomized Block Kaczmarz (iRBK):
Let = and let pick in each iteration the random matrix =_:C∼. In this setup the update rule of the iBasic simplifies to 

    x^k+1=x^k -ω_C:^⊤ (_C:_C:^⊤)^† (_C:x^k-b_C) + ϵ^k.


  * Inexact Randomized Block Coordinate Descent (iRBCD)[In the setting of solving linear systems Randomized Coordinate Descent is known also as Gauss-Seidel method. Its block variant can be also interpret as randomized coordinate Newton method (see <cit.>).]:
If the matrix  of the linear system is positive definite then we can choose =. Let also pick in each iteration the random matrix =_:C∼. In this setup the update rule of the iBasic simplifies to 

    x^k+1=x^k -ω_:C (_:C^⊤_:C)^†_:C^⊤ ( x^k-b) + ϵ^k.



For more papers related to Kaczmarz method (randomized, greedy, cyclic update rules) we refer the interested reader to <cit.>. 
For the coordinate descent method (a.k.a Gauss-Seidel for linear systems) and its block variant, Randomized Block Coordinate Descent we suggest <cit.>. 



 §.§ Other related work on inexact methods


One of the current trends in the large scale optimization problems is the introduction of inexactness in the update rules of popular deterministic and stochastic methods. The rational behind this is that an approximate/inexact step can often computed very efficiently and can have significant computational gains compare to its exact variants. 

In the area of deterministic algorithms, the inexact variant of the full gradient descent method, x^k+1 = x^k - ω_k [∇ f(x^k)+ϵ^k], has received a lot of attention <cit.>. It has been analyzed for the cases of convex and strongly convex functions under several meaningful assumptions on the inexactness error ϵ^k and its practical benefit  compared to the exact gradient descent is apparent.  For further deterministic inexact methods check <cit.> for Inexact Newton methods, <cit.> for Inexact Proximal Point methods and <cit.> for Inexact Fixed point methods.

In the recent years, with the explosion that happens in areas like machine learning and data science inexactness enters also the updating rules of several stochastic optimization algorithms and many new methods have been proposed and analyzed. 

In the large scale setting, stochastic optimization methods are preferred mainly because of their cheap per iteration cost (compared to their deterministic variants), their property to scale to extreme dimensions and their improved theoretical complexity bounds. In areas like machine learning and data science, where the datasets become larger rapidly, the development of faster and efficient stochastic algorithms is crucial. For this reason, inexactness has recently introduced to the update rules of several stochastic optimization algorithms and new methods have been proposed and analyzed. One of the most interesting work on inexact stochastic algorithms appears in the area of second order methods. In particular on inexact variants of the Sketch-Newton method and subsampled Newton Method for minimize convex and non-convex functions <cit.>. Note that our results are related also with this literature since our algorithm can be seen as inexact stochastic Newton method (see equation (<ref>)). To the best or our knowledge our work is the first that provide convergence analysis of inexact stochastic proximal point methods (equation (<ref>)) in any setting. From numerical linear algebra viewpoint inexact sketch and project methods for solving the best approximation problem and its dual problem where also never analyzed before. 

As we already mentioned our framework is quite general and many algorithms, like iRBK (<ref>) and iRBCD (<ref>) can be cast as special cases. As a result, our general convergence analysis includes the analysis of inexact variants of all of these more specific algorithms as special cases. In <cit.> an analysis of the exact randomized block Kacmzarz method has been proposed and in the experiments an inexact variant was used to speedup the method. However, no iteration complexity results were presented for the inexact variant and both the analysis and numerical evaluation have been made for linear systems with full rank matrices that come with natural partition of the rows (this is a much more restricted case than the one analyzed in our setting). For inexact variants of the randomized block coordinate descent algorithm in different settings than ours we suggest  <cit.>.

Finally an analysis of approximate stochastic gradient descent for solving the empirical risk minimization problem using quadratic constraints and sequential semi-definite programs has been presented in <cit.>. 



§ CONVERGENCE RESULTS UNDER GENERAL ASSUMPTIONS


In this section we consider scenarios in which the inexactness error ϵ^k  can be controlled, by specifying a per iteration bound σ_k on the norm of the error.  In particular, by making different assumptions on the bound σ_k we derive general convergence rate results. Our focus is on the abstract notion of inexactness described in Section <ref> and we make no assumptions on how this error is generated. 

An important assumption that needs to be hold in all of our results is exactness. A formal presentation of exactness was presented in the introduction of this thesis. We highlight that is a requirement for all of the convergence results of this chapter (It is also required in the analysis of the exact algorithms; see Theorems <ref> and <ref> in the introduction).



 §.§ Assumptions on inexactness error


In the convergence analysis of iBasic the following assumptions on the inexactness error are used.  We note that Assumptions <ref>, <ref> and <ref> are special cases of Assumption <ref>. Moreover Assumption <ref> is algorithmic dependent and can hold in addition of any of the other four assumptions. In our analysis, depending on the result we aim at, we will require either one of the first four Assumptions to hold by itself, or to hold together with Assumption <ref>. We will always assume exactness.
  
In all assumptions the expectation on the norm of error (ϵ^k^2) is conditioned on the value of the current iterate x^k and the random matrix _k. Moreover it is worth to mention that for the convergence analysis we never assume that the inexactness error has zero mean, that is [ϵ^k]=0. 

1

 
    [ϵ^k^2_ | x^k,_k]≤σ_k^2,

where the upper bound σ_k is a sequence of random variables (that can possibly depends on both the value of the current iterate x^k and the choice of the random _k at the k^th iteration).


The following three assumptions on the sequence of upper bounds are more restricted however as we will later see allow us to obtain stronger and more controlled results.

 1a
 
 
    [ϵ^k^2_ | x^k,_k]≤σ_k^2,

where the upper bound σ_k∈ is a sequence of real numbers. 


1b

 
    [ϵ^k^2_ | x^k,_k]≤σ_k^2 = q^2x^k-x^*^2_,

where the upper bound is a special sequence that depends on a non-negative inexactness parameter q and the distance to the optimal value x^k-x^*^2_.


1c

 
    [ϵ^k^2_ | x^k,_k]≤σ_k^2 = 2 q^2 f__k(x^k),

where the upper bound is a special sequence that depends on a non-negative inexactness parameter q and the value of the stochastic function f__k computed at the iterate x^k. Recall, that in our setting f_(x) = 1/2∇ f_(x)^2_  (<ref>). Hence, the upper bound can be equivalently expressed as σ_k^2=q^2 ∇ f_(x)^2_.



Finally the next assumption is more algorithmic oriented. It holds in cases where the inexactness error ϵ^k in the update rule is chosen to be orthogonal with respect to the -inner product to the vector  Π___k, (x^k) - x^* = ( - ω^-1_k) (x^k-x^*). This statement may seem odd at this point but its usefulness will become more apparent in the next section where inexact algorithms with structured inexactness error will be analyzed. As it turns out, in the case of structured inexactness error (Algorithm <ref>) this assumption is satisfied.
2
 
 
    [⟨ ( - ω^-1_k) (x^k-x^*), ϵ^k ⟩_]=0.






 §.§ Convergence results

In this section we present the analysis of the convergence rates of iBasic by assuming several combination of the previous presented assumptions. 

All convergence results are described only in terms of convergence of the iterates x^k, that is x^k-x^*^2_, and not the objective function values f(x^k). This is sufficient, because by f(x)≤λ_ max/2x-x^*^2_ (see Lemma <ref>) we can directly deduce a convergence rate for the function values. 

The exact Basic method (Algorithm <ref> with ϵ^k =0), has been analyzed in <cit.> and it was shown to converge with [x^k-x^*^2_] ≤ρ^kx^0-x^*^2_ where ρ=1-ω(2-ω) λ_min^+. Our analysis of iBasic is more general and includes the convergence of the exact Basic method as special case when we assume that the upper bound is σ_k=0,   ∀ k≥0. For brevity, in he convergence analysis results of this chapter we also use 
    ρ=1-ω(2-ω) λ_min^+.


Let us start by presenting the convergence of iBasic when only Assumption <ref> holds for the inexactness error.


Let assume exactness and let {x^k}_k=0^∞ be the iterates produced by iBasic with ω∈(0,2).  Set x^*= Π_,(x^0) and consider the error ϵ^k be such that it satisfies Assumption <ref>. Then,

    [x^k-x^*_] ≤ρ^k/2x^0-x^*_ +  ∑_i=0^k-1ρ^k-1-i/2σ_i.



See Section <ref>.



 In the special case that the upper bound σ_k in Assumption <ref> is fixed, that is σ_k=σ for all k>0 then inequality (<ref>) of Theorem <ref> takes the following form:

    [x^k-x^*_] ≤ρ^k/2x^0-x^*_ +  σρ^1/2/1-ρ.

This means that we obtain a linear convergence rate up to a solution level that is proportional to the upper bound σ[Several similar more specific assumptions can be made for the upper bound σ_k. For example if the upper bound satisfies σ_k=σ^k with σ∈ (0,1) for all k>0 then it can be shown that C∈ (0,1) exist such that inequality (<ref>) of Theorem <ref> takes the form: [x^k-x^*_] ≤(C^k) (see <cit.> for similar results). ].


See Section <ref>.


Inspired from <cit.>, let us now analyze iBasic using the sequence of upper bounds that described in Assumption <ref>. This construction of the upper bounds  allows us to obtain stronger and more controlled results. In particular using the upper bound of Assumption <ref> the sequence of expected errors converge linearly to the exact x^* (not in a potential neighborhood like the previous result). In addition Assumption <ref> guarantees that the distance to the optimal solution reduces with the increasing of the number of iterations. However for this stronger convergence a bound for λ_min^+ is required, a quantity that in many problems is unknown to the user or intractable to compute. Nevertheless, there are cases that this value has a closed form expression and can be computed before hand without any further cost. See for example <cit.> where methods for solving the average consensus were presented and the value of λ_min^+ corresponds to the algebraic connectivity of the network under study.


Assume exactness. Let {x^k}_k=0^∞ be the iterates produced by iBasic with ω∈(0,2). Set x^*= Π_, (x^0) and consider the inexactness error ϵ^k be such that it satisfies Assumption <ref>,
with 0≤ q < 1-√(ρ). Then 

    [x^k-x^*_^2]≤(√(ρ)+q)^2kx^0-x^*_^2.



See Section <ref>.


At Theorem <ref>, to guarantee linear convergence the inexact parameter q should live in the interval [0,1-√(ρ)). In particular, q is the parameter that controls the level of inexactness of Algorithm <ref>. Not surprisingly the fastest convergence rate is obtained when q=0; in such case the method becomes equivalent with its exact variant and the convergence rate simplifies to ρ=1- ω (2-ω)λ_min^+.  Note also that similar to the exact case the optimal convergence rate is obtained for ω=1 <cit.>.

Moreover, the upper bound σ_k of Assumption <ref> depends on two important quantities, the λ_min^+ (through the upper bound of the inexactness parameter q) and the distance to the optimal solution x^k-x^*^2_. Thus, it can have natural interpretation. In particular the inexactness error is allowed to be large either when the current iterate is far from the optimal solution (x^k-x^*^2_ large) or when the problem is well conditioned and λ_min^+ is large. In the opposite scenario, when we have ill conditioned problem or we are already close enough to the optimum x^* we should be more careful and allow less errors to the updates of the method. 

In the next theorem we provide the complexity results of iBasic in the case that the Assumption <ref> is satisfied combined with one of the previous assumptions.


Let assume exactness and let {x^k}_k=0^∞ be the iterates produced by iBasic with ω∈(0,2).  Set x^*= Π_,(x^0). Let also assume that the inexactness error ϵ^k be such that it satisfies Assumption <ref>. Then:


  (i) If Assumption <ref> holds: 

    [x^k-x^*_^2] ≤ρ^kx^0-x^*^2_ +  ∑_i=0^k-1ρ^k-1-iσ̅^2_i,

where σ̅_i^2=[σ_i^2], ∀ i ∈ [k-1].

  (ii) If Assumption <ref> holds with q ∈(0,√(ρ)): 

    [x^k-x^*_^2]    ≤    (ρ + q^2)^k x^0-x^*_^2.


  (iii) If Assumption <ref> holds with q ∈(0, √(ω(2-ω))): 

    [x^k-x^*_^2]≤(1- (ω (2-ω)-q^2 )λ_min^+ )^k x^0-x^*_^2=(ρ+q^2λ_min^+ )^k x^0-x^*_^2.

 

See Section <ref>.




In the case that Assumptions <ref> and <ref> hold simultaneously, the convergence of iBasic is similar to (<ref>) but in this case σ̅^2_i=σ_i^2, ∀ i ∈ [k-1] (due to Assumption <ref>, σ_k ∈ is a sequence of real numbers). In addition, note that for q ∈ (0, min{√(ρ), 1-√(ρ)}) having Assumption <ref> on top of Assumption <ref> leads to improvement of the convergence rate. In particular, from Theorem <ref>, iBasic converges with rate (√(ρ)+q)^2= ρ+q^2+2√(ρ)q while having both assumptions this is simplified to the faster ρ + q^2 (<ref>). 




§ IBASIC WITH STRUCTURED INEXACTNESS ERROR


Up to this point, the analysis of iBasic was focused in more general abstract cases where the inexactness error ϵ^k of the update rule satisfies several general assumptions.  In this section we are focusing on a more structured form of inexactness error and we provide convergence analysis in the case that a linearly convergent algorithm is used for the computation of the expensive key subproblem of the method. 



 §.§ Linear system in the update rule



As we already mentioned in Section <ref> the update rule of the exact Basic method (Algorithm <ref> with ϵ^k=0) can be expressed as x^k+1 = x^k+ω^-1^⊤_kλ^k_*, where λ^k_*= (_k^⊤^-1^⊤_k)^†_k^⊤(b- x^k).

Using this expression the exact Basic method can be equivalently interpreted as the following two step procedure:


  * Find the least norm solution[We are precisely looking for the least norm solution of the linear system _k  λ =d_k because this solution can be written down in a compact way using the Moore-Penrose pseudoinverse.  This is equivalent with the expression that appears in our update: λ^k_*=(_k^⊤^-1^⊤_k)^†_k^⊤(b- x^k)=_k^† d_k. However it can be easily shown that the method will still converge with the same rate of convergence even if we choose any other solution of the linear system _k  λ =d_k.] of _k^⊤^-1^⊤_k__kλ = _k^⊤(b- x^k)_d_k. That is find λ^k_*=min_λ∈_kλ where _k= {λ∈^q : _k  λ =d_k}.

  * Compute the next iterate:  x^k+1=x^k + ω^-1 ^⊤_kλ^k_*.


In the case that the random matrix _k is large (this is the case that we are interested in), solving exactly the linear system _k  λ=d_k in each step can be prohibitively expensive. To reduce this cost we allow the inner linear system _k  λ=d_k to be solved inexactly using an iterative method. In particular we propose and analyze the following inexact algorithm: 



For the computation of the inexact solution of the linear system (<ref>) any known iterative method for solving general linear systems can be used. In our analysis we focus on linearly convergent methods. For example based on the properties of the linear system (<ref>),  conjugate gradient (CG) or sketch and project method (SPM) can be used for the execution of step 3. In these cases, we name Algorithm <ref>,  InexactCG and InexactSP  respectively.

It is known that the classical CG can solve linear systems with positive definite matrices. In our approach matrix _k is positive definite only when the original linear system x= b has full rank matrix . On the other side SPM can solve any consistent linear system and as a result can solve the inner linear system _k  λ^k=d_k without any further assumption on the original linear system. In this case, one should be careful because the system has no unique solution. We are interested to find the least norm solution of _k  λ^k=d_k which means that the starting point of the sketch and project at the k^th iteration should be always λ^k_0=0. Recall that any special case of the sketch and project method (Section <ref>) solves the best approximation problem.

Let us now define λ^k_r to be the approximate solution λ^k_≈ of the q × q linear system (<ref>) obtained after r steps of the linearly convergent iterative method.  Using this, the update rule of Algorithm <ref>, takes the form:

    x^k+1=x^k+ ω^-1 ^⊤_kλ^k_r.




The update rule (<ref>) of Algorithm <ref> is equivalent to the update rule of iBasic (Algorithm <ref>) when the error ϵ^k is chosen to be,

    ϵ^k=ω^-1^⊤_k (λ^k_r-λ^k_*).

This is precisely the connection between the abstract and more concrete/structured notion of inexactness that first presented in Table <ref>.


Let us now define a Lemma that is useful for the analysis of this section and it verifies that Algorithm <ref> with unit stepsize satisfies the general Assumption <ref> presented in Section <ref>. 



Let us denote x^k_*=Π___k,(x^k) the projection of x^k onto __k in the -norm and x^*= Π_,(x^0). Let also assume that ω=1 (unit stepsize). Then for the updates of Algorithm <ref> it holds that:

    ⟨ x^k_*-x^*, ϵ^k ⟩_=⟨ ( - ω^-1_k) (x^k-x^*), ϵ^k ⟩_=0,   ∀ k ≥ 0.



Note that x^k_*-x^*=x^k-∇ f__k(x^k)-x^* ∈ Null (_k^⊤) . Moreover ϵ^k(<ref>)=^-1^⊤_k (λ^k_r - λ^k_*) ∈ Range(^-1^⊤_k). From the knowledge that the null space of an arbitrary matrix is the orthogonal complement of the range space of its transpose we have that Null (_k^⊤) is orthogonal with respect to the -inner product to Range(^-1^⊤_k). This completes the proof (see Figure <ref> for the graphical interpretation).






 §.§ Sketch and project interpretation


Let us now give a different interpretation of the inexact update rule of Algorithm <ref> using the sketch and project approach. That will make us appreciate more the importance of the dual viewpoint and make clear the connection between the primal and dual methods.










In general, execute a projection step is one of the most common task in numerical linear algebra/optimization literature. However in the large scale setting even this task can be prohibitively expensive and it can be difficult to execute inexactly.  For this reason we suggest to move to the dual space where the inexactness can be easily controlled. 

Observe that the update rule of the exact sketch and project method (<ref>) has the same structure as the best approximation problem (<ref>) where the linear system under study is the sketched system _k^⊤ x = _k^⊤ b and the starting point is the current iterate x^k. Hence we can easily compute its dual:

    max_λ∈^q D_k(λ)  (_k^⊤ b - _k^⊤ x^k)^⊤λ - 1/2^⊤_k λ^2_^-1.

where λ∈^q is the dual variable. The λ^k_* (possibly more than one) that solves the dual problem in each iteration k, is the one that satisfies ∇ D_k (λ^k_*)=0. By computing the derivative this is equivalent with finding the λ that satisfies the linear system _k^⊤^-1^⊤_kλ = _k^⊤(b- x^k). This is the same linear system we desire to solve inexactly in Algorithm <ref>. Thus, computing an inexact solution λ^k_≈ of the linear system is equivalent with computing an inexact solution of the dual problem (<ref>). Then by using the affine mapping (<ref>) that connects the primal and the dual spaces we can also evaluate an inexact solution of the original primal problem (<ref>).

The following result relates the inexact levels of these quantities. In particular it shows that dual suboptimality of λ^k in terms of dual function values is equal to the distance of the dual values λ^k in the _k-norm.


Let us define λ^k_* ∈^q be the exact solution of the linear system _k^⊤^-1^⊤_kλ = _k^⊤(b- x^k) or equivalently of dual problem (<ref>). Let us also denote with λ^k_≈∈^q the inexact solution. Then:

    D_k(λ^k_*)-D_k(λ^k_≈)= 1/2λ^k_≈-λ^k_*^2__k^⊤^-1^⊤_k.





    D_k(λ^k_*)-D_k(λ^k_≈)    (<ref>)=     [_k^⊤ b - _k^⊤ x^k]^⊤ [λ^k_*-λ^k_≈]-1/2 (λ^k_*)^⊤_k^⊤^-1^⊤_kλ^k_*
          +1/2 (λ^k_≈)^⊤_k^⊤^-1^⊤_kλ^k_≈
       (<ref>)=    (λ^k_*)^⊤_k^⊤^-1^⊤_k  [λ^k_*-λ^k_≈]-1/2 (λ^k_*)^⊤_k^⊤^-1^⊤_kλ^k_*
          + 1/2 (λ^k_≈)^⊤_k^⊤^-1^⊤_kλ^k_≈
        =    1/2(λ^k_≈-λ^k_*)^⊤_k^⊤^-1^⊤_k(λ^k_≈-λ^k_*) 
        =    1/2λ^k_≈-λ^k_*^2__k^⊤^-1^⊤_k

where in the second equality we use equation (<ref>) to connect the optimal solutions of (<ref>) and (<ref>) and obtain [_k^⊤ b - _k^⊤ x^k]^⊤=(λ^k_*)^⊤_k^⊤^-1^⊤_k.




 §.§ Complexity results


In this part we analyze the performance of Algorithm <ref> when a linearly convergent iterative method is used for solving inexactly the linear system (<ref>) in step 3 of Algorithm <ref> . We denote with λ^k_r the approximate solution of the linear system after we run the iterative method for r steps.

Before state the main convergence result let us present a lemma that summarize some observations that are true in our setting.

Let λ^k_*=(_k^⊤^-1^⊤_k)^†_k^⊤(b- x^k)  be the exact solution and λ^k_r be approximate solution of the linear system (<ref>). Then, λ^k_*^2__k=2 f__k(x^k) and ϵ^k_^2= λ^k_r -λ^k_*__k^2.


    λ^k_*^2__k   =   _k^†_k^⊤ (x^*-x^k) ^2__k = (x^k-x^*)^⊤^⊤_k _k^†_k _k^†__k^†_k^⊤  (x^k-x^*) 
       (<ref>)=    (x^k-x^*)^⊤_k (x^k-x^*) (<ref>)= 2 f__k(x^k).

Moreover,

    ϵ^k_^2    Remark <ref>=   ^-1^⊤_k (λ^k_r -λ^k_*)_^2 
    =  λ^k_r -λ^k_*__k^⊤^-1 ^⊤_k^2
    =   λ^k_r -λ^k_*__k^2.






Let us assume that for the computation of the inexact solution of the linear system (<ref>) in step 3 of Algorithm <ref>, a linearly convergent iterative method is chosen such that [In the case that deterministic iterative method is used, like CG, we have that λ^k_r-λ^k_*__k^2 ≤ρ__k^r λ^k_0-λ^k_*^2__k which is also true in expectation.]: 

    [λ^k_r-λ^k_*__k^2  | x^k,_k] ≤ρ__k^r λ^k_0-λ^k_*^2__k,

where λ^k_0=0 for any k>0 and ρ__k∈ (0,1) for every choice of _k ∼.
Let exactness hold and let {x^k}_k=0^∞ be the iterates produced by Algorithm <ref> with unit stepsize (ω=1).  Set x^*= Π_,(x^0). 
Suppose further that there exists a scalar θ <1 such that with
probability 1,  ρ__k≤θ. Then,  Algorithm <ref> converges linearly with:

    [x^k-x^*_^2] ≤[ 1- (1 -θ^r  ) λ_min^+ ] ^k x^0-x^*_^2.



Theorem <ref> can be interpreted as corollary of the general Theorem <ref>(iii). Thus, it is sufficient to show that Algorithm <ref> satisfies the two Assumptions <ref> and <ref>. Firstly, note that from Lemma <ref>, Assumption <ref> is true. Moreover, 

    [ϵ^k__k^2  | x^k,_k]    (<ref>)=   [λ^k_r-λ^k_*__k^2  | x^k,_k] (<ref>)≤ρ__k^r λ^k_0-λ^k_*^2__k
       ≤   θ^r λ^k_0-λ^k_*^2__kλ^k_0=0=θ^r λ^k_*^2__k(<ref>)=  2 θ^r f__k(x^k)

which means that Assumption <ref> also holds with q = θ^r/2∈(0,1). This completes the proof.


Having present the main result of this section let us now state some remarks that will help understand the convergence rate of the last Theorem.

From its definition θ^r ∈ (0,1) and as a result (1 -θ^r  ) λ_min^+ ≤λ_min^+. This means that the method converges linearly but always with worst rate than its exact variant. 



Let as assume that θ is fixed. Then as the number of iterations in step 3 of the algorithm (r→∞) increasing (1-θ^r) → 1 and as a result the method behaves similar to the exact case.



The λ_min^+ depends only on the random matrices ∼ and to the positive definite matrix  and is independent to the iterative process used in step 3. The iterative process of step 3 controls only the parameter θ of the convergence rate.



Let us assume that we run Algorithm <ref> two separate times for two different choices of the linearly convergence iterative method of step 3. Let also assume that the distribution  of the random matrices and the positive definite matrix  are the same for both instances and that for step 3 the iterative method run  for r steps for both algorithms. Let assume that θ_1< θ_2 then we have that ρ_1=1- (1 -θ_1^r  ) λ_min^+< 1- (1 -θ_2^r  ) λ_min^+=ρ_2. This means in the case that θ is easily computable, we should always prefer the inexact method with smaller θ.


The convergence of Theorem <ref> is quite general and it holds for any linearly convergent methods that can inexactly solve  (<ref>). However, in case that the iterative method is known we can have more concrete results. See below the more specified results for the cases of Conjugate gradient (CG) and Sketch and project method (SPM). 



  
Convergence of InexactCG:
CG is deterministic iterative method for solving linear systems x=b with symmetric and positive definite matrix ∈^n× n in finite number of iterations. In particular, it can be shown that converges to the unique solution in at most n steps. The worst case behavior of CG is given by <cit.> [A sharper convergence rate of CG <cit.> for solving x=b can be also used 

    x^k-x^*^2_≤( λ_n-k-λ_1/λ_n-k+λ_1)^2x^0-x^*^2_,

where matrix ∈^n× n has λ_1≤λ_2 ≤…≤λ_n eigenvalues.]:

    x^k-x^*_≤( √(κ())-1/√(κ())+1)^2kx^0-x^*_,

where x^k is the k^th iteration of the method and κ() the condition number of matrix .

Having present the convergence of CG for general linear systems, let us now return back to our setting. We denote λ^k_r ∈^q to be the approximate solution of the inner linear system (<ref>) after r conjugate gradient steps. Thus using (<ref>) we know that
λ^k_r-λ^k_*__k^2 ≤ρ__k^4rλ^k_0-λ^k_*^2__k,
where ρ__k= ( √(κ(_k))-1/√(κ(_k))+1). Now by making the same assumption as the general Theorem <ref> the InexactCG converges with
[x^k-x^*_^2] ≤[ 1- (1 -θ_CG^r  ) λ_min^+ ] ^k x^0-x^*_^2,
where θ_CG <1 such that ρ__k=( √(κ(_k))-1/√(κ(_k))+1)^4≤θ_CG with probability 1.



  
Convergence of InexactSP:
In this setting we suggest to run the sketch and project method (SPM) for solving inexactly the linear system (<ref>). This allow us to have no assumptions on the structure of the original system x=b and as a result we are able to solve more general problems compared to what problems InexactCG can solve[Recall that InexactCG requires the matrix _k to be positive definite (this is true when matrix  is a full rank matrix)]. Like before, by making the same assumptions as in Theorem <ref> the more specific convergence [x^k-x^*_^2] ≤[ 1- (1 -θ_SP^r  ) λ_min^+ ] ^k x^0-x^*_^2, for the InexactSP can be obtained. Now the quantity ρ__k denotes the convergence rate of the exact Basic method[Recall that iBasic and its exact variant (ϵ^k=0) can be expressed as sketch and project methods (<ref>).] when this applied to solve linear system (<ref>) and θ_SP <1 is a scalar such that ρ__k≤θ_SP with probability 1.



§ INEXACT DUAL METHOD


In the previous sections we focused on the analysis of inexact stochastic methods for solving the stochastic optimization problem (<ref>) and the best approximation (<ref>).  In this section we turn into the dual of the best approximation (<ref>) and we propose and analyze an inexact variant of the SDSA (<ref>). We call the new method iSDSA and is formalized as Algorithm <ref>. In the update rule ϵ_d^k indicates the dual inexactness error that appears in the k^th iteration of iSDSA.





 §.§ Correspondence between the primal and dual methods

With the sequence of the dual iterates {y^k}_k=0^∞ produced by the iSDSA we can associate a sequence of primal iterates  {x^k}_k=0^∞ using the affine mapping (<ref>).  In our first result we show that the random iterates produced by iBasic arise as an affine image of iSDSA under this affine mapping.

(Correspondence between the primal and dual methods)

Let {x^k}_k=0^∞  be the iterates produced by iBasic (Algorithm <ref>). Let y^0=0, and {y^k}_k=0^∞ the iterates of the iSDSA. Assume that the two methods use the same stepsize ω >0 and the same sequence of random matrices _k. Assume also that ϵ^k=^-1^⊤ϵ_d^k where ϵ^k and ϵ_d^k are the inexactness errors appear in the update rules of iBasic and iSDSA respectively. Then 
    x^k=ϕ(y^k)=x^0+^-1^⊤ y^k.
 for all k≥0. That is, the primal iterates arise as affine images of the dual iterates.



    ϕ(y^k+1)    (<ref>)=    x^0  + ^-1^⊤ y^k+1(<ref>),Alg.<ref>= x^0+^-1^⊤[y^k+ω_k λ^k +ϵ_d^k ]
       (<ref>), (<ref>)=   x^0+^-1^⊤ y^k_ϕ(y^k)+ω^-1_k (x^*-( x^0+^-1^⊤ y^k_ϕ(y^k)) )+^-1^⊤ϵ_d^k
       =   ϕ(y^k)- ω^-1_k( ϕ(y^k)-x^*) +^-1^⊤ϵ_d^k

Thus by choosing the inexactness error of the primal method to be ϵ^k=^-1^⊤ϵ_d^k the sequence of vectors  {ϕ(y^k)} satisfies the same recursion as the sequence {x^k} defined by iBasic.  It remains to check that the first element of both recursions coincide. Indeed, since y^0=0, we have x^0 = ϕ(0) =  ϕ(y^0).




 §.§ iSDSA with structured inexactness error

In this subsection we present Algorithm <ref>. It can be seen as a special case of iSDSA but with a more structured inexactness error. 



Similar to their primal variants, it can be easily checked that Algorithm <ref> is a special case of the iSDSA ( Algorithm <ref>) when the dual inexactness error is chosen to be ϵ^k_d= _k(λ^k_r-λ^k_*). Note that, using the observation of Remark <ref> that ϵ^k=ω^-1^⊤_k (λ^k_r-λ^k_*) and the above expression of ϵ^k_d we can easily verify that the expression
ϵ^k=^-1^⊤ϵ_d^k holds.
This is precisely the connection between the primal and dual inexactness errors that have already been used in the proof of Theorem <ref>.



 §.§ Convergence of dual function values

We are now ready to state a linear convergence result describing the behavior of the inexact dual method in terms of the function values D(y^k). The following result is focused on the convergence of iSDSA by making similar assumption to Assumption <ref>. Similar convergence results can be obtained using any other assumption of Section <ref>. The convergence of Algorithm <ref>, can be also easily derived using similar arguments with the one presented in Section <ref> and the convergence guarantees of Theorem <ref>.

(Convergence of dual objective).
Assume exactness. Let y^0=0 and let {y^k}_k=0^∞ to be the dual iterates of iSDSA (Algorithm <ref>) with ω∈ (0,2).  Set x^*= Π_,(x^0) and let y^* be any dual optimal solution. Consider the inexactness error ϵ^k_d be such that it satisfies 
 [^-1^⊤ϵ^k_d^2_ | y^k,_k]≤σ_k^2= q^2 2 [D(y^*)-D(y^k)]
where 0≤ q < 1-√(ρ). Then 

    [D(y^*)-D(y^k)] ≤(√(ρ)+q)^2k[ D(y^*)-D(y^0)].



The proof follows by applying Theorem <ref> together with Theorem <ref> and the identity 1/2x^k-x^*^2_ = D(y^*) - D(y^k) (<ref>).


Note that in the case that q=0, iSDSA simplifies to its exact variant SDSA and the convergence rate coincide with the one presented in Theorem <ref>. Following similar arguments to those in <cit.>, the same rate can be proved for the duality gap [P(x^k)-D(y^k)].



§ NUMERICAL EVALUATION


In this section we perform preliminary numerical tests for studying the computational behavior of iBasic with structured inexactness error when is used to solve the best approximation problem (<ref>) or equivalently the stochastic optimization problem (<ref>)[Note that from Section <ref> and the correspondence between the primal and dual methods, iSDSA will have similar behavior when is applied to the dual problem (<ref>).]. As we have already mentioned, iBasic can be interpreted as sketch-and-project method, and as a result a comprehensive array of well-known algorithms can be recovered as special cases by varying the main parameters of the methods (Section <ref>).  In particular, in our experiments we focus on the evaluation of two popular special cases, the inexact Randomized Block Kaczmarz (iRBK) (equation (<ref>)) and inexact randomized block coordinate descent method (iRBCD) (equation (<ref>))
We implement Algorithm <ref> presented in Section <ref> using CG [Recall that in order to use CG, the matrix _k that appears in linear system (<ref>) should be positive definite. This is true in the case that the matrix  of the original system has full column rank matrix. Note however that the analysis of Section <ref> holds for any consistent linear system x=b and without making any further assumption on its structure or the linearly convergence methods.] to inexactly solve the linear system of the update rule (equation (<ref>)). Recall that in this case we named the method InexactCG.

The convergence analysis of previous sections is quite general and holds for several combinations of the two main parameters of the method, the positive definite matrix  and the distribution  of the random matrices . For obtaining iRBK as special case we have to choose =∈^n × n (Identity matrix) and for the iRBCD the given matrix  should be positive definite and choose =. For both methods the distribution  should be over random matrices =_:C where _:C is the column concatenation of the m × m identity matrix indexed by a random subset C of [m]. In our experiments we choose to have one specific distribution over these matrices. In particular, we assume that the random matrix in each iteration is chosen uniformly at random to be =_:d with the subset d of [m] to have fixed pre-specified cardinality.

The code for all experiments is written in the Julia 0.6.3 programming language and run on a Mac laptop computer (OS X El Capitan), 2.7 GHz Intel Core i5 with 8 GB of RAM.

To coincide with the theoretical convergence results of Algorithm <ref> the relaxation parameter (stepsize) of the methods study in our experiments is chosen to be ω=1 (no relaxation). In all implementations, we use x^0=0 ∈^n as an initial point and in comparing the methods with their inexact variants we use the relative error measure x^k-x^*^2_ / x^0-x^*^2_x^0=0=x^k-x^*^2_ / x^*^2_. We run each method (exact and inexact) until the relative error is below 10^-5. For the horizontal axis we use either the number of iterations or the wall-clock time measured using the tic-toc Julia function. In the exact variants, the linear system (<ref>) in Algorithm <ref> needs to be solved exactly. In our experiments we follow the implementation of <cit.> for both exact RBCD and exact RBK where the built-in direct solver (sometimes referred to as "backslash") is used. 



  
Experimental setup:
For the construction of consistent linear systems x=b we use the setup described in Section <ref>. In particular, the linear systems used for the numerical evaluation of iRBK and iRBCD have been generated as described in Section <ref> for algorithms mRK and mRCD, respectively.








 §.§ Importance of large block size

Many recent works have shown that using larger block sizes can be very beneficial for the performance of randomized iterative algorithms <cit.>. In Figure <ref> we numerically verify this statement. We show that both RBK and RBCD (no inexact updates) outperform in number of iterations and wall clock time their serial variants where only one coordinate is chosen (block of size d=1) per iteration. This justify the necessity of choosing methods with large block sizes. Recall that this is precisely the class of algorithms that could have an expensive subproblem in their update rule which is required to be solved exactly and as a result can benefit the most from the introduction of inexactness.





 §.§ Inexactness and block size (iRBCD)

In this experiment, we first construct a positive definite linear system following the previously described procedure for iRBCD. We first generate a Gaussian matrix ∈^10000 × 7000 and then the positive definite matrix = ^⊤∈^7000 × 7000 is used to define a consistent liner system. We run iRBCD in this specific linear system and compare its performance with its exact variance for several block sizes d (numbers of column of matrix ). For evaluating the inexact solution of the linear system in the update rule we run CG for either 2, 5 or 10 iterations. In Figure <ref>, we plot the evolution of the relative error in terms of both the number of iterations and the wall-clock time.

We observe that for any block size the inexact methods are always faster in terms of wall clock time than their exact variants even if they require (as is expected) equal or larger number of iterations. Moreover it is obvious that the performance of the inexact method becomes much better than the exact variant as the size d increases and as a results the sub-problem that needs to be solved in each step becomes more expensive. It is worth to highlight that for the chosen systems, the exact RBCD behaves better in terms of wall clock time as the size of block increases (this coincides with the findings of the previous experiment).





 §.§ Evaluation of iRBK

In the last experiment we evaluate the performance of iRBK in both synthetic and real datasets. For computing the inexact solution of the linear system in the update rule we run CG for pre-specified number of iterations that can vary depending the datasets.  In particular, we compare iRBK and RBK on synthetic linear systems generated with the Julia Gaussian matrix functions “randn(m,n)" and “sprandn(m,n,r)" (input r of sprandn function indicates the density of the matrix). For the real datasets, we test the performance of iRBK and RBK using real matrices from the library of support vector machine problems LIBSVM <cit.>. Each dataset of the LIBSVM consists of a matrix ∈^m × n (m features and n characteristics) and a vector of labels b ∈^m. In our experiments we choose to use only the matrices of the datasets and ignore the label vectors [Note that the real matrices of the Splice and Madelon datasets are full rank matrices.]. As before, to ensure consistency of the linear system, we choose a Gaussian vector z∈^n and the right hand side of the linear system is set to b =  z (for both the synthetic and the real matrices). By observing Figure <ref> it is clear that for all problems under study the performance of iRBK in terms of wall clock time is much better than its exact variant RBK.





§ CONCLUSION



In this chapter we propose and analyze inexact variants of several stochastic algorithms for solving quadratic optimization problems and linear systems. We provide linear convergence rate under several assumptions on the inexactness error. The proposed methods require more iterations than their exact variants to achieve the same accuracy. However, as we show through our numerical evaluations, the inexact algorithms require significantly less time to converge. 

With the continuously increasing size of datasets, inexactness should  definitely be a tool that practitioners should use in their implementations even in the case of stochastic methods that have much cheaper-to-compute iteration complexity than their deterministic variants.  Recently, accelerated and parallel stochastic optimization methods <cit.> have been proposed for solving linear systems. We speculate that the addition of inexactness to these update rules will lead to methods faster in practice. We also believe that our approach and complexity results can be extended to the more general case of minimization of convex and non-convex functions in the stochastic setting.  



§ PROOFS OF MAIN RESULTS

In our convergence analysis we use several popular inequalities. Look Table <ref>  for the abbreviations and the relevant formulas.

A key step in the proofs of the theorems is to use the tower property of the expectation. We use it in the form

    [[[ X   |  x^k,  _k]  |  x^k]] = [X],

where X is some random variable. In all proofs we perform the three expectations in order, from the innermost to the outermost. Similar to the main part of this chapter we use ρ=1 - ω(2-ω)λ_ min^+.

The following remark on random variables is also used in our proofs.
 
Let x and y be random vectors and let σ positive constant. If we assume [x_^2 | y] ≤σ^2 then by using the variance inequality (check Table <ref>) we obtain [x_ | y] ≤σ.
In our setting if we assume [ϵ^k_^2 | x^k,_k] ≤σ_k^2 where ϵ^k is the inexactness error and x^k is the current iterate then by the variance inequality it holds that [ϵ^k_ | x^k,_k] ≤σ_k.


 



 §.§ Proof of Theorem <ref>



First we decompose:

    x^k+1-x^*_^2    =    ( - ω^-1_k) (x^k-x^*) +  ϵ^k_^2
       =   ( - ω^-1_k) (x^k-x^*)_^2+ϵ^k_^2 
          +2 ⟨ ( - ω^-1_k) (x^k-x^*), ϵ^k ⟩.

Applying the innermost expectation of (<ref>) to (<ref>), we get:

    [x^k+1-x^*_^2 | x^k, _k]    =   [ ( - ω^-1_k) (x^k-x^*)_^2 | x^k,_k ]_T1 + [ ϵ^k_^2 | x^k,_k ]_T2
          + 2[⟨ ( - ω^-1_k) (x^k-x^*), ϵ^k ⟩_ | x^k,_k ]_T3.

We now analyze the three expression T1,T2,T3 separately. 

Note that an upper bound for the expression T2 can be directly obtained from the assumption 

    T2= [ϵ^k^2_ | x^k,_k]≤σ_k^2.

The first expression can be written as: 

    T1= [ ( - ω^-1_k) (x^k-x^*)_^2 | x^k,_k]    =    ( - ω^-1_k) (x^k-x^*)_^2
       (<ref>)=   x^k-x^*_^2 - 2ω (2-ω)f__k(x^k).

For expression T3: 

    [⟨ ( - ω^-1_k) (x^k-x^*), ϵ^k ⟩_ | x^k,_k ]    =   ⟨ ( - ω^-1_k) (x^k-x^*), [ϵ^k | x^k,_k] ⟩_
        C.S.≤   ( - ω^-1_k) (x^k-x^*)_[ϵ^k | x^k,_k]_
        C.J.≤   ( - ω^-1_k) (x^k-x^*)_[ϵ^k _ | x^k,_k]
       (*)≤   ( - ω^-1_k) (x^k-x^*)_σ_k,

where in the inequality (*) we use Remark <ref> and  (<ref>).

By substituting the bounds (<ref>), (<ref>), and (<ref>) into (<ref>) we obtain:

    [x^k+1-x^*_^2 | x^k, _k ]    ≤   x^k-x^*_^2 - 2ω (2-ω)f__k(x^k)+ σ_k^2 
          +2 ( - ω^-1_k) (x^k-x^*)_σ_k.

We now take the middle expectation (see (<ref>)) and apply it to inequality (<ref>):

    [[x^k+1-x^*_^2 | x^k, _k] | x^k ]    ≤   x^k-x^*_^2 - 2ω (2-ω)f(x^k)+σ_k^2 
          +2 [( - ω^-1_k) (x^k-x^*)_ | x^k]σ_k.

Now let us find a bound on the quantity [( - ω^-1_k) (x^k-x^*)_ | x^k].
Note that from (<ref>) and (<ref>) we have that [( - ω^-1_k) (x^k-x^*)^2_ | x^k] ≤ρx^k-x^*^2_. By using Remark <ref> in the last inequality we obtain:

    [( - ω^-1_k) (x^k-x^*)_ | x^k]     =    √(ρ)x^k-x^*_.
 

By substituting (<ref>) in (<ref>):

    [[x^k+1-x^*_^2 | x^k, _k] | x^k]    ≤   x^k-x^*_^2 - 2ω (2-ω)f(x^k)+ σ_k^2 
          +2σ_k √(ρ)x^k-x^*_
       (<ref>)≤   ρx^k-x^*_^2 + σ_k^2 + 2σ_k √(ρ)x^k-x^*_

We take the final expectation (outermost expectation in the tower rule (<ref>)) on the above expression to find:

    [x^k+1-x^*_^2]   =   [[[x^k+1-x^*_^2 | x^k, _k] | x^k]]
       ≤   ρ[x^k-x^*_^2] + σ_k^2 + 2 σ_k √(ρ)  [x^k-x^*_] 
       V.I≤   ρ[x^k-x^*_^2] + σ_k^2 + 2 σ_k √(ρ)√([x^k-x^*_^2])

Using r_k = x^k-x^*_^2 equation (<ref>) takes the form: 

    r_k+1≤   ρ r_k  +  σ_k^2 + 2σ_k √(ρ)√(r_k) = ( √(ρ r_k)+σ_k )^2

 If we further substitute p_k=√(r_k) and ℓ=√(ρ) the recurrence simplifies to:
 
    p_k+1≤   ℓ p_k +σ_k

By unrolling the final inequality:

    p_k ≤ℓ^k r_0 + (ℓ^0σ_k-1+ℓσ_k-2+⋯ + ℓ^k-1σ_0) = ℓ^k p_0 +  ∑_i=0^k-1ℓ^k-1-iσ_i.

Hence,  
    √([x^k-x^*_^2])≤ρ^k/2x^0-x^*_ +  ∑_i=0^k-1ρ^k-1-i/2σ_i.

The result is obtained by using V.I in the last expression.




 §.§ Proof of Corollary <ref>


By denoting r_k=[x^k-x^*_] in (<ref>) we obtain:

    r_k ≤ρ^k/2 r_0 + ρ^1/2σ∑_i=0^k-1ρ^k-1-i =ρ^k/2 r_0 + ρ^1/2σ∑_i=0^k-1ρ^i=ρ^k/2 r_0 + ρ^1/2σ1-ρ^k/1-ρ.

Since 1-ρ^k≤ 1 the result is obtained. 



 §.§ Proof of Theorem <ref>


In order to prove Theorem <ref> we need to follow similar steps to the proof of Theorem <ref>. The main  differences of the two proofs appear at the points that we need to upper bound the norm of the inexactness error (ϵ^k^2). In particular instead of using the general sequence σ_k^2 ∈ we utilize the bound q^2x^k-x^*^2_ from Assumption <ref>. Thus, it is sufficient to focus at the parts of the proof that these bound is used.

Similar to the proof of Theorem <ref> we first decompose to obtain the equation (<ref>). There, the expression T1 can be upper bounded from (<ref>) but now using the Assumption <ref> the expression T2  and T3 can be upper bounded as follows:

    T2= [ϵ^k^2_ | x^k,_k]≤ q^2x^k-x^*^2_.


    T3   =   [⟨ ( - ω^-1_k) (x^k-x^*), ϵ^k ⟩_ | x^k,_k] 
        Remark <ref>  and (<ref>)≤   ( - ω^-1_k) (x^k-x^*)_ qx^k-x^*


As a result by substituting the bounds (<ref>), (<ref>), and (<ref>) into (<ref>) we obtain:

    [x^k+1-x^*_^2 | x^k, _k]    (<ref>)≤   x^k-x^*_^2 - 2ω (2-ω)f__k(x^k)+ q^2x^k-x^*^2_
          +2 ( - ω^-1_k) (x^k-x^*)_ qx^k-x^*_.


By following the same steps to the proof of Theorem <ref> the equation (<ref>) takes the form:

    [[x^k+1-x^*_^2 | x^k, _k] | x^k] 
       ≤   ρx^k-x^*_^2 + q^2x^k-x^*^2_
           +    2qx^k-x^*_√(ρ)x^k-x^*_
       =   (ρ+2q √(ρ)+q^2)x^k-x^*_^2.
        =    (√(ρ)+q)^2 x^k-x^*_^2

We take the final expectation (outermost expectation in the tower rule (<ref>)) on the above expression to find:

    [x^k+1-x^*_^2]   =   [[[x^k+1-x^*_^2 | x^k, _k] | x^k] ]
       ≤   (√(ρ)+q)^2 [x^k-x^*_^2].

The final result follows by unrolling the recurrence. 



 §.§ Proof of Theorem <ref>



Similar to the previous two proofs by decomposing the update rule and using the innermost expectation of (<ref>) we obtain equation (<ref>).  An upper bound of expression T1 is again given by inequality (<ref>). For the expression T2 depending the assumption that we have on the norm of the inexactness error different upper bounds can be used. In particular, 


  (i) If Assumption <ref> holds then:  
 T2=[ϵ^k^2_ | x^k,_k]≤σ_k^2.

  (ii) If Assumption <ref> holds then: T2=[ϵ^k^2_ | x^k,_k]≤σ_k^2 = q^2x^k-x^*^2_.

  (iii) If Assumption <ref> holds then:  
T2=[ϵ^k^2_ | x^k,_k]≤σ_k^2 = 2 q^2 f_S_k(x^k).


The main difference from the previous proofs, is that due to the Assumption <ref> and tower property (<ref>) the expression T3 will eventually be equal to zero. More specifically, we have that:

    [[[ ⟨ ( - ω^-1_k) (x^k-x^*), ϵ^k ⟩_ |  x^k,  _k ]  |  x^k] ]    =   [⟨ ( - ω^-1_k) (x^k-x^*), ϵ^k ⟩_] 
       =    T3=0,

Thus, in this case equation (<ref>) takes the form:

    [x^k+1-x^*_^2 | x^k, _k]    ≤   x^k-x^*_^2 - 2ω (2-ω)f__k(x^k) + σ_k^2.

Using the above expression depending the assumption that we have we obtain the following results:


  (i) 
By taking the middle expectation (see (<ref>)) and apply it to the above inequality:

    [[x^k+1-x^*_^2 | x^k, _k ] | x^k]    ≤   x^k-x^*_^2 - 2ω (2-ω)f(x^k)+[σ_k^2  | x^k] 
       (<ref>)≤   ρx^k-x^*_^2 +[σ_k^2  | x^k]


We take the final expectation (outermost expectation in the tower rule (<ref>)) on the above expression to find:

    [x^k+1-x^*_^2 ]   =   [[[x^k+1-x^*_^2 | x^k, _k] | x^k] ]
       ≤   ρ[x^k-x^*_^2] + [[σ_k^2  | x^k]] 
        =   ρ[x^k-x^*_^2] + [σ_k^2] 
        =   ρ[x^k-x^*_^2] + σ̅_k^2

Using r_k = x^k-x^*_^2 the last inequality takes the form
r_k+1≤ρ r_k  +   σ̅_k^2. By unrolling the last expression:
r_k ≤ρ^k r_0 + (ρ^0 σ̅_k-1^2+ρσ̅^2_k-2+⋯ + ρ^k-1σ̅^2_0) =ρ^k r_0 +  ∑_i=0^k-1ρ^k-1-iσ̅^2_i.
Hence,  
    [x^k-x^*_^2] ≤ρ^kx^0-x^*^2_ +  ∑_i=0^k-1ρ^k-1-iσ̅^2_i.


  (ii)
For the case (ii) inequality (<ref>) takes the form:

    [x^k+1-x^*_^2 | x^k, _k] ≤x^k-x^*_^2 - 2ω (2-ω)f__k(x^k) + q^2x^k-x^*^2_,

and by taking the middle expectation (see (<ref>)) we obtain:

    [[x^k+1-x^*_^2 | x^k, _k] | x^k]     ≤   x^k-x^*_^2 - 2ω (2-ω)f(x^k) + q^2x^k-x^*^2_
       (<ref>)≤   ρx^k-x^*_^2 + q^2x^k-x^*^2_
        =    (ρ + q^2) x^k-x^*_^2.

By taking the final expectation of the tower rule (<ref>) and apply it to the above inequality:

    [x^k+1-x^*_^2]    ≤    (ρ + q^2) [x^k-x^*_^2].

and the result is obtain by unrolling the last expression.

  (iii) 
For the case (iii) inequality (<ref>) takes the form:

    [x^k+1-x^*_^2 | x^k, _k]    ≤   x^k-x^*_^2 - 2(ω (2-ω)-q^2 ) f__k(x^k),

and by taking the middle expectation (see (<ref>)) we obtain:

    [[x^k+1-x^*_^2 | x^k, _k] | x^k]     ≤   x^k-x^*_^2 - 2(ω (2-ω)-q^2 ) f(x^k) 
       (<ref>)≤   x^k-x^*_^2 - (ω (2-ω)-q^2 )λ_min^+ x^k-x^*_^2 
        =    (1- (ω (2-ω)-q^2 )λ_min^+ ) x^k-x^*_^2.

By taking the final expectation of the tower rule (<ref>) to the above inequality:

    [x^k+1-x^*_^2]    ≤    (1- (ω (2-ω)-q^2 )λ_min^+ ) [x^k-x^*_^2].

and the result is obtain by unrolling the last expression.





CHAPTER: REVISITING RANDOMIZED GOSSIP ALGORITHMS





§ INTRODUCTION


Average consensus is a fundamental problem in distributed computing and multi-agent systems. It comes up in many real world applications such as coordination of autonomous agents, estimation, rumour spreading in social networks, PageRank and distributed data fusion on ad-hoc networks and decentralized optimization. Due to its great importance there is much classical <cit.> and recent   <cit.> work on the design of efficient algorithms/protocols for solving it.


In the average consensus (AC) problem we are given an undirected connected network =(,) with node set ={1,2,…,n} and edges . Each node i ∈ “knows” a private value c_i ∈. The goal of AC is for every node to compute the average of these private values, c̅1/n∑_i c_i, in a decentralized fashion. That is, the exchange of information can only occur between connected nodes (neighbors). 

Among the most attractive protocols for solving the average consensus problem are gossip algorithms. The development and design of gossip algorithms was studied extensively in the last decade. The seminal 2006 paper of Boyd et al. <cit.> motivated a fury of subsequent research and gossip algorithms now appear in many applications, including distributed data fusion in sensor networks <cit.>, load balancing <cit.> and clock synchronization <cit.>.  For a survey of selected relevant work prior to 2010, we refer the reader to the work of  Dimakis et al. <cit.>. For more recent results on randomized gossip algorithms we suggest <cit.>. See also  <cit.>. 



 §.§ Main contributions
 
In this chapter, we connect two areas of research which until now have remained remarkably disjoint in the literature: randomized iterative (projection) methods for solving linear systems and randomized gossip protocols for solving the average consensus. This connection enables us to make contributions by borrowing from each body of literature to the other and using it we propose a new framework for the design and analysis of novel efficient randomized gossip protocols.

The main contributions of our work include:



  * RandNLA. We show how classical randomized iterative methods for solving linear systems can be interpreted as gossip algorithms when applied to special systems encoding the underlying network and explain in detail their decentralized nature.
Through our general framework we recover a comprehensive array of well-known gossip protocols as special cases. In addition our approach allows for the development of novel block and dual variants of all of these methods. From a numerical analysis viewpoint our work is the first that explores in depth, the decentralized nature of randomized iterative methods for solving linear systems and proposes them as efficient methods for solving the average consensus problem (and its weighted variant). 


  * Weighted AC.  The methods presented in this chapter solve the more general weighted average consensus (Weighted AC) problem (Section <ref>) popular in the area of distributed cooperative spectrum sensing networks. The proposed protocols are the first randomized gossip algorithms that directly solve this problem with finite-time convergence rate analysis. In particular, we prove linear convergence of the proposed protocols and explain how we can obtain further acceleration using momentum. To the best of our knowledge, the existing decentralized protocols that solve the weighted average consensus problem show convergence but without convergence analysis. 


  * Acceleration. We present novel and provably accelerated randomized gossip protocols. In each step, of the proposed algorithms, all nodes of the network update their values using their own information but only a subset of them exchange messages. The protocols are inspired by the recently proposed accelerated variants of randomized Kaczmarz-type methods and use momentum terms on top of the sketch and project update rule (gossip communication) to obtain better theoretical and practical performance.
To the best of our knowledge, our accelerated protocols are the first randomized gossip algorithms that converge to a consensus with a provably accelerated linear rate without making any further assumptions on the structure of the network. Achieving an accelerated linear rate in this setting using randomized gossip protocols was an open problem.


  * Duality. We reveal a hidden duality of randomized gossip algorithms, with the dual iterative process maintaining variables attached to the edges of the network. We show how the randomized coordinate descent and randomized Newton methods work as edge-based dual randomized gossip algorithms.


  * Experiments. We corroborate our theoretical results with extensive experimental testing on typical wireless network topologies.
We numerically verify the linear convergence of the our protocols for solving the weighted AC problem. We explain the benefit of using block variants in the gossip protocols where more than two nodes update their values in each iteration. We explore the performance of the proposed provably accelerated gossip protocols and show that they significantly outperform the standard pairwise gossip algorithm and existing fast pairwise gossip protocols with momentum. An experiment showing the importance of over-relaxation in the gossip setting is also presented.


We believe that this work could potentially open up new avenues of research in the area of decentralized gossip protocols. 



 §.§ Structure of the chapter

This chapter is organized as follows. Section <ref> introduces the necessary background on basic randomized iterative methods for linear systems that will be used for the development of randomized gossip protocols.  Related work on the literature of linear system solvers, randomized gossip algorithms for averaging and gossip algorithms for consensus optimization is presented.   In Section <ref> the more general weighted average consensus problem is described and the connections between the two areas of research (randomized projection methods for linear systems and gossip algorithms) is established. In particular we explain how methods for solving linear systems can be interpreted as gossip algorithms when applied to special systems encoding the underlying network and elaborate in detail their distributed nature. Novel block gossip variants are also presented. In Section <ref> we describe and analyze fast and provably accelerated randomized gossip algorithms. In each step of these protocols all nodes of the network update their values but only a subset of them exchange their private values. Section <ref> describes dual randomized gossip algorithms that operate with values that are associated to the edges of the network and Section <ref> highlights further connections between methods for solving linear systems and gossip algorithms.
Numerical evaluation of the new gossip protocols is presented in Section <ref>. Finally, concluding remarks are given in Section <ref>.



 §.§ Notation

For convenience, a table of the most frequently used notation of this chapter is included in Section <ref>. 
In particular, with boldface upper-case letters denote matrices;  is the identity matrix. By · and ·_F we denote the Euclidean norm and  the Frobenius norm, respectively.  For a positive integer number n, we write [n]{1,2, … ,n}.  By  we denote the solution set of the linear system x=b, where ∈^m× n and b∈^m.

Vector x^k = (x^k_1,…,x^k_n) ∈^n represents the vector with the private values of the n nodes of the network at the k^th iteration while with x_i^k we denote the value of node i ∈ [n] at the k^th iteration. _i ⊆ denotes the set of nodes that are neighbors of node i ∈.  By () we denote the algebraic connectivity of graph . Throughout the chapter, x^* is the projection of x^0 onto  in the -norm. We write x^*=Π_,(x^0).

The complexity of all gossip protocols presented in this chapter is described by the spectrum of matrix 

    =^-1/2^⊤[] ^-1/2(<ref>)=^-1/2[]^-1/2,

where the expectation is taken over ∼.  
With λ_min^+ and λ_max we indicate the smallest nonzero and the largest eigenvalue of matrix , respectively. Recall that this is exactly the same matrix used in the previous chapters of this thesis.

Finally, with ∈^|| × n we define the incidence matrix and with ∈^n× n the Laplacian matrix of the network. Note that it holds that =^⊤. Further, with  we denote the degree matrix of the graph. That is, =Diag(d_1,d_2,…, d_n) ∈^n × n where d_i is the degree of node i ∈. 



§ BACKGROUND - TECHNICAL PRELIMINARIES



As we have already mentioned in this thesis, solving linear systems is a central problem in numerical linear algebra and plays an important role in computer science, control theory, scientific computing, optimization, computer vision, machine learning, and many other fields. With the advent of the age of big data, practitioners are looking for ways to solve linear systems of unprecedented sizes. In this large scale setting, randomized iterative methods are preferred mainly because of their cheap per iteration cost and because they can easily scale to extreme dimensions.



 §.§ Randomized iterative methods for linear systems



Recall that in the introduction of this thesis we presented the sketch and project method (<ref>) and we explained how this algorithm is identical to SGD (<ref>), SN (<ref>) and SPP (<ref>) for solving the stochastic quadratic optimization problem (<ref>). For the benefit of the reader and for the easier comparison to the gossip algorithms, a formal presentation of the Sketch and Project method for solving a consistent linear system x=b is presented in Algorithm <ref>. 



In this chapter, we are mostly interested in two special cases of the sketch and project framework— the randomized Kaczmarz (RK) method and its block variant, the randomized block Kaczmarz (RBK) method. In addition, in the following sections we present novel scaled and accelerated variants of these two selected cases and interpret their gossip nature. In particular, we focus on explaining how these methods can solve the average consensus problem and its more general version, the weighted average consensus (subsection <ref>).

Let  e_i ∈^m  be the i^th unit coordinate vector in ^m and let _:C be column submatrix of the m × m identity matrix with columns indexed by  C⊆ [m].  Then RK and RBK methods can be obtained as special cases of Algorithm <ref> as follows:


  * RK: Let = and _k=e_i, where i ∈ [m] is chosen independently at each iteration, with probability p_i>0. In this setup the update rule of Algorithm <ref> simplifies to  

    x^k+1=x^k - ω_i : x^k -b_i/_i :^2_i :^ ⊤  .


  * RBK: Let = and =_:C, where set C⊆ [m] is chosen independently at each iteration, with probability p_C≥ 0. In this setup the update rule of Algorithm <ref> simplifies to  

    x^k+1=x^k - ω_C:^⊤ (_C:_C:^⊤)^† (_C:x^k-b_C).



As we explained in Chapter <ref>, the sketch and project method, converges linearly to one particular solution of the linear system: the projection (on -norm) of the initial iterate x^0 onto the solution set of the linear system, x^*=Π_,(x^0). Therefore, the method solve the best approximation problem (<ref>).

The convergence performance of the Sketch and Project method (Algorithm <ref>) for solving the best approximation problem is described by the following theorem[For the proof of Theorem <ref> check Theorem <ref> in Section <ref> and recall that SGD and the sketch and project method are identical in this setting.].



Let assume exactness and let {x^k}_k=0^∞ be the iterates produced by the sketch and project method (Algorithm <ref>) with step-size ω∈ (0,2). Set, x^*=Π_,(x^0). Then,
 
    [x^k-x^*_^2]≤ρ^k x^0-x^*_^2,

where

    ρ 1 - ω (2-ω) λ_min^+ ∈ [0,1].



In other words, using standard arguments, from Theorem <ref> we observe that for a given ϵ>0 we have that:

    k ≥1/1-ρlog(1/ϵ)   ⇒  [x^k-x^*_^2]≤ϵx^0-x^*_^2




 §.§ Other related work







  
Gossip algorithms for average consensus The problem of average consensus has been extensively studied in the automatic control and signal processing literature for the past two decades <cit.>, and was first introduced for decentralized processing in the seminal work <cit.>. A clear connection between the rate of convergence and spectral characteristics of the underlying network topology over which message passing occurs was first established in <cit.> for pairwise randomized gossip algorithms. 

Motivated by network topologies with salient properties of wireless networks (e.g., nodes can communicate directly only with other nearby nodes), several methods were proposed to accelerate the convergence of gossip algorithms. For instance, <cit.> proposed averaging among a set of nodes forming a path in the network (this protocol can be seen as special case of our block variants in Section <ref>). Broadcast gossip algorithms have also been analyzed <cit.> where the nodes communicate with more than one of their neighbors by broadcasting their values.

While the gossip algorithms studied in <cit.> are all first-order (the update of x^k+1 only depends on x^k), a faster randomized pairwise gossip protocol was proposed in <cit.> which suggested to incorporate additional memory to accelerate convergence. The first analysis of this protocol was later proposed in <cit.> under strong condition. It is worth to mention that in the setting of deterministic gossip algorithms theoretical guarantees for accelerated convergence were obtained in <cit.>.
In Section <ref> we propose fast and provably accelerated randomized gossip algorithms with memory and compare them in more detail with the fast randomized algorithm proposed in <cit.>.



  
Gossip algorithms for multiagent consensus optimization. 
In the past decade there has been substantial interest in consensus-based mulitiagent optimization methods that use gossip updates in their update rule <cit.>. In multiagent consensus optimization setting , n agents or nodes, cooperate to solve an optimization problem. In particular, a local objective function f_i:^d → is associated with each node i ∈ [n] and the goal is for all nodes to solve the optimization problem 
    min_x ∈^d1/n∑_i=1^n f_i(x)
 by communicate only with their neighbors. In this setting gossip algorithms works in two steps by first executing some local computation followed by communication over the network <cit.>. Note that the average consensus problem with c_i as node i initial value can be case as a special case of the optimization problem (<ref>) when the function values are f_i(x)=(x- c_i)^2.

Recently there has been an increasing interest in applying mulitagent optimization  methods to solve convex and non-convex optimization problems arising in machine learning <cit.>. In this setting most consensus-based optimization methods make use of standard, first-order gossip, such as those described in <cit.>, and incorporating momentum into their updates to improve their practical performance. 



§ SKETCH AND PROJECT METHODS AS GOSSIP ALGORITHMS


In this section we show how by carefully choosing the linear system in the constraints of the best approximation problem (<ref>) and the combination of the parameters of the Sketch and Project method (Algorithm <ref>) we can design efficient randomized gossip algorithms. We show that the proposed protocols can actually solve the weighted average consensus problem, a more general version of the average consensus problem described in Section <ref>. In particular we focus, on a scaled variant of the RK method (<ref>) and on the RBK (<ref>) and understand the convergence rates of these methods in the consensus setting, their distributed nature and how they are connected with existing gossip protocols. 



 §.§ Weighted average consensus


In the weighted average consensus (Weighted AC) problem we are given an undirected connected network =(,) with node set ={1,2,…,n} and edges . Each node i ∈ holds a private value c_i ∈ and its weight w_i. The goal of this problem is for every node to compute the weighted average of the private values, 

    c̅∑_i=1^n w_i c_i/∑_i=1^n w_i,

in a distributed fashion. That is, the exchange of information can only occur between connected nodes (neighbors). 

Note that in the special case when the weights of all nodes are the same (w_i=r for all i ∈ [n]) the weighted average consensus is reduced to the standard average consensus problem. However, there are more special cases that could be interesting. For instance the weights can represent the degree of the nodes (w_i= d_i) or they can denote a probability vector and satisfy ∑_i^n w_i=1 with w_i>0. 

It can be easily shown that the weighted average consensus problem can be expressed as optimization problem as follows:

    min_x = (x_1,…, x_n) ∈^n1/2x-c_^2 
      subject to   x_1=x_2=…=x_n

where matrix =Diag(w_1, w_2,…, w_n) is a diagonal positive definite matrix (that is w_i>0 for all i ∈ [n]) and c=(c_1,…,c_n)^⊤ the vector with the initial values c_i of all nodes i ∈. The optimal solution of this problem is x^*_i=∑_i=1^n w_i c_i/∑_i=1^n w_i for all i ∈ [n] which is exactly the solution of the weighted average consensus.

As we have explained, the standard average consensus problem can be cast as a special case of weighted average consensus. However, in the situation when the nodes have access to global information related to the network, such as the size of the network (number of nodes n=||) and the sum of the weights ∑_i=1^n w_i, then any algorithm that solves the standard average consensus can be used to solve the weighted average consensus problem with the initial private values of the nodes changed from c_i to n w_i c_i/∑_i=1^n w_i. 

The weighted AC problem is popular in the area of distributed cooperative spectrum sensing networks <cit.>. In this setting, one of the goals is to develop decentralized protocols for solving the cooperative sensing problem in cognitive radio systems. The weights in this case represent a ratio related to the channel conditions of each node/agent <cit.>. The development of methods for solving the weighted AC problem is an active area of research (check <cit.> for a recent comparison of existing algorithms). However, to the best of our knowledge, existing analysis for the proposed algorithms focuses on showing convergence and not on providing convergence rates. Our framework allows us to obtain novel randomized gossip algorithms for solving the weighted AC problem. In addition, we provide a tight analysis of their convergence rates. In particular, we show convergence with a linear rate. See Section <ref> for an experiment confirming linear convergence of one of our proposed protocols on typical wireless network topologies.



 §.§ Gossip algorithms through sketch and project framework

We propose that randomized gossip algorithms should be viewed as special case of the Sketch and Project update to a particular problem of the form (<ref>). In particular, we let c=(c_1,…,c_n) be the initial values stored at the nodes of , and choose  and b so that the constraint x = b is equivalent to the requirement that x_i=x_j (the value stored at node i is equal to the value stored at node j) for all (i,j)∈.



 We say that  x = b is an “average consensus (AC) system” when x = b iff x_i = x_j for all (i,j) ∈.


It is easy to see that x = b is an AC system precisely when b=0 and the nullspace of  is {t 1_n : t∈}, where 1_n is the vector of all ones in ^n.  Hence,  has rank n-1. Moreover in the case that x^0=c, it is easy to see that for any AC system, the solution of (<ref>)  necessarily is x^* = c̅· 1_n — this is why we singled out AC systems. In this sense, any algorithm for solving (<ref>) will “find” the (weighted) average c̅. However, in order to obtain a distributed algorithm we need to make sure that only “local” (with respect to ) exchange of information is allowed.  



  
Choices of AC systems. It can be shown that many linear systems satisfy the above definition. 

For example, we can choose:


  * b=0 and =∈^|| × n to be the incidence matrix of  . That is, ∈^|| × n such that  x = 0 directly encodes the constraints x_i=x_j for (i,j)∈. That is, row  e=(i,j) ∈ of matrix  contains value 1 in column i, value -1 in column j (we use an arbitrary but fixed order of nodes defining each edge in order to fix ) and zeros elsewhere. 

  * A different choice is to pick b=0 and  ==^⊤, where  is the Laplacian matrix of network . 

Depending on what AC system is used, the sketch and project methods can have different interpretations as gossip protocols.  

In this work we mainly focus on the above two AC systems but we highlight that other choices are possible[Novel gossip algorithms can be proposed by using different AC systems to formulate the average consensus problem. For example one possibility is using the random walk normalized Laplacian ^rw=^-1. For the case of degree-regular networks the symmetric normalized Laplacian matrix ^sym=^-1/2^-1/2 can also being used.]. In Section <ref> for the provably accelerated gossip protocols we also use a normalized variant (_i:^2=1) of the Incidence matrix. 



  §.§.§ Standard form and mass preservation

Assume that x = b is an AC system. Note that since b=0, the update rule of Algorithm <ref> simplifies to:

    x^k+1= [ - ω^⊤_k] x^k=[ - ω_k ] x^k.

This is the standard form in which randomized gossip algorithms are written. What is new here is that the iteration matrix - ω_k has a specific structure which guarantees convergence to x^* under very weak assumptions (see Theorem <ref>). Note that if x^0=c, i.e., the starting primal iterate is the vector of private values (as should be expected from any gossip algorithm), then the iterates of (<ref>) enjoy a mass preservation property (the proof follows the fact that 1_n = 0):

If x =b is an AC system, then the iterates produced by (<ref>) satisfy:  1/n∑_i=1^nx_i^k=c̅, for all  k ≥ 0.
 
  Let fix k≥0 then,
 
    1/n 1_n^⊤ x^k+1=1/n 1_n^⊤ (  - ω^⊤_k ) x^k=1/n 1_n^⊤ x^k - 1/n 1_n^⊤ω^⊤_k  x^k 1_n = 0=1/n 1_n^⊤ x^k.

 
 


  §.§.§ ε-Averaging time


Let z^kx^k - x^*. The typical measure of convergence speed employed in the randomized gossip literature, called ε-averaging time and here denoted by T_ave(ε), represents the smallest time k for which  x^k gets within ε z^0 from x^*, with probability greater than 1-ε, uniformly over all starting values x^0=c. More formally, we define



    T_ave(ε)sup_c∈^ninf{k : (z^k > ε z^0 )≤ε}.


This definition differs slightly from the standard one in that we use z^0 instead of c.

Inequality (<ref>), together with Markov inequality, can be used  to give a bound on K(ε), formalized next:

 Assume x=b is an AC system. Let x^0=c and  be positive definite diagonal matrix. Assume exactness. Then for any 0<ε < 1 we have

    T_ave(ϵ) ≤ 3 log(1/ε)/log(1/ρ)≤ 3log(1/ϵ)/1-ρ,

where ρ is defined in (<ref>). 


See Section <ref>.


Note that under the assumptions of the above theorem, =^-1/2[] ^-1/2 only has a single zero eigenvalue, and hence λ_min^+ () is the second smallest eigenvalue of . Thus, ρ is the second largest eigenvalue of -. The bound on K(ε) appearing in Thm <ref> is often written with ρ replaced by λ_2( - ) <cit.>.

In the rest of this section we show how two special cases of the sketch and project framework, the randomized Kaczmarz (RK) and its block variant, randomized block Kaczmatz (RBK) work as gossip algorithms for the two AC systems described above. 



 §.§ Randomized Kaczmarz method as gossip algorithm

As we described before the sketch and project update rule of Algorithm <ref> has several parameters that should be chosen in advance by the user. These are the stepsize ω (relaxation parameter), the positive definite matrix  and the distribution  of the random matrices . 

In this section we focus on one particular special case of the sketch and project framework, a scaled/weighted variant of the randomized Kaczmarz method (RK) presented in (<ref>),  and we show how this method works as gossip algorithm when applied to special systems encoding the underlying network. In particular, the linear systems that we solve are the two AC systems described in the previous section where the matrix is either the incidence matrix  or the Laplacian matrix  of the network.

As we described in (<ref>) the standard RK method can be cast as special case of Algorithm <ref> by choosing = and =e_i. In this section, we focus on a small modification of this algorithm and we choose the positive definite matrix  to be  =Diag(w_1, w_2,…, w_n), the diagonal matrix of the weights presented in the weighted average consensus problem. 



  
Scaled RK: Let us have a general consistent linear system x= b with ∈^m × n. Let us also choose =Diag(w_1, w_2,…, w_n) and _k=e_i, where i∈ [m] is chosen in each iteration independently, with probability p_i>0. In this setup the update rule of Algorithm <ref> simplifies to  

    x^k+1=x^k - ωe_i^⊤ ( x^k -b) /e_i^⊤^-1^⊤ e_i^-1^ ⊤ e_i  =x^k - ω_i : x^k -b_i/^-1/2_i :^⊤^2_2^-1_i :^ ⊤.

This small modification of RK allow us to solve the more general weighted average consensus presented in Section <ref> (and at the same time the standard average consensus problem if = r where r ∈). To the best of our knowledge, even if this variant is special case of the general Sketch and project update, was never precisely presented before in any setting.



  §.§.§ AC system with incidence matrix 

Let us represent the constraints of problem (<ref>) as linear system with matrix =∈^|| × n be the Incidence matrix of the graph and right had side b=0. Lets also assume that the random matrices ∼ are unit coordinate vectors in ^m=^||. 

Let e=(i,j) ∈ then from the definition of matrix  we have that _e:^⊤=f_i-f_j where f_i, f_j are unit coordinate vectors in ^n. In addition, from the definition the diagonal positive definite matrix  we have that 

    ^-1/2_e :^⊤^2=^-1/2 (f_i-f_j)^2=1/w_1+1/w_j.


Thus in this case the update rule (<ref>) simplifies:

    x^k+1   b=0, =, (<ref>)=    x^k - ω_e : x^k/^-1/2_e :^⊤^2^-1_e :^ ⊤
       (<ref>)=   x^k - ω_e : x^k/1/w_1+1/w_j^-1_e :^ ⊤
       =   x^k-ω (x^k_i-x^k_j)/1/w_i+1/w_j(1/w_i f_i- 1/w_j  f_j ).


From (<ref>) it can be easily seen that only the values of coordinates i and j update their values. These coordinates correspond to the private values x^k_i and x^k_j of the nodes of the selected edge e=(i,j). In particular the values of x^k_i and x^k_j are updated as follows:


    x^k+1_i=(1-ωw_j/w_j+w_i) x^k_i + ω w_j/w_j+w_i x^k_j   and   x^k+1_j= ω w_i/w_j+w_i x^k_i + (1-ω w_i/w_j+w_i) x^k_j.



In the special case that =r where r∈ (we solve the standard average consensus problem) the update of the two nodes is simplified to

    x^k+1_i=(1-ω/2) x^k_i + ω/2 x^k_j   and   x^k+1_j= ω/2 x^k_i + (1-ω/2) x^k_j.

If we further select ω=1 then this becomes:

    x^k+1_i=x^k+1_j= x^k_i +x^k_j/2,

which is the update of the standard pairwise randomized gossip algorithm first presented and analyzed in <cit.>.




  §.§.§ AC system with Laplacian matrix 

The AC system takes the form x=0, where matrix ∈^n × n is the Laplacian matrix of the network. In this case, each row of the matrix corresponds to a node. Using the definition of the Laplacian, we have that _i:^⊤=d_if_i-∑_j ∈_if_j, where f_i, f_j are unit coordinate vectors in ^n and d_i is the degree of node i ∈. 

Thus, by letting =Diag(w_1, w_2,…, w_n) to be the diagonal matrix of the weights we obtain:


    ^-1/2_i :^⊤^2=^-1/2 (d_if_i-∑_j ∈_if_j) ^2=d_i^2/w_i+∑_j ∈_i1/w_j.


In this case, the update rule (<ref>) simplifies to:

    x^k+1   b=0, =, (<ref>)=    x^k - ω_i : x^k/^-1/2_i :^⊤^2_2^-1_i :^ ⊤
       (<ref>)=   x^k - ω_i : x^k/d_i^2/w_i+∑_j ∈_i1/w_j^-1_i :^ ⊤
       =   x^k-ω(d_i x^k_i- ∑_j∈_i x^k_j ) /d_i^2/w_i+∑_j ∈_i1/w_j(d_i/w_i f_i- ∑_j∈_i1/w_j  f_j ).


From (<ref>), it is clear that only coordinates {i}∪_i update their values. All the other coordinates remain unchanged. In particular, the value of the selected node i (coordinate i) is updated as follows:


    x^k+1_i = x^k_i-ω(d_i x^k_i- ∑_j∈_i x^k_j ) /d_i^2/w_i+∑_j ∈_i1/w_j d_i/w_i,

while the values of its neighbors j ∈_i are updated as:

    x^k+1_j =x^k_j+ω(d_i x^k_i- ∑_ℓ∈_i x^k_ℓ)  /d_i^2/w_i+∑_ℓ∈_i1/w_ℓ1/w_j .




Let  ω=1 and =r where r∈ then the selected nodes update their values as follows:

    x^k+1_i = ∑_ℓ∈{i ∪_i} x^k_ℓ/d_i+1  and  
    x^k+1_j =x^k_j+ (d_i x^k_i- ∑_ℓ∈_i x^k_ℓ)  /d_i^2+d_i.

That is, the selected node i updates its value to the average of its neighbors and itself, while all the nodes j ∈_i update their values using the current value of node i and all nodes in _i.


In a wireless network, to implement such an update, node i would first broadcast its current value to all of its neighbors. Then it would need to receive values from each neighbor to compute the sums over _i, after which node i would broadcast the sum to all neighbors (since there may be two neighbors j_1, j_2 ∈_i for which (j_1, j_2) ∉). In a wired network, using standard concepts from the MPI library, such an update rule could be implemented efficiently by defining a process group consisting of {i}∪_i, and performing one  in this group from i (containing x_i) followed by an  to sum x_ℓ over ℓ∈_i. Note that the terms involving diagonal entries of  and the degrees d_i could be sent once, cached, and reused throughout the algorithm execution to reduce communication overhead.



  §.§.§ Details on complexity results

Recall that the convergence rate of the sketch and project method (Algorithm <ref>) is equivalent to:

    ρ 1 - ω (2-ω) λ_min^+(),

where ω∈ (0,2) and = ^-1/2^⊤[] ^-1/2 (from Theorem <ref>).  In this subsection we explain how the convergence rate of the scaled RK method (<ref>) is modified for different choices of the main parameters of the method.

Let us choose ω=1 (no over-relaxation). In this case, the rate is simplified to ρ=1 - λ_min^+. 

Note that the different ways of modeling the problem (AC system) and the selection of the main parameters (weight matrix  and distribution ) determine the convergence rate of the method through the spectrum of matrix .

Recall that in the k^th iterate of the scaled RK method (<ref>) a random vector _k=e_i is chosen with probability p_i>0. 
For convenience, let us choose[Similar probabilities have been chosen in <cit.> for the convergence of the standard RK method (=). The distribution  of the matrices  used in equation (<ref>) is common in the area of randomized iterative methods for linear systems and is used to simplify the analysis and the expressions of the convergence rates. For more choices of distributions we refer the interested reader to <cit.>. It is worth to mention that the probability distribution that optimizes the convergence rate of the RK and other projection methods can be expressed as the solution to a convex semidefinite program <cit.>.]:

    p_i= ^-1/2_i:^⊤^2/^-1/2^⊤^2_F .

Then we have that:

    [ ]    =   [  (^⊤^-1^⊤)^†^⊤] 
       =   ∑_i=1^m p_i e_i e_i^⊤/e_i^⊤^-1^⊤ e_i=∑_i=1^m p_i e_i e_i^⊤/_i:^⊤_^-1^2=∑_i=1^m p_i e_i e_i^⊤/^-1/2_i:^⊤^2
       (<ref>)=   ∑_i=1^m e_i e_i^⊤/^-1/2^⊤^2_F=1/^-1/2^⊤^2_F,

and

    (<ref>), (<ref>)=^-1/2^⊤^-1/2/^-1/2^⊤^2_F.




  
Incidence Matrix: Let us choose the AC system to be the one with the incidence matrix =. Then ^-1/2^⊤^2_F= ∑_i=1^n d_i/b_i and we obtain 

    =, (<ref>)=^-1/2^-1/2/^-1/2^⊤^2_F=^-1/2^-1/2/∑_i=1^n d_i/_ii.


If we further have =, then =^-1/2^-1/2/n and the convergence rate simplifies to:

    ρ=1-λ_min^+(^-1/2^-1/2)/n=1-λ_min^+(^sym)/n.


If =r where r∈ (solve the standard average consensus problem), then =/^2_F= /∑_i=1^n d_i=/2m and the convergence rate simplifies to 

    ρ=1-λ_min^+()/2m=1-()/2m,

which is exactly the same convergence rate of the pairwise gossip algorithm presented in <cit.>. This was expected, since the gossip protocol in this case works exactly the same as the one proposed in <cit.>, see equation (<ref>).



  
Laplacian Matrix: If we choose to formulate the AC system using the Laplacian matrix , that is =, then ^-1/2^⊤^2_F= ∑_i=1^n d_i(d_i+1)/_ii and we have:

    =, (<ref>)=^-1/2^⊤^-1/2/∑_i=1^n d_i(d_i+1)/_ii.


If =, then the convergence rate simplifies to:

    ρ=1-λ_min^+(^-1/2^⊤^-1/2)/∑_i=1^n (d_i+1)=1-λ_min^+(^-1/2^2 ^-1/2)n+ ∑_i=1^n d_i∑_i=1^n d_i=2m=1-λ_min^+(^-1/2^2 ^-1/2)n+ 2m.


If =r, where r∈, then =^2/^2_F= ^2/∑_i=1^n d_i(d_i+1) and the convergence rate simplifies to 
    ρ=1-λ_min^+(^2)/∑_i=1^n d_i(d_i+1)=1-()^2/∑_i=1^n d_i(d_i+1).




 §.§ Block gossip algorithms


Up to this point we focused on the basic connections between the convergence analysis of the sketch and project methods and the literature of randomized gossip algorithms. We show how specific variants of the randomized Kaczmarz method (RK) can be interpreted as gossip algorithms for solving the weighted and standard average consensus problems. 

In this part we extend the previously described methods to their block variants related to randomized block Kaczmarz (RBK) method (<ref>). In particular, in each step of Algorithm <ref>, the random matrix  is selected to be a random column submatrix of the m × m identity matrix corresponding to columns indexed by a random subset C  ⊆ [m]. That is, =_:C, where a set C⊆ [m] is chosen in each iteration independently, with probability p_C≥ 0 (see equation (<ref>)). Note that in the special case that set C is a singleton with probability 1 the algorithm is simply the randomized Kaczmarz method of the previous section.

To keep things simple, we assume that = (standard average consensus, without weights) and choose the stepsize ω=1. In the next section, we will describe gossip algorithms with heavy ball momentum and explain in detail how the gossip interpretation of RBK change in the more general case of ω∈ (0,2).

Similar to the previous subsections, we formulate the consensus problem using either = or = as the matrix in the AC system.
In this setup, the iterative process of Algorithm <ref> has the form:

    x^k+1   (<ref>),(<ref>)=    x^k - ^ ⊤_:C(_:C^⊤^⊤_:C)^†_:C^⊤ x^k= x^k - _C:^⊤ (_C:_C:^⊤)^†_C:x^k,

which, as explained in the introduction, can be equivalently written as: 

    x^k+1=x ∈^nargmin{x-x^k^2  : _:C^⊤ x=0}.
 
Essentially in each step of this method the next iterate is evaluated to be the projection of the current iterate x^k onto the solution set of a row subsystem of x=0. 



  
AC system with Incidence Matrix: In the case that = the selected rows correspond to a random subset C ⊆ of selected edges. While (<ref>) may seem to be a complicated algebraic (resp. variational) characterization of the method, due to our choice of = we have the following result which gives a natural interpretation of RBK as a gossip algorithm (see also Figure <ref>).



Consider the AC system with the constraints being expressed using the Incidence matrix .  
Then each iteration of RBK (Algorithm (<ref>)) works as gossip algorithm as follows:


  * Select a random set of edges C ⊆, 

  * Form subgraph _k of  from the selected edges 

  * For each connected component of _k, replace node values with their average.



See Section <ref>.

Using the convergence result of general Theorem <ref> and the form of matrix  (recall that in this case we assume =, =_:C∼ and ω=1), we obtain the following complexity for the algorithm:

    [x^k-x^*^2]≤[1 - λ_min^+ ( [_C:^⊤ (_C:_C:^⊤)^†_C:] ) ]^k x^0-x^*^2.

For more details on the above convergence rate of randomized block Kaczmarz method with meaningfully bounds on the rate in a more general setting we suggest the papers <cit.>.

There is a very closed relationship between the gossip interpretation of RBK explained in Theorem <ref> and several existing randomized gossip algorithms that in each step update the values of more than two nodes. 
For example the path averaging algorithm porposed in <cit.> is a special case of RBK, when set C is restricted to correspond to a path of vertices. That is, in path averaging, in each iteration a path of nodes is selected and the nodes that belong to it update their values to their exact average.
A different example is the recently proposed clique gossiping <cit.> where the network is already divided into cliques and through a random procedure a clique is activated and the nodes of it update their values to their exact average.
In <cit.> a synchronous variant of gossip algorithm is presented where in each step multiple node pairs communicate exactly at the same time with the restriction that these simultaneously active node pairs are disjoint.

It is easy to see that all of the above algorithms can be cast as special cases of RBK if the distribution  of the random matrices is chosen carefully to be over random matrices  (column sub-matrices of Identity) that update specific set of edges in each iteration. As a result our general convergence analysis can recover the complexity results proposed in the above works.

Finally, as we mentioned, in the special case in which set C is always a singleton, Algorithm (<ref>) reduces to the standard randomized Kaczmarz method. This means that only a random edge is selected in each iteration and the nodes incident with this edge replace their local values with their average. This is the pairwise gossip algorithm of Boyd er al. <cit.> presented in equation (<ref>). Theorem <ref> extends this interpretation to the case of the RBK method. 





  
AC system with Laplacian Matrix:
For this choice of AC system the update is more complicated. To simplify the way that the block variant work as gossip we make an extra assumption. We assume that the selected rows of the constraint _:C^⊤ x=0 in update (<ref>) have no-zero elements at different coordinates. This allows to have a direct extension of the serial variant presented in Remark <ref>. Thus, in this setup, the RBK update rule (<ref>) works as gossip algorithm as follows:



  * |C| nodes are activated (with restriction that the nodes are not neighbors and they do not share common neighbors)

  * For each node i ∈ C we have the following update:

    x^k+1_i = ∑_ℓ∈{i ∪_i} x^k_ℓ/d_i+1  and  
    x^k+1_j =x^k_j+(d_i x^k_i- ∑_ℓ∈_i x^k_ℓ)  /d_i^2+d_i .


The above update rule can be seen as a parallel variant of update (<ref>). Similar to the convergence in the case of Incidence matrix, the RBK for solving the AC system with a Laplacian matrix converges to x^* with the following rate (using result of Theorem <ref>):

    [x^k-x^*^2]≤[1 - λ_min^+ ( [_C:^⊤ (_C:_C:^⊤)^†_C:] ) ]^k x^0-x^*^2.




§ FASTER AND PROVABLY ACCELERATED RANDOMIZED GOSSIP ALGORITHMS



The main goal in the design of gossip protocols is for the computation and communication to be done as quickly and efficiently as possible. In this section, our focus is precisely this. We design randomized gossip protocols which converge to consensus fast with provable accelerated linear rates. To the best of our knowledge, the proposed protocols are the first randomized gossip algorithms that converge to consensus with an accelerated linear rate. 

In particular, we present novel protocols for solving the average consensus problem where in each step all nodes of the network update their values but only a subset of them exchange their private values. The protocols are inspired from the recently developed accelerated variants of randomized Kaczmarz-type methods for solving consistent linear systems where the addition of momentum terms on top of the sketch and project update rule provides better theoretical and practical performance.

In the area of optimization algorithms, there are two popular ways to accelerate an algorithm using momentum. The first one is using the Polyak's heavy ball momentum <cit.> and the second one is using the theoretically much better understood momentum introduced by Nesterov <cit.>.  Both momentum approaches have been recently proposed and analyzed to improve the performance of randomized iterative methods for solving linear systems.

To simplify the presentation, the accelerated algorithms and their convergence rates are presented for solving the standard average consensus problem (=). Using a similar approach as in the previous section, the update rules and the convergence rates can be easily modified to solve the more general weighted average consensus problem. For the protocols in this section we use the incidence matrix = or its normalized variant to formulate the AC system.



 §.§ Gossip algorithms with heavy ball momentum


In Chapter <ref> of this thesis we have analyzed heavy ball momentum variants of several algorithms for solving the stochastic optimization problem (<ref>) and as we explained the best approximation problem (<ref>). In this section we revisit Algorithm <ref> of Chapter <ref> and we focus on its sketch and project viewpoint. In particular, we explain how it works as gossip algorithm when is applied to the AC system with the incidence matrix.



  §.§.§ Sketch and project with heavy ball momentum


The sketch and project method with heavy ball momentum is formally presented in the following algorithm. 



Using, = and the same choice of distribution  as in equations (<ref>) and (<ref>) we can now obtain momentum variants of the RK and RBK as special case of the above algorithm as follows: 


  * RK with momentum (mRK): 


    x^k+1=x^k -ω_i : x^k -b_i/_i :^2_i :^ ⊤ + β(x^k - x^k-1).


  * RBK with momentum (mRBK):

    x^k+1=x^k -ω_C:^⊤ (_C:_C:^⊤)^† (_C:x^k-b_C) + β(x^k - x^k-1).



For more details on the convergence analysis of Algorithm <ref> see Section <ref> and recall that in our setting the sketch and project update rule is identical to the SGD (Chapter <ref>). As a result Algorithm <ref> is identical to Algorithm <ref> (mSGD/mSN/mSPP).

Having presented Algorithm <ref>, let us now describe its behavior as a randomized gossip protocol when applied to the AC system x=0 with =∈ || × n (incidence matrix of the network).  

Note that since b=0 (from the AC system definition), the update rule (<ref>) of Algorithm <ref> is simplified to (and by having =):

    x^k+1=[ - ω^⊤_k (_k^⊤^⊤_k)^†_k^⊤] x^k + β(x^k - x^k-1).


In the rest of this section we focus on  two special cases of (<ref>): RK with heavy ball momentum (equation (<ref>) with b_i=0) and RBK with heavy ball momentum (equation (<ref>) with b_C=0).



  §.§.§ Randomized Kaczmarz gossip with heavy ball momentum

As we have seen in previous section when the standard RK is applied to solve the AC system x=0, one can recover the famous pairwise gossip algorithm <cit.>.  Algorithm <ref> describes how a relaxed variant of randomized Kaczmarz with heavy ball momentum (0<ω < 2 and 0 ≤β <1) behaves as a gossip algorithm. See also Figure (<ref>) for a graphical illustration of the method.




In the special case that β=0 (zero momentum) only the two nodes of edge e=(i,j) update their values. In this case the two selected nodes do not update their values to their exact average but to a convex combination that depends on the stepsize ω∈ (0,2). To obtain the pairwise gossip algorithm of <cit.>, one should further choose ω=1.


Distributed Nature of the Algorithm: Here we highlight a few ways to implement mRK in a distributed fashion.


  * Pairwise broadcast gossip:  In this protocol each node i ∈ of the network  has a clock that ticks at the times of a rate 1 Poisson process. The inter-tick times are exponentially distributed, independent across nodes, and independent across time. This is equivalent to a global clock ticking at a rate n Poisson process which wakes up an edge of the network at random. In particular, in this implementation mRK works as follows:  In the k^th iteration (time slot) the clock of node i ticks and node i randomly contact one of its neighbors and simultaneously broadcast a signal to inform the nodes of the whole network that is updating (this signal does not contain any private information of node i). The two nodes (i,j) share their information and update their private values following the update rule of Algorithm <ref> while all the other nodes update their values using their own information. In each iteration only one pair of nodes exchange their private values. 
 

  * Synchronous pairwise gossip: In this protocol a single global clock is available to all nodes. The time is assumed to be slotted commonly across nodes and in each time slot only a pair of nodes of the network is randomly activated and exchange their information following the update rule of Algorithm <ref>.  The remaining not activated nodes update their values using their own last two private values.  Note that this implementation of mRK comes with the disadvantage that it requires a central entity which in each step requires to choose the activated pair of nodes[We speculate that a completely distributed synchronous gossip algorithm that finds pair of nodes in a distributed manner without any additional computational burden can be design following the same procedure proposed in Section III.C of  <cit.>.].


  * Asynchronous pairwise gossip with common counter: 
Note that the update rule of the selected pair of nodes (i,j) in Algorithm <ref> can be rewritten as follows:

    x_i^k+1= x_i^k + β (x_i^k - x_i^k-1) + ω/2 (x_j^k -x_i^k),


    x_j^k+1= x_j^k + β (x_j^k - x_j^k-1) + ω/2 (x_i^k -x_j^k).

In particular observe that the first part of the above expressions x_i^k + β (x_i^k - x_i^k-1) (for the case of node i) is exactly the same with the update rule of the non activate nodes at k^th iterate (check step 5 of Algorithm <ref>) . Thus, if we assume that all nodes share a common counter that keeps track of the current iteration count and that each node i ∈ remembers the iteration counter k_i of when it was last activated, then step 5 of Algorithm <ref> takes the form:


  * x_i^k+1= i_k [x_i^k + β (x_i^k - x_i^k-1) ]+ ω/2 (x_j^k -x_i^k),

  * x_j^k+1= j_k [x_j^k + β (x_j^k - x_j^k-1) ] + ω/2 (x_i^k -x_j^k),

  * k_i = k_j =k+1,

  * Any other node ℓ: x_ℓ^k+1=x_ℓ^k,

where i_k=k-k_i (j_k=k-k_j) denotes the number of iterations between the current iterate and the last time that the i^th (j^th) node is activated. In this implementation only a pair of nodes communicate and update their values in each iteration (thus the justification of asynchronous), however it requires the nodes to share a common counter that keeps track the current iteration count in order to be able to compute the value of i_k=k-k_i.








  §.§.§ Connections with existing fast randomized gossip algorithms



In the randomized gossip literature there is one particular method closely related to our approach. It was first proposed in <cit.> and its analysis under strong conditions was presented in <cit.>. In this work local memory is exploited by installing shift registers at each agent. In particular we are interested in the case of two registers where the first stores the agent's current value and the second the agent's value before the latest update. The algorithm can be described as follows. Suppose that edge e=(i,j) is chosen at time k. Then,


  * Node i: x_i^k+1= ω(x_i^k+x_j^k/2)+(1-ω)x_i^k-1,

  * Node j: x_i^k+1=  ω(x_i^k+x_j^k/2)+(1-ω)x_j^k-1,

  * Any other node ℓ: x_ℓ^k+1=x_ℓ^k,

where ω∈ [1,2). The method was analyzed in <cit.> under a strong assumption on the probabilities of choosing the pair of nodes, that as the authors mentioned, is unrealistic in practical scenarios, and for networks like the random geometric graphs. At this point we should highlight that the results presented in Chapter <ref> hold for essentially any distribution  [The only restriction is the exactness condition to be satisfied. See Theorem <ref>.] and as a result in the proposed gossip variants with heavy ball momentum such problem cannot occur.

Note that, in the special case that we choose β=ω-1 in the update rule of Algorithm <ref> is simplified to:


  * Node i: x_i^k+1= ω(x_i^k+x_j^k/2)+(1-ω)x_i^k-1,

  * Node j: x_i^k+1=  ω(x_i^k+x_j^k/2)+(1-ω)x_j^k-1,

  * Any other node ℓ: x_ℓ^k+1=ω x_ℓ^k+(1-ω)x_ℓ^k-1.


Recall that in order to apply Theorem <ref>, we need to assume that 0< ω < 2 and β=ω-1 ≥ 0 which also means that ω∈ [1,2). Thus for ω∈ [1,2) and momentum parameter β=ω-1 it is easy to see that our approach is very similar to the shift-register algorithm. Both methods update the selected pair of nodes in the same way. However, in Algorithm <ref> the not selected nodes of the network do not remain idle but instead update their values using their own previous information.

By defining the momentum matrix =Diag(β_1,β_2,…, β_n), the above closely related algorithms can be expressed, in vector form, as:

    x^k+1 = x^k - ω/2 (x_i^k-x_j^k)(e_i-e_j) +  (x^k-x^k-1).


In particular, in mRK every diagonal element of matrix  is equal to ω-1, while in the algorithm of <cit.> all the diagonal elements are zeros except the two values that correspond to nodes i and j that are equal to β_i=β_j=ω-1.


The shift register algorithm of <cit.> and Algorithm <ref> of this work can be seen as the two limit cases of the update rule (<ref>). As we mentioned, the shift register method <cit.> uses only two non-zero diagonal elements in , while our method has a full diagonal. We believe that further methods can be developed in the future by exploring the cases where more than two but not all elements of the diagonal matrix  are non-zero. It might be possible to obtain better convergence if one carefully chooses these values based on the network topology. We leave this as an open problem for future research.




  §.§.§ Randomized block Kaczmarz gossip with heavy ball momentum

Recall that Theorem <ref> explains how RBK (with no momentum and no relaxation) can be interpreted as a gossip algorithm. In this subsection by using this result we explain how relaxed RBK with momentum works. Note that the update rule of RBK with momentum can be rewritten as follows:

    x^k+1(<ref>),(<ref>)=ω(- _C:^⊤ (_C:_C:^⊤)^†_C:) x^k+(1-ω)x^k +β(x^k-x^k-1),

and recall that x^k+1 =( - _C:^⊤ (_C:_C:^⊤)^†_C:) x^k is the update rule of the standard RBK  (<ref>).

Thus, in analogy to the standard RBK, in the k^th step, a random set of edges is selected and q ≤ n connected components are formed as a result. This includes the connected components that belong to both sub-graph _k and also the singleton connected components (nodes outside the _k). Let us define the set of the nodes that belong in the r ∈ [q] connected component at the k^th step _r^k, such that = ∪_r∈ [q]_r^k and ||=∑_r=1^q |_r^k| for any k>0. 

Using the update rule (<ref>), Algorithm <ref> shows how mRBK is updating the private values of the nodes of the network (see also Figure <ref> for the graphical interpretation).



Note that in the update rule of mRBK the nodes that are not attached to a selected edge (do not belong in the sub-graph _k) update their values via x_ℓ^k+1=x_ℓ^k+β (x_ℓ^k - x_ℓ^k-1). By considering these nodes as singleton connected components their update rule is exactly the same with the nodes of sub-graph _k. This is easy to see as follows:

    x_ℓ^k+1   (<ref>)=   ω∑_j ∈_r^k x_j^k/|_r^k| +(1-ω)x_ℓ^k+β (x_ℓ^k-x_ℓ^k-1)
       |_r^k|=1=   ω x_ℓ^k +(1-ω)x_ℓ^k+β (x_ℓ^k-x_ℓ^k-1)
       =    x_ℓ^k+β (x_ℓ^k - x_ℓ^k-1).



In the special case that only one edge is selected in each iteration (_k ∈^m × 1)  the update rule of mRBK is simplified to the update rule of mRK. In this case the sub-graph _k is the pair of the two selected edges.  



In previous section we explained how several existing gossip protocols for solving the average consensus problem are special cases of the RBK (Theorem <ref>). For example two gossip algorithms that can be cast as special cases of the standard RBK are the path averaging proposed in <cit.> and the clique gossiping <cit.>. In path averaging, in each iteration a path of nodes is selected and its nodes update their values to their exact average (ω=1). In clique gossiping, the network is already divided into cliques and through a random procedure a clique is activated and the nodes of it update their values to their exact average (ω=1). Since mRBK contains the standard RBK as a special case (when β=0), we expect that these special protocols can also be accelerated with the addition of momentum parameter β∈ (0,1).






  §.§.§ Mass preservation

One of the key properties of some of the most efficient randomized gossip algorithms is mass preservation. That is, the sum (and as a result the average) of the private values of the nodes remains fixed during the iterative procedure (∑_i=1^nx_i^k=∑_i=1^nx_i^0,   ∀ k ≥ 1). The original pairwise gossip algorithm proposed in <cit.> satisfied the mass preservation property, while exisiting fast gossip algorithms <cit.>  preserving a scaled sum.  
In this subsection we show that mRK and mRBK gossip protocols presented above satisfy the mass preservation property. In particular, we prove mass preservation for the case of the block randomized gossip protocol (Algorithm <ref>) with momentum. This is sufficient since the randomized Kaczmarz gossip with momentum (mRK), Algorithm <ref> can be cast as special case.


Assume that x^0=x^1=c. That is, the two registers of each node have the same initial value.  Then for the Algorithms <ref> and <ref> we have ∑_i=1^nx_i^k=∑_i=1^nc_i for any k≥ 0 and as a result, 1/n∑_i=1^nx_i^k=c̅. 


We prove the result for the more general Algorithm <ref>. Assume that in the k^th step of the method q connected components are formed.  Let the set of the nodes of each connected component be _r^k so that = ∪_r={1,2,...q}_r^k and ||=∑_r=1^q |_r^k| for any k>0.  Thus:

    ∑_i=1^nx_i^k+1=∑_i ∈_1^k x_i^k+1 +… + ∑_i ∈_q^k x_i^k+1.

Let us first focus, without loss of generality, on  connected component r ∈ [q] and simplify the expression for the sum of its nodes:

    ∑_i∈_r^k x_i^k+1   (<ref>)=   ∑_i ∈_r^kω∑_j ∈_r^k x_j^k/|_r^k| +   (1-ω) ∑_i ∈_r^k x_i^k  +β∑_i ∈_r^k  (x_i^k-x_i^k-1)
       =   |_r^k| ω∑_j ∈_r^k x_j^k/|_r^k|+ (1-ω) ∑_i ∈_r^k x_i^k 
     +β∑_i ∈_r^k  (x_i^k-x_i^k-1)
       =   (1+β) ∑_i ∈_r^kx_i^k-β∑_i ∈_r^kx_i^k-1.

 By substituting this for all r ∈ [q] into the right hand side of (<ref>) and from the fact that = ∪_r∈ [q]_r^k, we obtain:

    ∑_i=1^nx_i^k+1= (1+β) ∑_i=1^nx_i^k-β∑_i=1^n x_i^k-1.

Since x^0=x^1, we have ∑_i=1^nx_i^0=∑_i=1^nx_i^1, and as a result ∑_i=1^nx_i^k = ∑_i=1^nx_i^0 for all k ≥ 0.



 §.§ Provably accelerated randomized gossip algorithms



In this subsection we focus on one specific case of the Sketch and Project framework, the RK method (<ref>). We present two accelerated variants of RK where the Nesterov's momentum is used, for solving consistent linear systems and we describe their theoretical convergence results. Based on these methods we propose two provably accelerated gossip protocols, along with some remarks on their implementation.



  §.§.§ Accelerated Kaczmarz methods using Nesterov's momentum


There are two different but very similar ways to provably accelerate the randomized Kaczmarz method using Nesterov's acceleration. The first paper that proves asymptotic convergence with an accelerated linear rate is <cit.>. The proof technique is similar to the framework developed by Nesterov in <cit.> for the acceleration of coordinate descent methods.  In <cit.> a modified version for the selection of the parameters was proposed and a non-asymptotic accelerated linear rate was established. In Algorithm <ref>, pseudocode of the Accelerated Kaczmarz method (AccRK) is presented where both variants can be cast as special cases, by choosing the parameters with the correct way. 


There are two options for selecting the parameters of the AccRK for solving consistent linear systems with normalized matrices, which we describe next. 


  * From <cit.>: 
Choose λ∈ [0,λ_min^+(^⊤)] and set γ_-1=0.
Generate the sequence {γ_k: k=0,1,…, K+1} by choosing γ_k to be the largest root of 
    γ_k^2-γ_k/m=(1-γ_k/λm)γ_k-1^2,
 and generate the sequences {α	_k :  k=0,1,…,K+1} and {β_k :  k=0,1,…,K+1} by setting 
    α_k=m-γ_kλ/γ_k(m^2-λ),   β_k=1-γ_k λ/m.


  * From <cit.>: Let 

    ν= max_u ∈^⊤ u^⊤[∑_i=1^m  _i:^⊤_i: (^⊤)^†_i:^⊤_i:]u / u^⊤^⊤/mu .

Choose the three sequences to be fixed constants as follows:
β_k =β = 1-√(λ_min^+()/ν),  γ_k=γ = √(1/λ_min^+() ν),   α_k=α =  1/1+γν∈ (0,1) where =^⊤/m.



  §.§.§ Theoretical guarantees of AccRK

The two variants (Option 1 and Option 2) of AccRK are closely related, however their convergence analyses are different. Below we present the theoretical guarantees of the two options as presented in <cit.> and <cit.>.



Let {x^k}_k=0^∞ be the sequence of random iterates produced by Algorithm <ref> with the Option 1 for the parameters. Let  be normalized matrix and let λ∈ [0,λ_min^+(^⊤)]. Set σ_1=1+√(λ)/2m and σ_2=1-√(λ)/2m. Then for any k≥ 0 we have that:

    [x^k-x^*^2 ] ≤4 λ/(σ_1^k-σ^k_2)^2x^0-x^*^2_(^⊤)^†.




Note that as k→∞, we have that σ^k_2→ 0. This means that the decrease of the right hand side is governed mainly by the behavior of the term σ_1 in the denominator and as a result the method converge asymptotically with a decrease factor per iteration:
σ_1^-2=(1+√(λ)/2m)^-2≈ 1-√(λ)/m.
That is, as k→∞:

    [x^k-x^*^2 ] ≤(1- √(λ)/m )^k 4 λx^0-x^*^2_(^⊤)^†



Thus, by choosing λ=λ_min^+ and for the case that λ_min^+ is small, Algorithm <ref> will have significantly faster convergence rate than RK. Note that the above convergence results hold only for normalized matrices ∈^m × n, that is matrices that have _i:=1 for any i ∈ m. 

Using Corollary <ref>, Algorithm <ref> with the first choice of the parameters converges linearly with rate (1- √(λ)/m ). That is, it requires O ( m/ √(λ)log(1/ϵ) ) iterations to obtain accuracy [x^k-x^*^2 ]≤ϵ 4 λx^0-x^*^2_(^⊤)^†.



Let =^⊤/m and let assume exactness[Note that in this setting =, which means that =[], and the exactness assumption takes the form Null() =  Null().]. Let {x^k,y^k,v^k} be the iterates  of Algorithm <ref> with the Option 2 for the parameters.  Then 

    Ψ^k ≤(1-√(λ_min^+()/ν))^k Ψ^0,

where Ψ^k =[ v^k - x^*_^†^2+  1/μx^k-x^*^2 ].


The above result implies that Algorithm <ref> converges linearly with rate 1-√(λ_min^+()/ν), which translates to a total of (√(ν/λ_min^+())log(1/ϵ)) iterations to bring the quantity Ψ^k below ϵ>0. It can be shown that 1 ≤ν≤ 1/λ_min^+(), (Lemma 2 in <cit.>) where ν is as defined in (<ref>). Thus,
√(1/λ_min^+())≤√(ν/λ_min^+())≤1/λ_min^+(),
which means that the rate of AccRK (Option 2) is always better than that of the RK with unit stepsize which is equal to (1/λ_min^+()log(1/ϵ)) (see Theorem <ref>). 

In <cit.>, Theorem <ref> has been proposed for solving more general consistent linear systems (the matrix  of the system is not assumed to be normalized). In this case =[] and the parameter ν is slightly more complicated than the one of equation (<ref>). We refer the interested reader to <cit.> for more details.



  
Comparison of the convergence rates:
Before describe the distributed nature of the AccRK and explain how it can be interpreted as a gossip algorithm, let us compare the convergence rates of the two options of the parameters for the case of general normalized consistent linear systems (_i:=1 for any i ∈ [m]).

Using Theorems <ref> and <ref>, it is clear that the iteration complexity of AccRK is 

    O (m/√(λ)log(1/ϵ) )λ=λ_min^+(^⊤)= O ( m/√(λ_min^+(^⊤))log(1/ϵ) ),
 and 

    (√(ν m/λ_min^+(^⊤))log(1/ϵ)),
 for the Option 1 and Option 2 for the parameters, respectively.

In the following derivation we compare the iteration complexity of the two methods.



Let matrices ∈^n × n  and _i ∈^n × n where i ∈ [m] be positive semidefinite, and satisfying ∑_i=1^m _i =. Then 

    ∑_i=1^m _i  ^†_i ≼.




From the definition of the matrices it holds that _i ≼ for any i ∈ [m]. Using the properties of Moore-Penrose pseudoinverse,  this implies that

    _i^†≽^†.

Therefore

    _i= _i _i^†_i (<ref>)≽_i ^†_i.
 
From the definition of the matrices by taking the sum over all i ∈ [m] we obtain:

    = ∑_i=1^m _i (<ref>)≽∑_i=1^m _i ^†_i,

which completes the proof.


Let us now choose _i = _i:^⊤_i: and =^⊤. Note that from their definition the matrices are positive semidefinite and satisfy ∑_i=1^m _i:^⊤_i:= ^⊤.
Using Lemma <ref> it is clear that:

    ∑_i=1^m  _i:^⊤_i: (^⊤)^†_i:^⊤_i:≼^⊤,

or in other words, for any vector v ∉ Null() we set the inequality

    v^⊤[∑_i=1^m  _i:^⊤_i: (^⊤)^†_i:^⊤_i:] v/v^⊤ [^⊤] v≤ 1.

Multiplying both sides by m, we set:

    v^⊤[∑_i=1^m  _i:^⊤_i: (^⊤)^†_i:^⊤_i:] v/v^⊤ [^⊤/m] v≤ m.


Using the above derivation, it is clear from the definition of the parameter ν (<ref>), that ν≤ m.
By combining our finding with the bounds already obtained in  <cit.> for the parameter ν, we have that:

    1 ≤ν≤min{ m, 1/λ_min^+()}.

Thus, by comparing the two iteration complexities of equations (<ref>) and (<ref>) it is clear that Option 2 for the parameters <cit.> is always faster in theory than Option 1 <cit.>. To the best of our knowledge, such comparison of the two choices of the parameters for the AccRK was never presented before.



  §.§.§ Accelerated randomized gossip algorithms


Having presented the complexity analysis guarantees of AccRK for solving consistent linear systems with normalized matrices, let us now explain how the two options of AccRK behave as gossip algorithms when they are used to solve the linear system x=0 where ∈^|| × n is the normalized incidence matrix of the network. That is, each row e=(i,j) of  can be represented as (_e:)^⊤= 1/√(2)(e_i -e_j) where e_i (resp.e_j) is the i^th (resp. j^th) unit coordinate vector in ^n.

By using this particular linear system, the expression _i : y^k -b_i/_i :^2_i :^ ⊤ that appears in steps 7 and 8 of AccRK takes the following form when the row e=(i,j) ∈ is sampled:

    _e : y^k -b_i/_e :^2_e :^ ⊤b=0=_e : y^k/_e :^2_e :^ ⊤form of A= y_i^k - y_j^k/2(e_i-e_j).


Recall that with  we denote the Laplacian matrix of the network. For solving the above AC system (see Definition <ref>), the standard RK requires ( (2m/λ_min^+())log(1/ϵ)) iterations to achieve expected accuracy ϵ>0. To understand the acceleration in the gossip framework this should be compared to the 
    (m √(2/λ_min^+())log(1/ϵ))
 of AccRK (Option 1) and the 
    (√(2mν/λ_min^+())log(1/ϵ))
 of AccRK (Option 2).

Algorithm <ref> describes in a single framework how the two variants of AccRK of Section <ref> behave as gossip algorithms when are used to solve the above linear system. Note that each node ℓ∈ of the network has two local registers to save the quantities v^k_ℓ and x^k_ℓ. In each step using these two values every node ℓ∈ of the network (activated or not) computes the quantity y^k_ℓ =α_k v^k_ℓ + (1-α_k) x^k_ℓ. Then in the k^th iteration the activated nodes i and j of the randomly selected edge e=(i,j) exchange their values y^k_i and y^k_j and update the values of x^k_i,  x^k_j and v^k_i, v^k_j as shown in Algorithm <ref>. The rest of the nodes use only their own y^k_ℓ to update the values of v^k_i and x^k_i without communicate with any other node.

The parameter λ^+_min() can be estimated by all nodes in a decentralized manner using the method described in <cit.>. In order to implement this algorithm, we assume that all nodes have synchronized clocks and that they know the rate at which gossip updates are performed, so that inactive nodes also update their local values. This may not be feasible in all applications, but when it is possible (e.g., if nodes are equipped with inexpensive GPS receivers, or have reliable clocks) then they can benefit from the significant speedup achieved.





§ DUAL RANDOMIZED GOSSIP ALGORITHMS


An important tool in optimization literature is duality. In our setting, instead of solving the original minimization problem (primal problem) one may try to develop dual in nature methods that have as a goal to directly solve the dual maximization problem.  Then the primal solution can be recovered through the use of optimality conditions and the development of an affine mapping between the two spaces (primal and dual).

In this section, using existing dual methods and the connection already established between the two areas of research (methods for linear systems and gossip algorithms), we present a different viewpoint that allows the development of novel dual randomized gossip algorithms. 

Without loss of generality we focus on the case of = (no weighted average consensus). For simplicity, we formulate the AC system as the one with the incidence matrix of the network (=) and focus on presenting the distributed nature of dual randomized gossip algorithms with no momentum. While we focus only on no-momentum protocols, we note that accelerated variants of the dual methods could be easily obtained using tools from Section <ref>.



 §.§ Dual problem and SDSA

As we have already presented in Section <ref>, the Lagrangian dual of the best approximation problem (<ref>) is the (bounded) unconstrained concave quadratic maximization problem:

    max_y∈^m D(y)  (b- x^0)^⊤ y - 1/2^⊤ y_^-1^2.


A direct method for solving the dual problem is Stochastic Dual Subspace Accent (SDSA), a randomized iterative algorithm first proposed in <cit.>, which updates the dual vectors y^k as follows: 

    y^k+1= y^k + ω_k ( _k^⊤^-1^⊤_k )^†_k^⊤( b-(x^0+ ^-1^⊤ y^k) ).


In Section <ref> we showed that the iterates {x^k}_k≥0 of the sketch and project method (Algorithm <ref>) can be arised as affine images of the iterates {y^k}_k≥0 of the dual method (<ref>) through the mapping: 

    x^k = ϕ(y^k) =  x^0 + ^-1^⊤ y^k,

and we provided a proof for the linear convergence of SDSA (see Theorem <ref>). Recall that SDSA and the sketch and project method (Algorithm <ref>) converge to a solution of the dual problem and primal problem, respectively, with exactly the same convergence rate. 

Let us choose =. In the special case that the random matrix _k is chosen randomly from the set of unit coordinate/basis vectors in ^m,  the dual method (<ref>) is the randomized coordinate descent <cit.>, and the corresponding primal method is RK (<ref>).  More generally, if _k is a random column submatrix of the m × m identity matrix, the dual method is the randomized Newton method <cit.>, and the corresponding primal method is RBK (<ref>). Next we shall describe the more general block case in more detail.



 §.§ Randomized Newton method as a dual gossip algorithm


In this subsection we bring a new insight into the randomized gossip framework by presenting how the dual iterative process that is associated to RBK method solves the AC problem with = (incidence matrix). Recall that the right hand side of the linear system is b=0. For simplicity, we focus on the case of = and ω=1.

Under this setting (=, = and ω=1) the dual iterative process (<ref>) takes the form:

    y^k+1= y^k - _C:(_C:^⊤^⊤_C:)^†(x^0+^⊤ y^k),

and from Theorem <ref> converges to a solution of the dual problem as follows:

    [D(y^*)-D(y^k) ] ≤[1 - λ_min^+ ( [_C:^⊤ (_C:_C:^⊤)^†_C:] ) ]^k [D(y^*)-D(y^0)].

Note that the convergence rate is exactly the same with the rate of the RBK under the same assumptions (see (<ref>)).

This algorithm is a randomized variant of the Newton method applied to the problem of maximizing the quadratic function D(y) defined in (<ref>). Indeed, in each iteration we perform the update y^k+1 = y^k +_C:λ^k, where λ^k is chosen greedily so that D(y^k+1) is maximized. In doing so, we invert a random principal submatrix of the Hessian of D, whence the name.  

 Randomized Newton Method (RNM) was first proposed by Qu et al. <cit.>. RNM was first analyzed as an algorithm for minimizing smooth strongly convex functions. In <cit.> it was also extended to the case of a smooth but weakly convex quadratics.  This method was not previously associated with any gossip algorithm.

The most important distinction of RNM compared to  existing gossip algorithms is that it operates with values that are associated to the edges of the network. To the best of our knowledge, it is the first randomized dual gossip method. In particular, instead of iterating over values  stored at  the nodes, RNM uses these values to update “dual weights” y^k ∈^m that correspond to the edges  of the network. However, deterministic dual  distributed averaging algorithms were proposed before  <cit.>. Edge-based methods have also been proposed before; in particular  in <cit.> an asynchronous distributed ADMM algorithm presented for solving the more general consensus optimization problem with convex functions. 

Natural Interpretation. In  iteration k, RNM (Algorithm (<ref>)) executes the following steps:
1) Select a random set of edges _k ⊆, 2)  Form a subgraph _k of  from the selected edges, 3) The values of the edges in each connected component of _k are updated: their new values are a linear combination of the private values of the nodes belonging to the connected component and of the adjacent edges of their connected components. (see also example of Figure <ref>).

Dual Variables as Advice. The weights y^k of the edges  have a natural interpretation as advice that each selected node receives from the network in order to update its value (to one that will eventually converge to the desired average).

Consider RNM performing the k^th iteration and let _r denote the set of nodes of the selected connected component that node i belongs to. Then, from Theorem <ref> we know that x_i^k+1=∑_i ∈_r x_i^k / |_r|. Hence, by using (<ref>), we obtain the following identity:

    (^⊤ y^k+1)_i=1/|_r|∑_i ∈_r(c_i+(^⊤ y^k)_i)- c_i.

Thus in each step (^⊤ y^k+1)_i represents the term (advice) that must be added to the initial value c_i of node i in order to update its value to the average of the values of the nodes of the connected component  i belongs to.

Importance of the dual perspective: It was shown  in  <cit.> that when RNM (and as a result, RBK, through the affine mapping (<ref>)) is viewed as a family of methods indexed by the size τ = || (we choose  of fixed size in the experiments),   then τ→ 1/(1-ρ), where ρ is defined in (<ref>), decreases superlinearly fast in τ. That is, as τ increases by some factor, the iteration complexity  drops by a factor that is at least as large. Through preliminary numerical experiments in Section <ref> we experimentally show that this is true for the case of AC systems as well.




§ FURTHER CONNECTIONS BETWEEN METHODS FOR SOLVING LINEAR SYSTEMS AND GOSSIP ALGORITHMS


In this section we highlight some further interesting connections between linear systems solvers and gossip protocols for average consensus:


  * Eavesdrop gossip as special case of Kaczmarz-Motzkin method.
In <cit.> greedy gossip with eavesdropping (GGE), a novel randomized gossip algorithm for distributed computation of the average consensus problem was proposed and analyzed. 
In particular it was shown that that greedy updates of GGE lead to rapid convergence. In this protocol, the greedy updates are made possible by exploiting the broadcast nature of wireless communications. During the operation of GGE, when a node decides to gossip, instead of choosing one of its neighbors at random, it makes a greedy selection, choosing the node which has the value most different from its own. In particular the method behaves as follows:

At the k^th iteration of GGE, a node i_k is chosen uniformly at random from [n]. Then, i_k identifies a neighboring node j_k ∈_i satisfying:

    j_k ∈max_j ∈_i{1/2 (x_i^k-x_j^k)^2 }

which means that the selected node i_k identifies a neighbor that currently has the most different value from its own. This choice is possible because each node i∈  maintains not only its own local variable x_i^k, but also a copy of the current values at its neighbors x_j^k for j ∈_i. In the case that node i_k has multiple neighbors whose values are all equally (and maximally) different from  its current value, it chooses one of these neighbors at random. Then node i_k and j_k update their values to:

    x_i^k+1=x_j^k+1=1/2 (x_i^k+x_j^k).


In the area of randomized methods for solving large linear system there is one particular method,  the Kaczmarz-Motzkin algorithm <cit.> that can work as gossip algorithm with the same update as the GGE when is use to solve the homogeneous linear system with matrix the Incidence matrix of the network.

Update rule of Kaczmarz-Motzkin algorithm (KMA) <cit.>:


  * Choose sample of d_k constraints, P_k, uniformly at random from among the rows of matrix .

  * From among these d_k constraints, choose t_k=argmax_i ∈ P_k_i: x^k-b_i.

  * Update the value: x^k+1=x^k - _t_k : x^k -b_i/_t_k :^2_t_k :^ ⊤.


It is easy to verify that when the Kaczmarz-Motzkin algorithm is used for solving the AC system with = (incidence matrix) and in each step of the method the chosen constraints d_k of the linear system correspond to edges attached to one node it behaves exactly like the GGE. From numerical analysis viewpoint an easy way to choose the constraints d_k that are compatible to the desired edges is in each iteration to find the indexes of the non-zeros of a uniformly at random selected column (node) and then select the rows corresponding to these indexes.

Therefore, since GGE <cit.> is a special case of the KMA (when the later applied to special AC system with Incidence matrix) it means that we can obtain the convergence rate of GGE by simply use the tight conergence analysis presented in <cit.> [Note that the convergence theorems of <cit.> use d_k=d. However, with a small modification in the original proof the theorem can capture the case of different d_k.]. In <cit.> it was mentioned that analyzing the convergence behavior of GGE is non-trivial and not an easy task. By establishing the above connection the convergence rates of GGE can be easily obtained as special case of the theorems presented in <cit.>. 

In Section <ref> we presented provably accelerated variants of the pairwise gossip algorithm and of its block variant. Following the same approach one can easily develop accelerated variants of the GGE using the recently proposed analysis for the accelerated Kaczmarz-Motzkin algorithm presented in <cit.>. 


  * Inexact Sketch and Project Methods:

In Chapter <ref>, several inexact variants of the sketch and project method (<ref>) have been proposed. As we have already mentioned the sketch and project method is a two step procedure algorithm where first the sketched system is formulated and then the last iterate x^k is exactly projected into the solution set of the sketched system. In Chapter <ref>, we replace the exact projection with an inexact variant and we suggest to run a different algorithm (this can be the sketch and project method itself) in the sketched system to obtain an approximate solution.  It was shown that in terms of time the inexact updates can be faster than their exact variants.

In the setting of randomized gossip algorithms for the AC system with Incidence matrix (=) , = and ω =1 a variant of the inexact sketch and project method will work as follows (similar to the update proved in Theorem <ref>):


  * Select a random set of edges C ⊆.

  * Form subgraph _k of  from the selected edges.

  * Run the pairwise gossip algorithm of <cit.> (or any variant of the sketch and project method) on the subgraph _k until an accuracy ϵ is achieved (reach a neighborhood of the exact average).



  * Non-randomized gossip algorithms as special cases of Kaczmarz methods:

In the gossip algorithms literature there are efficient protocols that are not randomized<cit.>. Typically, in these algorithms the pairwise exchanges between nodes it happens in a deterministic, such as predefined cyclic, order. For example, T-periodic gossiping is a protocol which stipulates that each node must interact with each of its neighbours exactly once every T time units. It was shown that under suitable connectivity assumptions of the network 𝒢, the T-periodic gossip sequence will converge at a rate determined by the magnitude of the second largest eigenvalue of the stochastic matrix determined by the sequence of pairwise exchanges which occurs over a period. It has been shown that if the underlying graph is a tree, the mentioned eigenvalue is constant for all possible T-periodic gossip protocols.

In this work we focus only on randomized gossip protocols. However we speculate that the above non-randomized gossip algorithms would be able to express as special cases of popular non-randomized projection methods for solving linear systems   <cit.>.  Establishing connections like that is an interesting future direction of research and can possibly lead to the development of novel block and accelerated variants of many non-randomized gossip algorithms, similar to the protocols we present in Sections <ref> and <ref>.





§ NUMERICAL EVALUATION


In this section, we empirically validate our theoretical results and evaluate the performance of the proposed randomized gossip algorithms. The section is divided into four main parts, in each of which we highlight a different aspect of our contributions. 

In the first experiment, we numerically verify the linear convergence of the Scaled RK algorithm (see equation (<ref>))  for solving the weighted average consensus problem presented in Section <ref>. In the second part, we explain the benefit of using block variants in the gossip protocols where more than two nodes update their values in each iteration (protocols presented in Section <ref>).  In the third part, we explore the performance of the faster and provably accelerated gossip algorithms proposed in Section <ref>. In the last experiment, we numerically show that relaxed variants of the pairwise randomized gossip algorithm converge faster than the standard randomized pairwise gossip with unit stepsize (no relaxation). This gives a specific setting where the phenomenon of over-relaxation of iterative methods for solving linear systems is beneficial.

In the comparison of all gossip algorithms we use the relative error measure x^k-x^*_^2 / x^0-x^*_^2 where x^0 =c ∈^n is the starting vector of the values of the nodes and matrix  is the positive definite diagonal matrix with weights in its diagonal (recall that in the case of standard average consensus this can be simply =). Depending on the experiment, we choose the values of the starting vector c ∈^n to follow either a Gaussian distribution or uniform distribution or to be integer values such that c_i=i ∈.  In the plots, the horizontal axis represents the number of iterations except in the figures of subsection <ref>, where the horizontal axis represents the block size.

In our implementations we use three popular graph topologies from the area of wireless sensor networks. These are the cycle (ring graph), the 2-dimension grid and the random geometric graph (RGG) with radius r=√(log(n)/n). In all experiments we formulate the average consensus problem (or its weighted variant) using the incidence matrix. That is, = is used as the AC system. Code was written in Julia 0.6.3. 



 §.§ Convergence on weighted average consensus


As we explained in Section <ref>, the sketch and project method (Algorithm <ref>) can solve the more general weighted AC problem. In this first experiment we numerically verify the linear convergence of the Scaled RK algorithm (<ref>) for solving this problem in the case of =.  That is, the matrix  of the weights is the degree matrix  of the graph (_ii=d_i,  ∀ i ∈ [n]). In this setting the exact update rule of the method is given in equation (<ref>), where in order to have convergence to the weighted average the chosen nodes are required to share not only their private values but also their weight _ii (in our experiment this is equal to the degree of the node d_i).  In this experiment the starting vector of values x^0=c ∈^n is a Gaussian vector. The linear convergence of the algorithm is clear in Figure <ref>.






 §.§ Benefit of block variants


We devote this experiment to evaluate the performance of the randomized block gossip algorithms presented in Sections <ref> and <ref>.  In particular, we would like to highlight the benefit of using larger block size in the update rule of randomized Kaczmarz method and as a result through our established connection of the randomized pairwise gossip algorithm <cit.> (see equation (<ref>)).

Recall that in Section <ref> we show that both RBK and RNM converge to the solution of the primal and dual problems respectively with the same rate and that their iterates are related via a simple affine transform (<ref>). In addition note that an interesting feature of the RNM <cit.>, is that when the method viewed as algorithm indexed by the size τ=|C|, it enjoys superlinear speedup in τ. That is, as τ (block size) increases by some factor, the iteration complexity drops by a factor that is at least as large (see Section <ref>). Since RBK and RNM share the same rates this property naturally holds for RBK as well.

We show that for a  connected network , the complexity improves superlinearly in τ = |C|, where C is chosen as a subset of  of size τ, uniformly at random (recall the in the update rule of RBK the random matrix is =_:C). Similar to the rest of this section in comparing the number of iterations for different values of τ, we use the relative error  ε=x^k-x^*^2 / x^0-x^*^2. We let x^0_i=c_i=i for each node i ∈ (vector of integers). We run RBK until the relative error becomes smaller than 0.01.  The blue solid line in the figures denotes the actual number of iterations (after running the code) needed in order to achieve ε≤ 10 ^-2 for different values of τ. The green dotted line represents the function f(τ)ℓ/τ, where ℓ is the number of iterations of RBK with τ=1 (i.e., the pairwise gossip algorithm).  The green line depicts linear speedup;  the fact that the blue line (obtained through  experiments) is below the green line points to superlinear speedup.  In this experiment we use the Cycle graph with n=30 and n=100 nodes (Figure <ref>) and  the 4 × 4 two dimension grid graph (Figure <ref>). Note that, when |C|=m the convergence rate of the  method becomes ρ=0 and as a result it converges in one step.






 §.§ Accelerated gossip algorithms

We devote this subsection to experimentally evaluate the performance of the proposed accelerated gossip algorithms: mRK (Algorithm <ref>),  mRBK (Algorithm <ref>) and AccGossip with the two options of the parameters (Algorithm <ref>). In particular we perform four experiments. In the first two we focus on the performance of the mRK and how the choice of stepsize (relaxation parameter) ω and heavy ball momentum parameter β affect the performance of the method. In the next experiment we show that the addition of heavy ball momentum can be also beneficial for the performance of the block variant mRBK. In the last experiment we compare the standard pairwise gossip algorithm (baseline method) from <cit.>, the mRK and the AccGossip and show that the probably accelerated gossip algorithm, AccGossip outperforms the other algorithms and converge as predicted from the theory with an accelerated linear rate.



  §.§.§ Impact of momentum parameter on mRK

As we have already presented in the standard pairwise gossip algorithm (equation (<ref>)) the two selected nodes that exchange information update their values to their exact average while all the other nodes remain idle. In our framework this update can be cast as special case of mRK when β=0 and ω=1.

In this experiment we keep the stepsize fixed and equal to ω=1 which means that the pair of the chosen nodes update their values to their exact average and we show that by choosing a suitable momentum parameter β∈ (0,1) we can obtain faster convergence to the consensus for all networks under study. The momentum parameter β is chosen following the suggestions made in Chapter <ref> for solving general consistent linear systems. See Figure <ref> for more details. It is worth to point out that for all networks under study the addition of a heavy ball momentum term is beneficial in the performance of the method.





  §.§.§ Comparison of mRK and shift-Register algorithm <cit.>

In this experiment we compare mRK with the shift register gossip algorithm (pairwise momentum method, abbreviation: Pmom) analyzed in <cit.>. We choose the parameters ω and β of mRK in such a way in order to satisfy the connection established in Section <ref>. That is, we choose β=ω-1 for any choice of ω∈ (1,2). Observe that in all plots of Figure <ref> mRK outperforms the corresponding shift-register algorithm. 





  §.§.§ Impact of momentum parameter on mRBK

In this experiment our goal is to show that the addition of heavy ball momentum accelerates the RBK gossip algorithm presented in Section <ref>. Without loss of generality we choose the block size to be  equal to τ=5. That is, the random matrix _k∼ in the update rule of mRBK is a m × 5 column submatrix of the indetity m × m matrix. Thus, in each iteration 5 edges of the network are chosen to form the subgraph _k and the values of the nodes are updated according to Algorithm <ref>. Note that similar plots can be obtained for any choice of block size. We run all algorithms with fixed stepsize ω=1. From Figure <ref>, it is obvious that for all networks under study, choosing a suitable momentum parameter β∈ (0,1) gives faster convergence than having no momentum, β =0. 





  §.§.§ Performance of AccGossip

In the last experiment on faster gossip algorithms we evaluate the performance of the proposed provably accelerated gossip protocols of Section <ref>. In particular we compare the standard RK (pairwise gossip algorithm of <cit.>) the mRK (Algorithm <ref>) and the AccGossip (Algorithm <ref>) with the two options for the selection of the parameters presented in Section <ref>. 

The starting vector of values x^0=c is taken to be a Gaussian vector.  For the implementation of mRK we use the same parameters with the ones suggested in the stochastic heavy ball (SGB) setting in Chapter <ref>. For the AccRK (Option 1) we use λ=λ_min^+(^⊤) and for AccRK (Option 2) we select ν=m[For the networks under study we have
 m < 1/λ_min^+(). Thus, by choosing ν=m we select the pessimistic upper bound of the parameter (<ref>) and not its exact value (<ref>). As we can see from the experiments, the performance is still accelerated and almost identical to the performance of AccRK (Option 1) for this choice of ν.]. From Figure <ref> it is clear that for all networks under study the two randomized gossip protocols with Nesterov momentum are faster than both the pairwise gossip algorithm of <cit.> and the mRK/SHB (Algorithm <ref>). To the best of our knowledge Algorithm <ref> (Option 1 and Option 2) is the first randomized gossip protocol that converges with provably accelerated linear rate and as we can see from our experiment its faster convergence is also obvious in practice.





 §.§ Relaxed randomized gossip without momentum

In the area of randomized iterative methods for linear systems it is know that over-relaxation (using of larger step-sizes) can be particularly helpful in practical scenarios. However, to the best of our knowledge there is not theoretical justification of why this is happening.

In our last experiment we explore the performance of relaxed randomized gossip algorithms (ω≠ 1) without momentum and show that in this setting having larger stepsize can be particularly beneficial.

As we mentioned before (see Theorem <ref>) the sketch and project method (Algorithm <ref>) converges with linear rate when the step-size (relaxation parameter) of the method is ω∈ (0,2) and the best theoretical rate is achieved when ω=1.  In this experiment we explore the performance of the standard pairwise gossip algorithm when the step-size of the update rule is chosen in (1,2). Since there is no theoretical proof of why over-relaxation can be helpful we perform the experiments using different starting values of the nodes. In particular we choose the values of vector c ∈ R^n to follow (i) Gaussian distribution, (ii) Uniform Distribution and (iii) to be integers values such that c_i=i ∈ R. Our findings are presented in Figure <ref>. Note that for all networks under study and for all choices of starting values having larger stepsize, ω∈ (1,2) can lead to better performance. Interesting observation from Figure <ref> is that the stepsizes ω =1.8 and ω=1.9 give the best performance (among the selected choices of stepsizes) for all networks and for all choices of starting vector x^0=c.





§ CONCLUSION


In this chapter, we present a general framework for the analysis and design of randomized gossip algorithms.  Using tools from numerical linear algebra and the area of randomized projection methods for solving linear systems we propose novel serial, block and accelerated gossip protocols for solving the average consensus and weighted average consensus problems. 

We believe that this work could open up several future avenues for research. Using similar approach with the one presented in this manuscript, many popular projection methods can be interpreted as gossip algorithms when used to solve linear systems encoding the underlying network. This can lead to the development of novel distributed protocols for average consensus.  

In addition, we speculate that the gossip protocols presented in this work can be extended to the more general setting of multi-agent consensus optimization where the goal is to minimize the average of convex or non-convex functions 1/n∑_i=1^n f_i(x) in a decentralized way <cit.>. 



§ MISSING PROOFS
 
 

 §.§ Proof of Theorem <ref>

 
 
Let z^k x^k-x^* , x^0=c is the starting point and ρ is as defined in (<ref>).
From Theorem <ref> we know that sketch and project method converges with
    [x^k-x^*_^2]≤ρ^k x^0-x^*_^2,

where x^* is the solution of (<ref>). Inequality (<ref>), together with Markov inequality can be used to give the following bound

    (z^k / z^0 ≥ε^2) ≤(z^k/z^0)/ε^2≤ρ^k/ε^2.

Therefore, as long as k is large enough so that ρ^k ≤ε^3, we have
(z^k / z^0 ≥ε^2 ) ≤ε. 
That is, if 

    ρ^k ≤ε^3 ⇔ k ≥3logε/logρ⇔ k ≥3log(1/ε)/log(1/ ρ),

then:
    (x^k-c̅1/x^0-c̅1≥ε)≤ε.


Hence, an upper bound for value T_ave(ϵ) can be obtained as follows,

    T_ave(ϵ)   =   sup_c∈^ninf{k : (z^k > ε z^0 )≤ε}≤sup_c∈^ninf{k :  k ≥3log(1/ε)/log(1/ ρ)}
        =   sup_c∈^n3 log(1/ε)/log(1/ ρ)= 3 log(1/ε)/log(1/ ρ)≤3 log(1/ε)/1-ρ,

where in last inequality we use 1/log(1/ ρ) ≤ 1/1-ρ which is true because ρ∈ (0,1).  





 §.§ Proof of Theorem <ref>




The following notation conventions are used in this proof.
With q_k we indicate the number of connected components of subgraph _k, while with _r we denote the set of nodes of each connected component q_k (r ∈{1,2,…,q_k}). Finally, |_r| shows the cardinality of set _r. Notice that, if  is the set of all nodes of the graph then = r={1,2,...q}∪_r and ||=∑_r=1^q |_r|. 

Note that from equation (<ref>), the update of RBK for =(Incidence matrix) can be expressed as follows:

    xminimize      ϕ^k(x)x-x^k^2
       subject to      _:C^⊤ x=0

Notice that _:C^⊤ is a row submatrix of matrix  with rows those that correspond to the random set C ⊆ of the edges. From the expression of matrix  we have that

    (_:C^⊤)_e:^⊤=f_i-f_j ,   ∀ e=(i,j) ∈ C ⊆.


Now, using this, it can be seen that the constraint _:C^⊤ x=0 of problem (<ref>) is equivalent to q equations (number of connected components) where each one of them forces the values x_i^k+1 of the nodes i ∈_r to be equal. That is, if we use z_r to represent the value of all nodes that belong in the connected component r then:

    x_i^k+1= z_r   ∀ i ∈_r,

and the constrained optimization problem (<ref>) can expressed as unconstrained as follows:

    zminimize  ϕ^k(z)=∑_i ∈_1^(z_1-x_i^k)^2+...+∑_i ∈_q^(z_q-x_i^k)^2,

where z=(z_1,z_2,…,z_q) ∈^q is the vector of all values z_r when r ∈{1,2,…,q}.
Since our problem is unconstrained the minimum of equation (<ref>) is obtained when ∇ϕ^k(z)=0.

By evaluating partial derivatives of (<ref>) we obtain:

    ∂ϕ^k(z)/∂ z_r=0 ⟺∑_i ∈_r^ 2(z_r- x_i^k)=0.

As a result, 

    z_r=∑_i ∈_r^ x_i^k/|_r|,   ∀ r ∈{1,2,…,q}.

Thus from (<ref>), the value of each node i ∈_r will be updated to 

    x_i^k+1= z_r=∑_i ∈_r^ x_i^k/|_r|.
 .





CHAPTER: PRIVACY PRESERVING RANDOMIZED GOSSIP ALGORITHMS





§ INTRODUCTION


In this chapter, similar to Chapter <ref>, we consider the average consensus (AC) problem. 
In particular, we focus on randomized gossip algorithms for solving the AC problem and propose techniques for protecting the information of the initial values c_i, as these may be sensitive. We develop and analyze three privacy preserving variants of the randomized pairwise gossip algorithm (“randomly pick an edge (i,j)∈ and then replace the values stored at vertices i and j by their average”) first proposed in <cit.> for solving the average consensus problem.  While we shall not formalize the notion of privacy preservation in this work, it will be intuitively clear that our methods indeed make it harder for nodes to infer information about the private values of other nodes, which might be useful in practice. 



 §.§ Background

The literature on decentralized protocols for solving the average consensus problem is vast and has long history <cit.>. In particular, the algorithms for solving this problem can be divided into two broad categories: the average consensus algorithms <cit.> which work in a synchronous setting and the gossip algorithms <cit.> which they consider ideal protocols for the asynchronous time model <cit.>.  
In the average consensus algorithms, all nodes of the network update their values simultaneously by communicating with a set of their neighbours and in each iteration of the algorithmic procedure the same update occurs. On the other hand, in gossip protocols only one edge of the whole network is selected at each iteration and only the nodes, that this edge connects, exchange their private information and update their values to their average. 

In this chapter, we focus on modifying the basic algorithm of <cit.>, which we refer to as “Standard Gossip” algorithm. In the following, we review some of the most important gossip protocols for solving the average consensus proposed in the last decade. While we do not address any privacy considerations for these protocols, they can serve as inspiration for further work. For a survey of relevant work, we refer the interested reader to <cit.>.

The Geographic Gossip algorithm was proposed in <cit.>, in which the authors combine the gossip approach with a geographic routing towards a randomly chosen location with the main goal to improve the convergence rate of Standard Gossip algorithm. In each step, a node is activated, assuming that it is aware of its geographic location and some additional assumptions on the network topology, it chooses another node from the rest of the network (not necessarily one of its neighbours) and performs a pairwise averaging with this node. Later, using the same assumptions, this algorithm was extended into Geographic Gossip Algorithm with Path Averaging <cit.>, in which connected sequences of nodes were chosen in each step and they averaged their values. More recently, in <cit.> and <cit.> authors propose a geographic and path averaging methods which converge to the average consensus without the assumption that nodes are aware of their geographic location. Recall that, in Section <ref> we show how the path averaging gossip algorithm can be seen as special case of the Randomized Block Kaczmarz method for solving consistent linear systems.

Another important randomized gossip algorithm is the Broadcast Gossip algorithm, first proposed in <cit.> and then extended in <cit.>. The idea of this algorithm is simple: In each step, a node in the network is activated uniformly at random, following the asynchronous time model, and broadcasts its value to its neighbours. The neighbours receive this value and update their own values. It was experimentally shown that this method converges faster than the pairwise and geographic randomized gossip algorithms.

Alternative gossip protocols are the so-called non-randomized Gossip algorithms <cit.>. Typically, this class of algorithms executes the pairwise exchanges between nodes in a deterministic, such as predefined cyclic, order. T-periodic gossiping is a protocol which stipulates that each node must interact with each of its neighbours exactly once every T time units. Under suitable connectivity assumptions of the network 𝒢, the T-periodic gossip sequence will converge at a rate determined by the magnitude of the second largest eigenvalue of the stochastic matrix determined by the sequence of pairwise exchanges which occurs over a period. It has been shown that if the underlying graph is a tree, the mentioned eigenvalue is constant for all possible T-periodic gossip protocols.

 Accelerated Gossip algorithms have also been proposed for solving the average consensus problem. In this setting, the nodes of the network incorporate additional memory to accelerate convergence. In particular,  the nodes update their value using an update rule that involves not only the current values of the sampled nodes but also their previous values. This idea is closely related to the shift register methods studied in numerical linear algebra for improving the convergence rate of linear system solvers. The works <cit.> have shown theoretically and numerically, that under specific assumptions this idea can improve the performance of the Standard Gossip algorithm. For more details on these gossip protocols check also Section <ref> of this thesis.

Randomized Kaczmarz-type Gossip algorithms. In Chapter <ref> of this thesis we presented how popular randomized Kaczmarz-type methods for solving large linear systems can also solve the AC problem. We explained how these methods can be interpreted as randomized gossip algorithms when applied to special systems encoding the underlying network structure and present in detail their decentralized nature. 



  
Asynchronous Time Model: In this chapter, we are interested in the asynchronous time model <cit.>. More precisely, we assume that each node of our network has a clock which ticks at a rate of 1 Poisson process. This is equivalent of having available a global clock which ticks according to a rate n Poisson process and selects an edge of the network uniformly at random. In general, the synchronous setting (all nodes update the values of their nodes simultaneously using information from a set of their neighbours) is convenient for theoretical considerations but is not representative of some practical scenarios, such as the distributed nature of sensor networks. For more details on clock modeling we refer the reader to <cit.>, as the contribution of this chapter is orthogonal to these considerations.



  
Privacy and Average Consensus: 
Finally, the introduction of notions of privacy within the AC problem is relatively recent in the literature, and the existing works consider two different ideas.


  * In <cit.>, the concept of differential privacy <cit.> is used to protect the output value c̅ computed by all nodes. In this work, an exponentially decaying Laplacian noise is added to the consensus computation. This notion of privacy refers to protection of the final average, and formal guarantees are provided. 

  * A different approach with a more stricter goal is the design of privacy-preserving average consensus protocols that guarantee protection of the initial values c_i of the nodes <cit.>. In this setting each node should be unable to infer a lot about the initial values c_i of any other node. In the existing works, this is mainly achieved with the clever addition of noise through the iterative procedure that guarantees preservation of privacy and at the same time converges to the exact average. We shall however mention, that none of these works address any specific notion of privacy (no clear measure of privacy is presented) and it is still not clear how the formal concept of differential privacy <cit.> can be applied in this setting.


It is worth to highlight that all of the above-mentioned privacy preserving average consensus papers propose protocols which work on the synchronous setting (all nodes update their values simultaneously). To the best of our knowledge our proposed protocols are the first that solve the AC problem and at the same time protect the initial values of the nodes using the asynchronous time model (by having gossip updates).



 §.§ Main contributions

In this chapter, we present three different approaches for solving the Average Consensus problem while at the same time protecting the information about the initial values. To the best of our knowledge, this work is the first which combines the gossip framework with the privacy concept of protection of the initial values. It is important to stress that, we provide tools for protection of the initial values, but we do not address any specific notion of privacy or a threat model, nor how these quantitatively translate to any explicit measure. These would be highly application dependent, and we only provide theoretical convergence rates for the techniques we propose.

The methods we propose are all dual in nature. The dual setting of this chapter will be explained in detail in Section <ref>. Recall that in Chapters <ref>, <ref> and <ref> of this thesis, we have shown how duality and dual algorithms can be used for solving consistent linear systems. In addition, in Chapter <ref>, the dual viewpoint was extended to the concept of the average consensus problem and the first dual gossip algorithms were presented. As we have seen, the dual updates correspond to updates of the primal variables, via an affine mapping. Using this relationship of the primal and the dual spaces the convergence analysis of the dual methods can be easily obtained once the analysis of the primal methods is available (see for example the proof of Theorem <ref> in the introduction of this thesis). In this chapter, one of our contributions is a novel dual analysis of randomized pairwise gossip (without the use of rates that obtain first through a primal analysis) which exactly recovers existing convergence rates for the primal iterates. 

We now outline the three different techniques we propose in this chapter, which we refer to as “Binary Oracle”, “ϵ-Gap Oracle” and “Controlled Noise Insertion”. The first two are, to best of our knowledge, the first proposals of weakening the oracle used in the gossip framework. Privacy preservation is attained implicitly, as the nodes do not exchange the full information about their values. The last technique is inspired by the addition of noise proposed in <cit.> for the synchronous setting. We extend this technique by providing explicit finite time convergence guarantees.

Binary Oracle.
We propose to reduce the amount of information transmitted in each iteration to a single bit[We do not refer to the size of the object being transmitted over the network, but the binary information that can be inferred from the exchange. In practice, this might be achieved using secure multiparty protocols <cit.>, causing the overall network bandwidth to slightly increase compared to the usual implementation of standard gossip algorithm.]. More precisely, when an edge is selected, each corresponding node will only receive information whether the value on the other node is smaller or larger. Instead of setting the value on the selected nodes to their average, each node increases or decreases its value by a pre-specified step.

ϵ-Gap Oracle.
In this case, we have an oracle that returns one of three options and is parametrized by ϵ. If the difference in values of sampled nodes is larger than ϵ, an update similar to the one in Binary Oracle is taken. Otherwise, the values remain unchanged. An advantage compared to the Binary Oracle is that this approach will converge to a certain accuracy and stop there, determined by ϵ (Binary Oracle will oscillate around optimum for a fixed stepsize). However, in general, it will disclose more information about the initial values.

Controlled Noise Insertion.
This approach is inspired by the works of <cit.>, and protects the initial values by inserting noise in the process. Broadly speaking, in each iteration, each of the sampled nodes first adds a noise to its current value, and an average is computed afterward. Convergence is guaranteed due to the correlation in the noise across iterations. Each node remembers the noise it added last time it was sampled, and in the following iteration, the previously added noise is first subtracted, and a fresh noise of smaller magnitude is added. Empirically, the protection of initial values is provided by first injecting noise into the system, which propagates across the network, but is gradually withdrawn to ensure convergence to the true average.





  
Convergence Rates of our Methods: In Table <ref>, we present the summary of convergence guarantees for the above three techniques. By · we denote the standard Euclidean norm. 

The two approaches which restrict the amount of information disclosed, Binary Oracle and ϵ-Gap Oracle, converge slower than the standard Gossip. In particular, these algorithms have sublinear convergence rate. At first sight, this should not be surprising, since we indeed use much less information. However, in Theorem <ref>, we show that if we had in a certain sense perfect global information, we could use it to construct a sequence of adaptive stepsizes, which would push the capability of the binary oracle to a linear convergence rate. However, this rate is still m-times slower than the standard rate of the binary gossip algorithm.  We note, however, that having the global information at hand is an impractical assumption. Nevertheless, this result highlights that there is a potentially large scope for improvement, which we leave for future work.

The approach of Controlled Noise Insertion yields a linear convergence rate which is driven by the minimum of two factors. Without going into details, which of these is bigger depends on the speed by which the magnitude of the inserted noise decays. If the noise decays fast enough, we recover the convergence rate of the standard the gossip algorithm. In the case of slow decay, the convergence is driven by this decay. By () we denote the algebraic connectivity of graph  <cit.>. The parameter γ controls the decay speed of the inserted noise, see Corollary <ref>.

Measures of Success: Note that the convergence of each randomized gossip algorithm in Table <ref> naturally depends on a different measure of suboptimality. All of them converge to 0 as we approach the optimal solution. The details of these measures will be described later in the main body of this chapter. In particular a lemma that formally describes the key connections between these measures is presented in Section <ref>. For now lets us give a brief description of these results.
The standard Gossip and Controlled Noise Insertion essentially depend on the same quantity, but we present the latter in terms of dual values as this is what our proofs are based on. Lemma <ref> formally specifies this equivalence. The binary oracle depends on the average difference among directly connected nodes. The measure for the ϵ-Gap Oracle depends on quantities Δ^t(ϵ)=1/m|{ (i,j)∈: |x_i^t-x_j^t|≥ϵ}|, which is the number of edges that the values of their connecting nodes differ by more than ϵ. 



 §.§ Structure of the chapter

The remainder of this chapter is organized as follows: Section <ref> introduces the basic setup that is used through the chapter. A detailed explanation of the duality behind the randomized pairwise gossip algorithm is given. We also include a novel and insightful dual analysis of this method as it will make it easier for the reader to parse later development.   In Section <ref> we present our three private gossip algorithms as well as the associated iteration complexity results. Section <ref> is devoted to the numerical evaluation of our methods. Finally, conclusions are drawn in Section <ref>. 



§ DUAL ANALYSIS OF RANDOMIZED PAIRWISE GOSSIP
 
As we outlined in the introduction of this chapter, our approach for extending the (standard) randomized pairwise gossip algorithm to privacy preserving variants utilizes duality. The purpose of this section is to formalize this duality. In addition, we provide a novel and self-contained dual analysis of randomized pairwise gossip. While this is of an independent interest, we include the proofs as their understanding aids in the understanding of the more involved proofs of our private gossip algorithms developed in the remainder of the chapter.

The main problems under study are the best approximation problem (<ref>) and its dual (<ref>) that we have seen multiple times throughout the thesis. However, similar to Chapter <ref> we focus on the more specific setting of the average consensus. To keep the chapter self-contained and for the benefit of the reader we present the definitions of these problems and we explain again how they are related to the average consensus problem. 



 §.§ Primal and dual problems



Consider solving the (primal) problem of projecting a given vector c=x^0∈^n onto the solution space of a linear system: 

    min_x∈^n P(x) 1/2x-x^0^2   subject to   x=b,
 
where ∈^m× n, b∈^m, x^0∈^n. Note that this the best approximation problem (<ref>) with = (Identity matrix).
We assume the problem is feasible, i.e., that the system x = b is consistent. With the above optimization problem we associate the dual problem

    max_y∈^m D(y) (b- x^0)^⊤ y - 1/2^⊤ y^2.

As we have explained in the previous chapters, the dual is an unconstrained concave (but not necessarily strongly concave) quadratic maximization problem. It can be seen that as soon as the system x = b is feasible, the dual problem is bounded. Moreover, all bounded concave quadratics in ^m can be written in the as D(y) for some matrix  and vectors b and x^0 (up to an additive constant).

With any dual vector y we associate the primal vector via an affine transformation: ϕ(y) = x^0 + ^⊤ y. It can be shown that if y^* is dual optimal, then x^*=ϕ(y^*) is primal optimal <cit.>. Hence, any dual algorithm producing a sequence of dual variables y^t → y^* gives rise to a corresponding primal algorithm producing the sequence x^t ϕ(y^t) → x^*. We shall now consider one such dual algorithm. 




 §.§ Stochastic dual subspace ascent


Stochastic dual subspace ascent (SDSA) is a stochastic method for solving the dual problem (<ref>). In Section <ref> we have already described how by choosing appropriately the main parameters of SDSA we can recover many known algorithms as special cases. In this chapter we focus only on one special case of the general algorithm. For the more general update rule of SDSA check equations (<ref>) and (<ref>). In particular, following the notation of the rest of the thesis, we select ω=1 (stepsize of the method) and = (positive definite matrix that defines the geometry of the space). 
If we further use the fact that AC linear systems (see Definition <ref>) have zero right hand side (b=0), then the update rule of SDSA takes the form:


    y^t+1 = y^t - _t(_t^⊤^⊤_t)^†_t^⊤(x^0 + ^⊤ y^t),

where _t is a random matrix drawn independently at each iteration t from an arbitrary but fixed distribution , and † denotes the Moore-Penrose pseudoinverse. 
 
The corresponding primal iterates are defined via:

    x^tϕ(y^t) = x^0 + ^⊤ y^t.


The relevance of this all to average consensus follows through the observation that for a specific choice of matrix  and distribution , the primal method produced by combining (<ref>) and (<ref>) is equivalent to the (standard) randomized pairwise gossip method (see discussion in Chapter <ref>). In that case, SDSA is a dual variant of randomized pairwise gossip. In particular, in this chapter, we define  as follows: _t is a  unit basis vector in ^m, chosen uniformly at random from the collection of all such unit basis vectors, denoted {f_e  |  e∈}. In this case, SDSA is the randomized coordinate ascent method applied to the dual problem.



 §.§ Randomized gossip setup: choosing 


We wish (,b) to be an average consensus (AC) system (see Definition <ref>).  As we explained in Chapter <ref> if x = b is an AC system, then the solution of the primal problem (<ref>) is necessarily
x^* = c̅·, where c̅ = 1/n∑_i=1^n x^0_i is the value that each node needs to compute in the standard average consensus problem (x^*_i = c̅ for all i ∈). 

In the rest of this chapter we focus on a specific AC system; the one  in which the matrix   is the incidence matrix of the graph .  In particular, we let ∈^m× n be the matrix defined as follows. Row e=(i,j)∈  of  is given by _ei = 1, _ej=-1 and _el=0 if l∉{i,j}.  Notice that the system x=0 encodes the constraints x_i=x_j for all (i,j)∈, as desired.
 



 §.§ Randomized pairwise gossip



We provide both primal and dual form of the (standard) randomized pairwise gossip algorithm. 

The primal form is standard and needs no lengthy commentary. At the beginning of the process, node i contains private information c_i = x^0_i. In each iteration we sample a pair of connected nodes (i,j)∈ uniformly at random, and update x_i and x_j to their average. We let the values at the remaining nodes intact.




The dual form of the standard randomized pairwise gossip method is a specific instance of SDSA, as described in (<ref>), with x^0=c and _t being a randomly chosen standard unit basis vector f_e in ^m (e is a randomly selected edge). It can be seen <cit.> that in that case, (<ref>) takes the following form:




The following lemma is useful for the analysis of all our methods. It describes the increase in the dual function value after an arbitrary change to a single dual variable e.

 
Define z=y^t + λ f_e, where e=(i,j) and λ∈. Then
 
    D(z) - D(y^t) = -λ(x^t_i - x^t_j) - λ^2.



The claim follows by direct calculation:

    D(y^t+ λ f_e) - D(y^t)
       =    -( c)^⊤ (y^t + λ f_e) - 1/2^⊤ (y^t+ λ f_e)^2 + ( c)^⊤ y^t + 1/2^⊤ y^t^2
       =     - λ f_e^⊤(c +  ^⊤ y^t)_x^t - 1/2λ^2 ^⊤ f_e^2_=2   =     -λ (x^t_i-x^t_j) - λ^2.




The maximizer in λ of the expression in (<ref>) leads to the exact line search formula
λ^t = (x_j^t-x_i^t)/2
used in the dual form of the method. 



 §.§ Complexity results




With graph = {,} we now associate a certain quantity, which we shall denote β = β(). It is the smallest nonnegative number β such that the following inequality[We write ∑_(i,j) to indicate sum over all unordered pairs of vertices. That is, we do not count (i,j) and (j,i) separately, only once. By ∑_(i,j)∈ we denote a sum over all edges of . On the other hand, by writing ∑_i ∑_j, we are summing over all (unordered) pairs of vertices twice.] holds for all x∈^n:


    ∑_(i,j) (x_j  -x_i)^2 ≤β∑_(i,j)∈ (x_j  -x_i)^2.



The Laplacian matrix of graph  is given by = ^⊤. Let λ_1()≥λ_2() ≥…≥λ_n-1()≥λ_n() be the eigenvalues of . The algebraic connectivity of  is the second smallest eigenvalue of :

    () = λ_n-1().

We have λ_n()=0. Since we assume  to be connected, we have ()>0. 
Thus,  () is the smallest nonzero eigenvalue of the Laplacian: ()=λ_min^+() = λ_min^+(^⊤ ).
As the next result states, the quantities β() and () are inversely proportional. 




 
β() = n/().


See Section <ref>.


The following theorem gives a complexity result for (standard) randomized gossip. Our analysis is dual in nature.

 Consider the randomized gossip algorithm (Algorithm ) with uniform edge-selection probabilities: p_e=1/m. Then:

    D(y^*) - D(y^k) ≤(1-()/2m )^k[D(y^*) - D(y^0) ].



See Section <ref>


Theorem <ref>  yields the complexity estimate
O(2m/()log(1/ϵ)),
which exactly matches the complexity result obtained from the primal analysis (see (<ref>) in Chapter <ref>). Hence, the primal and dual analyses give the same rate. 

Randomized coordinate descent methods were first analyzed in <cit.>. For a recent treatment, see <cit.>. Duality in randomized coordinate descent methods was studied in <cit.>. Acceleration was studied in <cit.>. These methods extend to nonsmooth problems of various flavours <cit.>.

With all of this preparation, we are now ready to formulate and analyze our private gossip algorithms; we do so in Section <ref>.



§ PRIVATE GOSSIP ALGORITHMS
 

In this section, we introduce three novel private gossip algorithms, complete with iteration complexity guarantees. In Section <ref> the key relationships between the measures of success (see Table <ref>) of all proposed algorithms are presented.  In Section <ref> the privacy is protected via a binary communication protocol. In Section <ref> we communicate more: besides binary information, we allow for the communication of a bound on the gap, introducing the ϵ-gap oracle. In Section <ref> we introduce a privacy-protection mechanism based on a procedure we call controlled noise insertion.



 §.§ Measures of success


We devote this subsection to present Lemma <ref> that formally specifies the connections between the different measures of suboptimality of the privacy preserving algorithms, firstly presented in Table <ref>. 

 

(Relationship between convergence measures)
Suppose that x is primal variable corresponding to the dual variable y as defined in (<ref>).
Dual suboptimality can be expressed as the following <cit.>:

    D(y^*)-D(y) = 1/2c̅ - x^2.

Moreover, for any x∈^n we have : 

    1/2n∑_i=1^n ∑_j=1^n  (x_j  -x_i)^2
       =   c̅ - x^2 
    
    ∑_e=(i,j)∈|x_i-x_j|
       ≤   √(mn)c̅ -x ,
    
    ∑_e=(i,j)∈|x_i-x_j|
       ≥   √(())c̅-x ,
    
    ∑_e=(i,j)∈|x_i-x_j|
       ≥   ϵ|{ (i,j)∈: |x_i-x_j|≥ϵ}|.



See Section <ref>.





 §.§ Private gossip via binary oracle
 


We now present the gossip algorithm with Binary Oracle in detail and provide theoretical convergence guarantee. The information exchanged between sampled nodes is constrained to a single bit, describing which of the nodes has the higher value. As mentioned earlier, we only present the conceptual idea, not how exactly would the oracle be implemented within a secure multiparty protocol between participating nodes <cit.>.

We will first introduce the dual version of the algorithm.




The update of primal variables above is equivalent to set x^t+1 as primal point corresponding to dual iterate: x^t+1 = c+ ^⊤ y^t+1 = x^t + ^⊤ (y^t+1-y^t). In other words, the primal iterates {x^t} associated with the dual iterates {y^t} can be written in the form:

    x^t+1 =  x^t + λ^t _e:^⊤,        x^t_i < x^t_j,
    
     x^t - λ^t _e:^⊤,       x^t_i≥ x^t_j.
 
It is easy to verify that due to the structure of , this is equivalent to the updates above.

Since the evolution of dual variables {y^k} serves only the purpose of the analysis, the method can be written in the primal-only form as follows:




Given a sequence of stepsizes {λ^t}, it will be convenient to define α^k∑_t=0^kλ^t and β^k ∑_t=0^k(λ^t)^2.
In the following theorem, we study the convergence of the quantity

    L^t1/m∑_e=(i,j)∈ |x^t_i - x^t_j|.
 

  For all k≥ 1 we have

    min_t=0,1,…,kL^t≤∑_t=0^kλ^t/α^kL^t≤ U^k D(y^*)-D(y^0)/α^k + β^k/α^k.

Moreover:


  (i) If we set λ^t = λ^0>0 for all t, then  U^k = D(y^*)-D(y^0)/λ^0 (k+1) + λ^0.

  (ii) Let R be any constant such that R≥ D(y^*)-D(y^0). If we fix  k≥ 1, then the choice of stepsizes {λ^0,…,λ^k} which minimizes U^k correspond to the constant stepsize rule λ^t = √(R/k+1) for all t=0,1,…,k, and U^k = 2√(R/k+1).


  (iii) If we set λ^t=a/√(t+1) for all t=0,1,…,k, then 

    U^k≤D(y^*)-D(y^0)+a^2 (log(k+3/2)+log(2) )/2a( √(k+2) -1)=(log(k)/√(k))




See Section <ref>


The part (ii) of Theorem <ref> is useful in the case that we know exactly the number of iterations before running the algorithm, providing in a sense optimal stepsizes and rate (1/√(k)). However, this might not be the case in practice. Therefore part (iii) is also relevant, which yields the rate (log(k)/√(k)). These bounds are significantly weaker than the standard bound in Theorem <ref>. This should not be surprising though, as we use significantly less information than the Standard Gossip algorithm.

Nevertheless, there is a potential gap in terms of what rate can be practically achievable. The following theorem can be seen as a form of a bound on what convergence rate is possible to be attained by the Binary Oracle. However, this rate can be attained with access to very strong information. It requires a specific sequence of stepsizes λ^t which is likely unrealistic in practical scenarios. This result points to a gap in the analysis which we leave open. We do not know whether the sublinear convergence rate in Theorem <ref> is necessary or improvable without additional information about the system.



For Algorithm  with stepsizes chosen in iteration t adaptively to the current values of x^t as λ^t = 1/2m∑_e∈|x_i^t-x_j^t|, we have

    c-x^k^2≤(1-()/2m^2)^kc-x^0^2



See Section <ref>

Comparing Theorem <ref> with the result for standard Gossip in Theorem <ref>, the convergence rate is worse by factor of m, which is the price we pay for the weaker oracle.

An alternative to choosing adaptive stepsizes is the use of adaptive probabilities <cit.>. We leave such a study for future work.



 §.§ Private gossip via  ϵ-gap oracle
 


Here we present the gossip algorithm with ϵ-Gap Oracle in detail and provide theoretical convergence guarantees. The information exchanged between the sampled nodes is restricted to be one of three cases, based on the difference of their values. As mentioned earlier, we only present the conceptual idea, not how exactly would the oracle be implemented within a secure multiparty protocol between participating nodes <cit.>.

We will first introduce the dual version of the algorithm.




Note that the primal iterates {x^t} associated with the dual iterates {y^t} can be written in the form:

    x^t+1 =  x^t + ϵ/2_e:^⊤,        x^t_i-x^t_j < -ϵ
    
     x^t - ϵ/2_e:^⊤,       x^t_j-x^t_i < -ϵ,
    
    x^t,      otherwise.

The above is equivalent to setting
x^t+1 = x^t + ^⊤ (y^t+1-y^t)=c+ ^⊤ y^t+1. 

Since the evolution of dual variables {y^t} serves only the purpose of the analysis, the method can be written in the primal-only form as follows:




Before stating the convergence result, let us define a quantity the convergence will naturally depend on. For each edge e=(i,j)∈ and iteration t ≥ 0 define the random variable 
    Δ^t_e(ϵ)      1,      |x^t_i-x^t_j|≥ϵ,
    
    0,     otherwise.

Moreover, let 

    Δ^t (ϵ)    1/m∑_e∈Δ^t_e(ϵ).


The following Lemma bounds the expected increase in dual function value in each iteration.

 For all t ≥ 0 we have
D(y^t+1)-D(y^t) ≥ϵ^2/4Δ^t(ϵ).


See Section <ref>


Our complexity result will be expressed in terms of the quantity:

    δ^k(ϵ)    1/k∑_t=0^k-1Δ^t(ϵ) = 1/k∑_t=0^k-1Δ^t (ϵ).


 For all k≥ 1 we have 
    δ^k(ϵ) ≤4(D(y^*)-D(y^0))/kϵ^2.



See Section <ref>


Note that if Δ^k(ϵ)=0, it does not mean the primal iterate x^k is optimal. This only implies that the values of all pairs of directly connected nodes differ by less than ϵ.





 §.§ Private gossip via controlled noise insertion



In this section, we present the gossip algorithm with Controlled Noise Insertion. As mentioned in the introduction of this chapter, the approach is similar to the technique proposed in <cit.>. Those works, however, address only algorithms in the synchronous setting, while our work is the first to use this idea in the asynchronous setting. Unlike the above, we provide finite time convergence guarantees and allow each node to add the noise differently, which yields a stronger result.

In our approach, each node adds noise to the computation independently of all other nodes. However, the noise added is correlated between iterations for each node. We assume that every node owns two parameters — the initial magnitude of the generated noise σ_i^2 and rate of decay of the noise ϕ_i. The node inserts noise w_i^t_i to the system every time that an edge corresponding to the node was chosen, where variable t_i carries an information how many times the noise was added to the system in the past by node i. Therefore, if we denote by t the current number of iterations, we have ∑_i=1^nt_i = 2t.

In order to ensure convergence to the optimal solution, we need to choose a specific structure of the noise in order to guarantee the mean of the values x_i converges to the initial mean. In particular, in each iteration a node i is selected, we subtract the noise that was added last time, and add a fresh noise with smaller magnitude:

    w_i^t_i = ϕ_i^t_iv_i^t_i-ϕ_i^t_i-1v_i^t_i-1,

where 0 ≤ϕ_i<1, v_i^-1 = 0 and v_i^t_i∼ N(0,σ_i^2) for all iteration counters k_i ≥ 0 is independent to all other randomness in the algorithm. This ensures that all noise added initially is gradually withdrawn from the whole network.

After the addition of noise, a standard Gossip update is made, which sets the values of sampled nodes to their average. Hence, we have

    lim_t→∞(
    c-
    1/n∑_i=1^n x_i^t
    )^2   =   lim_t→∞(
    1/n∑_i=1^n ϕ_i^t_i-1v_i^t_i-1)^2
       ≤   lim_t→∞1/n∑_i=1^n (
     ϕ_i^t_i-1v_i^t_i-1)^2
       =   1/nlim_t→∞∑_i=1^n (
     ϕ_i^t_i-1v_i^t_i-1)^2
       =   1/nlim_t→∞∑_i=1^n ϕ_i^2t_i-2(v_i^t_i-1)^2
    
       =   1/nlim_t→∞∑_i=1^n ϕ_i^2t_i-2σ_i^2
     =
     1/n∑_i=1^n σ_i^2 lim_t→∞ϕ_i^2t_i-2
       =   
     0,

as desired. 

It is not the purpose of this work to define any quantifiable notion of protection of the initial values formally. However, we note that it is likely the case that the protection of private value c_i will be stronger for bigger σ_i and for ϕ_i closer to 1.

For simplicity, we provide only the primal algorithm below.




We now provide results of dual analysis of Algorithm . The following lemma provides us the expected decrease in dual suboptimality for each iteration.


Let d_i denote the number of neighbours of node i. Then, 

    D(y^*)- D(y^t+1) ≤   ( 1-()/2m)D(y^*)- D(y^t) +1/4m∑_i=1^n d_iσ^2_i ϕ_i^2t_i
           -
    1/2m∑_e∈( ϕ_i^t_i-1v_i^t_i-1 x_j^t+ ϕ_j^t_j-1v_j^t_j-1 x_i^t
    ).



See Section <ref>


We use the lemma to prove our main result, in which we show linear convergence for the algorithm. For notational simplicity, we decided to have ρ^t=(ρ)^t, i.e. superscript of ρ denotes its power, not an iteration counter.


Let us define the following quantities:

    ρ       1-()/2m,
    
    ψ^t      1/∑_i=1^n(d_iσ_i^2)∑_i=1^n d_i σ_i^2(1-d_i/m(1-ϕ_i^2) )^t.


Then for all k≥ 1 we have the following bound

    D(y^*)- D(y^k) ≤ρ^k ( D(y^*)- D(y^0) )  + ∑(d_iσ_i^2)/4m∑_t=1^k ρ^k-tψ^t.




See Section <ref>


Note that ψ^t is a weighted sum of t-th powers of real numbers smaller than one.  For large enough t, this quantity will depend  on the largest of these numbers. This brings us to define M as the set of indices i for which the quantity 1-d_i/m(1-ϕ_i^2) is maximized:  

    M=max_i{ 1-d_i/m(1-ϕ_i^2)}.

Then for any i_max∈ M we have

    ψ^t  
    ≈1/∑_i=1^n (d_iσ_i^2)∑_i∈ M d_i σ_i^2(1-d_i/m(1-ϕ_i^2) )^t
     =
      ∑_i∈ M d_i σ_i^2/∑_i=1^n (d_iσ_i^2)(1-d_i_max/m(1-ϕ_i_max^2) )^t,

which means that increasing ϕ_j for j∉M will not substantially influence convergence rate. 

Note that as soon as we have

    ρ>
     1-d_i/m(1-ϕ_i^2)
 
for all i, the rate from theorem <ref> will be driven by ρ^k (as k →∞) and we will have


    D(y^*)- D(y^k)  = Õ(ρ^k)


One can think of the above as a threshold: if there is i such that ϕ_i is large enough so that the inequality (<ref>) does not hold, the convergence rate is driven by ϕ_i_max. Otherwise, the rate  is not influenced by the insertion of noise. Thus, in theory, we do not pay anything in terms of performance as long as we do not hit the threshold. One might be interested in choosing ϕ_i so that the threshold is attained for all i, and thus M={1,…,n}. This motivates the following result:



Let us choose 

    ϕ_i √(1-γ/d_i)

for all i, where  γ≤ d_min.
Then

    D(y^*)- D(y^k)    ≤   ( 1-min( ()/2m,γ/m) )^k ( D(y^*)- D(y^0)+∑_i=1^n (d_iσ_i^2)/4mk ).


As a consequence, ϕ_i=√(1-()/2d_i) is the largest decrease rate of noise for node i such that the guaranteed convergence rate of the algorithm is not violated.



See Section <ref>



While the above result clearly states the important threshold, it is not always practical as () might not be known. However, note that if we choose nd_min/2(n-1)≤γ≤ d_min, we have 
min( ()/2m,γ/m) = ()/2m
since 
()/2≤n/n-1d_min/2≤γ,
where e() denotes graph edge connectivity: the minimal number of edges to be removed so that the graph becomes disconnected.   Inequality () ≤n/n-1d_min is a well known result in spectral graph theory <cit.>. As a consequence, if for all i we have

    ϕ_i≤√(1-(n-1)d_min/2nd_i),

then the convergence rate is not driven by the noise.




§ NUMERICAL EVALUATION


We devote this section to experimentally evaluate the performance of the Algorithms ,  and  we proposed in the previous sections, applied to the Average Consensus problem. In the experiments, we used two popular graph topologies the cycle graph (ring network) and the random geometric graph (see Figure <ref> for an illustration of the two graphs). 


  * Cycle graph with n nodes: 𝒞(n). In our experiments we choose n=10. This small simple graph with regular topology is chosen for illustration purposes.

  * Random geometric graph with n nodes and radius r:  𝒢(n,r). Random geometric graphs <cit.> are very important in practice because of their particular formulation which is ideal for modeling wireless sensor networks <cit.>. In our experiments we focus on a 2-dimensional random geometric graph (n,r) which is formed by placing n nodes uniformly at random in a unit square with edges between nodes which are having euclidean distance less than the given radius r. We set this to be to be r = r(n) = √(log(n)/n) — it is well know that the connectivity is preserved in this case <cit.>. We set n = 100.




Setup:
In all experiments we generate a vector with of initial values c_i from a uniform distribution over [0,1]. We run several experiments and present two kinds of figures that help us to understand how the algorithms evolve and verify the theoretical results of the previous sections. These figures are:


  * The evolution of the initial values of the nodes. In these figures, we plot how the trajectory of the values x_i^t of each node i evolves throughout iterations. The black dotted horizontal line represents the exact average consensus value which all nodes should approach, and thus all other lines should approach this level.

  * The evolution of the relative error measure x^t-x^*^2 / x^0-x^*^2 where x^0 =c ∈^n is the starting vector of the values of the nodes. In these figures we choose to have the relative error, both in normal and logarithmic scale on the vertical axis and the number of iterations on the horizontal axis.

For our evaluation we run each privacy preserving algorithm for several parameters and for a pre-specified number of iterations not necessarily the same for each experiment. 

To illustrate the first concept (trajectories of the values x_i^t) , we provide a simple example of the evolution of the initial values x_i^t for the case of the Standard Gossip algorithm <cit.> in Figure <ref>. The horizontal black dotted line represents the average consensus value. It is the exact average of the initial values c_i of the nodes in the network.


In the rest of this section we evaluate the performance of the three privacy preserving randomized gossip algorithms of Section <ref>, and contrast with the above Standard Gossip algorithm, which we refer to as “Baseline” in the following figures labels.



 §.§ Private gossip via binary oracle


In this section, we evaluate the performance of Algorithm  presented in Section <ref>. In the algorithm, the input parameters are the positive stepsizes {λ^t}_t=0^∞. The goal of the experiments is to compare the performance of the proposed algorithm using different choices of λ^t.

In particular, we use decreasing sequences of stepsizes λ^t=1/t and λ^t=1/ √(t), and three different fixed values for the stepsizes λ^t=λ∈{0.001, 0.01, 0.1}. We also include the adaptive choice λ^t = 1/4m∑_e∈|x_i^t-x_j^t| which we have proven to converge with linear rate in Theorem <ref>. We compare these choices in Figures <ref> and <ref>, along with the Standard Gossip algorithm for clear comparison.




 
In general, we clearly see what is expected with the constant stepsizes — that they converge to a certain neighbourhood and oscillate around optimum. With smaller stepsize, this neighbourhood is more accurate, but it takes longer to reach. With decreasing stepsizes, Theorem <ref> suggests that λ^t of order 1 / √(t) should be optimal. Figure <ref> demonstrates this, as the choice of λ^t = 1/t decreases the stepsizes too quickly. However, this is not the case in Figure <ref> in which we observe the opposite effect. This is due to the cycle graph being small and simple, and hence the diminishing stepsize becomes a problem only after a relatively large number of iterations. With the adaptive choice of stepsizes, we recover the linear convergence rate as predicted by Theorem <ref>.

The results in Figure <ref> show one surprising comparison. The adaptive choice of stepsizes does not seem to perform better than λ^t = 1 / √(t). However, we verified that when running for more iterations, the linear rate of adaptive stepsize is present and converges significantly faster to higher accuracies. We chose to present the results for 6000 iterations since we found it overall cleaner.







 §.§ Private gossip via ϵ-gap oracle



In this section, we evaluate the performance of the Algorithm  presented in Section <ref>. In the algorithm, the input parameter is the positive error tolerance variable ϵ. For experimental evaluation. we choose three different values for the input, ϵ∈{0.2, 0.02, 0.002}, and again use the same cycle and random geometric graphs. The trajectories of the values x_i^t are presented in Figures <ref> and <ref>, respectively. The performance of the algorithm in terms of the relative error is presented in Figures <ref> and <ref>.

The performance is exactly matching the expectation — with larger ϵ, the method converges very fast to a wide neighbourhood of the optimum. For a small value, it converges much closer to the optimum, but it requires more iterations.












 §.§ Private gossip via controlled noise insertion


In this section, we evaluate the performance of Algorithm  presented in Section <ref>. This algorithm has two different parameters for each node i. These are the initial variance σ_i^2 ≥ 0 and the rate of decay, ϕ_i,  of the noise.

To evaluate the impact of these parameters, we perform several experiments. As earlier, we use the same graph structures for evaluation: cycle graph and random geometric graph. The algorithm converges with a linear rate depending on the minimum of two factors — see Theorem <ref> and Corollary <ref>. We will verify that this is indeed the case, and for values of ϕ_i above a certain threshold, the convergence is driven by the rate at which the noise decays. This is true for both identical values of ϕ_i for all i, and for varying values as per (<ref>). We further demonstrate the latter is superior in the sense that it enables insertion of more noise, without sacrificing the convergence speed. Finally, we study the effect of various magnitudes of the noise inserted initially.



  §.§.§ Fixed variance, identical decay rates


In this part, we run Algorithm  with σ_i = 1 for all i, and set ϕ_i = ϕ for all i and some ϕ. We study the effect of varying the value of ϕ on the convergence of the algorithm. 

In both Figures <ref>b and <ref>b, we see that for small values of ϕ, we eventually recover the same rate of linear convergence as the Standard Gossip algorithm. If the value of ϕ is sufficiently close to 1 however, the rate is driven by the noise and not by the convergence of the Standard Gossip algorithm. This value is ϕ = 0.98 for cycle graph, and ϕ=0.995 for the random geometric graph in the plots we present.

Looking at the individual runs for small values of ϕ in Figure <ref>b, we see some variance in terms of when the asymptotic rate is realized. We would like to point out that this does not provide additional insight into whether specific small values of ϕ are in general better for the following reason. The Standard Gossip algorithm is itself a randomized algorithm, with an inherent uncertainty in the convergence of any particular run. If we ran the algorithms multiple times, we observe variance in the evolution of the suboptimality of similar magnitude, just as what we see in the figure. Hence, the variance is expected, and not significantly influenced by the noise.












  §.§.§ Variance 1 and different decay rates


In this section, we perform a similar experiment as above, but the values ϕ_i are not all the same. We rather control them by the choice of γ as in (<ref>). Note that by decreasing γ, we increase ϕ_i, and thus smaller γ means the noise decays at a slower rate. Here, due to the regular structure of the cycle graph, we present only results for the random geometric graph.

It is not straightforward to compare this setting with the setting of identical ϕ_i, and we return to it in the next section. Here we only remark that we again see the existence of a threshold predicted by theory, beyond which the convergence is dominated by the inserted noise. Otherwise, we recover the rate of the Standard Gossip algorithm.









  §.§.§ Impact of varying ϕ_i


In this experiment, we demonstrate the practical utility of letting the rate of decay ϕ_i to be different on each node i. In order to do so, we run the experiment on the random geometric graph and compare the settings investigated in the previous two sections — the noise decay rate driven by ϕ, or by γ.

In first place, we choose the values of ϕ_i such that that the two factors in Corollary <ref> are equal. For the particular graph we used, this corresponds to γ≈ 0.17 with ϕ_i=√(1-()/2d_i). Second, we make the factors equal, but with constraint of having ϕ_i to be equal for all i. This corresponds to ϕ_i ≈ 0.983 for all i.

The performance for a large number of iterations is displayed in the left side of Figure <ref>. We see that the above two choices indeed yield very similar practical performance, which also eventually matches the rate predicted by theory. For a complete comparison, we also include the performance of the Standard Gossip algorithm.

The important message is conveyed in the histogram in the right side of Figure <ref>. The histogram shows the distribution of the values of ϕ_i for different nodes i. The minimum of these values is what we needed in the case of identical ϕ_i for all i. However, most of the values are significantly higher. This means, that if we allow the noise decay rates to depend on the number of neighbours, we are able to increase the amount of noise inserted, without sacrificing practical performance. This is beneficial, as more noise will likely be beneficial for any formal notion of protection of the initial values.





§ CONCLUSION



In this chapter, we addressed the Average Consensus problem via novel asynchronous privacy preserving randomized gossip algorithms. In particular, we propose three different algorithmic tools for the protection of the initial private values of the nodes.

The first two proposed algorithms “Private Gossip via Binary Oracle" and “Private Gossip via  ϵ-Gap Oracle" are based on the same idea of weakening the oracle used in the gossip update rule. In these two protocols the chosen pair of nodes of each gossip step instead of share their exact values they provide only categorical (or even binary) information to each other. 

In the third protocol “Private Gossip via Controlled Noise Insertion", we systematically inject and withdraw noise throughout the iterations, so as to ensure convergence to the average consensus value and at the same time protect the private information of the nodes. 

In all cases, we provide explicit convergence rates and evaluate practical convergence on common simulated network topologies. 

Future work inludes the design of privacy preserving variants of several popular and fast gossip protocols  <cit.>.  One can also investigate more challenging types of consensus problems like the finite step consensus or consensus on
networks with time-varying topology, and design gossip protocols that preserve the privacy of the participating agents.  Designing the optimal network structure for information preservation is also an interesting research direction. 

As we have already mentioned the gossip algorithms of this chapter do not address any specific notion of privacy (no clear measure of privacy is presented) and it is still not clear how the formal concept of differential privacy <cit.> can be applied in protocols for solving the average consensus problem. Propose efficient differential privacy guarantees for gossip protocols in general graphs is an interesting open problem.





§ PROOFS OF MAIN RESULTS

 



 §.§ Proof of Lemma <ref>


Let us first present a lemma that we use in the proof of Lemma <ref>.

The eigenvalues of =n - ^⊤ are {0,n,n,…,n}



Clearly, =0. Consider some vector x such that ⟨ x, ⟩=0. Then, x= n x-^⊤ x=nx+^⊤ x=nx thus x is an eigenvector corresponding to eigenvalue n. Thus, we can pick n-1 linearly independent eigenvectors of  corresponding to eigenvalue n,  which concludes the proof. 
 
Having established the above lemma let us present the proof Lemma <ref>.

The Laplacian matrix of  is the matrix =^⊤. We have 
_ii = d_i (degree of vertex i), _ij = _ji = -1 if (i,j)∈ and _ij = 0 otherwise. A simple computation reveals that for any x∈^n we have

    x^⊤ x = ∑_e=(i,j)∈ (x_i-x_j)^2.


Let  be the n(n-1)/2 × n  matrix corresponding to the complete graph  on . Let = ^⊤ be its Laplacian. We have _ii=n-1 for all i and _ij =-1 for i≠ j. So, = n  - ^⊤.  Then

    x^⊤ x = nx^2 -(∑_i=1^n x_i)^2=∑_(i,j) (x_i - x_j)^2.


Inequality (<ref>) can therefore be recast as follows:

    x^⊤ (n  - ^⊤) x ≤  x^⊤β()  x,      x∈^n.


Let β=β(). Note that both  and β are Hermitian thus have real eigenvalues and there exist an orthonormal basis of their eigenvectors. Suppose that {x_1,… x_n} are eigenvectors of β L corresponding to eigenvalues λ_1(β),λ_2(β) … ,λ_n(β). Without loss of generality assume that these eigenvectors form an orthonormal basis and λ_1(β)≥…≥λ_n(β).

Clearly, λ_n(β)=0, x_n=/√(n), and λ_n-1(β)=n. Lemma <ref>
states that eigenvalues of  are {0,n,n,…,n}.

 One can easily see that eigenvector corresponding to zero eigenvalue of  is x_n. Note that eigenvectors x_1,…,x_n-1 generate an eigenspace corresponding to eigenvalue n of . 

Consider some x=∑_i=1^n c_ix_i, c_i ∈ for all i. Then we have

    x^⊤x=
    ∑_i=1^n λ_i()c_i^2≤∑_i=1^n λ_i(β)c_i^2=
     x^⊤β x,

which concludes the proof. 


 §.§ Proof of Theorem <ref>


We first establish two lemmas which will be needed to prove Theorem <ref>.
 
Assume that edge e=(i,j) is selected in iteration t of Algorithm . Then 
    D(y^t+1) - D(y^t) = 1/4(x^t_i-x^t_j)^2.


 We have y^t+1 = y^t + λ^t f_e where λ^t is chosen so that D(y^t+1)-D(y^t) is maximized. Applying Lemma <ref>, we have

    D(y^t+1) - D(y^t) =max_λ -λ (x^t_i-x^t_j) - λ^2
    
    =  1/4(x_i^t - x_j^t)^2.



  Let  x∈^n such that 1/n∑_i x_i = c̅. Then 
    1/2c̅ - x^2 ≤1/2()∑_e=(i,j)∈ (x_i-x_j)^2.





    1/2c̅ - x^2 
       (<ref>)=   1/4n∑_i=1^n ∑_j=1^n  (x_j  -x_i)^2
       =   1/2n∑_(i,j) (x_j  -x_i)^2
       (<ref>)≤   β()/2n∑_e=(i,j)∈ (x_i-x_j)^2
      Lemma <ref>=  1/2()∑_e=(i,j)∈ (x_i-x_j)^2



Having established Lemmas <ref> and <ref>, we can now proceed with the proof of Theorem <ref>:

    D(y^*) - D(y^t+1)  |  y^t   =    D(y^*) - D(y^t) - D(y^t+1) - D(y^t)  |  y^t
       (<ref>)=     D(y^*) - D(y^t) - ∑_e=(i,j)∈1/4m (x^t_i-x^t_j)^2
       (<ref>)=   1/2c̅ - x^t^2- ∑_e=(i,j)∈1/4m (x^t_i-x^t_j)^2
       (<ref>)≤   (1-()/2m )1/2c̅ - x^t^2
       (<ref>)=   (1-()/2m)(D(y^*) - D(y^t) )  .

Taking expectation again, we get the recursion

    D(y^*) - D(y^t+1)   ≤  (1-()/2m)D(y^*) - D(y^t) .



 §.§ Proof of Lemma <ref>


Let us first present a Lemma that we use in the proof of Lemma <ref>.


    ∑_i=1^n ( ∑_j=1^n (x_j-x_i) )^2
    =
    n/2∑_i=1^n ∑_j=1^n(x_j-x_i)^2



Using simple algebra we have

    ∑_i=1^n ( ∑_j=1^n (x_j-x_i) )^2
       =   ∑_i=1^n ( ∑_j=1^n x_j-nx_i )^2
    
       =   ∑_i=1^n ( ( ∑_j=1^n x_j )^2 +n^2x_i^2-2nx_i ( ∑_j=1^n x_j ) )
    
       =   
    n(∑_j=1^n x_j )^2
    +
    n^2∑_i=1^nx_i^2
    -2n(  ∑_j=1^n x_j )^2
    
       =   
    n^2∑_i=1^nx_i^2
    -n( ∑_i=1^n x_i )^2 .

Manipulating right hand side of (<ref>) we obtain

    n/2∑_i=1^n ∑_j=1^n(x_j-x_i)^2    =   n/2∑_i=1^n ∑_j=1^n(x_j^2+x_i^2-2x_ix_j)
    
       =    
    n^2∑_i=1^nx_i^2 -n∑_i=1^n ∑_j=1^nx_ix_j
       =   
    n^2∑_i=1^nx_i^2
    -n( ∑_i=1^n x_i )^2 .

Clearly, LHS and RHS of (<ref>) are equal.

In order to show (<ref>) it is enough to notice that

    c̅ - x^2 
       =   ∑_i=1^n (c̅-x_i)^2 
       =   ∑_i=1^n (1/n∑_j=1^nx_j  -x_i)^2 
     
       =   ∑_i=1^n (∑_j=1^n 1/n (x_j  -x_i))^2 
       (<ref>)=  1/2∑_i=1^n ∑_j=1^n 1/n (x_j  -x_i)^2 
     
       =   1/2n∑_i=1^n ∑_j=1^n  (x_j  -x_i)^2.

Note that we have 

    1/nm(∑_e=(i,j)∈|x_i-x_j|)^2 ≤1/n∑_e∈(x_i-x_j)^2≤1/n∑_(i,j)(x_i-x_j)^2 (<ref>)=c̅ -x^2,

which proves (<ref>).  On the other hand, we have

    1/()(∑_e=(i,j)∈|x_i-x_j|)^2
    ≥1/()∑_e∈(x_i-x_j )^2
    (<ref>)≥c̅ -x ^2,

which concludes (<ref>). Inequality (<ref>) holds trivially.  



 §.§ Proof of Theorem <ref>


The following lemma is used in the proof of Theorem <ref>.
 
 Fix k≥ 0 and let R >0. Then

    min_λ = (λ^0,…,λ^k)∈^k+1R + β^k/α^k = 2√(R/k+1),

and the optimal solution is given by λ^t = √(R/k+1) for all t.

 Define ϕ(λ) = R+ β^k/α^k. If we write λ = r x, where r=λ and x is of unit norm, then ϕ(tx)=R+r^2/r ⟨, x⟩. Clearly, for any fixed r, the x∈^k+1  minimizing x↦ϕ(rx) is x=/, where  is the vector of ones in ^k+1. It now only remains to minimize the function r↦R+r^2/r. This function is convex and differentiable. Setting the derivative to zero leads to r = √(R). Combining the above, we get the optimal solution λ = r / = √(R)/.


Let e=(i,j) be the edge selected at iteration t≥ 0. Applying Lemma <ref>, we see that
D(y^t+1) - D(y^t)
= λ^t |x^t_i-x^t_j| - (λ^t)^2. Taking expectation with respect to edge selection, we get 

    D(y^t+1)-D(y^t)  |  y^t = -(λ^t)^2 +  λ^t ·1/m∑_e=(i,j)∈ |x^t_i - x^t_j|,

and taking expectation again and using the tower property, we get the identity

    D(y^t+1)-D(y^t)  = -(λ^t)^2 +  λ^t·L^t.

Therefore,

    D(y^*) - D(y^0)   ≥   D(y^k+1)-D(y^0)
       =   ∑_t=0^kD(y^t+1) - D(y^t)
       =   ∑_t=0^kD(y^t+1) - D(y^t)   =    -∑_t=0^k(λ^t)^2 + ∑_t=0^kλ^t ·L^t.

It remains to reshuffle the resulting inequality to obtain (<ref>).

We can see that part (i) follows directly. Optimality of stepsizes in (ii) is due to Lemma <ref>. To show (iii) we should state that

    α^k=∑_t=0^k λ^k=∑_1^k+1a/√(t)≥ a ∫_t=1^k+2t^-1/2 dt=2a( √(k+2) -1)


    β^k=∑_t=0^k (λ^k)^2=∑_t=1^k+1a^2/t≤ a^2 ∫_1/2^k+3/2t^-1 dt=a^2 (log(k+3/2)+log(2) )

The inequality above holds due to the fact that for t>1/2 we have t^-1≤∫_t-1/2^t+1/2x^-1dx since x^-1 is convex function. 


 §.§ Proof of Theorem <ref>


Using Lemma <ref> with we have

    D(y^t+1)-D(y^t) |  y^t   =   -(λ^t)^2+λ^t 1/m∑_e∈|x_i^t-x_j^t|
    
       =   1/4m^2(∑_e∈|x_i^t-x_j^t|)^2
      ≥  1/4m^2∑_e∈(x_i^t-x_j^t)^2.

Taking the expectation again we obtain

    D(y^t+1)-D(y^t)≥1/4m^2∑_e∈(x_i^t-x_j^t)^2.

On the other hand, we have

    D(y^t+1)-D(y^t)    =    (D(y^t+1)-D(y^*) ) +(D(y^*)-D(y^t) )
       =   1/2c-x^t^2
    -
    1/2c-x^t+1^2 
    
       =   ()/4m^2c-x^t^2+
    (1-()/2m^2)1/2c-x^t^2
    
            -    1/2c-x^t+1^2 
    
       (<ref>)≤   1/4m^2∑_e=(i,j)∈(x_i^t-x_j^t)^2+
    (1-()/2m^2)1/2c-x^t^2
    
            -   1/2c -x^t+1^2 .

Taking the expectation of the above and combining with (<ref>) we obtain the desired recursion

    c-x^t+1^2  ≤  (1-()/2m^2)
    c-x^t^2
    .




 §.§ Proof of Lemma <ref>


Let e=(i,j) be the edge selected at iteration t. Applying Lemma <ref>, we see that

    D(y^t+1) - D(y^t)
    = -ϵ/2 (x^t_i-x^t_j) - ϵ^2/4 ,        x^t_i-x^t_j ≤ -ϵ
    ϵ/2 (x^t_i-x^t_j) - ϵ^2/4,       x^t_j-x^t_i ≤ -ϵ,
    
    0,      otherwise.


This implies that 
    D(y^t+1)-D(y^t) ≥ϵ^2/4,        if Δ^t_e = 1,
    
     = 0,        if Δ^t_e = 0.


Taking expectation in the selection of e, we get 

    D(y^t+1)-D(y^t)  |  y^t≥ϵ^2/4·(Δ^t_e = 1  |  y^t) + 0  ·(Δ^t_e = 0  |  y^t)=ϵ^2/4Δ^t.

It remains to take expectation again.


 §.§ Proof of Theorem <ref>


Since for all k≥ 0 we have D(y^k) ≤ D(y^*), it follows that 
    D(y^*) - D(y^0) ≥D(y^k)-D(y^0) = ∑_t=0^k-1D(y^t+1) - D(y^t)= ∑_t=0^k-1D(y^t+1) - D(y^t).

It remains to apply Lemma <ref>.



 §.§ Proof of Lemma <ref>


Let us first present three lemmas that we use in the proof of Lemma <ref>.

Suppose that we run Algorithm  for t iterations and t_i denotes the number of times that some edge corresponding to node i was selected during the algorithm.  


  * v_i^t_i and t_j are independent  for all (i.e., not necessarily distinct) i,j.

  * v_i^t_i and ϕ_j^t_j are independent for all (i.e., not necessarily distinct)  i,j.

  * w_i^t_i and w_j^t_j have zero correlation for all i≠ j.

  * x_j^t and ϕ_i^t_iv_i^t_i have zero correlation for all (i.e., not necessarily distinct) i,j.





  * Follows from the definition of v_i^t.

  * Follows from the definition of v_i^t.

  * Note that we have w_i^t_i = ϕ_i^t_iv_i^t_i - ϕ_i^t_i-1v_i^t_i-1 and w_j^t_j = ϕ_j^t_jv_j^t_j - ϕ_j^t_j-1v_j^t_j-1. 
Clearly, v_i^t_i and w_j^t_j have zero correlation. Similarly v_i^t_i-1 and w_j^t_j have zero correlation. Thus, w_i^t_i and w_j^t_j have zero correlation.

  * Clearly, x_j^t is a function initial state and all instances of random variables up to the iteration t. Thus, v_i^t_i is independent to x_j^t from the definition. Thus, x_j^t and ϕ_i^t_iv_i^t_i have zero correlation.




    ϕ_i^t_i-1v_i^t_i-1x_i^t = 1/2(ϕ_i^t_i-1v_i^t_i-1)^2.




    ϕ_i^t_i-1v_i^t_i-1x_i^t
          
    =
     ϕ_i^t_i-1v_i^t_i-1((x_i^t - ϕ_i^t_i-1v_i^t_i-1/2)+ϕ_i^t_i-1v_i^t_i-1/2)
          
    =
    ϕ_i^t_i-1v_i^t_i-1(x_i^t - ϕ_i^t_i-1v_i^t_i-1/2)+1/2(ϕ_i^t_i-1v_i^t_i-1)^2
          (*)=ϕ_i^t_i-1v_i^t_i-1(x_i^t_i-1 + x_l^t_l^0 +w^t_i-1+w^t_l^0 -ϕ_i^t_i-1v_i^t_i-1/2)
     +
     1/2(ϕ_i^t_i-1v_i^t_i-1)^2
          (<ref>)=ϕ_i^t_i-1v_i^t_i-1(x_i^t_i-1 + x_l^t_l^0 +ϕ_i^t_i-1v_i^t_i-1- ϕ_i^t_i-2v_i^t_i-2/2)
               +
    ϕ_i^t_i-1v_i^t_i-1(ϕ_i^t_l^0v_l^t_l^0 - ϕ_i^t_l^0-1v_l^t_l^0-1 -ϕ_i^t_i-1v_i^t_i-1/2)
               +
    1/2(ϕ_i^t_i-1v_i^t_i-1)^2
          
    =
    ϕ_i^t_i-1v_i^t_i-1(x_i^t_i-1 + x_l^t_l^0 +ϕ_i^t_l^0v_l^t_l^0- ϕ_i^t_l^0-1v_l^t_l^0-1-ϕ_i^t_i-2v_i^t_i-2/2)
               
     +1/2(ϕ_i^t_i-1v_i^t_i-1)^2
          L.<ref>=0ϕ_i^t_i-1v_i^t_i-1(x_i^t_i-1 + x_l^t_l^0 +ϕ_i^t_l^0v_l^t_l^0-ϕ_i^t_l^0-1v_l^t_l^0-1-ϕ_i^t_i-2v_i^t_i-2/2)
               +
    1/2(ϕ_i^t_i-1v_i^t_i-1)^2
          
    =
    1/2(ϕ_i^t_i-1v_i^t_i-1)^2,

where in the first equality we add and subtracting ϕ_i^t_i-1v_i^t_i-1/2. In step (*) we denote by l a node such that that the noise ϕ_i^t_i-1v_i^t_i-1 was added to the system when the edge (i,l) was chosen (we do not consider t_i=0 since in this case the Lemma <ref> trivially holds).



    (ϕ_i^t_iv_i^t_i+
    ϕ_j^t_jv_j^t_j)^2
    |x^t,e^t
     = σ^2_iϕ_i^2t_i+σ^2_jϕ_j^2t_j.



Since we have (ϕ_i^t_iv_i^t_i+
ϕ_j^t_jv_j^t_j)|x^t,e^t = 0, and also for any random variable X: X^2 = (X)+X^2, we only need to compute the variance:

    (ϕ_i^t_iv_i^t_i+
    ϕ_j^t_jv_j^t_j)
     = 
     (ϕ_i^t_iv_i^t_i)+(
    ϕ_j^t_jv_j^t_j) 
    =
    (ϕ_i^t_i)^2(v_i^t_i)+(ϕ_j^t_j)^2(
    v_j^t_j) .



Having presented the above lemmas we can now proceed with the proof of Lemma <ref>.

Firstly,  let us compute the increase of the dual function value at iteration t:

    D(y^t+1)-D(y^t)   =   1/2c-x^t ^2-1/2c-x^t+1^2
    
       =   1/2( 
    (c-x^t_j)^2+(c-x^t_i)^2-(c-x^t+1_j)^2-(c-x^t+1_i)^2
    )
    
       =   
    -c(
     x^t+1_j+x^t+1_i-x^t_j-x^t_i
    )
    +1/2(
    (x^t_j)^2+(x^t_i)^2
    -(x^t+1_j)^2-(x^t+1_i)^2
    )
    
       =   
    -c( w_j^t_j+w_i^t_i)
    +1/2(
    (x^t_j)^2+(x^t_i)^2
    )
    
               
    -1/4( 
    (x_j^t+x_i^t  )^2
    + 
    2(x_j^t+x_i^t )( w_j^t_j+w_i^t_i)
    +( w_j^t_j+w_i^t_i)^2
    )
    
       =   1/4(x_j^t-x_i^t)^2-
    (w_i^t_i+w_j^t_j)
    (    
    c+1/2(x_j^t+x_i^t)
    )
    -1/4(w_i^t_i+w_j^t_j)^2.

Our goal is to estimate an upper bound of the quantity D(y^t+1) - D(y^t). There are three terms in (<ref>). Since the expectation is linear, we will evaluate the expectations of these three terms separately and merge them at the end. 

Taking the expectation over the choice of edge and inserted noise in iteration t we obtain

    1/4(x_i^t - x_j^t)^2|x^t = 1/4m∑_e∈(x_i^t-x_j^t)^2.

Thus we have

    D(y^t+1)-D(y^t) - 1/4(x_i^t-x_j^t)^2  |  x^t
              (<ref>)=D(y^t+1)-D(y^t)  |  x^t - 1/4m∑_e∈(x_i^t-x_j^t)^2
    
              
    =
    D(y^*)-D(y^t) |  x^t +D(y^t+1)-D(y^*) |  x^t  - 1/4m∑_e∈(x_i^t-x_j^t)^2
    
              (<ref>)=1/2c-x^t ^2
    -1/2c-x^t+1^2  |  x^t
    - 1/4m∑_e=(i,j)∈(x_i^t-x_j^t)^2
    
              (<ref>)≤1/2c-x^t ^2
    -1/2c-x^t+1^2  |  x^t
    - ()/4mc-x^t ^2
    
              
    =
    (1-()/2m)1/2c-x^t ^2
    -1/2c-x^t+1^2  |  x^t
              (<ref>)=(1-()/2m)( D(y^*)- D(y^t) )
    -D(y^*)- D(y^t+1)   |  y^t.

Taking the full expectation of the above and using tower property, we get

    D(y^t+1)-D(y^t) - 1/4(x_i^t-x_j^t)^2≤( 1-()/2m)D(y^*)- D(y^t) 
    - D(y^*)- D(y^t+1) .


Now we are going to take the expectation of the second term of (<ref>). We will use the “tower rule" of expectations  in the form

    X |   Y,Z | Y = X

where X,Y,Z are random variables. In particular, we get

    -(w_i^t_i+w_j^t_j)
    (    
    c+1/2(x_j^t+x_i^t)
    )
              =-(w_i^t_i+w_j^t_j)
    (    
    c+1/2(x_j^t+x_i^t)
    )  |   e^t,x^t |   x^t.

In equation (<ref>), e^t denotes the edge selected at iteration t.

Let us first calculate the inner most expectation of the right hand side of (<ref>):

    -(w_i^t_i+w_j^t_j)
    (    
    c+1/2(x_j^t+x_i^t)
    ) |  e^t,x^t
          (<ref>)=(ϕ_i^t_i-1v_i^t_i-1+
    ϕ_j^t_j-1v_j^t_j-1- ϕ_i^t_iv_i^t_i-
    ϕ_j^t_jv_j^t_j)(    
    c+1/2(x_j^t+x_i^t)
    ) |  e^t,x^t
          
    =
    (ϕ_i^t_i-1v_i^t_i-1+
    ϕ_j^t_j-1v_j^t_j-1)^constant(    
    c+1/2(x_j^t+x_i^t
    )
    )^constant |  e^t,x^t
               
    +(- ϕ_i^t_iv_i^t_i-
    ϕ_j^t_jv_j^t_j)
    (    
    c+1/2(x_j^t+x_i^t)
    )^constant |  e^t,x^t
          
    =
    (ϕ_i^t_i-1v_i^t_i-1+
    ϕ_j^t_j-1v_j^t_j-1)(    
    c+1/2(x_j^t+x_i^t)
    )
    
              
    +0(- ϕ_i^t_iv_i^t_i-
    ϕ_j^t_jv_j^t_j)|e,x^t(    
    c+1/2(x_j^t+x_i^t)
    )
    
          L. <ref>=(ϕ_i^t_i-1v_i^t_i-1+
    ϕ_j^t_j-1v_j^t_j-1)(    
    c+1/2(x_j^t+x_i^t)
    )
    
          
    =
    (ϕ_i^t_i-1v_i^t_i-1+
    ϕ_j^t_j-1v_j^t_j-1)   
    c
    +1/2(ϕ_i^t_i-1v_i^t_i-1+
    ϕ_j^t_j-1v_j^t_j-1)  (x_j^t+x_i^t
    ).
  

Now we take the expectation of the last expression above with respect to the choice of an edge at t-th iteration. We obtain

    (ϕ_i^t_i-1v_i^t_i-1+
    ϕ_j^t_j-1v_j^t_j-1)   
    c
    +1/2(ϕ_i^t_i-1v_i^t_i-1+
    ϕ_j^t_j-1v_j^t_j-1)  (x_j^t+x_i^t
    )|x^t
          
    =
    1/2(ϕ_i^t_i-1v_i^t_i-1+
    ϕ_j^t_j-1v_j^t_j-1)  (x_j^t+x_i^t) |x^t
              
    + 0(ϕ_i^t_i-1v_i^t_i-1+ϕ_j^t_j-1v_j^t_j-1)  c |x^t
          L. <ref>=1/2(ϕ_i^t_i-1v_i^t_i-1+
    ϕ_j^t_j-1v_j^t_j-1)  (x_j^t+x_i^t) |x^t
          
    =
    1/2m∑_e∈(ϕ_i^t_i-1v_i^t_i-1+
    ϕ_j^t_j-1v_j^t_j-1)  (x_j^t+x_i^t) 
    
          
    =
    1/2m∑_e∈( ϕ_i^t_i-1v_i^t_i-1 x_i^t +ϕ_j^t_j-1v_j^t_j-1 x_j^t)
    
              
    +1/2m∑_e∈( ϕ_i^t_i-1v_i^t_i-1 x_j^t+ ϕ_j^t_j-1v_j^t_j-1 x_i^t
    )
    
          
    =
    1/2m∑_i=1^n d_i ϕ_i^t_i-1v_i^t_i-1 x_i^t
    +1/2m∑_e∈( ϕ_i^t_i-1v_i^t_i-1 x_j^t+ ϕ_j^t_j-1v_j^t_j-1 x_i^t
    ),

where in the last step we change the summation order.

Taking the expectation with respect to the algorithm we obtain

    -(w_i^t_i+w_j^t_j)
    (   
    c+1/2(x_j^t+x_i^t)
    )
          (<ref>)=-(w_i^t_i+w_j^t_j)(   
    c+1/2(x_j^t+x_i^t)
    ) |x^t,e|x^t
           
    =
    1/2m∑_i=1^n d_i ϕ_i^t_i-1v_i^t_i-1 x_i^t
    + 1/2m∑_e∈( ϕ_i^t_i-1v_i^t_i-1 x_j^t+ ϕ_j^t_j-1v_j^t_j-1 x_i^t
    )
    
           
    =
     1/2m∑_i=1^n d_i ϕ_i^t_i-1v_i^t_i-1 x_i^t
    + 1/2m∑_e∈( ϕ_i^t_i-1v_i^t_i-1 x_j^t+ ϕ_j^t_j-1v_j^t_j-1 x_i^t
    )
    
          L.<ref>=1/4m∑_i=1^n d_i (ϕ_i^t_i-1v_i^t_i-1)^2 
    + 1/2m∑_e∈( ϕ_i^t_i-1v_i^t_i-1 x_j^t+ ϕ_j^t_j-1v_j^t_j-1 x_i^t
    )
    .


Taking an expectation of the third term of (<ref>) with respect to lastly added noise, the expression (w_i^t_i+w_j^t_j)^2 |x^t,e^t is equal to

    (w_i^t_i+w_j^t_j)^2 |x^t,e^t   (<ref>)=   (
    ϕ_i^t_iv_i^t_i+
    ϕ_j^t_jv_j^t_j
    -
    ϕ_i^t_i-1v_i^t_i-1-
    ϕ_j^t_j-1v_j^t_j-1)^2 |x^t,e^t
            
    =
    (ϕ_i^t_iv_i^t_i+
    ϕ_j^t_jv_j^t_j)^2
    |x^t,e^t
    
             -
    2
    (ϕ_i^t_iv_i^t_i+ϕ_j^t_jv_j^t_j)
    ^ϕ_i^t_iv_i^t_i+ϕ_j^t_jv_j^t_j=0(ϕ_i^t_i-1v_i^t_i-1+ϕ_j^t_j-1v_j^t_j-1)
    ^constant
    |x^t,e^t
    
              +
     (    
    ϕ_i^t_i-1v_i^t_i-1+
    ϕ_j^t_j-1v_j^t_j-1)^2
    |x^t,e^t^constant
          (<ref>)=σ_i^2ϕ_i^2t_i+σ^2_jϕ_j^2t_j+(
    ϕ_i^t_i-1v_i^t_i-1+
    ϕ_j^t_j-1v_j^t_j-1)^2.

Taking the expectation over e^t we obtain:

    (w_i^t_i+w_j^t_j)^2 |x^t   (<ref>)=   (w_i^t_i+w_j^t_j)^2 |e^t, x^t|x^t
       =   σ_i^2ϕ_i^2t_i+σ^2_jϕ_j^2t_j
    +(
    ϕ_i^t_i-1v_i^t_i-1+
    ϕ_j^t_j-1v_j^t_j-1)^2
    |x^t
       =   1/m∑_e ∈( σ_i^2ϕ_i^2t_i+σ^2_jϕ_j^2t_j)
             +
    1/m∑_e∈(
    ϕ_i^t_i-1v_i^t_i-1+
    ϕ_j^t_j-1v_j^t_j-1)^2
    
       =   1/m∑_i=1^n d_iσ^2_i ϕ_i^2t_i+
    1/m∑_e∈(
    ϕ_i^t_i-1v_i^t_i-1+
    ϕ_j^t_j-1v_j^t_j-1)^2.

where in the last step we change the summation order.
 
Finally, taking the expectation with respect to the algorithm we get

    (w_i^t_i+w_j^t_j)^2 
          (<ref>)=(w_i^t_i+w_j^t_j)^2|x^t 
          
    =
    1/m∑_i=1^n d_iσ^2_i ϕ_i^2t_i+
    1/m∑_e∈(
    ϕ_i^t_i-1v_i^t_i-1+
    ϕ_j^t_j-1v_j^t_j-1)^2
          
    =
    1/m∑_i=1^n d_iσ^2_i ϕ_i^2t_i
              +
    1/m∑_e∈(ϕ_i^t_i-1v_i^t_i-1)^2
    +
    (ϕ_j^t_j-1v_j^t_j-1)^2
    + 2ϕ_i^t_i-1v_i^t_i-1ϕ_j^t_j-1v_j^t_j-1
          (*)=1/m∑_i=1^n d_iσ^2_i ϕ_i^2t_i+
    1/m∑_i=1^n d_i(ϕ_i^t_i-1v_i^t_i-1)^2
    + 2/m∑_e∈ϕ_i^t_i-1v_i^t_i-1ϕ_j^t_j-1v_j^t_j-1
          L.<ref>=1/m∑_i=1^n d_iσ^2_i ϕ_i^2t_i+
    1/m∑_i=1^n d_i(ϕ_i^t_i-1v_i^t_i-1)^2
              + 2/m∑_e∈0ϕ_i^t_i-1v_i^t_i-10ϕ_j^t_j-1v_j^t_j-1
          
    =
    1/m∑_i=1^n d_iσ^2_i ϕ_i^2t_i+
    1/m∑_i=1^n d_i(ϕ_i^t_i-1v_i^t_i-1)^2
    ,

where in step (*) we change the summation order.

Combining (<ref>) with (<ref>), (<ref>) and (<ref>) we obtain

    D(y^*)- D(y^t+1)    ≤   ( 1-()/2m)D(y^*)- D(y^t) 
    -1/4m∑_i=1^n d_i (ϕ_i^t_i-1v_i^t_i-1)^2 
              - 1/2m∑_e∈( ϕ_i^t_i-1v_i^t_i-1 x_j^t+ ϕ_j^t_j-1v_j^t_j-1 x_i^t
    )
              +1/4m∑_i=1^n d_i σ^2_i ϕ_i^2t_i+
    1/4m∑_i=1^n d_i(ϕ_i^t_i-1v_i^t_i-1)^2
       =   ( 1-()/2m)D(y^*)- D(y^t) +1/4m∑_i=1^n d_iσ^2_i ϕ_i^2t_i
              -
    1/2m∑_e∈( ϕ_i^t_i-1v_i^t_i-1 x_j^t+ ϕ_j^t_j-1v_j^t_j-1 x_i^t
    ),

which concludes the proof.


 §.§ Proof of Theorem <ref>


Let us present two lemmas that we use in the proof of Theorem <ref>.

After t iterations of algorithm  we have

    ϕ_i^2t_i=(1-d_i/m(1-ϕ_i^2))^t.





    ϕ_i^2t_i   =   ∑_j=0^t (t_i=j)ϕ_i^2j   =   ∑_j=0^ttj(m-d_i/m)^t-j(d_i/mϕ_i^2)^j
    
       =   (m-d_i/m +d_i/mϕ_i^2)^t   =   (1-d_i/m(1-ϕ_i^2))^t.



Random variables ϕ_i^t_i-1v_i^t_i-1 and x_j^t are nonegatively correlated, i.e. 

    ϕ_i^t_i-1v_i^t_i-1x_j^t≥ 0.




Denote R_i,j to be a random variable equal to 1 if the noise w_i^t_i was added to the system when edge (i,j) was chosen and equal to 0 otherwise. We can rewrite the expectation in the following way:

    ϕ_i^t_i-1v_i^t_i-1x_j^t   =   ϕ_i^t_i-1v_i^t_i-1x_j^t  |  R_i,j=1^≥ 0(R_i,j=1)
             +ϕ_i^t_i-1v_i^t_i-1x_j^t  |  R_i,j=0^0(R_i,j=0)   ≥   0.

The inequality  ϕ_i^t_i-1v_i^t_i-1x_j^t  |  R_i,j=1≥ 0
holds due to the fact that ϕ_i^t_i-1v_i^t_i-1 was added to x_j with the positive sign.

Combining (<ref>) with the results of Lemmas <ref> and <ref> we obtain:

    D(y^*)- D(y^t+1)    (<ref>)≤   ( 1-()/2m)D(y^*)- D(y^t) +1/4m∑_i=1^n d_i σ^2_i ϕ_i^2t_i
              -
    1/2m∑_e∈( ϕ_i^t_i-1v_i^t_i-1 x_j^t+ ϕ_i^t_j-1v_j^t_j-1 x_i^t
    )
       (<ref>)≤   ( 1-()/2m)D(y^*)- D(y^t) +1/4m∑_i=1^n d_iσ^2_i  ϕ_i^2t_i
       (<ref>)=   ( 1-()/2m)D(y^*)- D(y^t)
              +
    1/4m∑_i=1^n d_iσ^2_i  (1-d_i/m(1-ϕ_i^2) )^t
       =   ( 1-()/2m)D(y^*)- D(y^t) +∑(d_iσ_i^2)/4mψ^t.


The recursion above gives us inductively the following

    D(y^*)- D(y^k) ≤ρ^k ( D(y^*)- D(y^0) )  + ∑(d_iσ_i^2)/4m∑_t=1^k ρ^k-tψ^t,

which concludes the proof of the theorem.


 §.§ Proof of Corollary <ref>


Note that we have

    ψ^t
       =   1/∑_i=1^n d_iσ_i^2 ∑_i=1^n d_i σ^2_i (1-d_i/m(1-(1-γ/d_i)) )^t
       =   1/∑_i=1^n d_iσ_i^2 ∑_i=1^n d_i σ^2_i(1-γ/m)^t   =   (1-γ/m)^t.


In view of Theorem <ref>, this gives us the following:

    D(y^*)- D(y^k)    ≤   ( D(y^*)- D(y^0) ) ρ^k + ∑(d_iσ_i^2)/4m∑_t=1^k ρ^k-tψ^t
       ≤   ( D(y^*)- D(y^0) ) ρ^k + ∑(d_iσ_i^2)/4m∑_t=1^k ρ^k-t(1-γ/m)^t
       ≤   ( D(y^*)- D(y^0) ) ρ^k + ∑(d_iσ_i^2)/4mk max(ρ,1-γ/m)^k
    
       ≤   ( D(y^*)- D(y^0)+∑(d_iσ_i^2)/4mk ) max(ρ,1-γ/m)^k.




CHAPTER: CONCLUSION AND FUTURE WORK




§ CONCLUSIONS

In this thesis we studied the design and analysis of novel efficient randomized iterative methods for solving large scale linear systems, stochastic quadratic optimization problems, the best approximation problem and quadratic optimization problems. Using these methods we also proposed and analyzed efficient gossip protocols for solving the average consensus problem on large scale networks.

In Chapter <ref>, we studied the convergence of several stochastic optimization algorithms enriched with heavy ball momentum for solving  stochastic optimization problems of special structure.  We proved global, non-asymptotic linear convergence rates of all of these methods as well as accelerated linear rate for the case of the norm of expected iterates. We also introduced a new momentum strategy called stochastic momentum which is beneficial in the case of sparse data, and proved linear convergence in this setting. We corroborated our theoretical results with extensive experimental testing.

In Chapter <ref>, we proposed and analyzed inexact variants of several stochastic algorithms for solving quadratic optimization problems and linear systems. We provided linear convergence rate under several assumptions on the inexactness error. The proposed methods require more iterations than their exact variants to achieve the same accuracy. However, as we show through our numerical evaluations, the inexact algorithms require significantly less time to converge. 

In Chapter <ref>, we presented a general framework for the analysis and design of randomized gossip algorithms for solving the average consensus problem.  Using tools from numerical linear algebra and the area of randomized projection methods for solving linear systems, we proposed novel serial, block and accelerated gossip protocols for solving the average consensus and weighted average consensus problems. 

In Chapter <ref>, we addressed the average consensus problem via novel asynchronous privacy preserving randomized gossip algorithms. In particular, we proposed three different algorithmic tools for the protection of the initial private values of the nodes. The first two proposed algorithms “Private Gossip via Binary Oracle" and “Private Gossip via  ϵ-Gap Oracle" are based on the same idea of weakening the oracle used in the gossip update rule. Instead of sharing their exact values, in these two protocols the chosen pair of nodes of each gossip step provide only categorical (or even binary) information to each other.  In the third protocol, “Private Gossip via Controlled Noise Insertion", we systematically inject and withdraw noise throughout the iterations, so as to ensure convergence to the average consensus value, and at the same time protect the private information of the nodes. For all proposed protocols, we provide explicit convergence rates and evaluate practical convergence on common simulated network topologies.  



§ FUTURE WORK


Perhaps the most exciting direction for future work is to extend the analysis of the proposed randomized iterative methods to more general settings. In particular, the more natural extension of our results is the analysis of heavy ball momentum variants and inexact variants of the proposed methods (SGD, SN, SPP, SPM and SDSA) in the case of general convex or strongly convex functions. 

From numerical linear algebra viewpoint, we believe that the proposed randomized iterative methods of Chapters <ref> and <ref> could have great potential to make a practical difference to iterative solvers for large-scale linear systems. In this aspect, a future effort needs to be devoted to the practical development and implementations of the algorithms. For example, one promising direction is to use new sophisticated sketching matrices , such as the Walsh-Hadamard matrix <cit.> in the update rules of the proposed methods.

In this thesis we focused on algorithms with a fixed constant step-size. An interesting extension will be to study the effect of decreasing or adaptive choice for the relaxation parameter. This might provide novel insights, even in the case of quadratic functions and  (not necessarily consistent) linear systems. As we have mentioned in several parts of the thesis, the obtained results hold under the exactness condition, which as we explained, is very weak, allowing for virtually arbitrary distributions  from which the random matrices are drawn. A different future direction will be the design of optimized distributions in order to improve further the convergence rates and the overall complexity of the proposed algorithms. 

In addition, we believe that the gossip protocols proposed in Chapters <ref> and <ref> would be particularly useful in the development of efficient decentralized protocols.

Using the novel framework presented in this thesis, many popular projection methods can be interpreted as gossip algorithms when used to solve linear systems encoding the underlying network. This can lead to the development of novel distributed protocols for average consensus.  

Our work on gossip algorithms is amenable to further extensions. For instance, the proposed novel gossip protocols (block, accelerated, privacy-preserving) can be extended to the more general setting of multi-agent consensus optimization, where the goal is to minimize the average of convex or non-convex functions (1/n) ∑_i=1^n f_i(x) in a decentralized way. Such protocols will be particularly useful in settings where the data describing a given optimization problem is so big that it becomes impossible to store it on a single machine. These situations often arise in modern machine learning and deep learning applications.

plain




CHAPTER: NOTATION GLOSSARY



§ NOTATION USED IN CHAPTERS <REF>, <REF> AND <REF>









§ NOTATION USED IN CHAPTER <REF>









§ NOTATION USED IN CHAPTER <REF>





