Network consensus optimization has received increasing attention in recent years and has found important applications in many scientific and engineering fields. To solve network consensus optimization problems, one of the most well-known approaches is the distributed gradient descent method (DGD). However, in networks with slow communication rates, DGD's performance is unsatisfactory for solving high-dimensional network consensus problems due to the communication bottleneck. This motivates us to design a communication-efficient DGD-type algorithm based on compressed information exchanges. Our contributions in this paper are three-fold: i) We develop a communication-efficient algorithm called amplified-differential compression DGD (ADC-DGD) and show that it converges under any unbiased compression operator; ii) We rigorously prove the convergence performances of ADC-DGD and show that they match with those of DGD without compression; iii) We reveal an interesting phase transition phenomenon in the convergence speed of ADC-DGD. Collectively, our findings advance the state-of-the-art of network consensus optimization theory.
@cite_14: Recently, there has been growing interest in solving consensus optimization problems in a multiagent network. In this paper, we develop a decentralized algorithm for the consensus optimization problem @math which is defined over a connected network of @math agents, where each function @math is held privately by agent @math and encodes the agent's data and objective. All the agents shall collaboratively find the minimizer while each agent can only communicate with its neighbors. Such a computation scheme avoids a data fusion center or long-distance communication and offers better load balance to the network. This paper proposes a novel decentralized exact first-order algorithm (abbreviated as EXTRA) to solve the consensus optimization problem. “Exact” means that it can converge to the exact solution. EXTRA uses a fixed, large step size, which can be determined independently of the network size or topology. The local variable of every a...
@cite_7: Consider the consensus problem of minimizing @math , where @math and each @math is only known to the individual agent @math in a connected network of @math agents. To solve this problem and obtain the solution, all the agents collaborate with their neighbors through information exchange. This type of decentralized computation does not need a fusion center, offers better network load balance, and improves data privacy. This paper studies the decentralized gradient descent method [A. Nedic and A. Ozdaglar, IEEE Trans. Automat. Control, 54 (2009), pp. 48--61], in which each agent @math updates its local variable @math by combining the average of its neighbors' with a local negative-gradient step @math . The method is described by the iteration @math where @math is nonzero only if @math and @math are neighbors or @math and the matrix...
@cite_23: Consensus optimization has received considerable attention in recent years. A number of decentralized algorithms have been proposed for convex consensus optimization. However, to the behaviors or consensus optimization, our understanding is more limited. When we lose convexity, we cannot hope our algorithms always return global solutions though they sometimes still do sometimes. Somewhat surprisingly, the decentralized consensus algorithms, DGD and Prox-DGD, retain most other properties that are known in the convex setting. In particular, when diminishing (or constant) step sizes are used, we can prove convergence to a (or a neighborhood of) consensus stationary solution under some regular assumptions. It is worth noting that Prox-DGD can handle nonconvex nonsmooth functions if their proximal operators can be computed. Such functions include SCAD and @math quasi-norms, @math . Similarly, Prox-DGD can take the constraint to a nonconvex set with an easy projection. To establish these properties, we have to introduce a completely different line of analysis, as well as modify existing proofs that were used the convex setting.
@cite_25: This paper considers the problem of distributed optimization over time-varying graphs. For the case of undirected graphs, we introduce a distributed algorithm, referred to as DIGing, based on a combination of a distributed inexact gradient method and a gradient tracking technique. The DIGing algorithm uses doubly stochastic mixing matrices and employs fixed step-sizes and, yet, drives all the agents' iterates to a global and consensual minimizer. When the graphs are directed, in which case the implementation of doubly stochastic mixing matrices is unrealistic, we construct an algorithm that incorporates the push-sum protocol into the DIGing structure, thus obtaining the Push-DIGing algorithm. Push-DIGing uses column stochastic matrices and fixed step-sizes, but it still converges to a global and consensual minimizer. Under the strong convexity assumption, we prove that the algorithms converge at R-linear (geometric) rates as long as the step-sizes do not exceed some upper bounds. We establish explicit est...
@cite_12: 
@cite_17: Abstract : This thesis investigates certain fundamental problems in decentralized decision making and computation. We study the problem of whether a set of decision makers (or processors) with different (but related) information may make compatible decisions without communication and we characterize the computational complexity of this problem. We also analyze the complexity of other basic problems of decentralized decision making, such as decentralized detection (hypothesis testing). We then consider a scheme whereby a set of decision makers (processors) exchange and update tentative decisions which minimize a common cost function, given information they possess; we show that they are guaranteed to converge to consensus. Finally, we consider a broad class of asynchronous distributed (deterministic and stochastic) iterative optimization algorithms tolerating communication delays. We associate communication requirements with such algorithms and show that they converge appropriately under certain conditions which are no more severe than those required for their centralized counterparts. Several applications in human organizations, parallel computation and distributed signal processing are indicated. Additional keywords: decision theory. (Author).
1) DGD-Based Algorithms for Network Consensus: Network consensus optimization can trace its roots to the seminal work by Tsitsiklis @cite_17 , where the system model and the analysis framework were first developed. As mentioned earlier, a well-known method for solving network consensus optimization is the distributed (sub)gradient descent (DGD) method, which was proposed by Nedic and Ozdaglar in @cite_12 . DGD was recently reexamined in @cite_7 by Yuan using a new Lyapunov technique, which offers further mathematical understanding of its convergence performance. In their follow-up work @cite_23 , the convergence behavior of DGD was further analyzed for non-convex problems. Recently, several DGD variants have been proposed to enhance the convergence performance (e.g., achieving the same @math convergence rate with constant step-size @cite_14 or even under time-varying network graphs @cite_25 ).