Title: Robust Decentralized Learning Using ADMM with Unreliable Agents,
Abstract: Many machine learning problems can be formulated as consensus optimization
problems which can be solved efficiently via a cooperative multi-agent system.
However, the agents in the system can be unreliable due to a variety of
reasons: noise, faults and attacks. Providing erroneous updates leads the
optimization process in a wrong direction, and degrades the performance of
distributed machine learning algorithms. This paper considers the problem of
decentralized learning using ADMM in the presence of unreliable agents. First,
we rigorously analyze the effect of erroneous updates (in ADMM learning
iterations) on the convergence behavior of multi-agent system. We show that the
algorithm linearly converges to a neighborhood of the optimal solution under
certain conditions and characterize the neighborhood size analytically. Next,
we provide guidelines for network design to achieve a faster convergence. We
also provide conditions on the erroneous updates for exact convergence to the
optimal solution. Finally, to mitigate the influence of unreliable agents, we
propose \textsf{ROAD}, a robust variant of ADMM, and show its resilience to
unreliable agents with an exact convergence to the optimum.