Title: Control Interpretations for First-Order Optimization Methods,
Abstract: First-order iterative optimization methods play a fundamental role in large
scale optimization and machine learning. This paper presents control
interpretations for such optimization methods. First, we give loop-shaping
interpretations for several existing optimization methods and show that they
are composed of basic control elements such as PID and lag compensators. Next,
we apply the small gain theorem to draw a connection between the convergence
rate analysis of optimization methods and the input-output gain computations of
certain complementary sensitivity functions. These connections suggest that
standard classical control synthesis tools may be brought to bear on the design
of optimization algorithms.