Title: Beyond Backprop: Online Alternating Minimization with Auxiliary Variables,
Abstract: We propose a novel online alternating minimization (AltMin) algorithm for
training deep neural networks, provide theoretical convergence guarantees and
demonstrate its advantages on several classification tasks as compared both to
standard backpropagation with stochastic gradient descent (backprop-SGD) and to
offline alternating minimization. The key difference from backpropagation is an
explicit optimization over hidden activations, which eliminates gradient chain
computation in backprop, and breaks the weight training problem into
independent, local optimization subproblems; this allows to avoid vanishing
gradient issues, simplify handling non-differentiable nonlinearities, and
perform parallel weight updates across the layers. Moreover, parallel local
synaptic weight optimization with explicit activation propagation is a step
closer to a more biologically plausible learning model than backpropagation,
whose biological implausibility has been frequently criticized. Finally, the
online nature of our approach allows to handle very large datasets, as well as
continual, lifelong learning, which is our key contribution on top of recently
proposed offline alternating minimization schemes (e.g., (Carreira-Perpinan
andWang 2014), (Taylor et al. 2016)).