Title: Decoupled Greedy Learning of CNNs,
Abstract: A commonly cited inefficiency of neural network training by back-propagation
is the update locking problem: each layer must wait for the signal to propagate
through the network before updating. We consider and analyze a training
procedure, Decoupled Greedy Learning (DGL), that addresses this problem more
effectively and at scales beyond those of previous solutions. It is based on a
greedy relaxation of the joint training objective, recently shown to be
effective in the context of Convolutional Neural Networks (CNNs) on large-scale
image classification. We consider an optimization of this objective that
permits us to decouple the layer training, allowing for layers or modules in
networks to be trained with a potentially linear parallelization in layers. We
show theoretically and empirically that this approach converges. In addition,
we empirically find that it can lead to better generalization than sequential
greedy optimization and even standard end-to-end back-propagation. We show that
an extension of this approach to asynchronous settings, where modules can
operate with large communication delays, is possible with the use of a replay
buffer. We demonstrate the effectiveness of DGL on the CIFAR-10 datasets
against alternatives and on the large-scale ImageNet dataset, where we are able
to effectively train VGG and ResNet-152 models.