Title: Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters,
Abstract: Deep learning models can take weeks to train on a single GPU-equipped
machine, necessitating scaling out DL training to a GPU-cluster. However,
current distributed DL implementations can scale poorly due to substantial
parameter synchronization over the network, because the high throughput of GPUs
allows more data batches to be processed per unit time than CPUs, leading to
more frequent network synchronization. We present Poseidon, an efficient
communication architecture for distributed DL on GPUs. Poseidon exploits the
layered model structures in DL programs to overlap communication and
computation, reducing bursty network communication. Moreover, Poseidon uses a
hybrid communication scheme that optimizes the number of bytes required to
synchronize each layer, according to layer properties and the number of
machines. We show that Poseidon is applicable to different DL frameworks by
plugging Poseidon into Caffe and TensorFlow. We show that Poseidon enables
Caffe and TensorFlow to achieve 15.5x speed-up on 16 single-GPU machines, even
with limited bandwidth (10GbE) and the challenging VGG19-22K network for image
classification. Moreover, Poseidon-enabled TensorFlow achieves 31.5x speed-up
with 32 single-GPU machines on Inception-V3, a 50% improvement over the
open-source TensorFlow (20x speed-up).