Title: Evolution in Groups: A deeper look at synaptic cluster driven evolution of deep neural networks,
Abstract: A promising paradigm for achieving highly efficient deep neural networks is
the idea of evolutionary deep intelligence, which mimics biological evolution
processes to progressively synthesize more efficient networks. A crucial design
factor in evolutionary deep intelligence is the genetic encoding scheme used to
simulate heredity and determine the architectures of offspring networks. In
this study, we take a deeper look at the notion of synaptic cluster-driven
evolution of deep neural networks which guides the evolution process towards
the formation of a highly sparse set of synaptic clusters in offspring
networks. Utilizing a synaptic cluster-driven genetic encoding, the
probabilistic encoding of synaptic traits considers not only individual
synaptic properties but also inter-synaptic relationships within a deep neural
network. This process results in highly sparse offspring networks which are
particularly tailored for parallel computational devices such as GPUs and deep
neural network accelerator chips. Comprehensive experimental results using four
well-known deep neural network architectures (LeNet-5, AlexNet, ResNet-56, and
DetectNet) on two different tasks (object categorization and object detection)
demonstrate the efficiency of the proposed method. Cluster-driven genetic
encoding scheme synthesizes networks that can achieve state-of-the-art
performance with significantly smaller number of synapses than that of the
original ancestor network. ($\sim$125-fold decrease in synapses for MNIST).
Furthermore, the improved cluster efficiency in the generated offspring
networks ($\sim$9.71-fold decrease in clusters for MNIST and a $\sim$8.16-fold
decrease in clusters for KITTI) is particularly useful for accelerated
performance on parallel computing hardware architectures such as those in GPUs
and deep neural network accelerator chips.