Title: Random active path model of deep neural networks with diluted binary synapses,
Abstract: Deep learning has become a powerful and popular tool for a variety of machine
learning tasks. However, it is challenging to understand the mechanism of deep
learning from a theoretical perspective. In this work, we propose a random
active path model to study collective properties of deep neural networks with
binary synapses, under the removal perturbation of connections between layers.
In the model, the path from input to output is randomly activated, and the
corresponding input unit constrains the weights along the path into the form of
a $p$-weight interaction glass model. A critical value of the perturbation is
observed to separate a spin glass regime from a paramagnetic regime, with the
transition being of the first order. The paramagnetic phase is conjectured to
have a poor generalization performance.