Title: Sparse Neural Networks Topologies,
Abstract: We propose Sparse Neural Network architectures that are based on random or
structured bipartite graph topologies. Sparse architectures provide compression
of the models learned and speed-ups of computations, they can also surpass
their unstructured or fully connected counterparts. As we show, even more
compact topologies of the so-called SNN (Sparse Neural Network) can be achieved
with the use of structured graphs of connections between consecutive layers of
neurons. In this paper, we investigate how the accuracy and training speed of
the models depend on the topology and sparsity of the neural network. Previous
approaches using sparcity are all based on fully connected neural network
models and create sparcity during training phase, instead we explicitly define
a sparse architectures of connections before the training. Building compact
neural network models is coherent with empirical observations showing that
there is much redundancy in learned neural network models. We show
experimentally that the accuracy of the models learned with neural networks
depends on expander-like properties of the underlying topologies such as the
spectral gap and algebraic connectivity rather than the density of the graphs
of connections.