Synthesis and Pruning as a Dynamic Compression Strategy for Efficient
  Deep Neural Networks
  The brain is a highly reconfigurable machine capable of task-specific
adaptations. The brain continually rewires itself for a more optimal
configuration to solve problems. We propose a novel strategic synthesis
algorithm for feedforward networks that draws directly from the brain's
behaviours when learning. The proposed approach analyses the network and ranks
weights based on their magnitude. Unlike existing approaches that advocate
random selection, we select highly performing nodes as starting points for new
edges and exploit the Gaussian distribution over the weights to select
corresponding endpoints. The strategy aims only to produce useful connections
and result in a smaller residual network structure. The approach is
complemented with pruning to further the compression. We demonstrate the
techniques to deep feedforward networks. The residual sub-networks that are
formed from the synthesis approaches in this work form common sub-networks with
similarities up to ~90%. Using pruning as a complement to the strategic
synthesis approach, we observe improvements in compression.
