Title: Sparse Deep Neural Network Exact Solutions,
Abstract: Deep neural networks (DNNs) have emerged as key enablers of machine learning.
Applying larger DNNs to more diverse applications is an important challenge.
The computations performed during DNN training and inference are dominated by
operations on the weight matrices describing the DNN. As DNNs incorporate more
layers and more neurons per layers, these weight matrices may be required to be
sparse because of memory limitations. Sparse DNNs are one possible approach,
but the underlying theory is in the early stages of development and presents a
number of challenges, including determining the accuracy of inference and
selecting nonzero weights for training. Associative array algebra has been
developed by the big data community to combine and extend database, matrix, and
graph/network concepts for use in large, sparse data problems. Applying this
mathematics to DNNs simplifies the formulation of DNN mathematics and reveals
that DNNs are linear over oscillating semirings. This work uses associative
array DNNs to construct exact solutions and corresponding perturbation models
to the rectified linear unit (ReLU) DNN equations that can be used to construct
test vectors for sparse DNN implementations over various precisions. These
solutions can be used for DNN verification, theoretical explorations of DNN
properties, and a starting point for the challenge of sparse training.