Title: Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach,
Abstract: The Fisher information matrix (FIM) is a fundamental quantity to represent
the characteristics of a stochastic model, including deep neural networks
(DNNs). The present study reveals novel statistics of FIM that are universal
among a wide class of DNNs. To this end, we use random weights and large width
limits, which enables us to utilize mean field theories. We investigate the
asymptotic statistics of the FIM's eigenvalues and reveal that most of them are
close to zero while the maximum takes a huge value. This implies that the
eigenvalue distribution has a long tail. Because the landscape of the parameter
space is defined by the FIM, it is locally flat in most dimensions, but
strongly distorted in others. We also demonstrate the potential usage of the
derived statistics through two exercises. First, small eigenvalues that induce
flatness can be connected to a norm-based capacity measure of generalization
ability. Second, the maximum eigenvalue that induces the distortion enables us
to quantitatively estimate an appropriately sized learning rate for gradient
methods to converge.