Title: Spectral Dynamics of Learning Restricted Boltzmann Machines,
Abstract: The Restricted Boltzmann Machine (RBM), an important tool used in machine
learning in particular for unsupervized learning tasks, is investigated from
the perspective of its spectral properties. Starting from empirical
observations, we propose a generic statistical ensemble for the weight matrix
of the RBM and characterize its mean evolution. This let us show how in the
linear regime, in which the RBM is found to operate at the beginning of the
training, the statistical properties of the data drive the selection of the
unstable modes of the weight matrix. A set of equations characterizing the
non-linear regime is then derived, unveiling in some way how the selected modes
interact in later stages of the learning procedure and defining a deterministic
learning curve for the RBM.