Title: Critical Learning Periods in Deep Neural Networks,
Abstract: Critical periods are phases in the early development of humans and animals
during which experience can irreversibly affect the architecture of neuronal
networks. In this work, we study the effects of visual stimulus deficits on the
training of artificial neural networks (ANNs). Introducing well-characterized
visual deficits, such as cataract-like blurring, in the early training phase of
a standard deep neural network causes a permanent performance loss that closely
mimics critical period behavior in humans and animal models. Deficits that do
not affect low-level image statistics, such as vertical flipping of the images,
have no lasting effect on the ANNs' performance and can be rapidly overcome
with further training. In addition, the deeper the ANN is, the more pronounced
the critical period. To better understand this phenomenon, we use Fisher
Information as a measure of the strength of the network's connections during
the training. Our information-theoretic analysis suggests that the first few
epochs are critical for the creation of strong connections across different
layers, optimal for processing the input data distribution. Once such strong
connections are created, they do not appear to change during additional
training. These findings suggest that the initial rapid learning phase of ANN
training, under-scrutinized compared to its asymptotic behavior, plays a key
role in defining the final performance of networks. Our results also show how
critical periods are not restricted to biological systems, but can emerge
naturally in learning systems, whether biological or artificial, due to
fundamental constrains arising from learning dynamics and information
processing.