Title: On the Origin of Deep Learning,
Abstract: This paper is a review of the evolutionary history of deep learning models.
It covers from the genesis of neural networks when associationism modeling of
the brain is studied, to the models that dominate the last decade of research
in deep learning like convolutional neural networks, deep belief networks, and
recurrent neural networks. In addition to a review of these models, this paper
primarily focuses on the precedents of the models above, examining how the
initial ideas are assembled to construct the early models and how these
preliminary models are developed into their current forms. Many of these
evolutionary paths last more than half a century and have a diversity of
directions. For example, CNN is built on prior knowledge of biological vision
system; DBN is evolved from a trade-off of modeling power and computation
complexity of graphical models and many nowadays models are neural counterparts
of ancient linear models. This paper reviews these evolutionary paths and
offers a concise thought flow of how these models are developed, and aims to
provide a thorough background for deep learning. More importantly, along with
the path, this paper summarizes the gist behind these milestones and proposes
many directions to guide the future research of deep learning.