Title: Transportation analysis of denoising autoencoders: a novel method for analyzing deep neural networks,
Abstract: The feature map obtained from the denoising autoencoder (DAE) is investigated
by determining transportation dynamics of the DAE, which is a cornerstone for
deep learning. Despite the rapid development in its application, deep neural
networks remain analytically unexplained, because the feature maps are nested
and parameters are not faithful. In this paper, we address the problem of the
formulation of nested complex of parameters by regarding the feature map as a
transport map. Even when a feature map has different dimensions between input
and output, we can regard it as a transportation map by considering that both
the input and output spaces are embedded in a common high-dimensional space. In
addition, the trajectory is a geometric object and thus, is independent of
parameterization. In this manner, transportation can be regarded as a universal
character of deep neural networks. By determining and analyzing the
transportation dynamics, we can understand the behavior of a deep neural
network. In this paper, we investigate a fundamental case of deep neural
networks: the DAE. We derive the transport map of the DAE, and reveal that the
infinitely deep DAE transports mass to decrease a certain quantity, such as
entropy, of the data distribution. These results though analytically simple,
shed light on the correspondence between deep neural networks and the
Wasserstein gradient flows.