Title: Optimization Landscape and Expressivity of Deep CNNs,
Abstract: We analyze the loss landscape and expressiveness of practical deep
convolutional neural networks (CNNs) with shared weights and max pooling
layers. We show that such CNNs produce linearly independent features at a
"wide" layer which has more neurons than the number of training samples. This
condition holds e.g. for the VGG network. Furthermore, we provide for such wide
CNNs necessary and sufficient conditions for global minima with zero training
error. For the case where the wide layer is followed by a fully connected layer
we show that almost every critical point of the empirical loss is a global
minimum with zero training error. Our analysis suggests that both depth and
width are very important in deep learning. While depth brings more
representational power and allows the network to learn high level features,
width smoothes the optimization landscape of the loss function in the sense
that a sufficiently wide network has a well-behaved loss surface with almost no
bad local minima.