Are deep learning models way over the required capacity for their datasets' estimated entropies?
The current thinking is that it is easier to fit an overparameterized neural network, since the local extrema are different ways of expressing the same thing, whereas in a minimal neural network you have to worry about getting to the global extremum: 


  The subtle reason behind this is that smaller networks are harder to
  train with local methods such as Gradient Descent: It’s clear that
  their loss functions have relatively few local minima, but it turns
  out that many of these minima are easier to converge to, and that they
  are bad (i.e. with high loss). Conversely, bigger neural networks
  contain significantly more local minima, but these minima turn out to
  be much better in terms of their actual loss. Since Neural Networks
  are non-convex, it is hard to study these properties mathematically,
  but some attempts to understand these objective functions have been
  made, e.g. in a recent paper The Loss Surfaces of Multilayer Networks.
  In practice, what you find is that if you train a small network the
  final loss can display a good amount of variance - in some cases you
  get lucky and converge to a good place but in some cases you get
  trapped in one of the bad minima. On the other hand, if you train a
  large network you’ll start to find many different solutions, but the
  variance in the final achieved loss will be much smaller. In other
  words, all solutions are about equally as good, and rely less on the
  luck of random initialization.


CS231n Convolutional Neural Networks for Visual Recognition