the size of training data set in the context of computer vision
Your second hypothesis is on the right track. Try comparing the information content of the training set with the information content of the network parameters. Of course most images are compressible, but they don't compress down to a single floating-point number, which is how network parameters are usually encoded.