Invariances in Neural Networks I have a question about invariances in neural networks. 


In general, neural nets with enough layers can learn arbitrarily complicated nonlinear functions. Therefore, it's not hard to understand how a neural net can learn a Fourier transform or convolution: (3rd to last paragraph of this page: http://www.dspguide.com/ch26/3.htm).
Fourier transforms have a useful translation (shift) invariance property, i.e. the magnitude of the FT of a function/image remains the same after you translate it.
Other transforms also have useful invariance properties, e.g. the Mellin transform is scale invariant, i.e. the magnitude of the Mellin transform of a function/image remains the same after you scale it.


So my question is about why neural networks don't already have built in invariances. Is it purely a matter of number of neurons? If I have an enormously large number of neurons and/or layers, do I start to get built in invariances? 

(E.g. as described in above link, you need 2 layers of N nodes to learn convolution of an N sample signal. Also, not every single neuron is going to be weighted in such a way to participate in learning convolution, so you'll need many more neurons than just 2N.)
The networks generally learn to discriminate without the information of a generative model. This is why, I would say, they tend to learn not to be invariant. This is also the reason why they perform very well at fine-grained classification tasks. Their level of invariance is largely influenced by the data, which is provided. So if the data is augmented so as to cover all rotations, then the network starts to be invariant against rotations. This is also similar for other types of non-trivial transformations. Such a property is desirable, as we do not want to be invariant against the transformations that are not shown to us.

Having said that, they do, intrinsically develop certain invariance. This is mostly due to the pooling mechanism. So if you were to design your pooling method (corresponding to certain norm), some more invariance can be achieved.

Also, scattering transform of Mallat construct translation-invariant operators on $L^2(R^d)$, which are Lipschitz-continuous to the action of diffeomorphisms. They can be made to be theoretically invariant to actions of compact
Lie groups.

Please check Goodfellow et. al. for the degree of invariance, the networks could develop.