Invariances in Neural Networks
The networks generally learn to discriminate without the information of a generative model. This is why, I would say, they tend to learn not to be invariant. This is also the reason why they perform very well at fine-grained classification tasks. Their level of invariance is largely influenced by the data, which is provided. So if the data is augmented so as to cover all rotations, then the network starts to be invariant against rotations. This is also similar for other types of non-trivial transformations. Such a property is desirable, as we do not want to be invariant against the transformations that are not shown to us.

Having said that, they do, intrinsically develop certain invariance. This is mostly due to the pooling mechanism. So if you were to design your pooling method (corresponding to certain norm), some more invariance can be achieved.

Also, scattering transform of Mallat construct translation-invariant operators on $L^2(R^d)$, which are Lipschitz-continuous to the action of diffeomorphisms. They can be made to be theoretically invariant to actions of compact
Lie groups.

Please check Goodfellow et. al. for the degree of invariance, the networks could develop.