How do you make an NN for image classification invariant to translation **and** rotation?
It is not possible to have general rotationally-invariant neural network architecture for a CNN*. In fact CNNs are not strongly translation invariant, except due to pooling - instead they combine a little bit of translation invariance with translation equivariance. There is no equivalent to pooling layers that would reduce the effect of rotation this way (although for very small rotations the translation invariance will still help).

You can however construct features and create a pipeline that reduces incoming rotation differences in your inputs. For example, this answer on Signal Processing Stack Exchange suggests calculating dominant gradient of an image, then rotating the image so that this is always oriented the same way before further processing. If your image has strong straight edges, you could do similar by detecting those (e.g. by using a Hough transform) and rotating the input so that these are always oriented the same way. These approaches work only within certain image tasks, but can save on processing time and potentially increase accuracy if they are possible. Effectively they are a form of input normalisation.

A more radical idea might be to perform a map to polar co-ordinates in your image before processing. This would effectively convert rotational (and radial) variance into translation variance in your image. A CNN processing this mapped image would effectively convert its translation invariance into rotational invariance on the original unmapped image. But the cost would be losing all translation invariance, so only worth considering if your inputs have high variance in rotation but low variance in translation.



* Never say never. This is caveated by special cases, for example the paper Group Equivariant Convolutional Networks explains an architecture that adds support for multiples of 90 degree rotation (taking advantage of grid structure in computer images and the weight matrices that construct neural network layers). However, if you want to support free rotation of values other than 90, 180, 270 degrees, then as far as I know, there is no way to do that architecturally within the network.