anchor.diff:  [.text-center]
   .All 8 shifts of the `L` shaped filter. Each row contains a clockwise rolled elements from the previous.
   image::alt_neural2/rotations_L.png[rotations_of_filter, 600]
  -{empty} +
   
   Despite the setback, all \(90^{\circ}\) rotations are accurate and \(45^{\circ}\) rotations are only slightly skewed. Skew might be a blessing in disguise as it might endow the neuron to be _deform_ invariant as well (wishful thinking). In either case, the deformations are not too bad for \(3 \times 3\) filters, and it is at-least worth experimenting with vggnet which uses all \(3 \times 3\) conv filters.
   
  
positive.code: [.text-center]
  .Visual representation of \(W\) template
  image::alt_neural2/vertical_line_filter.png[vertical_line_filter, 300]
  {empty} +
  
  When a neuron tries to match this pattern in a \(3 \times 3\) input image patch, it will only output a high value if a vertical edge is found in the center of the image patch. i.e., dot product is not invariant to shifts. My first thought was to use link:https://en.wikipedia.org/wiki/Pearson_product-moment_correlation_coefficient[Pearson coefficient] as it provides shift invariance. However, this not a problem in practice as the filter is slid across all possible \(3 \times 3\) patches within the input image (see fig2).
  
  [.text-center]
  .Sliding filter over input image to generate the output activation area (image borrowed from link:http://intellabs.github.io/RiverTrail/tutorial/[river trail documentation])
  image::alt_neural2/conv_illustration.png[conv_illustration, 400]
  {empty} +
  
  In a way, the sliding window operation is quite clever as it communicates information about _parts of the image_ matching filter template without increasing the number of _trainable_ parameters.  As a concrete example, consider a conv-net with filters as shown in fig3footnote:[First layer filters generally detect basic lines and textures, this is just a simplified view to illustrate the point]. The first layer filters (green) detect eyes, nose etc., second layer filters (yellow) detect face, leg etc., by aggregating features from first layer and so onfootnote:[In reality, convolution filters may detect objects that have no meaningful interpretation].
  
  [.text-center]
  .Hypothetical conv-net filters to illustrate how a _human_ might be detected. (image borrowed from quora link:https://www.quora.com/How-is-a-convolutional-neural-network-able-to-learn-invariant-features[post])
  image::alt_neural2/conv_net1.png[conv_illustration, 600]
  {empty} +
  
  Now suppose the face (represented by two red and one magenta point) is moved to another corner as shown in fig4. The same activations occur, leading to the same end result.
  
  [.text-center]
  .Illustration of activations when an input object is shifted. (image borrowed from quora link:https://www.quora.com/How-is-a-convolutional-neural-network-able-to-learn-invariant-features[post])
  image::alt_neural2/conv_net2.png[conv_illustration, 600]
  {empty} +
  
  So far so good. How about rotational invariance? We know that a lot of learned conv filters are identical, but rotated by some non-random factor. Instead of learning rotational invariance in this manner, we will try to build it directly into the computational model. The motivation is simple - the neuron should output high value even if some rotated version of pattern is present in the input. Hopefully this eliminates redundant filters and improves test time accuracy.
  
  [.text-center]
  .A few conv filters of a vggnet (image borrowed from keras link:https://blog.keras.io/how-convolutional-neural-networks-see-the-world.html[blog post]). Notice how some filters are identical but rotated.
  image::alt_neural2/conv_filters.jpg[conv_filters, 600]
  {empty} +
  
  == Endowing rotational invariance
  
  
negative.code: [.text-center]
  .Prediction probability of the correct output class as a test input image is rotated from \(\theta = \{0, 360\}\) by step of \(1^{\circ}\) across models. Legend shows the percentage correct classifications across rotations (ascending order).
  image::alt_neural2/rotations_all.png[rotations_all, 800]
  
  Instead of checking random images, lets compute these values for 1000 random test images and use a box plot to see _mean, std, min, max_ of percentage correct classifications across all rotations. Interestingly, _8pass:[_]rotpass:[_]4_, _4pass:[_]rotpass:[_]4_ show the best rotational invariance (higher mean, large _std_ in positive direction). It is also somewhat mysterious why _8pass:[_]rotpass:[_]2_, _4pass:[_]rotpass:[_]2_ is slightly worse than `baseline` compared to 1, 3, 4.
  
  [.text-center]
  .Box plot of percentage correct classifications across all rotations for 1000 random test images (in ascending order of _mean_).
  image::alt_neural2/rotations_box_all.png[rotations_box_all, 800]
  
  Can we get the benefits of 8/4 rot models without the slower convergence? What if we applied new conv model (8/4 rotations) to the _baseline_ model weights? Fig 11. shows the results for a single image. It is very curious to note that _8pass:[_]rotpass:[_]4_ and _4pass:[_]rotpass:[_]4_ models yield zero accuracy. This seems to indicate that weights from the usual conv layer won't translate well into rotated conv model (the errors across various layers add up). The error rates seem to recover as we remove rotational conv layers from the early layers, but there is a lot of noise since this is for a single test image.
  
  [.text-center]
  .Prediction probability of the correct output class as a test input image is rotated from \(\theta = \{0, 360\}\) by step of \(1^{\circ}\) across models with _baseline_ trained weights. Legend shows the percentage correct classifications across rotations (ascending order).
  image::alt_neural2/rot_baseline_all.png[rot_baseline_all, 800]
  
  To get a clearer picture, lets compute the box plot by considering 1000 random test images as described earlier. Fig. 12 makes it pretty clear that we cannot just add rotation operations to a _regular_ model after training. On the flip side, it provides evidence that rotational conv affects the weights.
  
  [.text-center]
  .Box plot of percentage correct classifications across all rotations for 1000 random test images (in ascending order of _mean_). Note that all models use trained _baseline_ weights.
  image::alt_neural2/rot_baseline_box_all.png[rot_baseline_box_all, 800]
  
  == Conclusions
  
  