anchor.diff:  [.text-center]
   .Convergence graphs for #{8/4 rotations}_rot_{last 1, 2, 3, 4 conv with new conv layer} and the baseline model. Models in legend are ordered in descending order of accuracy.
   image::alt_neural2/convergence.png[convergence, 800]
  -{empty} +
   
   It seems evident that using rotationally invariant convolutional layers everywhere is not a good idea (_4pass:[_]rotpass:[_]4_ and _8pass:[_]rotpass:[_]4_ have the lowest values). In line with hypothesis 2, the accuracy improved as we get rid of earlier rotationally invariant conv layers, with 4_rot_1 and 8_rot_1 on par/slightly better than the baseline model. The skew (according to hypothesis 3) does not seem to the issue as _8pass:[_]rotpass:[_][4, 3, 2, 1]_ closely followed _4pass:[_]rotpass:[_][4, 3, 2, 1]_ accuracies.
   
  
positive.code: [.text-center]
  .All 8 shifts of the `L` shaped filter. Each row contains a clockwise rolled elements from the previous.
  image::alt_neural2/rotations_L.png[rotations_of_filter, 600]
  {empty} +
  
  Despite the setback, all \(90^{\circ}\) rotations are accurate and \(45^{\circ}\) rotations are only slightly skewed. Skew might be a blessing in disguise as it might endow the neuron to be _deform_ invariant as well (wishful thinking). In either case, the deformations are not too bad for \(3 \times 3\) filters, and it is at-least worth experimenting with vggnet which uses all \(3 \times 3\) conv filters.
  
  
negative.code: [.text-center]
  .Prediction probability of the correct output class as a test input image is rotated from \(\theta = \{0, 360\}\) by step of \(1^{\circ}\) across models. Legend shows the percentage correct classifications across rotations (ascending order).
  image::alt_neural2/rotations_all.png[rotations_all, 800]
  
  Instead of checking random images, lets compute these values for 1000 random test images and use a box plot to see _mean, std, min, max_ of percentage correct classifications across all rotations. Interestingly, _8pass:[_]rotpass:[_]4_, _4pass:[_]rotpass:[_]4_ show the best rotational invariance (higher mean, large _std_ in positive direction). It is also somewhat mysterious why _8pass:[_]rotpass:[_]2_, _4pass:[_]rotpass:[_]2_ is slightly worse than `baseline` compared to 1, 3, 4.
  
  [.text-center]
  .Box plot of percentage correct classifications across all rotations for 1000 random test images (in ascending order of _mean_).
  image::alt_neural2/rotations_box_all.png[rotations_box_all, 800]
  
  Can we get the benefits of 8/4 rot models without the slower convergence? What if we applied new conv model (8/4 rotations) to the _baseline_ model weights? Fig 11. shows the results for a single image. It is very curious to note that _8pass:[_]rotpass:[_]4_ and _4pass:[_]rotpass:[_]4_ models yield zero accuracy. This seems to indicate that weights from the usual conv layer won't translate well into rotated conv model (the errors across various layers add up). The error rates seem to recover as we remove rotational conv layers from the early layers, but there is a lot of noise since this is for a single test image.
  
  [.text-center]
  .Prediction probability of the correct output class as a test input image is rotated from \(\theta = \{0, 360\}\) by step of \(1^{\circ}\) across models with _baseline_ trained weights. Legend shows the percentage correct classifications across rotations (ascending order).
  image::alt_neural2/rot_baseline_all.png[rot_baseline_all, 800]
  
  To get a clearer picture, lets compute the box plot by considering 1000 random test images as described earlier. Fig. 12 makes it pretty clear that we cannot just add rotation operations to a _regular_ model after training. On the flip side, it provides evidence that rotational conv affects the weights.
  
  [.text-center]
  .Box plot of percentage correct classifications across all rotations for 1000 random test images (in ascending order of _mean_). Note that all models use trained _baseline_ weights.
  image::alt_neural2/rot_baseline_box_all.png[rot_baseline_box_all, 800]
  
  == Conclusions
  
  