What does it mean to rotate an image by 0.1 degrees?
From the article:


  First the image is rotated by an infinitesimal amount $\alpha$. This is done by computing the rotated coordinates of each pixel and interpolating the gray level values at the new coordinates.


Interpolation is key to very small rotations. It allows to sample the image between the pixel center points as if it was a smooth 2-dimensional function. You can choose the interpolation method in skimage.transform.rotate using the parameter order. It controls the order of the interpolation method which (I think) is an interpolating cardinal B-spline. Higher orders give better results but are slower to compute.

If the tangent vector calculated by rotation, subtraction and division is increasingly non-zero as you go further away from the center, then that is an expected result and you can conclude that some rotation took place.