Optical Flow: Method to estimate a very small displacement
I was confronted to the same problem and I can recommend to use particle filtering. The idea is that instead of using a discretization of the space (which would favor only a limited range of motion detection), the optic flow is computed as a probability density function which is represented by a limited number of samples. These samples (or particles) are precise up to the precision of your computer and can thus detect very small displacments.

Some resources:


the original paper that led to this idea of applying the particle filter method to computer vision: http://www.robots.ox.ac.uk/~misard/condensation.html
a first implementation with details on the method: http://invibe.net/LaurentPerrinet/Publications/Perrinet12pred
a recent implementation : https://github.com/laurentperrinet/Khoei_2017_PLoSCB