Avoiding false optical rotary encoder readings
You have a very high resolution optical encoder and it's detecting real movements just as it's meant to. What's the problem?

Why do you have such a high resolution encoder if you don't want to measure such tiny movements? These two things seem like a contradiction. If you don't want to be able to measure such small movements, I have three suggestions:


Improve your mechanical set up so that small vibrations aren't transmitted to the encoder so much. Are you using proper bearings everywhere?
In software, take your calculated encoder position and right shift it by 2 or 3. (I.E. divide it by 4 or 8). This will basically simulate a much lower resolution encoder, which won't suffer from these problems so much.
Send me the encoder, and I'll send you a much lower resolution one, for free!