Unusual servo reaction
Assuming this is a standard size servo, put about 200 uF across the servo power leads, as close to the servo as possible.  Put something similar across your control circuit.

Servos draw a LOT of power when they move.  They draw even more in the startup spike when they start to move.  This puts nasty glitches on your power bus, and those glitches CAN get back into your control circuit.

Yes, this answer IS written in my personal blood, thank you for asking...