PID control at small PWM values for a DC motor
You have backlash in the gear train which is like electrical hysteresis. That is causing "hunting". You can minimize the backlash by preloading as in anti-backlash gears. You can adopt a simpler control scheme such as P (perhaps with some control deadband) or P+D. The integral term, if present, will integrate any remaining error until motion occurs (unless the I term has deliberate or accidental deadband itself - say due to low ADC resolution).

You might want to read this paper by Tim Wescott. Tim does a lot of work with precision electromechanical systems used in military and similar applications.