Motor speed control using arduino and quadrature encoders
It looks to me that your control loop is essentially:

if speed is too slow:
    increase PWM duty cycle one unit
if speed is too fast:
    descreate PWM duty cycle one unit


As you have observed, this doesn't work so well. Your control loop cyclically overshoots the target speed.

The canonical solution to this sort of problem is a PID controller. The concept is essentially the same, measure a thing and compare it to a target to calculate an error. Then adjust something (in your case, PWM duty cycle) based on the error.

However, a PID controller has three error terms:


P: the current error
I: the integral of the error
D: the derivative of the error


For each of these terms, the controller has programmed some gain. It then multiplies each term by the respective gain to calculate how much the control input (PWM duty cycle) should be changed. Properly tuned, this allows you to get a feedback loop that at first ramps up the duty cycle, then as your vehicle approaches the target speed, backs off smoothly so you get a minimum of overshoot.