Servo Motion Direction
The standard answer is to use a microcontroller to read the pot and make the decisions.

You don't say what the drawbacks of the current circuit are. A comparator with some positive feedback or hysteresis, and an inverter, will do the trick, but likely it sounds awful when reversing, and takes a huge spike of current.

The micro will be able to drive the H-bridge with some sort of PWM, and thus gently slow the motor down and reverse it again. You will define a voltage profile in software, or even make your own PD control loop and have the servo follow an internally generated position profile.

With some programming experience, the learning curve with an Arduino will be short. And if you need to produce several of them, you could use a small 8-pin microcontroller with internal clock, not very expensive.