How to prevent PF M-motor from springing back
It sounds like the motor is "winding up" from drivetrain deflection.  Lego parts "give" a little when stressed and therefore store some energy like a spring.  This is likely being caused by the high speed of the PF "M" motor.  A picture of the assembly would be very useful in determining a solution.  Without seeing it, the simplest fix would be to gear down the motor.  Given the fairly high speed of the "M" motor output, I would suggest doing this with a worm gear assembly.  A worm gear will greatly reduce your output ratio to the driving ring while also giving you more precise control that is less susceptible to back-tracking from deflection wind-up.  

Here is a photo demonstrating how a worm gear can greatly reduce output ratio in very little space:  

  

Another potential solution, if you have the part, is to affix a linear actuator to the "M" motor.  It would have the same benefits as the standard worm gear:  

   

One last thing... Which version of part# 6538 are you using?  6538a &amp; 6538b have ridges that allow the drive ring to "click' into place.  If you're using part# 6538c then swapping it with one of the others may be all you need to fix your problem.  

  

A picture of your build would be appreciated to help reduce speculation.  It would also be nice to see this just for the "neat-o" factor :)