How to instantly stop the rotation of an object in BGE?
This appears to be a reported bug

Workaround: use a very small value like 0.00000012

def rotation():
    if (mX &lt; 0.10) and (mY &lt; 0.90):
        cam.setAngularVelocity([0, 0, 0.1], 0)
    elif (mX &gt; 0.90) and (mY &lt; 0.90):
        cam.setAngularVelocity([0, 0, -0.1], 0)
    else:       
        cam.setAngularVelocity([0, 0, 0.00000012], 0)