How to stop viewing controls to be too sensitive to mouse movements?
Maybe focusing on your object does what you want?

Select a object and press the . on your numeric pad, if you have one.

If you like that feature and want it to be always active, you can in the preferences menu set it up: