Text unselected when clicking to activate Terminal window (OS X Lion)
If you're perturbed every time you move the mouse (like me), you could alway try switching to keystrokes.

For example, to switch between open applications, use CMD+Tab, and then between windows using CMD+~

Another useful way to get around with the keyboard is to use Tab to move between all controls using the Keyboard control panel in System Preferences like so: