Red & white dotted line on outside of view
The border looks like a Render Border (activated by Ctrl B in the 3D view in Object Mode). It can be deactivated with CtrlAltB.

I don't think it's connected with navigation being disabled while in brush (circle) select mode. The scroll-wheel is taken over to determine the size of the brush, and Shift is occupied to switch from select to deselect.