How do I move the camera in any perspective mode so the reference image doesn't vanish?
Your background image will vanish as soon as you leave orthographic view or rotate the camera to an angle the background image isn´t set for.

Just stay in orthographic view (Numpad 5) and use Front View (Numpad 1), Side View (Numpad 3) and Top View (Numpad 7) to change the directions; for the opposite of those views press Ctrl as well.

To move your current view without rotating just press Shift and drag your view with Middle Mouse Button