Should I Remove the Power When Unplugging Components From an Arduino?
It is best to remove power while re-wiring a circuit, so, yes, disconnect the USB cable, and any other sources of power.

While disconnecting and re-connecting things, you may introduce short circuits, or put the circuit in undesirable states (eg disconnecting power to a chip while leaving power on its I/O pins, thus powering the chip throough protection diodes.)