Unamplified AND Amplified input on the same speaker
Mordern amps are usually bridged, that means each speaker is connected in a H-bridge setup. This allows higher output currents due to polarity reversal.

In your case that means you must switch all 4 wires - the speaker "minus" wires are not GND.

In contrast, the headphone minus wires are connected together as GND. The forth wire on the connector is for the mic.