This question is primarily concerned with audio but it's probably applicable to other areas as well.

As far as I understand, EMI is caused by electromagnetic waves adding power to wires as they transmit a signal. Let's say you are transmitting an audio signal and the reference voltage wire is at zero watts and the signal wire is at one watt.

Now imagine you have the world's worst EMI and it causes an extra watt to be added to the signal wire and the reference voltage wire. The difference between the wires is still two so the signal hasn't changed. What gives? Why does EMI cause nasty noises at the other end when it would presumably be changing both wires' power equally?