Headphone volume control works for one speaker only
A stereo headphone audio circuit has two low-impedance paths: from signal left to ground, and from signal right to ground. However, it also has a third high-impedance path from signal left to signal right.

Normally the parasitic third path does not affect the overall system very much, since signal left and signal right prefer to return on the much lower-impedance path to ground. But when you increase the impedance on ground, say by adding a potentiometer and subsequently turning it up, you cause the third path to become much more prominent. The exact effect from doing so varies, but in general it could be said that it "turns the audio to sh*t".

Using a double-gang potentiometer instead will also increase the impedance on the normal return paths, but the added impedance on the third path will be double that added to the normal paths, reducing its effect overall and maintaining audio fidelity.