Differential output to headphones
The LM386 isn't quite suitable for what you need. Firstly it has a minimum gain of 20 so it's going to be pounding out loud and distorting badly. The second thing is that this type of amplifier benefits from a bypass capacitor from pin 7 to ground - it greatly enhances rejection of power supply noise - take a look at the top middle graph on page 4 to see the effects.

Thirdly, and quite importantly you need an output capacitor to stop dc voltages being shorted by the low impedance of your speaker - take a look at the application diagrams on page 5 onwards to see what I mean - it's mostly shown as a 250uF capacitor for your reference.

Fourthly, in most applications where the device is being used as a normal amplifier, it is operated with a single ended input - I'm not sure the data sheet adequately informs anyone that operating differentially (as you have it) is ideal.

You can get this device to work as a headphone amp but you'll probably need to reduce the signals going into it.