Join independent amp grounds to common headphone ground?
Browsing some Adafruit's products I think that the used chip here is: MAX98357 - datatsheet. Like most of the mini Class D amplifiers this one also has a bridged output. Because of that you should not connect R- with L- in any case. Shorting them you make a short on amplifier's output transistors and this way on power supply. 

Internal short circuit protection limits the curent so that the amplifier is not fried, but this excessive current can make a power glitch on Raspberry's supply and make the CPU freeze or reset.

On  this product picture:

can be seen that both positive and negative outputs go trough SMD ferrite beads (right next to the 4-pin connector) which confirms that they are HF switched.

So the conclusion is that this amplifer board cannot be used with headphones with common ground wiring.