Matching ground references between computer and small DIY amplifier
You probably won't need to worry about the common since it will be already common with the 0V from the USB port. However, this is not a very nice situation- it means that power supply noise is going to be injected directly into your amplifier input. It may not matter so much for a relatively high level signal, but if there is noise in your headphones, this is probably why.  

Options include adding a differential front end to your amplifier or isolating the power supply (for example, with a DC-DC converter) or adding signal isolation (for example, with an audio transformer).