One USB-port on MacBook Pro late 2011 doesn't work.
So, the first thing that different devices will tax the USB port differently. The Jaybird headphones likely requires less wattage than the iPhone to charge which is why the damaged port works for some devices but not others. 

Given that you are not getting full power output and data transfer is not working it sounds like the connectors on the inside of the port are bad (as you alluded to in your question). 

You did not state a year for your MacBook Pro but on pretty much all modern models the USB ports contacts reside embedded in a bit of plastic, and the actual connectors are soldered directly onto the main board. If you had skills with soldering etc you might be able to take the machine apart and fix the port, but this would void any warranty you may have.

My recommendation would be to take it into the Apple Store and have them take a look at it. I would guess you will need a new logic board. Apple will give you a price and you can decide from there if you want to:


Have Apple fix it
Take to a 3rd party to fix (cheaper but not Apple parts)
Try and fix it yourself
Buy a new machine
Live with the issue and just use the other ports


Hope this helps!