Increase headphone impedance using resistors
You can try putting two 2.2 ohms resistors in series with the left-audio the right-audio connections - likely it will do the trick for you and should be pretty simple to do.