TDA2030A gain reduction for Headphone Amp
Lower the signal going in with an attenuator - just a couple of resistors should do the job. 

A lot of headphone outputs on hi-fi amplifiers do have resistor attenuators despite what you said about destroying damping factor and reducing quality.

The dynamic impedance of speakers (headphone or otherwise) are mainly resistive but if you have some article or other that explains why this should be a problem I'd love to read it.