Audio Amplifier Clipping
Note: This answer is predicated on the fact that you can verify the clipping by putting a scope across the speaker.  And the clipping is happening near the 6.1V mark.



If the output of the amp is clipping, then you only have three choices, 


Increase the voltage to the amplifier (if you can) 
Reduce the gain of the amplifier if you can.
Reduce the amplitude of the input.



  I would like not to sacrifice for volume


After you do one (or more) of the above to regain audio quality, you can add more speakers in parallel, based on your output power description.  Two speakers should give you twice the output power 3dB(assuming that the amplifier drives them equally as well as it drives one speaker), then 4 speakers would double the output power of the two speakers to 6dB.

Turns out that for humans that 3dB changes in sound are marginally perceptible, 6dB is pretty noticeable and 10dB is considered double the volume (or half if going down).

Doubling Power vs. Doubling Output

Thanks @ Marcus Müller for your comment.