Increasing speakers impedance by adding resistors
It'll change the sound quality a little, but at such a low volume, not enough to worry about. 6 ohms would probably be about right. 

I would add a shorting switch across the resistors for daytime use.