Can a speaker be damaged but not blown?
Sure. Speakers are essentially large coils of wire, and when they "blow", it's generally that the coil is taking too much current and isn't being cooled enough, and is damaged. If the speaker is completely destroyed, this may be that the wire in the coil is broken or shorted, so that it doesn't operate. 

However, one can imagine a scenario where something else in the speaker is damaged, such as the surround (the rubber coil that allows the cone to vibrate freely) or the spider (cloth "guide" inside the speaker, behind the cone) or leads leading into the cone are damaged, but the coil still operates. Any of these issues have the ability to reduce the volume level of the speaker.

It's worth noting that it's usually not the "loud sound" that damages the speaker, it's the large inrush of current that the voice coil cannot handle, the excursion of the cone (moving farther than designed for), or natural wear and tear on components.