Can parallel addition of a bigger capacitor damage the smaller one?
Unlikely, but yes. 

A capacitor and the coil of the speaker form an oscillator, which will resonate to some frequencies more than at others. Resonance means you get larger voltages.

Adding resonance to a system can hence introduce voltages that you didn't design for. That can, in turn, break voltage-sensitive components such as capacitors.

Why it's unlikely: The smaller capacitor you used is probably sufficiently robust against overvoltage; also, speaker capacitors tend to be of the self-healing film type, so only a reduction in capacity would be noticable, with should shift the frequency response, but not lead to a reduction in output power with a sensible test signal (I assume that's what you mean with "2x low", whatever that exactly means).

More likely are cabling problems or a damaged speaker.