Is it current (and power dissipation) which damages things, not voltage?
It's primarily because the insulating layers in the device can only withstand a certain voltage. The insulation breaks down with excessive voltage and causes internal shorts.