Why exactly is connecting/disconnecting circuits with power on dangerous?
Wires and inputs act as inductors and capacitances. Often, for filtering reasons, you will also find the real components (inductors and capacitors) in addition to the system's parasitic elements. When plugging or unlplugging a device, you "hit" a resonant LC tank circuit that will cause voltage and current swings beyond the steady-state levels and that might be destructive to sensitive components like microcontrollers or memory ICs.

This happens as a step response upon connecting and disconnecting the plug. When connecting (turning on) the LC resonant tank, it will swing to twice the applied voltage. When you disconnect it, you may also get a freewheeling pulse from the energy stroed in the inductance. To make things worse, connecting or disconnecting a plug is never a clean event. If you look closely with an oscilloscope, you will notice that each connection or disconnection consists of many fast pulses, comparable but often far worse than the so-called bouncing of a switch.

There are a bunch of application notes out there about preventing circuits from this type of "hot plugging" damage, for example Linear's AN-88, this design feature by LTC or this app'note.

This mechanism is true for power wires and data wires.

Also, unprotected analog or digital inputs may cause latch-up events if they are connected before the power wires are connected. This may also cause permanent damage.

This just explains how damage to physical components might occur. In addition, the software that is needed to establish the communication across hot-pluggable interfaces must be able to recognize and support hot-plugging without becoming hung-up.