What happens if an input is applied to an output enabled pin of a microcontroller
Stop doing this -- you're creating a short.

Since the pin is configured as an output, but presumably the port bit is set at 0, this is the same as ground (or very nearly ground).  So this is essentially the same as short +5 to ground -- except in this case the short is going through the output pin's CMOS transistor.  Not good.

This is temporarily dropping the supply voltage to the microcontroller, which is causing it to reboot.  Keep this up, and you will probably burn out the chip (it will overheat).

(edit, from my comment) The processor reboots because the short between the output pin and +5 shorts the power supply, causing the power supply to drop below the minimum supply voltage necessary to keep the processor running. As soon as this happens, the short is removed because the pin is no longer configured as an output. So the power supply recovers momentarily, and the processor starts back up again. But if the switch is still held down, this will just repeat as soon as the code runs that sets the pin as an output.

Even if the output was configured high before you set it as an output, then you would still have a partial short because the voltage of an output pin is usually not quite equal to the supply voltage.  So you would still be sinking some current.

The only case where you wouldn't have a short is if the pin is set high, and the pin is configured as an open-drain output.  In this case, the pin is normally driven high externally, but always through a resistor, and not a direct connection to the supply voltage