What is the impact of unhandled exception on microcontroller peripherals?
You should handle all exceptions/fault interrupts, so there should not be any &quot;unhandled&quot; ones. Unhandled means that the exception and/or interrupt table has an instance which sends the program counter nowhere/to the wrong address.
Please note that in circuit debuggers might be designed to halt upon encountering such problems, to give you a chance to spot one. While the real program will hopefully just reset the MCU and start over - at watchdog reset if not sooner.

Can I assume the peripheral will stay configured as is?
Can I assume the GPIOs will stay in their current state?

No. Hardware peripheral registers are to be regarded as RAM, which in turn is traditionally regarded as less reliable than NVM. In systems where very long up-times are expected, it is custom to initialize those registers on regular basis, by copying down the default settings from flash.
Additionally, in safety-critical applications, I usually recommend to implement a vector table where every unhandled interrupt that you shouldn't be getting results in a defensive ISR. This ISR does nothing but shutting off the source of its interrupt and then returns. (This of course depending on the nature of the application: is the safe state a stop or is it &quot;keep running as well as you can&quot;.) That is, if you aren't using UART in your application, still write a UART ISR which does nothing but shutting off the UART interrupt, that you shouldn't be getting.