How do simple low power devices store setup when switched off?
Many (most?) of these items have a small microcontroller inside to supply the functionality the device requires.  Some small microcontrollers have a small amount of EEPROM built into them.  The Microchip PIC10F322 is one such but there are many others now available.

Another method uses a capacitor to keep a few SRAM cells alive within the controller.  However, this method is not used much any more because it is more expensive than using a controller with built-in EEPROM.