What's the simplest way to store 1 bit after a device has been turned off?
The most simplest way to store a bit I can think of is a literal, mechanical switch. Switch closed - Current can flow: 1, Switch open - No current: 0. Yes this requires human input to flip the bit if necessary but it isn't going to change if the power is turned off or on (unless you change it yourself) - acting like a mechanical register.
Mechanical solutions are easy to understand, and implement, but the drawback is for the amount of information one switch can hold (one bit), a regular EEPROM chip could hold hundreds to thousands of MBs.
Hope this helps!