Firmware backup and restore I need to have the firmware stored somewhere on the board in order to get back to it after flashing a new firmware.

Is there a way to store a backup of the firmware somewhere, and use a button to erase a newer firmware and get back to the old one, without re-flashing the old firmware again ?
Yes. You need to create a factory partition and upload your bin file to there.

You can read more about ESP32 partitions here

Note that none of the partition schemes in the ESP32 core have factory partitions, so you will need to create a custom one and link it to a menu entry (look at the boards.txt file to see how that is done).

The existing partition files are just .csv files within the ESP32 core. Copy one and modify it, then link that to a menu entry using the new file name.