Adding to a circuit the ability to "remember" the last chosen settings when turned off or disconnected
Assuming it doesn't display anything when then is no audio input, then leave the circuitry on all the time and have a SPDT switch (or DPDT if stereo) that either passes through the audio, or shorts out the input so nothing is displayed.  That becomes your on/off switch.