If my brain lost all of its activity for a moment, would my memory be gone too?
Memories are dynamic; that is not equivalent to "memories are evanescent". Cellular structure and composition controls memory, not just a continuous passage of signals. This point is already discussed in the comments. 

Loss of energy (in the form of ATP) would cause memory deterioration and in some cases neuronal cell death too— strokes, which happen due to lack of blood supply to brain (and hence unavailability of nutrients and oxygen) can cause amnesia. 

But one second will practically not affect anything.