Experience Replay, must return minibatch back to Memory Bank? During Experience Replay, we are randomly gathering a minibatch from the Memory bank. We then use the minibatch to correct our NeuralNetwork q-value function approximator.

When done, should we return the minibatch back to the memory bank, or should we throw it out?

Doing the former seems to grow the memory indefinitely, and might result in some state being sampled a few extra times in the future, if the state is lucky and ends up in future mini batches.
If you throw it out, training might suffer in the beginning as you would have even less data at that time.

If you do throw it out, then you have to be aware of your fit interval, your expected episode length, and batch size so you aren't throwing out data faster than you generate it.

As for the memory growing indefinitely, I've used a deque to limit its size.

from collections import deque
memory = deque(maxlen=10000000)