Experience Replay, must return minibatch back to Memory Bank?
If you throw it out, training might suffer in the beginning as you would have even less data at that time.

If you do throw it out, then you have to be aware of your fit interval, your expected episode length, and batch size so you aren't throwing out data faster than you generate it.

As for the memory growing indefinitely, I've used a deque to limit its size.

from collections import deque
memory = deque(maxlen=10000000)