Q learning - how to use experience replay, when playing against other agent?
I am pretty sure you have to:

wait for enough data in the memory (100 entries, etc), then take out a minibatch (for example, 10 random elements), run backprop 10 times - once for each selected element.

You then take an average of resulted gradients and finally, correct your network with that averaged gradient.

After,


put the elements back in the memory (where you've taken them from) 


  Samples that have high priority are likely to be used in training many times. Reducing the weights on these oft-seen samples basically tells the network, "train on these samples, but without much emphasis; they'll be seen again soon." 

or throw it out, substituting it with new memories instead.


Then keep playing the game, adding say, another 30 examples before doing another weights-correction session using a batch of 10 elements

I refer to the "Session" meaning a sequence of backprops, where the result is the average gradient used to finally correct the network.


  EDIT: Another question I have in terms of training a neural network against a neural network, is that do you train it against a completely separate network that trains itself, or do you train it against a previous version of itself. And when training it against the other neural network, do you turn the epsilon greedy down to make the opposing neural network not use any random moves.


Consider using just one network.  Let's say our memory bank contains several elements:

...
{...}
{stateFrom, takenActionIndex, takenActionQval, immediateReward, stateNext }  &lt;-- a single element
{...}
{...}
{...}
...


When using each element in your memory during the Correction session (one element after the other), you need to:


pick an element. As seen, it contains stateFrom, taken action (action X), qvalue of action X, reward you received, state that it led to.
Run forward prop as if you were in "next state" (mentioned in that element).  Get its best [action Y, qvalue]. In fact, your action Y doesn't have to be the action with highest q-value. Instead, it could be the "epsilon-greedy decided action" from the "next state" - you would then have SARSA instead of Q-learning 
obtain delta by which the "qvalue of action X" differs from the immediate reward + "qvalue of action Y".
get deltas of several other elements that you've chosen from the bank
Using each delta, compute a gradient tensor for your network's weights. Average those gradients into a single tensor. Punish your network's weights with this tensor.


I've intentionally circumvented 1 thing - you don't actually store takenActionQval, because they might become obsolete by the time you fetch its element from MemoryBank. You have to re-compute these scores during backprop.



Therefore, you are training against the version of your network from the similar but previous correction session.

Notice, you don't store ["next state", action Y] because by the time you select it to train (maybe you didn't select it for several minibatches), the network might have a different q-value for that action Y.

You could also copy your network to the second network (target network), but only, say, every 200 timesteps. You would still proceed to punish your network for any differences with the target network in the meantime, after every 30 timesteps.

Notice, the intuition of why this works is because: the Q-values are sort-of "flowing" from finish back to the beginning, by a little with every new journey. And you are always updating the current state towards the successor state. And train the successor state against its own (even further) successor.