Quick question on basic Basic concept of experience replay
The 2nd one is the correct one. We update the weights according to the mean squared error between real value (in case of DQN it is a bootstrapped value) and the current value prediction, obtained for every sample ($s,a,R_t$) in the minibatch. In other words your network is going to get &quot;corrected&quot;, for every prediction it has made, by the difference between estimation and ground truth values.