Quick question on basic Basic concept of experience replay Due to my admitted newbie's understanding on the field, I'm about to ask a dummy question.
While sampling batches, for example experience replay buffer which contains number of samples, after getting n (size of a batch) of loss values through forward propagation, what way will the weights updates follow:

Update all weights based on all losses separately, like iterating through that n losses to update
Update all weights based on the mean (or maybe standard deviation) of all losses
The 2nd one is the correct one. We update the weights according to the mean squared error between real value (in case of DQN it is a bootstrapped value) and the current value prediction, obtained for every sample ($s,a,R_t$) in the minibatch. In other words your network is going to get &quot;corrected&quot;, for every prediction it has made, by the difference between estimation and ground truth values.