How is weighted average computed in Deep Q networks I was going through the Sutton book and they said the update formula for Q learning comes from the weighted average of the returns
I.e
New estimate= old estimate +alpha*[returns- old estimate]
So by the law of large numbers this will converge to the optimal true q value
Now when we go to Deep Q networks,how exactly is the weighted average computed, all they simply did was try to reduce the error between the target and the estimate, and keep in mind this isn’t the true target, it’s just an unbiased estimate,since it’s an unbiased estimate how is the weighted average computed , which is the expectation?
Can someone help me out here??
Thanks in advance
Let's say $Q$ is the old estimate, $Q'$ the new estimate, and $R$ is the return.
We have
$$ 
Q' = Q + \alpha(R-Q)
$$
This can be rewritten as
$$ 
Q' = (1-\alpha)Q + \alpha R
$$
When $\alpha$ is a constant, this is an exponential weighted average of returns. If $n$ is the number of samples we get and $\alpha=1/n$ (so it decreases with each sample), we get
$$
Q' = \frac{n-1}{n}Q + \frac{1}{n}R
$$
This simply represents the average return. So, playing with $\alpha$ tunes the weighting of the estimate.