What is a weighted average in a non-stationary k-armed bandit problem?
The weighted average stands for a linear combination of all values, such that the sum of all weights is 1.

More specifically, if you denote the rewards by a vector $X$, the weighted average will be taking the dot product between $X$ and a vector $W$ such that $0 \le W_i \le 1$ and the sum of all $W_i$ is 1.

If each $W_i = 1/n$ it will be a weighted average (a.k.a the mean). Using the exponential decay $W_i = \alpha^i/ (\sum W_i)$ is also a weighted average.

Then both strategies to compute the Q value use weighted average of previous reward.