Larger values of γ favor rewards in the future, rather than immediate rewards which are given by smaller values of γ. Although the update in Eq. ( ) appears to be a gradient descent method, for approximate state-action value functions such as that used in Eq. ( ), this is no longer the case. This has been pointed out in Ref 45, Appendix I. To see this, assume that the update in Eq. ( ) has been mapped onto the general gradient descent expression