v is of the same dimension as w and may be initialized to zero at the beginning of the RLCI algorithm. The update for v in Eq. ( ) is derived from the Least Mean Square (LMS) rule that seeks v so as to minimize the squared error f v -δ 2 . This modifies the update scheme to account for the fact that the linear approximation to Q(s, a) is not complete. Note that upon setting the parameter β to zero the algorithm will recover the classic approximate Q-learning algorithm. In this gradient corrected method, the update in Eq. ( ) implicitly assumes that the update in Eq. ( ) is at, or near, steady-state. In other words, the secondary learning rate β needs to be greater than the learning rate α. Because α and β take values between 0 and 1, in our experience a reasonable value for β is √ α, where α is the learning rate. This eliminates the need to set an additional hyperparameter, while maintaining the condition that β > α.