Title: TIDBD: Adapting Temporal-difference Step-sizes Through Stochastic Meta-descent,
Abstract: In this paper, we introduce a method for adapting the step-sizes of temporal
difference (TD) learning. The performance of TD methods often depends on well
chosen step-sizes, yet few algorithms have been developed for setting the
step-size automatically for TD learning. An important limitation of current
methods is that they adapt a single step-size shared by all the weights of the
learning system. A vector step-size enables greater optimization by specifying
parameters on a per-feature basis. Furthermore, adapting parameters at
different rates has the added benefit of being a simple form of representation
learning. We generalize Incremental Delta Bar Delta (IDBD)---a vectorized
adaptive step-size method for supervised learning---to TD learning, which we
name TIDBD. We demonstrate that TIDBD is able to find appropriate step-sizes in
both stationary and non-stationary prediction tasks, outperforming ordinary TD
methods and TD methods with scalar step-size adaptation; we demonstrate that it
can differentiate between features which are relevant and irrelevant for a
given task, performing representation learning; and we show on a real-world
robot prediction task that TIDBD is able to outperform ordinary TD methods and
TD methods augmented with AlphaBound and RMSprop.