Title: Sample Complexity of Estimating the Policy Gradient for Nearly Deterministic Dynamical Systems,
Abstract: Reinforcement learning is a promising approach to learning robot controllers.
It has recently been shown that algorithms based on finite-difference estimates
of the policy gradient are competitive with algorithms based on the policy
gradient theorem. We propose a theoretical framework for understanding this
phenomenon. Our key insight is that many dynamical systems (especially those of
interest in robot control tasks) are \emph{nearly deterministic}---i.e., they
can be modeled as a deterministic system with a small stochastic perturbation.
We show that for such systems, finite-difference estimates of the policy
gradient can have substantially lower variance than estimates based on the
policy gradient theorem. We interpret these results in the context of
counterfactual estimation. Finally, we empirically evaluate our insights in an
experiment on the inverted pendulum.