title: Continuous reinforcement learning optimization explodes
body: <p>I'm implementing continuous reinforcement learning (<em>Continuous Control with Deep Reinforcement Learning</em> <a href="https://arxiv.org/abs/1509.02971" rel="nofollow noreferrer">https://arxiv.org/abs/1509.02971</a>) but I have ran into problems when optimizing the policy-neural network.</p>
  
  <p>As recommended in the paper I maximize policy's Q-value: max Q(state,action=policy(state)) by calculating and following the gradient (gradient ascent). However, because the Q-neural network is not (initially) perfect approximation of Q(state,action) and becomes very large/infinite when policy's action becomes large/infinite I ran into problems. Q->infinite when action->infinite and this means policy's weights are optimized to extremely large or infinite values. Morover, when policy starts outputting near infinite values this means Q-values become too large and reinforcement values becomes irrelevant as "r + gamma*Q" is dominated by extremely large Q-values.</p>
  
  <p>What is your recommended solutions for this? I'm planning to restrict policy's outputs to [0,1] by using sigmoidal output layer instead of linear one but this seems to seriously hamper the optimization process and the algorithm cannot learn good policies anymore..</p>