title: How to manage negative rewards in policy gradient in keras?
body: <p>The same basic question here, but 3 years old and no definitive answer:</p>
  
  <p><a href="https://ai.stackexchange.com/questions/2405/negative-reward-penalty-in-policy-gradient-reinforcement-learning">https://ai.stackexchange.com/questions/2405/negative-reward-penalty-in-policy-gradient-reinforcement-learning</a></p>
  
  <p>The question is, if I'm doing policy gradient in keras, using a loss of the form:</p>
  
  <p><code>rewards*cross_entropy(action_pdf, selected_action_one_hot)</code></p>
  
  <p>How do I manage negative rewards?  </p>
  
  <p>I've had success with this form in cases where the reward is always positive, but it does not train with negative rewards. The failure mode is for it to drive itself to very confident predictions all the time, which results in very large negative losses due to induced deviation for exploration. I can get it to train by clipping rewards at zero, but this throws a lot of valuable information on the table (only carrots, no sticks).</p>