Entropy term in Proximal Policy Optimization (PPO) becomes undefined after few training epochs I have implemented the total loss of my PPO objective as follows:-

total_loss = critic_discount * critic_loss + actor_loss -  entropy_beta * K.mean(-(newpolicy_probs * K.log(newpolicy_probs)))


After training for a few epochs, the entropy term becomes "nan" for some reason. I used tf.Print() to see the new policy probabilities when the entropy becomes undefined, it is as follows-


  new policy probs: [[6.1029973e-06 1.93471514e-08
  0.000299338106...]...]


I am not clear as to why taking log of these small probabilities is coming out as nan. Any idea how to prevent this?
I browsed through some other implementations of PPO and they all add small offset (1e-10) to prevent undefined log(0). I did that and the training works now.