Entropy term in Proximal Policy Optimization (PPO) becomes undefined after few training epochs
I browsed through some other implementations of PPO and they all add small offset (1e-10) to prevent undefined log(0). I did that and the training works now.