title: Understanding tensorboard plots for PPO in RLLIB
body: <p>I am beginner in Deep RL and would like to train my own gym environment in RLLIB with the PPO algorithm. However, I am having some difficulties seeing if my hyperparameter settings are being successful. Apart from the obvious episode_reward_mean metric which should rise we have many other plots.</p>
  
  <p>I am especially interested in how entropy should evolve during a successful training. In my case it looks like this:</p>
  
  <p><a href="https://i.stack.imgur.com/mOgou.jpg" rel="nofollow noreferrer">entropy.jpg</a></p>
  
  <p>It is usually dropping below 0 and then converging. I understand that entropy as part of the loss function is enforcing exploration and can therefore speedup learning. But why is it getting negative? Shouldn't it be always greater or equal to 0?</p>
  
  <p>What are other characteristics of a successful training (vf_explained_var, vf_loss, kl,...)?</p>