title: Reinforcement Learning Agent Policy Won't Update
body: <p>I am currently training a deep reinforcement learning model in a continuous environment using Ray.</p>
  
  <p>The environment I am using was coded up in OpenAI Gym using baselines by another person who's research I am trying to replicate in Ray so that it may be parallelized. The model converges about 5% of the time in baselines after thousands of episodes though I have not been able to replicate this.</p>
  
  <p>My problem is that the AI stops changing its policy very quickly and the mean reward then stays roughly the same for the rest of the run. This is independent of every parameter I've tried to vary. So far I've tried changing:</p>
  
  <ul>
  <li>Learning Algorithm: I've used PPO, A3C, and DDPG</li>
  <li>Learning Rate: I've done hyperparameter sweeps from 1e-6 up to 10</li>
  <li>Episode Length: I've used <code>complete_episodes</code> as well as <code>truncated_episodes</code> in <code>batch_mode</code>. I've varied <code>rollout_fragment_length</code> and made sure <code>train_batch_size</code> was always equal to <code>rollout_fragment_length</code> times the number of workers</li>
  <li><code>entropy_coeff</code> in A3C</li>
  <li><code>kl_target</code>, <code>kl_coeff</code>, and <code>sgd_minibatch_size</code> in PPO</li>
  <li>Network Architecture: I've tried both wide and deep networks. The default Ray network is 2x256 layers. I've tried changing it to 4x2048 and 10x64. The baselines network that succeeded was 8x64.</li>
  <li>Changing activation function: ReLU returns NaNs but tanh lets the program run</li>
  <li>Implementing L2 loss to prevent connection weights from exploding. This kept tanh activation from returning NaNs after a certain amount of time (the higher the learning rate the faster it got to the point it returned NaNs)</li>
  <li>Normalization: I normalized the state that is returned to the agent from 0 to 1. Actions are not normalized because they both go from -6 to 6.</li>
  <li>Scaling the reward: Ive had rewards in the range of 1e-5 to 1e5. This had no effect on the agent's behavior.</li>
  <li>Changing reward functions</li>
  <li>Making the job easier (more lenient conditions)</li>
  </ul>
  
  <p>Note: I am running this in a docker image and have verified that this occurs on another machine so it's not a hardware issue.</p>
  
  <p><strong>Here is a list of things I believe are red flags that could be interrelated:</strong></p>
  
  <ul>
  <li>I was originally getting NaNs (Ray returned NaN actions) when using ReLU but switching my activation function to tanh.</li>
  <li>The KL divergence coefficient <code>cur_kl_coeff</code> in PPO goes to zero almost immediately when I include L2 loss. From what I understand, this means that the model is not making meaningful weight changes after the first few iterations.
  <a href="https://i.stack.imgur.com/KFhDY.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/KFhDY.png" alt="enter image description here"></a></li>
  </ul>
  
  <p>However, when the loss is just policy loss, <code>cur_kl_coeff</code> varies normally.</p>
  
  <ul>
  <li><p>Loss doesn't substantially change on average and doesn't ever converge
  <a href="https://i.stack.imgur.com/fqjW8.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/fqjW8.png" alt="enter image description here"></a></p></li>
  <li><p>Reward doesn't change substantially <code>reward_mean</code> always converges to roughly the same value. This occurs even when <code>reward_mean</code> starts above average or initially increases to above average due to random weight initialization.
  <a href="https://i.stack.imgur.com/8jVWD.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8jVWD.png" alt="Here, mean iteration reward starts around average but quickly increases in the first few iterations before falling back. This may look like coincidence on this graph but I have observed the phenomenon countless times."></a></p></li>
  </ul>
  
  <p>Note: This is not an issue of the agent not finding a better path or a poor reward function not valuing good actions. </p>
  
  <p><a href="https://i.stack.imgur.com/Rj5zI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Rj5zI.png" alt="Note that the max reward is an order of magnitude higher than the mean reward"></a></p>
  
  <p>Zooming out shows why mean reward peaked previously. The agent performed phenomenally but was never able to repeat anywhere near its success.</p>
  
  <p><a href="https://i.stack.imgur.com/UGwPi.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/UGwPi.png" alt="enter image description here"></a></p>
  
  <p>And, I cannot stress this enough, this is not an issue of me not letting the agent run for long enough. I have observed this same behavior across dozens of runs and cumulatively tens of thousands of iterations. I understand that it can take models a long time to converge, but the models consistently make <strong>zero</strong> progress.</p>
  
  <p><strong>tldr;</strong> My model doesn't change it's policy in any meaningful way despite getting very high rewards on certain runs.</p>