title: DQN cannot converge while the example with the same graph works
body: <p>Here is the problem. I try to rewrite a dpn example of the CartPole-v0. The example converges with less than 2000 episodes, However, my version could not converge. </p>
  
  <p>Therefore I use TensorBoard as the tool to help me figure out the problem. I find that these two codes have the same graph</p>
  
  <p>What's more, I find that the graph of the loss is very different.
  <a href="https://i.stack.imgur.com/dBBM5.png" rel="nofollow noreferrer">loss of the example</a>
  <a href="https://i.stack.imgur.com/afliU.png" rel="nofollow noreferrer">loss of mine</a></p>
  
  <p>I wonder what's the problem. I will be grateful if anyone could help me with it! Thank you!</p>
  
  <p>P.S. If the codes are needed. I could upload them. Both of them are shorter than 200 lines.</p>