Each training run for DDQN agent takes 2 days, and still ends up with -13 avg score, but OpenAi baseline DQN needs only an hour to converge to +18? Status:

For a few weeks now, I have been working on a Double DQN agent for the PongDeterministic-v4 environment, which you can find here.

A single training run lasts for about 7-8 million timesteps (about 7000 episodes) and takes me about 2 days, on Google Collab (K80 Tesla GPU and 13 GB RAM). At first, I thought this was normal because I saw a lot of posts talking about how DQNs take a long time to train for Atari games.

Revelation:

But then after cloning the OpenAI baselines repo, I tried running python -m baselines.run --alg=deepq --env=PongNoFrameskip-v4  and this took about 500 episodes and an hour or 2 to converge to a nice score of +18, without breaking a sweat. Now I'm convinced that I'm doing something terribly wrong but I don't know what exactly.

Investigation:

After going through the DQN baseline code by OpenAI, I was able to note a few differences:


I use the PongDeterministic-v4 environment but they use the PongNoFrameskip-v4 environment 
I thought a larger replay buffer size was important, so I struggled (with the memory optimization) to ensure it was set to 70000 but they set it to a mere 10000, and still got amazing results.
I am using a normal Double DQN, but they seem to be using a Dueling Double DQN.


Results/Conclusion

I have my doubts about such a huge increase in performance with just these few changes. So I know there is probably something wrong with my existing implementation. Can someone point me in the right direction?

Any sort of help will be appreciated. Thanks!
Dueling architectures create bigger differences in the values of actions in the state space. This is because the state-value V(s) function is estimated separately from the state-action value Q(s, a). A new quantity, the advantage of an action, can then be defined as A(s, a) = Q(s, a) - V(s). 


  The Q function, however, measures the the value
  of choosing a particular action when in this state. The advantage
  function subtracts the value of the state from the Q
  function to obtain a relative measure of the importance of
  each action.
  
  Dueling Network Architectures for Deep Reinforcement Learning




To better direct you, here are 2 resources that could really help you understand why those differences are important.

Speeding up DQN on PyTorch: how to solve Pong in 30 minutes

The main points of the blog are:


Use a larger batch size and play several steps before updating
Play and train in a separate process
Use asyncronous cuda transfers


RL Adventure

This github library has easy to follow jupyter notebooks and links to all of the papers.
It includes:


DQN
Double DQN
Dueling DQN
Prioritized Experience Replay
Noise Networks for Exploration
Distributional RL
Rainbow (That network that deepmind made that had so many things in it they couldn't find a good name)
Distributional RL with Quantile Regression
Hierarchical Deep RL