Rainbow vs A3C ...too unfair?
Keep in mind that the DQN based method have replay memory that is sampled and used to optimize the model at every time step while A3C is only optimized when the episode finishes. So for 4000 timesteps, and 1000 timestep per episode, A3C optimized 4 times while DQN could have optimized 3000+ times depending on the implementation. This may explain why given the same number of timesteps, DQN is learning faster.