Why isn't my implementation of A2C for the the atari pong game converging?
Here is the commit

I fixed few minor errors, but the major one was when I saw what the line histories = [deque(maxlen=self.reward_steps)] * len(self.env.envs) was doing. It was just repeating the same queue.

In [2]: histories = [deque(maxlen=5)] * 4                                       

In [3]: histories                                                               
Out[3]: [deque([]), deque([]), deque([]), deque([])]

In [4]: histories[0].append(1)                                                  

In [5]: histories                                                               
Out[5]: [deque([1]), deque([1]), deque([1]), deque([1])]


So I just replace it by histories = [deque(maxlen=self.reward_steps) for i in range(len(self.env.envs))]. That fixed my problem.

In [7]: histories = [deque(maxlen=5) for i in range(4)]                         

In [8]: histories                                                               
Out[8]: [deque([]), deque([]), deque([]), deque([])]

In [9]: histories[0].append(1)                                                  

In [10]: histories                                                              
Out[10]: [deque([1]), deque([]), deque([]), deque([])]


The curve representing the mean reward looks like