Simulating successful trajectories in Montezuma's Revenge turns out to be unsuccessful I have written code in OpenAI's gym to simulate a random playing in Montezuma's Revenge where the agent randomly samples actions from the action space and tries to play the game. A success for me is defined as the case when the agent is atleast able to successfully retrieve the key (Gets a reward of 100). And such cases I dump in a pickle file. I got 44 successful cases when I kept it to run for a day or so. Here is the code I use to generate the training set :

import numpy
import gym
import cma
import random
import time
import pickle as pkl

env   = gym.make('MontezumaRevenge-ram-v0')

observation = env.reset()

#print(observation)
#print(env.action_space.sample())

obs_dict    = []
action_dict = []
success_ctr = 0

for i in range(0, 1000000):
    print('Reward for episode',i+1)
    done = False
    rew = 0
    action_list = []
    obs_list    = []
    while not done:
        action = env.action_space.sample()
        observation, reward, done, _ = env.step(action)
        action_list.append(action)
        obs_list.append(observation)

        rew += reward
        env.render()
        time.sleep(0.01)
        if done:
            env.reset()
            if rew &gt; 0:
                success_ctr += 1
                print(action_list)
                action_dict.append(action_list)
                obs_dict.append(obs_list)
                pkl.dump(obs_dict, open("obslist.pkl", "wb"))
                pkl.dump(action_dict, open("action.pkl", "wb"))

    print(rew)
    time.sleep(1)

try:
    print(obs_dict.shape)
except:
    pass

print("Took key:", success_ctr)


I loaded the successful cases from my generated pickle file, and simulated the agent's playing using those exact same cases. But, the agent never receives a reward of 100. I dont understand why this is happening. A little search online suggested it could be because of noise in the game. So, I gave a sleep time, before running each episode. Still, doesn't work. Can someone please explain why is this happening? And suggest a way I could go about generating the training set?
What's exactly the point of time.sleep() in this code? I don't really understand it, you're simply stopping the execution of the program for $0.01$ seconds, how will that affect the simulator in any way ? It's not running in parallel, it does one step of the simulation when you call env.step function and returns the next state and reward. Calling sleep function only slows down the program here.  

The reason for the failure of successful trajectories, when repeated, is probably because the environment isn't stationary. That means that there are enemies or obstacles moving in the environment. If you simply repeat the trajectory that was successful once the enemies and obstacles might be in different positions and the agent will die. The reason why it succeeded the first time is because the agent got lucky. These are still valid learning trajectories because they were successful. The agent should learn reasons (features) why those trajectories were successful but not learn the trajectories themselves because, as you saw, they don't generalize well. If you plan on doing some kind of supervised learning approach you should also generate variety of unsuccessful trajectories so that the agent can learn what are correct and what are not correct actions depending on the current state in the environment.