title: Q-values exploding when training DQN
body: <p>I'm training a DQN to play OpenAI's Atari environment, but the Q-values of my network quickly explode far above what is realistic.</p>
  
  <p>Here's the relevant portion of the code:</p>
  
  <pre><code>for state, action, reward, next_state, done in minibatch:
          if not done:
              # To save on memory, next_state is just one frame
              # So we have to add it to the current state to get the actual input for the network
              next_4_states = np.array(state)
              next_4_states = np.roll(next_4_states, 1, axis=3)
              next_4_states[:, :, :, 0] = next_state
              target = reward + self.gamma * \
                  np.amax(self.target_model.predict(next_4_states))
          else:
              target = reward
          target_f = self.target_model.predict(state)
          target_f[0][action] = target
  
          self.target_model.fit(state, target_f, epochs=1, verbose=0)
  </code></pre>
  
  <p>The discount factor is 0.99 (it doesn't happen with discount factor 0.9, but also doesn't converge because it can't think far enough ahead).</p>
  
  <p>Stepping through the code, the reason it's happening is all the Q values that aren't meant to be updated (the ones for actions we didn't take) increase slightly. It's my understanding that passing the networks own output to the network during training should keep the output the same, not increase or decrease it. Is there something wrong with my model? Is there some way I can mask the update so it only updates the relevant Q value?</p>
  
  <p>EDIT: My model creation code is here:</p>
  
  <pre><code>def create_model(self, input_shape, num_actions, learning_rate):
          model = Sequential()
          model.add(Convolution2D(32, 8, strides=(4, 4),
                                  activation='relu', input_shape=(input_shape)))
          model.add(Convolution2D(64, 4, strides=(2, 2), activation='relu'))
          model.add(Convolution2D(64, 3, strides=(1, 1), activation='relu'))
          model.add(Flatten())
          model.add(Dense(512, activation='relu'))
          model.add(Dense(num_actions))
  
          model.compile(loss='mse', optimizer=Adam(lr=learning_rate))
  
          return model
  </code></pre>
  
  <p>I create two of these. One for the online network and one for the target.</p>