title: Keras Error: "Optimization loop failed: Cancelled: Operation was cancelled" When Calling predict_on_batch
body: <p>I had some older working code that used keras. I recently dusted it off and tried to use it but with current versions of keras/tensorflow. I am getting a warning/error when calling predict_on_batch:</p>
  <p>W tensorflow/core/data/root_dataset.cc:167] Optimization loop failed: Cancelled: Operation was cancelled</p>
  <p>I tried to google this and, to my surprise, there doesn't seem to be a good explanation on the web as to what causes this or how to fix it. Here is what I did find:</p>
  <p><a href="https://github.com/tensorflow/tensorflow/issues/48689" rel="nofollow noreferrer">https://github.com/tensorflow/tensorflow/issues/48689</a></p>
  <p><a href="https://discuss.tensorflow.org/t/optimization-loop-failed-cancelled-operation-was-cancelled/1524" rel="nofollow noreferrer">https://discuss.tensorflow.org/t/optimization-loop-failed-cancelled-operation-was-cancelled/1524</a></p>
  <p>The one answer it lists is to make sure the batch size isn't larger than the whole set. That isn't the case here.</p>
  <p>The code is a bit long, so I can't easily show it all. It's a deep reinforcement learning app so the DL code is split up across two main functions which I'll show here:</p>
  <p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
  <div class="snippet-code">
  <pre class="snippet-code-html lang-html prettyprint-override"><code>class DQN(QContract):
      def __init__(self, states, actions, lr, DDQN=False):
          self.history = []
          act_relu = activations.relu
          act_linear = activations.linear
          top_layer = 150
          middle_layer = 120
          # Create Network: Default Parameters from https://towardsdatascience.com/solving-lunar-lander-openaigym-reinforcement-learning-785675066197
          model = Sequential()
          layer = layers.Dense(top_layer, input_dim=states, activation=act_relu)
          model.add(layer)
          layer = layers.Dense(middle_layer, activation=act_relu)
          model.add(layer)
          layer = layers.Dense(actions, activation=act_linear)
          model.add(layer)
          opt = optimizers.Adam(learning_rate=lr)
          model.compile(loss='mse', optimizer=opt)
          # Create DDQN-like networks
          self.modelA = model
          #self.modelB = copy.deepcopy(model)
          self.batch_size = 100
          self.current = "A"
          self.count = 0</code></pre>
  </div>
  </div>
  </p>
  <p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
  <div class="snippet-code">
  <pre class="snippet-code-html lang-html prettyprint-override"><code>    def Update(self, state, action, reward, new_state, gamma, alpha=None):
          # Preform Replay
          row_count = self.batch_size
          if len(self.history) &lt; row_count: return
  
          # Column names
          state = 0
          action = 1
          reward = 2
          next_state = 3
          done = 4
          # Get samples in mini-batches
          samples = random.sample(self.history, row_count)
          # Separate into separate arrays
          states_array = [sample[state] for sample in samples]
          actions_array = [sample[action] for sample in samples]
          rewards_array = [sample[reward] for sample in samples]
          next_states_array = [sample[next_state] for sample in samples]
          done_array = [sample[done] for sample in samples]
          # Turn into arrays
          states_array = np.array(states_array)
          actions_array = np.array(actions_array)
          rewards_array = np.array(rewards_array)
          next_states_array = np.array(next_states_array)
          done_array = (1.0 - np.array(done_array))
  
          # train on states_array
          X = states_array
  
          # Create y (i.e. labels for supervised learning)
          if self.current == "A":
              model1 = self.modelA
              model2 = self.modelA
          else:
              model1 = self.modelA
              model2 = self.modelA
  
          predicted_values = self.modelA.predict_on_batch(states_array)
          next_predicted_values = self.modelA.predict_on_batch(next_states_array)
          actual_values = rewards_array + gamma * np.amax(next_predicted_values, axis=1) * done_array
  
          predicted_values[list(range(row_count)), actions_array] = actual_values
          y = predicted_values
  
          # Update network
          self.current = "A"
          if self.current == "A":
              print('Do fit'+str(self.count))
              self.count += 1
              self.modelA.fit(X, y, epochs=1, verbose=0)
              self.current = "B"
          else:
              self.modelA.fit(X, y, epochs=1, verbose=0)
              self.current = "A"</code></pre>
  </div>
  </div>
  </p>
  <p>At one point I was trying to do a DQN and I'm not right now, so ignore the attempt to have two models. It's currently disabled.</p>
  <p>It seems like this would be a fairly straightforward issue, but I can't seem to figure it out. I even tried stepping through the code and I found it didn't happen when stepping through the debugger.</p>