title: keras gradients computation fails "Invalid index from the dimension: 3, 0, C"
body: <p>When I try to compute the gradient of X w.r.t Y (doesn't really matter what X nor Y is) in a network with a conv1d layer I'm getting the message "Invalid index from the dimension: 3, 0, C" and the process dies.</p>
  
  <p>Minimal working example:</p>
  
  <pre><code>import numpy as np
  
  from tensorflow.python.keras import models
  from tensorflow.python.keras import layers
  from tensorflow.python.keras import backend as K
  
  inp = layers.Input(shape=(10, 20,))
  conv = layers.Conv1D(filters=10, kernel_size=2)(inp)
  pool = layers.GlobalMaxPool1D()(conv)
  output = layers.Dense(1, activation="sigmoid")(pool)
  
  m = models.Model(inp, output)
  
  m.summary()
  
  m.compile(optimizer="adam", loss="binary_crossentropy")
  </code></pre>
  
  <p>Seems to work:</p>
  
  <pre><code>m.fit(x=np.random.randn(100, 10, 20), y=np.random.randn(100))
  </code></pre>
  
  <p>This breaks:</p>
  
  <pre><code>loss = K.mean(m.output)
  grads = K.gradients(loss, m.input)[0]
  f = K.function([m.input], [grads])
  print(f([np.random.randn(10, 20)]))
  </code></pre>
  
  <p>My python, keras, tf versions:</p>
  
  <pre><code>import tensorflow as tf
  import sys
  from tensorflow.python import keras
  
  print(tf.__version__)
  print(keras.__version__)
  print(sys.version)
  
  1.12.0
  2.1.6-tf
  3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 14:01:38) 
  [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]
  </code></pre>
  
  <p>It doesn't really matter of what I calculate the gradient wrt to what.. The error message is </p>
  
  <pre><code>2019-04-19 17:00:58.249788: F ./tensorflow/core/util/tensor_format.h:420] Check failed: index &gt;= 0 &amp;&amp; index &lt; dimension_attributes.size() Invalid index from the dimension: 3, 0, C
  </code></pre>
  
  <p>I see that it is related to the conv 1d layer based on the error message, but I don't quite understand what I'm missing here. Thanks for any hints.</p>