Converting sparse IndexedSlices to a dense Tensor
I've had the same issue with the Embedding layer in Keras. The solution is to explicitly use a TensorFlow optimizer, like here:

model.compile(loss='mse',
               optimizer=TFOptimizer(tf.train.GradientDescentOptimizer(0.1)))