title: Model training hangs forever when using a Tensorboard callback with an LSTM layer (cuDNN implementation) - Tensorflow 2.3
body: <p>We decided to migrate our codebase from Tensorflow 1.13.1 to Tensorflow 2.3.</p>
  <p>Everything seems to work properly, except when training a LSTM model with a Tensorboard callback. The training process hangs forever after the first batch (maybe Tensorboard waits for never coming summary ops ?).</p>
  <p>After some investigations, it seems that the issue comes from the LSTM layer relying on the cuDNN implementation. Indeed, everything works as intended when forcing Tensorflow to fall back on the vanilla LSTM layer implementation (giving it incompatible arguments with the cuDNN implementation, c.f. <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM" rel="nofollow noreferrer">https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM</a>).</p>
  <p>Any suggestions to fix this ? Does anyone has any idea on why Tensorboard does not work with cuDNN implementation of the LSTM layer ?</p>
  <pre class="lang-py prettyprint-override"><code># Callbacks
  log_dir = './logs'
  callbacks = [TensorBoard(log_dir=log_dir)] # Training works without this callback
  
  # Model
  words = Input(shape=(max_sequence_length,))
  x = Embedding(input_dim, embedding_size, weights=[embedding_matrix], trainable=False)(words)
  x = Bidirectional(LSTM(40, return_sequences=True))(x)  # Setting recurrent_dropout = 0.1 makes the training work (using the vanilla LSTM implementation)
  hidden = concatenate([
      GlobalMaxPooling1D()(x),
      GlobalAveragePooling1D()(x),
  ])
  hidden = add([hidden, Dense(40, activation='relu')(hidden)])
  out = Dense(3, activation='softmax', kernel_initializer='glorot_uniform')(hidden)
  model = Model(inputs=words, outputs=[out])
  model.compile(optimizer=Adam(lr=0.01), loss='categorical_crossentropy')
  
  # Fit
  fit_history = model.fit(
      x_train,
      y_train,
      batch_size=batch_size,
      epochs=epochs,
      validation_data=validation_data,
      callbacks=callbacks
  )
  </code></pre>
  <p>UPDATE :
  Tried it on another computer (Windows 10) and everything seems to work correctly...<br />
  However, computers at work are mostly on Windows 7 (issue happened on 2 separate PC), and we are using virtual envs.<br />
  Another hint : can the fact that python and our library are installed on different hard drives may cause the issue? (that's one of the similarities of the setups where the issue arises)</p>