title: Keras - GRU layer with recurrent dropout - loss: 'nan', accuracy: 0
body: <h2>Problem description</h2>
  
  <p>I am going through "Deep Learning in Python" by François Chollet (<a href="https://www.manning.com/books/deep-learning-with-python" rel="nofollow noreferrer">publisher webpage</a>, <a href="https://github.com/fchollet/deep-learning-with-python-notebooks" rel="nofollow noreferrer">notebooks on github</a>). Replicating examples from Chapter 6 I encountered problems with (I believe) GRU layer with recurrent dropout.</p>
  
  <p>The code in which I had first observed those errors is quite long, so I decided to stick to the simplest problem, which could replicate the error: classifying IMDB reviews into "positive" and "negative" categories.</p>
  
  <p>When I use a GRU layer with recurrent dropout training loss (after couple of batches of first epoch) takes "value" of <code>nan</code>, while training accuracy (from the start of second epoch) takes the value of 0. </p>
  
  <pre><code>   64/12000 [..............................] - ETA: 3:05 - loss: 0.6930 - accuracy: 0.4844
    128/12000 [..............................] - ETA: 2:09 - loss: 0.6926 - accuracy: 0.4766
    192/12000 [..............................] - ETA: 1:50 - loss: 0.6910 - accuracy: 0.5573
  (...) 
   3136/12000 [======&gt;.......................] - ETA: 59s - loss: 0.6870 - accuracy: 0.5635
   3200/12000 [=======&gt;......................] - ETA: 58s - loss: 0.6862 - accuracy: 0.5650
   3264/12000 [=======&gt;......................] - ETA: 58s - loss: 0.6860 - accuracy: 0.5650
   3328/12000 [=======&gt;......................] - ETA: 57s - loss: nan - accuracy: 0.5667   
   3392/12000 [=======&gt;......................] - ETA: 57s - loss: nan - accuracy: 0.5560
   3456/12000 [=======&gt;......................] - ETA: 56s - loss: nan - accuracy: 0.5457
  (...)
  11840/12000 [============================&gt;.] - ETA: 1s - loss: nan - accuracy: 0.1593
  11904/12000 [============================&gt;.] - ETA: 0s - loss: nan - accuracy: 0.1584
  11968/12000 [============================&gt;.] - ETA: 0s - loss: nan - accuracy: 0.1576
  12000/12000 [==============================] - 83s 7ms/step - loss: nan - accuracy: 0.1572 - val_loss: nan - val_accuracy: 0.0000e+00
  Epoch 2/20
  
     64/12000 [..............................] - ETA: 1:16 - loss: nan - accuracy: 0.0000e+00
    128/12000 [..............................] - ETA: 1:15 - loss: nan - accuracy: 0.0000e+00
    192/12000 [..............................] - ETA: 1:16 - loss: nan - accuracy: 0.0000e+00
  (...)
  11840/12000 [============================&gt;.] - ETA: 1s - loss: nan - accuracy: 0.0000e+00
  11904/12000 [============================&gt;.] - ETA: 0s - loss: nan - accuracy: 0.0000e+00
  11968/12000 [============================&gt;.] - ETA: 0s - loss: nan - accuracy: 0.0000e+00
  12000/12000 [==============================] - 82s 7ms/step - loss: nan - accuracy: 0.0000e+00 - val_loss: nan - val_accuracy: 0.0000e+00
  Epoch 3/20
  
     64/12000 [..............................] - ETA: 1:18 - loss: nan - accuracy: 0.0000e+00
    128/12000 [..............................] - ETA: 1:18 - loss: nan - accuracy: 0.0000e+00
    192/12000 [..............................] - ETA: 1:16 - loss: nan - accuracy: 0.0000e+00
  (...)
  </code></pre>
  
  <h2>Localizing the problem</h2>
  
  <p>To find out the solution I wrote the code presented below, which goes through several models (GRU/LSTM, {no dropout, only "normal" dropout, only recurrent dropout, "normal" and recurrent dropout, rmsprop/adam}) and presents loss and accuracy of all those models. (It also creates smaller, separate graphs for each model.)</p>
  
  <pre><code># Based on examples from "Deep Learning with Python" by François Chollet:
  ## Constants, modules:
  VERSION = 2
  
  import os
  from keras import models
  from keras import layers
  import matplotlib.pyplot as plt
  import pylab
  
  ## Loading data:
  from keras.datasets import imdb
  (x_train, y_train), (x_test, y_test) = \
      imdb.load_data(num_words=10000)
  
  from keras.preprocessing import sequence
  x_train = sequence.pad_sequences(x_train, maxlen=500)
  x_test = sequence.pad_sequences(x_test, maxlen=500)
  
  
  ## Dictionary with models' hyperparameters:
  MODELS = [
      # GRU:
      {"no": 1,
       "layer_type": "GRU",
       "optimizer": "rmsprop",
       "dropout": None,
       "recurrent_dropout": None},
  
      {"no": 2,
       "layer_type": "GRU",
       "optimizer": "rmsprop",
       "dropout": 0.3,
       "recurrent_dropout": None},
  
      {"no": 3,
       "layer_type": "GRU",
       "optimizer": "rmsprop",
       "dropout": None,
       "recurrent_dropout": 0.3},
  
      {"no": 4,
       "layer_type": "GRU",
       "optimizer": "rmsprop",
       "dropout": 0.3,
       "recurrent_dropout": 0.3},
  
      {"no": 5,
       "layer_type": "GRU",
       "optimizer": "adam",
       "dropout": None,
       "recurrent_dropout": None},
  
      {"no": 6,
       "layer_type": "GRU",
       "optimizer": "adam",
       "dropout": 0.3,
       "recurrent_dropout": None},
  
      {"no": 7,
       "layer_type": "GRU",
       "optimizer": "adam",
       "dropout": None,
       "recurrent_dropout": 0.3},
  
      {"no": 8,
       "layer_type": "GRU",
       "optimizer": "adam",
       "dropout": 0.3,
       "recurrent_dropout": 0.3},
  
      # LSTM:
      {"no": 9,
       "layer_type": "LSTM",
       "optimizer": "rmsprop",
       "dropout": None,
       "recurrent_dropout": None},
  
      {"no": 10,
       "layer_type": "LSTM",
       "optimizer": "rmsprop",
       "dropout": 0.3,
       "recurrent_dropout": None},
  
      {"no": 11,
       "layer_type": "LSTM",
       "optimizer": "rmsprop",
       "dropout": None,
       "recurrent_dropout": 0.3},
  
      {"no": 12,
       "layer_type": "LSTM",
       "optimizer": "rmsprop",
       "dropout": 0.3,
       "recurrent_dropout": 0.3},
  
      {"no": 13,
       "layer_type": "LSTM",
       "optimizer": "adam",
       "dropout": None,
       "recurrent_dropout": None},
  
      {"no": 14,
       "layer_type": "LSTM",
       "optimizer": "adam",
       "dropout": 0.3,
       "recurrent_dropout": None},
  
      {"no": 15,
       "layer_type": "LSTM",
       "optimizer": "adam",
       "dropout": None,
       "recurrent_dropout": 0.3},
  
      {"no": 16,
       "layer_type": "LSTM",
       "optimizer": "adam",
       "dropout": 0.3,
       "recurrent_dropout": 0.3},
  ]
  
  ## Adding name:
  for model_dict in MODELS:
      model_dict["name"] = f"{model_dict['layer_type']}"
      model_dict["name"] += f"_d{model_dict['dropout']}" if model_dict['dropout'] is not None else f"_dN"
      model_dict["name"] += f"_rd{model_dict['recurrent_dropout']}" if model_dict['recurrent_dropout'] is not None else f"_rdN"
      model_dict["name"] += f"_{model_dict['optimizer']}"
  
  ## Fucntion - defing and training model:
  def train_model(model_dict):
      """Defines and trains a model, outputs history."""
  
      ## Defining:
      model = models.Sequential()
      model.add(layers.Embedding(10000, 32))
  
      recurrent_layer_kwargs = dict()
      if model_dict["dropout"] is not None:
          recurrent_layer_kwargs["dropout"] = model_dict["dropout"]
      if model_dict["recurrent_dropout"] is not None:
          recurrent_layer_kwargs["recurrent_dropout"] = model_dict["recurrent_dropout"]
  
      if model_dict["layer_type"] == 'GRU':
          model.add(layers.GRU(32, **recurrent_layer_kwargs))
      elif model_dict["layer_type"] == 'LSTM':
          model.add(layers.LSTM(32, **recurrent_layer_kwargs))
      else:
          raise ValueError("Wrong model_dict['layer_type'] value...")
      model.add(layers.Dense(1, activation='sigmoid'))
  
      ## Compiling:
      model.compile(
          optimizer=model_dict["optimizer"],
          loss='binary_crossentropy',
          metrics=['accuracy'])
  
      ## Training:
      history = model.fit(x_train, y_train,
                          epochs=20,
                          batch_size=64,
                          validation_split=0.2)
  
      return history
  
  ## Multi-model graphs' parameters:
  graph_all_nrow = 4
  graph_all_ncol = 4
  graph_all_figsize = (20, 20)
  
  assert graph_all_nrow * graph_all_nrow &gt;= len(MODELS)
  
  ## Figs and axes of multi-model graphs:
  graph_all_loss_fig, graph_all_loss_axs = plt.subplots(graph_all_nrow, graph_all_ncol, figsize=graph_all_figsize)
  graph_all_acc_fig, graph_all_acc_axs = plt.subplots(graph_all_nrow, graph_all_ncol, figsize=graph_all_figsize)
  
  ## Loop trough all models:
  for i, model_dict in enumerate(MODELS):
      history = train_model(model_dict)
  
      ## Metrics extraction:
      loss = history.history['loss']
      val_loss = history.history['val_loss']
      acc = history.history['accuracy']
      val_acc = history.history['val_accuracy']
  
      epochs = range(1, len(loss) + 1)
  
      ## Single-model grph - loss:
      graph_loss_fname = fr"{os.path.basename(__file__).replace('.py', '')}"
      graph_loss_fname += fr"_v{VERSION}_{model_dict['no']}_{model_dict['name']}_loss_graph.png"
  
      graph_loss_fig, graph_loss_ax = plt.subplots()
      graph_loss_ax.plot(epochs, loss, 'bo', label='Training loss')
      graph_loss_ax.plot(epochs, val_loss, 'b', label='Validation loss')
      graph_loss_ax.legend()
      graph_loss_fig.suptitle("Training and validation loss")
      graph_loss_fig.savefig(graph_loss_fname)
      pylab.close(graph_loss_fig)
  
  
      ## Single-model grph - accuracy:
      graph_acc_fname = fr"{os.path.basename(__file__).replace('.py', '')}"
      graph_acc_fname += fr"_v{VERSION}_{model_dict['no']}_{model_dict['name']}_acc_graph.png"
  
      graph_acc_fig, graph_acc_ax = plt.subplots()
      graph_acc_ax.plot(epochs, acc, 'bo', label='Training accuracy')
      graph_acc_ax.plot(epochs, val_acc, 'b', label='Validation accuracy')
      graph_acc_ax.legend()
      graph_acc_fig.suptitle("Training and validation acc")
      graph_acc_fig.savefig(graph_acc_fname)
      pylab.close(graph_acc_fig)
  
      ## Position of axes on multi-model graph:
      i_row = i // graph_all_ncol
      i_col = i % graph_all_ncol
  
      ## Adding model metrics to multi-model graph - loss:
      graph_all_loss_axs[i_row, i_col].plot(epochs, loss, 'bo', label='Training loss')
      graph_all_loss_axs[i_row, i_col].plot(epochs, val_loss, 'b', label='Validation loss')
      graph_all_loss_axs[i_row, i_col].set_title(fr"{model_dict['no']}. {model_dict['name']}")
  
      ## Adding model metrics to multi-model graph - accuracy:
      graph_all_acc_axs[i_row, i_col].plot(epochs, acc, 'bo', label='Training acc')
      graph_all_acc_axs[i_row, i_col].plot(epochs, val_acc, 'b', label='Validation acc')
      graph_all_acc_axs[i_row, i_col].set_title(fr"{model_dict['no']}. {model_dict['name']}")
  
  
  ## Saving multi-model graphs:
  # Output files are quite big (8000x8000 PNG), you may want to decrease DPI.
  graph_all_loss_fig.savefig(fr"{os.path.basename(__file__).replace('.py', '')}_ALL_loss_graph.png", dpi=400)
  graph_all_acc_fig.savefig(fr"{os.path.basename(__file__).replace('.py', '')}_ALL_acc_graph.png", dpi=400)
  </code></pre>
  
  <p>Please find two main graphs below: <a href="https://i.stack.imgur.com/QnINp.png" rel="nofollow noreferrer">Loss - binary crossentropy</a>, <a href="https://i.stack.imgur.com/8SUaW.png" rel="nofollow noreferrer">Accuracy</a> (I am not allowed te embed images in post due to low reputation).</p>
  
  <p>I have also obtained similarly strange problems in regression model - the MAE was in range of several <em>thousands</em> - in the problem where $y$ range was maybe of several <em>tens</em>. (I decided not to include this model here, because it would make this question even longer.)</p>
  
  <h2>Versions of modules and libraries, hardware</h2>
  
  <ul>
  <li>Modules:</li>
  </ul>
  
  <pre><code>Keras                    2.3.1
  Keras-Applications       1.0.8
  Keras-Preprocessing      1.1.0
  matplotlib               3.1.3
  tensorflow-estimator     1.14.0
  tensorflow-gpu           2.1.0
  tensorflow-gpu-estimator 2.1.0
  </code></pre>
  
  <ul>
  <li><code>keras.json</code> file:</li>
  </ul>
  
  <pre><code>{
      "floatx": "float32",
      "epsilon": 1e-07,
      "backend": "tensorflow",
      "image_data_format": "channels_last"
  }
  </code></pre>
  
  <ul>
  <li>CUDA - I have CUDA 10.0 and CUDA 10.1 installed on my system.</li>
  <li>CUDnn - I have three versions: cudnn-10.0 v7.4.2.24, cudnn-10.0 v7.6.4.38, cudnn-9.0 v7.4.2.24</li>
  <li>GPU: Nvidia GTX 1050Ti 4gb</li>
  <li>Windows 10 Home</li>
  </ul>
  
  <h2>Questions</h2>
  
  <ol>
  <li>Do you know what may be the reason of this behavior?</li>
  <li>Is it possible that this is caused by multiple CUDA and CUDnn installations? Before observing the problem I have trained several models (both from book and my own ones) and the seemed to behave mor or less as expected, while having 2 CUDA and 2 CUDnn versions (those above without cudnn-10.0 v7.6.4.38) installed.</li>
  <li>Is there any official/good source of adequate combinations of keras, tensorflow, CUDA, CUDnn (and  other relevant things e.g. maybe Visual Studio)? I cannot really find any authoritative and up-to-date source.</li>
  </ol>
  
  <p>I hope I've described everything clearly enough. If you have any questions, please ask.</p>