title: How to solve “nan loss” in Keras NN, arising with high number of features?
body: <p>I am training a neural network using Keras, but get a nan loss when adding more features. I have tried the following solutions, mentioned in other discussions: lower learning rate, a different optimizer, adding epsilon=1e-08 / clipnorm=1. argument for optimizer. However, this did not solve it. </p>
  
  <p>The dataset is small, with 1000 elements. I use different feature sets. When using 25 features there is no problem and good performance. But when using all 53 features, I get only nan losses. (When using the feature sets separately, there is no problem either so I think the problem is not in the feature sets themselves but in their high number). The network is below. Any suggestions to solve this? </p>
  
  <pre><code>embedding_layer = Embedding(nb_words,
                              EMBEDDING_DIM,
                              weights=[embedding_matrix],
                              input_length=300,
                              trainable=False)
  lstm_layer = LSTM(75, recurrent_dropout=0.2)
  
  sequence_1_input = Input(shape=(300,), dtype="int32")
  embedded_sequences_1 = embedding_layer(sequence_1_input)
  x1 = lstm_layer(embedded_sequences_1)
  
  sequence_2_input = Input(shape=(300,), dtype="int32")
  embedded_sequences_2 = embedding_layer(sequence_2_input)
  y1 = lstm_layer(embedded_sequences_2)
  
  features_input = Input(shape=(f_train.shape[1],), dtype="float32")
  features_dense = BatchNormalization()(features_input)
  features_dense = Dense(200, activation="relu")(features_dense)
  features_dense = Dropout(0.2)(features_dense)
  
  addition = add([x1, y1])
  minus_y1 = Lambda(lambda x: -x)(y1)
  merged = add([x1, minus_y1])
  merged = multiply([merged, merged])
  merged = concatenate([merged, addition])
  merged = Dropout(0.4)(merged)
  
  merged = concatenate([merged, features_dense])
  merged = BatchNormalization()(merged)
  merged = GaussianNoise(0.1)(merged)
  
  merged = Dense(150, activation="relu")(merged)
  merged = Dropout(0.2)(merged)
  merged = BatchNormalization()(merged)
  
  out = Dense(1, activation="sigmoid")(merged)
  optimizer = nadam(epsilon=1e-08, lr = 0.00001)
  model = Model(inputs=[sequence_1_input, sequence_2_input, features_input], 
    outputs=out) 
  model.compile(loss="binary_crossentropy", optimizer=optimizer)
  early_stopping = EarlyStopping(monitor="val_loss", patience=5)
  best_model_path = "best_model" + str(model_count) + ".h5"
  model_checkpoint = ModelCheckpoint(best_model_path, save_best_only=True, 
  save_weights_only=True)
  
  hist = model.fit([data_1_train, data_2_train, f_train], labels_train,
                   validation_data=([data_1_val, data_2_val, f_val], labels_val),
                   epochs=15, batch_size=BATCH_SIZE, shuffle=True,
                   callbacks=[early_stopping, model_checkpoint], verbose=1)
  </code></pre>