title: Keras parameter search and log loss
body: <p>I'm building a Keras simple classifier model for a multiclass problem where classes are not exclusive. In addition, I would like to add sklearn exhaustive parameter search with cross-validation. This is the trivial model:</p>
  
  <pre class="lang-python prettyprint-override"><code>def build_model(embedding_size):
      model = Sequential()
      model.add(Embedding(10000, embedding_size, input_length=100))
      model.add(Flatten())
      model.add(Dense(100, activation="relu"))
      model.add(Dense(10, activation="sigmoid"))
      model.compile(loss="binary_crossentropy", optimizer=Adam(), metrics=["accuracy"])
      return model
  
  batch_size = 64
  epochs = 2
  </code></pre>
  
  <p>Here's my first solution with manual cross-validation (without parameter search).</p>
  
  <pre class="lang-python prettyprint-override"><code>embedding_size=32
  
  skf = KFold(n_splits=3, shuffle=True)
  valid_evals = []
  for train_indices in skf.split(train_full_x):
      kfold_train_x = train_full_x[train_indices[0]]
      kfold_valid_x = train_full_x[train_indices[1]]
      kfold_train_y = train_full_y[train_indices[0]]
      kfold_valid_y = train_full_y[train_indices[1]]
      model = build_model(embedding_size)
      model.fit(kfold_train_x, kfold_train_y, batch_size=batch_size, epochs=epochs)
      valid_evals.append(model.evaluate(kfold_valid_x, kfold_valid_y))
  
  valid_evals = np.array(valid_evals)
  print(valid_evals)
  print("LogLoss: %.4f +/- %.4f" % (valid_evals.mean(axis=0)[0], valid_evals.std(axis=0)[0]))
  print("Accuracy: %.2f%% +/- %.2f%%" % (valid_evals.mean(axis=0)[1] * 100, valid_evals.std(axis=0)[1] * 100))
  </code></pre>
  
  <p>The results are quite consistent:</p>
  
  <pre><code>[[0.05730336 0.98051361]
   [0.0606665  0.98065738]
   [0.05717109 0.9801999 ]]
  LogLoss: 0.0584 +/- 0.0016
  Accuracy: 98.05% +/- 0.02%
  </code></pre>
  
  <p>This code works fine, returns a validation accuracy of ~98% and a log loss of about <code>0.06</code>. I then tried adding <code>GridSearchCV</code> instead of rolling my own solution for parameter search (in the sample below the search space is trivially reduced to one choice only).</p>
  
  <pre><code>embedding_size = [32]
  classifier = KerasClassifier(build_fn=build_model, epochs=epochs, batch_size=batch_size)
  kfold = KFold(n_splits=3, shuffle=True)
  param_grid = dict(embedding_size=embedding_size)
  grid = GridSearchCV(estimator=classifier, param_grid=param_grid, return_train_score=True, cv=kfold)
  results = grid.fit(train_x, train_y)
  print("Best: %f using %s" % (results.best_score_, results.best_params_))
  print(results.cv_results_)
  </code></pre>
  
  <p>This returns </p>
  
  <pre><code>Best: 0.980600 using {'embedding_size': 32}
  0.980600 (0.000187) with: {'embedding_size': 32}
  {'split2_train_score': array([0.9882294]), 'mean_train_score': array([0.98768258]), 'split0_train_score': array([0.98707933]), 'split1_train_score': array([0.98773903]), 'std_train_score': array([0.00047121]), 'mean_fit_time': array([8.7320834]), 'split1_test_score': array([0.98079212]), 'split0_test_score': array([0.98066088]), 'std_fit_time': array([0.19671295]), 'rank_test_score': array([1], dtype=int32), 'std_test_score': array([0.00018667]), 'param_embedding_size': masked_array(data=[32], mask=[False], fill_value='?', dtype=object), 'std_score_time': array([0.00984431]), 'split2_test_score': array([0.98034717]), 'mean_score_time': array([0.58589784]), 'params': [{'embedding_size': 32}], 'mean_test_score': array([0.98060006])}
  </code></pre>
  
  <p>Here I get an accuracy compatible with Keras manually splitting the training/validation. However, in my problem the relevant scoring function is the log loss, so I tried adding a <code>scoring="neg_log_loss"</code> as a parameter to <code>GridSearchCV</code>. <strong>But in this case I get a score like <code>-0.29</code>.</strong> </p>
  
  <pre><code>Best: -0.292518 using {'embedding_size': 32}
  -0.292518 (0.002988) with: {'embedding_size': 32}
  {'split1_train_score': array([-0.27363595]), 'std_score_time': array([0.01245312]), 'rank_test_score': array([1], dtype=int32), 'mean_test_score': array([-0.29251778]), 'std_fit_time': array([0.17412529]), 'split0_train_score': array([-0.27816725]), 'split1_test_score': array([-0.28917072]), 'mean_score_time': array([0.40924891]), 'params': [{'embedding_size': 32}], 'split0_test_score': array([-0.29195843]), 'split2_test_score': array([-0.2964242]), 'split2_train_score': array([-0.26628012]), 'mean_fit_time': array([8.75646718]), 'mean_train_score': array([-0.27269444]), 'std_test_score': array([0.00298751]), 'std_train_score': array([0.00489836]), 'param_embedding_size': masked_array(data=[32], mask=[False], fill_value='?', dtype=object)}
  </code></pre>
  
  <p>I expected a difference in sign (due to how <code>neg_log_loss</code> works) but not in value. </p>
  
  <p>What am I doing wrong? Or... is there another way to get <code>GridSearchCV</code> working with cross-validation and log loss?</p>
  
  <p>Thank you</p>
  
  <p>N.B. I did not set a random seed, but instead run both examples many times, so that randomness should not be the cause of the difference shown.</p>