title: lightGBM: Valid_sets predictions are not reproducible
body: <p>I am creating a model with lightGBM v3.1.1 in Python.<br />
  I used Optuna to explore the parameters and got the following values:</p>
  <pre><code>params = {'objective': 'regression',
            'metric': 'rmse',
            'verbosity': -1,
            'boosting_type': 'gbdt',
            'max_bin': 11,
            'max_depth': 5,
            'learning_rate': 0.1,
            'min_child_samples': 10,
            'seed': 123,
            'feature_pre_filter': False,
            'lambda_l1': 2.7055300945385866e-08,
            'lambda_l2': 1.1875908147914059e-07,
            'num_leaves': 31,
            'feature_fraction': 0.9272727272727272,
            'bagging_fraction': 0.4377408640572196,
            'bagging_freq': 6}
  </code></pre>
  <p>The datasets for training and validation were created as follows:</p>
  <pre><code>lgb_train = lgb.Dataset(data=X_train, label=y_train)
  lgb_test = lgb.Dataset(data=X_test, label=y_test, reference=lgb_train)
  </code></pre>
  <p>The training was done as follows.</p>
  <pre><code>evaluation_results  = {}
  gbm = lgb.train(
      params=params,
      train_set = lgb_train,
      valid_sets=[lgb_train, lgb_test],
      valid_names=['Train', 'Valid'],
      num_boost_round=2000,
      early_stopping_rounds=200,
      verbose_eval=10,
      feval = my_r2_score,
      evals_result = evaluation_results,
      keep_training_booster=True,
  )
  </code></pre>
  <p>The following function is used to calculate the R2 score.</p>
  <pre><code>def my_r2_score(preds, data):
      y_true = data.get_label()
      y_pred = np.where(preds &lt; 0, 0, preds)
      R2 = r2_score(y_true, y_pred)
  
      return 'R2', R2, True
  </code></pre>
  <p>The verbose output was as follows. Note that the r2 score of the validation sets is 0.908...</p>
  <pre><code>[LightGBM] [Warning] Accuracy may be bad since you didn't explicitly set num_leaves OR 2^max_depth &gt; num_leaves. (num_leaves=31).
  Training until validation scores don't improve for 200 rounds
  [10]    Train's rmse: 52.1459   Train's R2: -2.80291    Valid's rmse: 53.1739   Valid's R2: -2.20632
  [20]    Train's rmse: 52.1459   Train's R2: -2.80291    Valid's rmse: 53.1739   Valid's R2: -2.20632
  [30]    Train's rmse: 52.1459   Train's R2: -2.80291    Valid's rmse: 53.1739   Valid's R2: -2.20632
  [40]    Train's rmse: 52.1459   Train's R2: -2.80291    Valid's rmse: 53.1739   Valid's R2: -2.20632
  [50]    Train's rmse: 52.1459   Train's R2: -2.80291    Valid's rmse: 53.1739   Valid's R2: -2.20632
  [60]    Train's rmse: 52.1459   Train's R2: -2.80291    Valid's rmse: 53.1739   Valid's R2: -2.20632
  [70]    Train's rmse: 32.017    Train's R2: -0.433623   Valid's rmse: 32.2842   Valid's R2: -0.18192
  [80]    Train's rmse: 24.1125   Train's R2: 0.186874    Valid's rmse: 25.5191   Valid's R2: 0.261519
  [90]    Train's rmse: 18.4517   Train's R2: 0.523848    Valid's rmse: 20.2805   Valid's R2: 0.53359
  [100]   Train's rmse: 15.1617   Train's R2: 0.678508    Valid's rmse: 18.6286   Valid's R2: 0.606477
  ...
  [1000]  Train's rmse: 2.46019   Train's R2: 0.991535    Valid's rmse: 9.35062   Valid's R2: 0.900851
  [1010]  Train's rmse: 2.42361   Train's R2: 0.991785    Valid's rmse: 9.34548   Valid's R2: 0.90096
  [1020]  Train's rmse: 2.38906   Train's R2: 0.992018    Valid's rmse: 9.33087   Valid's R2: 0.901269
  [1030]  Train's rmse: 2.39676   Train's R2: 0.991966    Valid's rmse: 9.27854   Valid's R2: 0.902373
  [1040]  Train's rmse: 2.39676   Train's R2: 0.991966    Valid's rmse: 9.27854   Valid's R2: 0.902373
  [1050]  Train's rmse: 2.32587   Train's R2: 0.992434    Valid's rmse: 9.26247   Valid's R2: 0.902711
  [1060]  Train's rmse: 2.22859   Train's R2: 0.993054    Valid's rmse: 9.48976   Valid's R2: 0.897878
  Early stopping, best iteration is:
  [867]   Train's rmse: 2.98704   Train's R2: 0.987522    Valid's rmse: 8.97622   Valid's R2: 0.908631
  </code></pre>
  <p>However, when the data used for validation is used in the predict function, the scores are very different.
  This problem also occurs when the training data is predicted.</p>
  <pre><code>y_train_pred = gbm.predict(X_train, num_iteration=gbm.best_iteration)
  y_test_pred = gbm.predict(X_test, num_iteration=gbm.best_iteration)
  
  print('Train_RMSE : ',np.sqrt(mean_squared_error(y_train,np.where(y_train_pred &lt; 0, 0, y_train_pred))))
  print('Train_R2 : ',r2_score(y_train,np.where(y_train_pred &lt; 0, 0, y_train_pred)))
  print('Test_RMSE : ',np.sqrt(mean_squared_error(y_test,np.where(y_test_pred &lt; 0, 0, y_test_pred))))
  print('Test_R2 : ',r2_score(y_test,np.where(y_test_pred &lt; 0, 0, y_test_pred)))
  
  # Train_RMSE :  36.38169190538878
  # Train_R2 :  -0.8511459283386191
  # Test_RMSE :  40.49274115487656
  # Test_R2 :  -0.8593606256352719
  </code></pre>
  <p>In fact, contrary to my expectation, the prediction is not doing well either.<br />
  Considering the fact that the order of the true values and the predicted values are the same, I think it might be a problem with the output of the predict function.</p>
  <pre><code>print(list(y_train))
  
  # [0.31, 0.51, 1.63, 2.51, 15.64, 15.88, 16.34, 18.57, 22.57, 25.23, 25.72, 26.76, 34.16, 34.62, 34.89, 41.85, 47.35, 51.42, 53.77, 54.13, 54.53, 61.27, 63.4, 64.33, 65.02, 65.09, 65.26, 68.45, 70.34, 76.24, 81.44, 82.0, 90.08, 90.8]
  
  print(y_train_pred)
  
  # [-42.21137062 -42.5876803  -43.31970629 -40.15572661 -30.1829544
   -28.4649289  -28.28191554 -20.12112511 -23.34965759 -21.16999658
   -17.48387272 -15.26889047  -9.46770264 -18.03107688 -12.03409617
    -2.74651152   3.32581136   6.90444345   8.22034329  10.2525597
     9.66497184  17.06179049  18.95290265  18.91957477  21.54298163
    21.96186824  18.76801441  21.47902163  24.07296686  34.21581489
    35.69352109  39.98879279  44.43212373  44.57402187]
  
  print(list(y_test))
  
  # [2.16, 7.45, 27.17, 30.43, 46.2, 54.21, 65.93, 86.86, 88.43]
  
  print(y_test_pred)
  
  # [-34.33166873 -31.0656334   -0.34253524 -15.95319274  -2.61945177
    13.90965341  13.17142033  28.19411463  33.21328962]
  </code></pre>
  <p>I have also confirmed that the results of validation and prediction are slightly different even when the parameters are changed and training is done. (Even though I was using the same data.)</p>
  <p>I am unable to determine if this problem is due to the settings of the Dataset, train, and predict functions or if it is a bug.
  I would like to know why the results of validation and prediction are so different even though the same data is used, and how to deal with this problem.</p>