title: Lightgbm scores for custom RMSE loss function and build-in RMSE are different
body: <p>To start with custom objective functions for lightgbm I started to reproduce standard objective RMSE. Unfortunately, the scores are different.
  My the example is based on this <a href="https://towardsdatascience.com/custom-loss-functions-for-gradient-boosting-f79c1b40466d" rel="nofollow noreferrer">post</a> or <a href="https://github.com/manifoldai/mf-eng-public/blob/master/notebooks/custom_loss_lightgbm.ipynb" rel="nofollow noreferrer">github</a>.<br>
  Grad and hess are the same as in lightgbm <a href="https://github.com/microsoft/LightGBM/blob/1c27a15e42f0076492fcc966b9dbcf9da6042823/src/objective/regression_objective.hpp#L702-L732" rel="nofollow noreferrer">source</a> or as given in the answer to following <a href="https://stackoverflow.com/questions/60290304/reproduce-lightgbm-custom-loss-function-for-regression">question</a>. <br>
  What is wrong with the custom RMSE function? <br>
  Remark: In this example the final loss seems to be close but the trajectory is totally different. I experience even bigger differences in final loss in other (bigger) examples. </p>
  
  <pre><code>import numpy as np
  import matplotlib.pyplot as plt
  from lightgbm import LGBMRegressor
  import lightgbm 
  from sklearn.datasets import  make_friedman1
  from sklearn.model_selection import train_test_split
  
  X, y = make_friedman1(n_samples=10000, n_features=7, noise=10.0, random_state=11)
  X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.20, random_state=42)
  
  gbm2 = lightgbm.LGBMRegressor(objective='rmse', random_state=33, early_stopping_rounds = 5, n_estimators=10000)
  gbm2.fit(X_train,  y_train, eval_set=[(X_valid, y_valid)], eval_metric='rmse', verbose=10)
  gbm2eval = gbm2.evals_result_
  
  def custom_RMSE(y_true, y_pred):
      residual = (y_pred - y_true)
      grad = residual
      hess = np.ones(len(y_true))
      return grad, hess
  
  gbm3 = lightgbm.LGBMRegressor(random_state=33, early_stopping_rounds = 5, n_estimators=10000)
  gbm3.set_params(**{'objective': custom_RMSE})
  gbm3.fit(X_train,  y_train, eval_set=[(X_valid, y_valid)], eval_metric='rmse', verbose=10)
  gbm3eval = gbm3.evals_result_
  
  plt.plot(gbm2eval['valid_0']['rmse'],label='rmse')
  plt.plot(gbm3eval['valid_0']['rmse'],label='custom rmse')
  plt.legend()
  </code></pre>
  
  <p>eval_results for gbm2:</p>
  
  <pre><code>Training until validation scores don't improve for 5 rounds
  [10]    valid_0's rmse: 10.214
  [20]    valid_0's rmse: 10.044
  [30]    valid_0's rmse: 10.0348
  Early stopping, best iteration is:
  [28]    valid_0's rmse: 10.028
  </code></pre>
  
  <p>eval_results for gbm3:</p>
  
  <pre><code>Training until validation scores don't improve for 5 rounds
  [10]    valid_0's rmse: 11.5991 valid_0's l2: 134.539
  [20]    valid_0's rmse: 10.2721 valid_0's l2: 105.516
  [30]    valid_0's rmse: 10.0801 valid_0's l2: 101.608
  [40]    valid_0's rmse: 10.0424 valid_0's l2: 100.849
  Early stopping, best iteration is:
  [44]    valid_0's rmse: 10.0351 valid_0's l2: 100.703
  </code></pre>
  
  <p>and here as plot:
  <a href="https://i.stack.imgur.com/5bEp0.png" rel="nofollow noreferrer">losses for standard RMSE and custom RMSE</a></p>