title: How to calculate leaf values in LightGBM Classifier?
body: <p>I'm trying to understand how to calculate leaf values in LightGBM Classifier. I built a simple model with n_estimator=1 and max_depth=1, that means it has just one decision tree and one splitting point. I compared the scores output by the model and the scores which I calculated by myself using Python. As a result, there is a slight difference between them: -0.000457 in leaf 0 and -0.001548 in leaf 1. Is that due to the difference that LightGBM is written in C++ and I used Python for calculation? Or, is there anything wrong with my calculation steps?</p>
  
  <p>references:</p>
  
  <p><a href="https://stackoverflow.com/questions/50188797/what-is-leaf-values-from-python-lightgbm">What is leaf_values from Python LightGBM?</a></p>
  
  <p><a href="https://github.com/microsoft/LightGBM/issues/2283" rel="nofollow noreferrer">[Question] How leaf output determined for LambdaMART?</a></p>
  
  <pre><code>import pandas as pd
  import lightgbm as lgb
  from sklearn import datasets
  
  ## load data
  data = datasets.load_breast_cancer()
  df = pd.DataFrame(data.data, columns=data.feature_names)
  df['label'] = data.target
  
  ## build a simple model
  params = {'max_depth':1, 'n_estimators': 1, 'learning_rate':0.1}
  model = lgb.LGBMClassifier(**params)
  model.fit(df.drop('label', axis=1), df['label'])
  
  ## calculate scores by the model
  df_pred = df[['label']].copy()
  
  df_pred['score_raw'] = model.predict_proba(df.drop('label', axis=1), raw_score=True)
  df_pred['score_proba'] = model.predict_proba(df.drop('label', axis=1))[:, 1]
  df_pred['leaf'] = model.predict_proba(df.drop('label', axis=1), pred_leaf=True)
  
  ## calculate scores by myself
  start_value = len(df[df['label']==1])/len(df)
  df_pred['gradient'] = start_value - df_pred['label']
  df_pred['hessian'] = 1
  
  def calc_score_proba_(leaf):
      sum_gradients = df_pred[df_pred['leaf']==leaf]['gradient'].sum()
      sum_hessiants = df_pred[df_pred['leaf']==leaf]['hessian'].sum()
      delta = -sum_gradients / sum_hessiants * model.learning_rate
      df_pred.loc[df_pred['leaf']==leaf, 'score_proba_'] = start_value + delta
  
  calc_score_proba_(0)
  calc_score_proba_(1)
  
  ## calculate the gap between the scores
  df_pred['score_proba_gap'] = df_pred['score_proba'] - df_pred['score_proba_']
  
  df_pred
      label   score_raw   score_prob  leaf    gradient    hessian score_prob_ score_prob_gap
  0   0   0.275629    0.568474    1   0.627417    1   0.570022    -0.001548
  1   0   0.275629    0.568474    1   0.627417    1   0.570022    -0.001548
  2   0   0.275629    0.568474    1   0.627417    1   0.570022    -0.001548
  3   0   0.641339    0.655056    0   0.627417    1   0.655513    -0.000457
  4   0   0.275629    0.568474    1   0.627417    1   0.570022    -0.001548
  ... ... ... ... ... ... ... ... ...
  564 0   0.275629    0.568474    1   0.627417    1   0.570022    -0.001548
  565 0   0.275629    0.568474    1   0.627417    1   0.570022    -0.001548
  566 0   0.275629    0.568474    1   0.627417    1   0.570022    -0.001548
  567 0   0.275629    0.568474    1   0.627417    1   0.570022    -0.001548
  568 1   0.641339    0.655056    0   -0.372583   1   0.655513    -0.000457
  569 rows × 8 columns
  
  lgb.create_tree_digraph(model)
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/k3ZzP.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/k3ZzP.png" alt="enter image description here"></a></p>