title: How to create custom eval metric for catboost?
body: <p>Similar SO questions:</p>
  <ul>
  <li><a href="https://stackoverflow.com/questions/61354410/python-catboost-multiclass-f1-score-custom-metric">Python Catboost: Multiclass F1 score custom metric</a></li>
  </ul>
  <p>Catboost tutorials</p>
  <ul>
  <li><a href="https://catboost.ai/docs/concepts/python-usages-examples.html#user-defined-loss-function" rel="noreferrer">https://catboost.ai/docs/concepts/python-usages-examples.html#user-defined-loss-function</a></li>
  </ul>
  <h1>Question</h1>
  <p>In this question, I have a binary classification problem. After modelling we get the test model predictions <code>y_pred</code> and we already have true test labels <code>y_true</code>.</p>
  <p>I would like to get the custom evaluation metric defined by following equation:</p>
  <pre><code>profit = 400 * truePositive - 200*fasleNegative - 100*falsePositive
  </code></pre>
  <p>Also, since higher profit is better I would like to maximize the function instead of minimize it.</p>
  <p>How to get this eval_metric in catboost?</p>
  <h1>Using sklearn</h1>
  <pre class="lang-py prettyprint-override"><code>def get_profit(y_true, y_pred):
      tn, fp, fn, tp = sklearn.metrics.confusion_matrix(y_true,y_pred).ravel()
      loss = 400*tp - 200*fn - 100*fp
      return loss
  
  scoring = sklearn.metrics.make_scorer(get_profit, greater_is_better=True)
  </code></pre>
  <h1>Using catboost</h1>
  <pre class="lang-py prettyprint-override"><code>class ProfitMetric(object):
      def get_final_error(self, error, weight):
          return error / (weight + 1e-38)
  
      def is_max_optimal(self):
          return True
  
      def evaluate(self, approxes, target, weight):
          assert len(approxes) == 1
          assert len(target) == len(approxes[0])
  
          approx = approxes[0]
  
          error_sum = 0.0
          weight_sum = 0.0
  
          ** I don't know here**
  
          return error_sum, weight_sum
  </code></pre>
  <h1>Question</h1>
  <p>How to complete the custom eval metric in catboost?</p>
  <h1>UPDATE</h1>
  <p>My update so far</p>
  <pre class="lang-py prettyprint-override"><code>import numpy as np
  import pandas as pd
  import seaborn as sns
  import sklearn
  
  from catboost import CatBoostClassifier
  from sklearn.model_selection import train_test_split
  
  def get_profit(y_true, y_pred):
      tn, fp, fn, tp = sklearn.metrics.confusion_matrix(y_true,y_pred).ravel()
      profit = 400*tp - 200*fn - 100*fp
      return profit
  
  
  class ProfitMetric:
      def is_max_optimal(self):
          return True # greater is better
  
      def evaluate(self, approxes, target, weight):
          assert len(approxes) == 1
          assert len(target) == len(approxes[0])
  
          approx = approxes[0]
  
          y_pred = np.rint(approx)
          y_true = np.array(target).astype(int)
  
          output_weight = 1 # weight is not used
  
          score = get_profit(y_true, y_pred)
   
          return score, output_weight
  
      def get_final_error(self, error, weight):
          return error
  
  
  df = sns.load_dataset('titanic')
  X = df[['survived','pclass','age','sibsp','fare']]
  y = X.pop('survived')
  
  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=100)
  
  
  model = CatBoostClassifier(metric_period=50,
    n_estimators=200,
    eval_metric=ProfitMetric()
  )
  
  model.fit(X, y, eval_set=(X_test, y_test)) # this fails
  </code></pre>