title: How to calculate gini index in pyspark classification model using spark ML?
body: <p>I am trying to calculate the gini index for a classification model done using GBTClassifier from the pyspark ml models. I cant seem to find a metrics which gives the roc_auc_score like the one in python sklearn. </p>
  
  <p>Below is the code that I have used so far on databricks. I am currently using a dataset from the databricks </p>
  
  <pre><code>%fs ls databricks-datasets/adult/adult.data
  
  from pyspark.sql.functions import *
  from pyspark.ml.classification import  RandomForestClassifier, GBTClassifier
  from pyspark.ml.feature import StringIndexer, OneHotEncoderEstimator, VectorAssembler, VectorSlicer
  from pyspark.ml import Pipeline
  from pyspark.ml.evaluation import BinaryClassificationEvaluator,MulticlassClassificationEvaluator
  from pyspark.mllib.evaluation import BinaryClassificationMetrics
  from pyspark.ml.linalg import Vectors
  from pyspark.ml.tuning import ParamGridBuilder, TrainValidationSplit
  
  dataset = spark.table("adult")
  # spliting the train and test data frames 
  splits = dataset.randomSplit([0.7, 0.3])
  train_df = splits[0]
  test_df = splits[1]
  
  def churn_predictions(train_df,
                       target_col, 
  #                      algorithm, 
  #                      model_parameters = conf['model_parameters']
                      ):
    """
    #Function attributes
    dataframe        - training df
    target           - target varibale in the model
    Algorithm        - Algorithm used 
    model_parameters - model parameters used to fine tune the model
    """
  
    # one hot encoding and assembling
    encoding_var = [i[0] for i in train_df.dtypes if (i[1]=='string') &amp; (i[0]!=target_col)]
    num_var = [i[0] for i in train_df.dtypes if ((i[1]=='int') | (i[1]=='double')) &amp; (i[0]!=target_col)]
  
    string_indexes = [StringIndexer(inputCol = c, outputCol = 'IDX_' + c, handleInvalid = 'keep') for c in encoding_var]
    onehot_indexes = [OneHotEncoderEstimator(inputCols = ['IDX_' + c], outputCols = ['OHE_' + c]) for c in encoding_var]
    label_indexes = StringIndexer(inputCol = target_col, outputCol = 'label', handleInvalid = 'keep')
    assembler = VectorAssembler(inputCols = num_var + ['OHE_' + c for c in encoding_var], outputCol = "features")
    gbt = GBTClassifier(featuresCol = 'features', labelCol = 'label',
                       maxDepth = 5, 
                       maxBins  = 45,
                       maxIter  = 20)
  
    pipe = Pipeline(stages = string_indexes + onehot_indexes + [assembler, label_indexes, gbt])
    model = pipe.fit(train_df)
  
    return model  
  
  gbt_model = churn_predictions(train_df = train_df,
                       target_col = 'income')
  
  #### prediction in test sample ####
  gbt_predictions = gbt_model.transform(test_df)
  # display(gbt_predictions)
  gbt_evaluator = MulticlassClassificationEvaluator(
      labelCol="label", predictionCol="prediction", metricName="accuracy")
  
  accuracy = gbt_evaluator.evaluate(gbt_predictions) * 100
  print("Accuracy on test data = %g" % accuracy)
  
  gini_train = 2 * metrics.roc_auc_score(Y, pred_prob) - 1
  </code></pre>
  
  <p>as you can see in the last line of code there is clearly no metric called roc_auc_score to calculate the gini. </p>
  
  <p>Really appreciate any help on this. </p>