title: How to evaluate the xgboost classification model stability
body: <p>I have:</p>
  
  <ol>
  <li>Python xgboost classification model</li>
  <li>Weekly datasets (basis of classification) since the begining of 2018. Each of dataset has about 100 thousand rows and 70 columns (features).</li>
  <li>weekly prediction results on datasets via xgboost model (using logistic regression) in the format:</li>
  </ol>
  
  <pre><code>- date of modelling
  - items
  - test_auc_mean for each item (in percentage).
  </code></pre>
  
  <p>In total there are about 100 datasets and 100 prediction_results since January 2018.</p>
  
  <p>To assess the model I use such metrics as:</p>
  
  <p>-auc</p>
  
  <p>-confusion matrix</p>
  
  <p>-accuracy</p>
  
  <pre><code>param = {
      'num_parallel_tree':num_parallel_tree,
      'subsample':subsample,
      'colsample_bytree':colsample_bytree,
      'objective':objective, 
      'learning_rate':learning_rate, 
      'eval_metric':eval_metric, 
      'max_depth':max_depth,
      'scale_pos_weight':scale_pos_weight,
      'min_child_weight':min_child_weight,
      'nthread':nthread,
      'seed':seed
  }
  
  bst_cv = xgb.cv(
      param, 
      dtrain,  
      num_boost_round=n_estimators, 
      nfold = nfold,
      early_stopping_rounds=early_stopping_rounds,
      verbose_eval=verbose,
      stratified = stratified
  )
  
  test_auc_mean = bst_cv['test-auc-mean']
  best_iteration = test_auc_mean[test_auc_mean == max(test_auc_mean)].index[0]
  
  bst = xgb.train(param, 
                  dtrain, 
                  num_boost_round = best_iteration)
  
  best_train_auc_mean = bst_cv['train-auc-mean'][best_iteration]
  best_train_auc_mean_std = bst_cv['train-auc-std'][best_iteration]
  
  best_test_auc_mean = bst_cv['test-auc-mean'][best_iteration]
  best_test_auc_mean_std = bst_cv['test-auc-std'][best_iteration]
  
  print('''XGB CV model report
  Best train-auc-mean {}% (std: {}%) 
  Best test-auc-mean {}% (std: {}%)'''.format(round(best_train_auc_mean * 100, 2), 
                                            round(best_train_auc_mean_std * 100, 2), 
                                            round(best_test_auc_mean * 100, 2), 
                                            round(best_test_auc_mean_std * 100, 2)))
  
  y_pred = bst.predict(dtest)
  tn, fp, fn, tp = confusion_matrix(y_test, y_pred&gt;0.9).ravel()
  
  
  print('''
       | neg | pos |
  __________________
  true_| {}  | {}  |
  false| {}  | {}  |
  __________________
  
  '''.format(tn, tp, fn, fp))
  
  predict_accuracy_on_test_set = (tn + tp)/(tn + fp + fn + tp)
  print('Test Accuracy: {}%'.format(round(predict_accuracy_on_test_set * 100, 2)))
  </code></pre>
  
  <p>The model gives me general picture (as usually, auc is between .94 and .96)
  The problem is that the variability of predicting of some specific items is very high (today an item is positive, tomorrow an item is negative, the day after tomorrow - positive again)</p>
  
  <p>I wanna evaluate the model' stability. In other words, I wanna know, how many items with variable results does it generate. In the end, I wanna be ensured, that the model will generate stable results with minimal fluctuation.
  Do you have some thoughts how to do this?</p>