from sklearn metrics import roc_auc_score rf_auc roc_auc_score(y_testrf predict_proba(x_test)[: ]svc_auc roc_auc_score(y_testsvc decision_function(x_test)print("auc for random forest{ }format(rf_auc)print("auc for svc{ }format(svc_auc)out[ ]auc for random forest auc for svc comparing the random forest and svm using the auc scorewe find that the random forest performs quite bit better than the svm recall that because average precision is the area under curve that goes from to average precision always returns value between (worstand (bestpredicting randomly always produces an auc of no matter how imbalanced the classes in dataset are this makes auc much better metric for imbalanced classification problems than accuracy the auc can be interpreted as evaluating the ranking of positive samples it' equivalent to the probability that randomly picked point of the positive class will have higher score according to the classifier than randomly picked point from the negative class soa perfect auc of means that all positive points have higher score than all negative points for classification problems with imbalanced classesusing auc for model selection is often much more meaningful than using accuracy let' go back to the problem we studied earlier of classifying all nines in the digits dataset versus all other digits we will classify the dataset with an svm with three different settings of the kernel bandwidthgamma (see figure - )in[ ] digits target = x_trainx_testy_trainy_test train_test_splitdigits datayrandom_state= plt figure(for gamma in [ ]svc svc(gamma=gammafit(x_trainy_trainaccuracy svc score(x_testy_testauc roc_auc_score(y_testsvc decision_function(x_test)fprtpr_ roc_curve(y_test svc decision_function(x_test)print("gamma { faccuracy { fauc { }formatgammaaccuracyauc)plt plot(fprtprlabel="gamma={ }format(gamma)plt xlabel("fpr"plt ylabel("tpr"plt xlim(- plt ylim( plt legend(loc="best"evaluation metrics and scoring