title: Which accuracy score to use for the Mean Decrease Accuracy with the scikit RandomForestClassifier
body: <p>I've been running the implementation the '<strong>Mean Decrease Accuracy</strong>' measure that is shown on this <a href="http://blog.datadive.net/selecting-good-features-part-iii-random-forests/" rel="nofollow">website</a>: </p>
  
  <p>In the example the author is using the random forest regressor <code>RandomForestRegressor</code>, but I am using the random forest classifier <code>RandomForestClassifier</code>. Thus, my question is, if I should also use the <code>r2_score</code> for measuring accuracy or if I should switch to classic accuracy <code>accuracy_score</code> or matthews correlation coefficient <code>matthews_corrcoef</code>?. </p>
  
  <p>Does anybody here if I should switch or not. And why?</p>
  
  <p>Thanks for any help!</p>
  
  <hr>
  
  <p>Here is the code from the website in case you are too lazy to click :)</p>
  
  <pre><code>from sklearn.cross_validation import ShuffleSplit
  from sklearn.metrics import r2_score
  from collections import defaultdict
  
  X = boston["data"]
  Y = boston["target"]
  
  rf = RandomForestRegressor()
  scores = defaultdict(list)
  
  #crossvalidate the scores on a number of different random splits of the data
  for train_idx, test_idx in ShuffleSplit(len(X), 100, .3):
      X_train, X_test = X[train_idx], X[test_idx]
      Y_train, Y_test = Y[train_idx], Y[test_idx]
      r = rf.fit(X_train, Y_train)
      acc = r2_score(Y_test, rf.predict(X_test))
      for i in range(X.shape[1]):
          X_t = X_test.copy()
          np.random.shuffle(X_t[:, i])
          shuff_acc = r2_score(Y_test, rf.predict(X_t))
          scores[names[i]].append((acc-shuff_acc)/acc)
  print "Features sorted by their score:"
  print sorted([(round(np.mean(score), 4), feat) for
                feat, score in scores.items()], reverse=True)
  </code></pre>