title: Random Forest is overfitting
body: <p>I'm using scikit-learn with a stratified CV to compare some classifiers.
  I'm computing: accuracy, recall, auc. </p>
  
  <p>I used for the parameter optimization GridSearchCV with a 5 CV.</p>
  
  <pre><code>RandomForestClassifier(warm_start= True, min_samples_leaf= 1, n_estimators= 800, min_samples_split= 5,max_features= 'log2', max_depth= 400, class_weight=None)
  </code></pre>
  
  <p>are the best_params from the GridSearchCV.</p>
  
  <p>My problem, I think I really overfit. For example:</p>
  
  <blockquote>
    <p>Random Forest with standard deviation (+/-)</p>
    
    <ul>
    <li>precision: 0.99    (+/- 0.06) </li>
    <li>sensitivity: 0.94  (+/- 0.06)    </li>
    <li>specificity: 0.94  (+/- 0.06) </li>
    <li>B_accuracy: 0.94   (+/- 0.06)</li>
    <li>AUC: 0.94 (+/- 0.11)</li>
    </ul>
    
    <p>Logistic Regression with standard deviation (+/-)</p>
    
    <ul>
    <li>precision: 0.88(+/- 0.06) </li>
    <li>sensitivity: 0.79  (+/- 0.06)    </li>
    <li>specificity: 0.68  (+/- 0.06) </li>
    <li>B_accuracy: 0.73   (+/- 0.06)</li>
    <li>AUC: 0.73 (+/- 0.041)</li>
    </ul>
  </blockquote>
  
  <p>And the others also look like logistic regression (so they are not looking overfitted).</p>
  
  <p>My code for CV is:</p>
  
  <pre><code>for i,j in enumerate(data):
      X.append(data[i][0])
      y.append(float(data[i][1]))
  x=np.array(X)
  y=np.array(y)
  
  def SD(values):
  
      mean=sum(values)/len(values)
      a=[]
      for i in range(len(values)):
          a.append((values[i]-mean)**2)
      erg=sum(a)/len(values)
      SD=math.sqrt(erg)
      return SD,mean
  
      for name, clf in zip(titles,classifiers):
      # go through all classifiers, compute 10 folds 
      # the next for loop should be 1 tab indent more, coudlnt realy format it here, sorry
      pre,sen,spe,ba,area=[],[],[],[],[]
      for train_index, test_index in skf:
          #print train_index, test_index
          #get the index from all train_index and test_index
          #change them to list due to some errors
          train=train_index.tolist()
          test=test_index.tolist()
          X_train=[]
          X_test=[]
          y_train=[]
          y_test=[]
          for i in train:
              X_train.append(x[i])
  
          for i in test:
              X_test.append(x[i]) 
  
          for i in train:
              y_train.append(y[i])
  
          for i in test:
              y_test.append(y[i]) 
  
  
          #clf=clf.fit(X_train,y_train)
          #predicted=clf.predict_proba(X_test)
          #... other code, calculating metrics and so on...
      print name 
      print("precision: %0.2f \t(+/- %0.2f)" % (SD(pre)[1], SD(pre)[0]))
      print("sensitivity: %0.2f \t(+/- %0.2f)" % (SD(sen)[1], SD(pre)[0]))
      print("specificity: %0.2f \t(+/- %0.2f)" % (SD(spe)[1], SD(pre)[0]))
      print("B_accuracy: %0.2f \t(+/- %0.2f)" % (SD(ba)[1], SD(pre)[0]))
      print("AUC: %0.2f \t(+/- %0.2f)" % (SD(area)[1], SD(area)[0]))
      print "\n"
  </code></pre>
  
  <p>If I'm using the <code>scores = cross_validation.cross_val_score(clf, X, y, cv=10, scoring='accuracy')</code>  method, I don't get this "overfitting" values. So maybe there is something wrong in the CV method I'm using? But it is only for RF...</p>
  
  <p>I did my own due to the lag of specificity score function in the cross_val_function.</p>