Why does the votingClassifier in sklearn.ensemble gives higher accuracy than each of the classifiers that make it? and should i depend on it? I am running an ML classifier on my data.
I used SVM, RF and KNN. I used GScv for each of them and then used votingclassifier.The accuracy i got in each classifier independently was low, but from the hard and soft vote of the voting classifier is much higher! Why is that?
Here is my code
Fitting Kernel SVM to training set
from sklearn.svm import SVC
clf = SVC(C=1, cache_size=200, class_weight=None, coef0=0.0,
  decision_function_shape='ovr', degree=3, gamma=0.005, kernel='rbf',
  max_iter=-1, probability=True, random_state=1, shrinking=True,
  tol=0.001, verbose=False)

Checking accurancy, Best score of GV
print('Checking accurancy, Best score of GV')
best_accuracy =  grid_search.best_score_ 
print (best_accuracy)

[out] Checking accurancy, Best score of GV
0.5169491525423728

Applying RandomForest (RF) Classification
from sklearn.ensemble import RandomForestClassifier

clf_rf = RandomForestClassifier(random_state=42)
clf_rf.fit(X_train, y_train)
clf_rf.score(X_test, y_test)
print (&quot;Mean accuracy is&quot;, (clf_rf.score(X_test, y_test)))

y_pred2 = clf_rf.predict (X_test)

from sklearn.metrics import accuracy_score
accuracy_score(y_test, y_pred2)

[out] accuracy 0.21

Applying KNN
from sklearn.neighbors import KNeighborsClassifier

neigh = KNeighborsClassifier(n_neighbors=10)
neigh.fit(X_train, y_train) 

print (neigh.score(X_train, y_train))

[out] accuracy 0.58

Predicting the test set results
y_pred3 = neigh.predict (X_test)

from sklearn.ensemble import VotingClassifier

eclf1 = VotingClassifier (estimators= [('svm', clf), ('rf', clf_rf), ('KNN', neigh)], voting = 'hard')

eclf1 = eclf1.fit(X_train, y_train)

print 'Hard vote accuracy =', eclf1.score(X_train, y_train)


eclf2 = VotingClassifier (estimators= [('svm', clf), ('rf', clf_rf), ('KNN', neigh)], voting = 'soft')

eclf2 = eclf2.fit(X_train, y_train)

print 'Soft vote accuracy =', eclf2.score(X_train, y_train)

[out]
Hard vote accuracy = 0.9830508474576272
Soft vote accuracy = 0.7711864406779662
Basically I would say that your individual classifiers might be overfitting the training data. Ensemble voting is known to avoid overfitting, so it does not seem unlikely that the ensemble classifier behaves better than each classifier individually.