How to use Cohen's Kappa as the evaluation metric in GridSearchCV in Scikit Learn?
Cohen's kappa was introduced in scikit-learn 0.17.

You can wrap it in make_scorer for use in GridSearchCV.

from sklearn.metrics import cohen_kappa_score, make_scorer
from sklearn.grid_search import GridSearchCV
from sklearn.svm import LinearSVC

kappa_scorer = make_scorer(cohen_kappa_score)
grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]}, scoring=kappa_scorer)