to keep our experimentation agilelet us wrap everything together in train_ model(functionwhich takes function as parameter that creates the classifierfrom sklearn metrics import precision_recall_curveauc from sklearn cross_validation import shufflesplit def train_model(clf_factoryxy)setting random_state to get deterministic behavior cv shufflesplit( =len( )n_iter= test_size= indices=truerandom_state= scores [pr_scores [for traintest in cvx_trainy_train [train] [trainx_testy_test [test] [testclf clf_factory(clf fit(x_trainy_traintrain_score clf score(x_trainy_traintest_score clf score(x_testy_testscores append(test_scoreproba clf predict_proba(x_testprecisionrecallpr_thresholds precision_recall_curve (y_testproba[:, ]pr_scores append(auc(recallprecision)summary (np mean(scores)np std(scores)np mean(pr_scores)np std(pr_scores)print " \ \ \ "%summary xy load_sanders_data(pos_neg_idx=np logical_or( =="positive" =="negative" [pos_neg_idxy [pos_neg_idxy =="positivetrain_model(create_ngram_model