Cross Validation for Logistic Regression
Question: I am wondering how to use cross validation in python to improve the accuracy
of my logistic regression model. The dataset being used is called 'iris'. I
have already successfully used cross validation for a SVM model but I am
struggling to adjust my code to do the same for the logistic regression model.
Here's my code so far:

    
    
    from sklearn import cross_validation
    from sklearn import datasets, linear_model
    iris = datasets.load_iris()
    x_iris = iris.data
    y_iris = iris.target
    svc = svm.SVC(C=1, kernel='linear')
    k_fold = cross_validation.StratifiedKFold(y_iris, n_folds=10)  
    # labels, the     number of folders
    #for train, test in k_fold:
    #    print train, test
    scores = cross_validation.cross_val_score(svc, x_iris, y_iris, cv=k_fold,     scoring='accuracy') 
    # clf.fit() is repeatedly called inside the cross_validation.cross_val_score()
    print scores
    print 'average score = ', np.mean(scores)
    print 'std of scores = ', np.std(scores)
    

What adjustments must I make to the code to achieve successful cross
validation for my logistic regression model?

Thanks for any help.


Answer: 
    lg = LogisticRegression()
    scores = cross_validation.cross_val_score(lg, x_iris, y_iris, cv=k_fold,scoring='accuracy') 
    print scores
    print 'average score = ', np.mean(scores)
    print 'std of scores = ', np.std(scores)
    

Creating the `LogisticRegression` with default values classifier works fine
for me. The output is slightly lower than the `SVM` machine approach,
`0.953333333333` vs. `0.973333333333`.

But for **parameter adjustment** you can always use `GridSearchCV` which
automatically performs a cross-validation of `cv` folds (in the next example
I'll use `10` as you did before) trying all possible combinations of
parameters. Example:

    
    
    from sklearn import grid_search
    parameters = {
                'penalty':['l2'],
                'C':[1,10,100],
                'solver': ['newton-cg', 'lbfgs', 'liblinear', 'sag'],
            }
    GS = grid_search.GridSearchCV(lg, parameters,cv=10,verbose=10)
    GS.fit(x_iris,y_iris)
    print GS.best_params_ # output: {'penalty': 'l2', 'C': 100, 'solver': 'liblinear'}
    print GS.best_score_ # output: 0.98
    

By doing this, creating your classifier with best params
`LogisticRegression(penalty='l2',C=100,solver='liblinear')` will give you a
`0.98` accuracy.

> **Gentle warning** : when performing cross validation you'd better save a
> portion of your data for testing purposes that has not been included in the
> learning process. Otherwise, one way or another your learning algorithm has
> seen all data and you could easily fall into overfitting.

