Scikit-learn: Getting SGDClassifier to predict as well as a Logistic Regression A way to train a Logistic Regression is by using stochastic gradient descent, which scikit-learn offers an interface to.

What I would like to do is take a scikit-learn's SGDClassifier and have it score the same as a Logistic Regression here. However, I must be missing some machine learning enhancements, since my scores are not equivalent.

This is my current code. What am I missing on the SGDClassifier which would have it produce the same results as a Logistic Regression?

from sklearn import datasets
from sklearn.linear_model import LogisticRegression
from sklearn.linear_model import SGDClassifier
import numpy as np
import pandas as pd
from sklearn.cross_validation import KFold
from sklearn.metrics import accuracy_score

# Note that the iris dataset is available in sklearn by default.
# This data is also conveniently preprocessed.
iris = datasets.load_iris()
X = iris["data"]
Y = iris["target"]

numFolds = 10
kf = KFold(len(X), numFolds, shuffle=True)

# These are "Class objects". For each Class, find the AUC through
# 10 fold cross validation.
Models = [LogisticRegression, SGDClassifier]
params = [{}, {"loss": "log", "penalty": "l2"}]
for param, Model in zip(params, Models):
    total = 0
    for train_indices, test_indices in kf:

        train_X = X[train_indices, :]; train_Y = Y[train_indices]
        test_X = X[test_indices, :]; test_Y = Y[test_indices]

        reg = Model(**param)
        reg.fit(train_X, train_Y)
        predictions = reg.predict(test_X)
        total += accuracy_score(test_Y, predictions)
    accuracy = total / numFolds
    print "Accuracy score of {0}: {1}".format(Model.__name__, accuracy)


My output:

Accuracy score of LogisticRegression: 0.946666666667
Accuracy score of SGDClassifier: 0.76
The comments about iteration number are spot on. The default SGDClassifier n_iter is 5 meaning you do 5 * num_rows steps in weight space. The sklearn rule of thumb is ~ 1 million steps for typical data. For your example, just set it to 1000 and it might reach tolerance first. Your accuracy is lower with SGDClassifier because it's hitting iteration limit before tolerance so you are "early stopping"

Modifying your code quick and dirty I get:



# Added n_iter here
params = [{}, {"loss": "log", "penalty": "l2", 'n_iter':1000}]

for param, Model in zip(params, Models):
    total = 0
    for train_indices, test_indices in kf:
        train_X = X[train_indices, :]; train_Y = Y[train_indices]
        test_X = X[test_indices, :]; test_Y = Y[test_indices]
        reg = Model(**param)
        reg.fit(train_X, train_Y)
        predictions = reg.predict(test_X)
        total += accuracy_score(test_Y, predictions)

    accuracy = total / numFolds
    print "Accuracy score of {0}: {1}".format(Model.__name__, accuracy)

Accuracy score of LogisticRegression: 0.96
Accuracy score of SGDClassifier: 0.96