How to increase the model accuracy of logistic regression in Scikit python?
Question: I am trying to predict the admit variable with predictors such as gre,gpa and
ranks.But the prediction accuracy is very less(0.66).The dataset is given
below. <https://gist.github.com/abyalias/3de80ab7fb93dcecc565cee21bd9501a>

Please find the codes below:

    
    
     In[73]: data.head(20)
     Out[73]: 
    
       admit  gre   gpa  rank_2  rank_3  rank_4
    0      0  380  3.61     0.0     1.0     0.0
    1      1  660  3.67     0.0     1.0     0.0
    2      1  800  4.00     0.0     0.0     0.0
    3      1  640  3.19     0.0     0.0     1.0
    4      0  520  2.93     0.0     0.0     1.0
    5      1  760  3.00     1.0     0.0     0.0
    6      1  560  2.98     0.0     0.0     0.0
    
    y = data['admit']
    x = data[data.columns[1:]]
    
    from sklearn.cross_validation import  train_test_split
    xtrain,xtest,ytrain,ytest  = train_test_split(x,y,random_state=2)
    
    ytrain=np.ravel(ytrain)
    
    #modelling 
    clf = LogisticRegression(penalty='l2')
    clf.fit(xtrain,ytrain)
    ypred_train = clf.predict(xtrain)
    ypred_test = clf.predict(xtest)
    
    In[38]: #checking the classification accuracy
    accuracy_score(ytrain,ypred_train)
    Out[38]: 0.70333333333333337
    In[39]: accuracy_score(ytest,ypred_test)
    Out[39]: 0.66000000000000003
    
    In[78]: #confusion metrix...
    from sklearn.metrics import confusion_matrix
    confusion_matrix(ytest,ypred)
    
    Out[78]: 
    array([[62,  1],
           [33,  4]])
    

The ones are wrongly predicting.How to increase the model accuracy?


Answer: Since machine learning is more about experimenting with the features and the
models, there is no correct answer to your question. Some of my suggestions to
you would be:

**1\. Feature Scaling and/or Normalization** \- Check the scales of your _gre_
and _gpa_ features. They differ on 2 orders of magnitude. Therefore, your
_gre_ feature will end up dominating the others in a classifier like Logistic
Regression. You can normalize all your features to the same scale before
putting them in a machine learning model.[This](http://scikit-
learn.org/stable/modules/preprocessing.html) is a good guide on the various
feature scaling and normalization classes available in scikit-learn.

**2\. Class Imbalance** \- Look for class imbalance in your data. Since you
are working with admit/reject data, then the number of rejects would be
significantly higher than the admits. Most classifiers in SkLearn including
[`LogisticRegression`](http://scikit-
learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html)
have a `class_weight` parameter. Setting that to `balanced` might also work
well in case of a class imbalance.

**3\. Optimize other scores** \- You can optimize on other metrics also such
as _Log Loss_ and _F1-Score_. The F1-Score could be useful, in case of class
imbalance. [This](http://scikit-
learn.org/stable/modules/model_evaluation.html) is a good guide that talks
more about scoring.

**4\. Hyperparameter Tuning - Grid Search** \- You can improve your accuracy
by performing a Grid Search to tune the hyperparameters of your model. For
example in case of `LogisticRegression`, the parameter `C` is a
hyperparameter. Also, you should avoid using the test data during grid search.
Instead perform cross validation. Use your test data only to report the final
numbers for your final model. Please note that GridSearch should be done for
all models that you try because then only you will be able to tell what is the
best you can get from each model. Scikit-Learn provides the
[`GridSearchCV`](http://scikit-
learn.org/stable/modules/generated/sklearn.grid_search.GridSearchCV.html)
class for this. [This](http://scikit-
learn.org/stable/modules/grid_search.html) article is also a good starting
point.

**5\. Explore more classifiers** \- Logistic Regression learns a linear
decision surface that separates your classes. It could be possible that your 2
classes may not be linearly separable. In such a case you might need to look
at other classifiers such [**Support Vector Machines**](http://scikit-
learn.org/stable/modules/generated/sklearn.svm.SVC.html) which are able to
learn more complex decision boundaries. You can also start looking at Tree-
Based classifiers such as [**Decision Trees**](http://scikit-
learn.org/stable/modules/tree.html) which can learn rules from your data.
Think of them as a series of If-Else rules which the algorithm automatically
learns from the data. Often, it is difficult to get the right [Bias-Variance
Tradeoff](http://scott.fortmann-roe.com/docs/BiasVariance.html) with Decision
Trees, so I would recommend you to look at [Random Forests](http://scikit-
learn.org/stable/modules/ensemble.html#forest) if you have a considerable
amount of data.

**6\. Error Analysis** \- For each of your models, go back and look at the
cases where they are failing. You might end up finding that some of your
models work well on one part of the parameter space while others work better
on other parts. If this is the case, then [Ensemble Techniques](http://scikit-
learn.org/stable/modules/ensemble.html) such as
[`VotingClassifier`](http://scikit-
learn.org/stable/modules/ensemble.html#votingclassifier) techniques often give
the best results. Models that win Kaggle competitions are many times ensemble
models.

**7\. More Features** _ If all of this fails, then that means that you should
start looking for more features.

Hope that helps!

