How to store predicted classes matching the pre-vectorized X in Python Scikit-learn?
Question: I would like to use name to predict gender. And not just name but name
features like extracting the "last name" as a feature derived from a name. My
code's flow is as such, get data into df > specify lr classifier and dv
dictVectorizer > use functions to create features > perform dictVectorization
> training. I would like to do the followings but can't find any resources of
how.

1) I would like to add the predicted classes (0 and 1) back into the original
data set or the data set that I can see both the names and the predicted
gender classes. Currently my y_test_predictions correspond only to X_test
which is a sparse matrix.

2) How can I retain the trained classifier and use it to predict genders of a
different data set with a bunch of names? And how can I just insert a name
"Rick Grime" and have the classifier tells me what gender it predicts?

I have done something like this with nltk, but can't find any example or
references to do this in Scikit-learn.

Codes:

    
    
        import pandas as pd
        from pandas import DataFrame, Series
        import numpy as np
        import re
        import random
        import time
        from random import randint
        import csv
        import sys
        from sklearn.metrics import classification_report
        from sklearn.linear_model import LogisticRegression
        from sklearn.svm import LinearSVC
        from sklearn.tree import DecisionTreeClassifier
        from sklearn.naive_bayes import MultinomialNB
        from sklearn.feature_extraction import DictVectorizer
        from sklearn.metrics import confusion_matrix as sk_confusion_matrix
        from sklearn.metrics import roc_curve, auc
        import matplotlib.pyplot as plt
        from sklearn.metrics import precision_recall_curve
        from sklearn import cross_validation 
    
        data = pd.read_csv("file.csv", header=0, encoding="utf-8")
        df = DataFrame(data)
        dv = DictVectorizer()
        lr = LogisticRegression()
    
        X = df.raw_name.values
        X2 = df.name.values
        y = df.gender.values
    
        def feature_full_name(nameString):
            try:
                full_name = nameString
                if len(full_name) > 1: # not accept name with only 1 character
                    return full_name
                else: return '?'
            except: return '?'
    
        def feature_full_last_name(nameString):
            try:
                last_name = nameString.rsplit(None, 1)[-1]
                if len(last_name) > 1: # not accept name with only 1 character
                    return last_name
                else: return '?'
            except: return '?'
    
        def feature_name_entity(nameString2):
            space = 0
            try:
                for i in nameString2:
                    if i == ' ':
                        space += 1
                return space+1
            except: return 0
    
        my_dict = [{'last-name': feature_full_last_name(i)} for i in X]
        my_dict2 = [{'name-entity': feature_name_entity(feature_full_name(i))} for i in X2]
    
    
        all_dict = []
        for i in range(0, len(my_dict)):
            temp_dict = dict(
                my_dict[i].items() + my_dict2[i].items()
                )
            all_dict.append(temp_dict)
    
        newX = dv.fit_transform(all_dict)
    
        X_train, X_test, y_train, y_test = cross_validation.train_test_split(newX, y, test_size=0.3)
    
        lr.fit(X_train, y_train)
    
        y_test_predictions = lr.predict(X_test)
    


Answer: I would use some of scikit-learn's built-in tools to split the dataframe,
vectorize the names, and predict the results. Then you can add the predicted
results back into the test dataframe. For example, using a small set of names
as an example:

    
    
    data = {'Bruce Lee': 'Male',
            'Bruce Banner': 'Male',
            'Bruce Springsteen': 'Male',
            'Bruce Willis': 'Male',
            'Sarah McLaughlin': 'Female',
            'Sarah Silverman': 'Female',
            'Sarah Palin': 'Female',
            'Sarah Hyland': 'Female'}
    
    import pandas as pd
    df = pd.DataFrame.from_dict(data, orient='index').reset_index()
    df.columns = ['name', 'gender']
    print(df)
                    name  gender
    0    Sarah Silverman  Female
    1        Sarah Palin  Female
    2  Bruce Springsteen    Male
    3       Bruce Banner    Male
    4          Bruce Lee    Male
    5       Sarah Hyland  Female
    6   Sarah McLaughlin  Female
    7       Bruce Willis    Male
    

Now we can use scikit-learn's `CountVectorizer` to count the words in the
names; this produces essentially the same output as what you've done above,
except it doesn't filter on length of name, etc. For ease of use, we'll put
this in a pipeline with a cross-validated logistic regression:

    
    
    from sklearn.feature_extraction.text import CountVectorizer
    from sklearn.linear_model import LogisticRegressionCV
    from sklearn.pipeline import make_pipeline
    
    clf = make_pipeline(CountVectorizer(), LogisticRegressionCV(cv=2))
    

Now we can split our data into a train/test set, fit the pipeline, and then
assign the results:

    
    
    from sklearn.cross_validation import train_test_split
    
    df_train, df_test = train_test_split(df, train_size=0.5, random_state=0)
    
    clf.fit(df_train['name'], df_train['gender'])
    
    df_test = df_test.copy() # so we can modify it
    df_test['predicted'] = clf.predict(df_test['name'])
    
    print(df_test)
                    name  gender predicted
    6   Sarah McLaughlin  Female    Female
    2  Bruce Springsteen    Male      Male
    1        Sarah Palin  Female    Female
    7       Bruce Willis    Male      Male
    

Similarly, we can just pass a list of names to the pipeline and get a
prediction:

    
    
    >>> clf.predict(['Bruce Campbell', 'Sarah Roemer'])
    array(['Male', 'Female'], dtype=object)
    

If you want to do more sophisticated logic in your text vectorization, you can
create a custom transformer for your input data: a web search for "scikit-
learn custom transformer" should give you a decent set of examples to work
from.

* * *

Edit: here's an example of using a custom transformer to generate dicts from
the input names:

    
    
    from sklearn.base import TransformerMixin
    
    class ExtractNames(TransformerMixin):
        def transform(self, X, *args):
            return [{'first': name.split()[0],
                     'last': name.split()[-1]}
                    for name in X]
    
        def fit(self, *args):
            return self
    
    trans = ExtractNames()
    
    >>> trans.fit_transform(df['name'])
    [{'first': 'Bruce', 'last': 'Springsteen'},
     {'first': 'Bruce', 'last': 'Banner'},
     {'first': 'Sarah', 'last': 'Hyland'},
     {'first': 'Sarah', 'last': 'Silverman'},
     {'first': 'Sarah', 'last': 'Palin'},
     {'first': 'Bruce', 'last': 'Lee'},
     {'first': 'Bruce', 'last': 'Willis'},
     {'first': 'Sarah', 'last': 'McLaughlin'}]
    

Now you can put this in a pipeline with a `DictVectorizer` to generate sparse
features:

    
    
    from sklearn.feature_extraction import DictVectorizer
    from sklearn.pipeline import make_pipeline
    
    pipe = make_pipeline(ExtractNames(), DictVectorizer())
    
    >>> pipe.fit_transform(df['name'])
    <8x10 sparse matrix of type '<class 'numpy.float64'>'
        with 16 stored elements in Compressed Sparse Row format>
    

Finally, you could make a pipeline which combines these with a cross-validated
logistic regression and proceed as above:

    
    
    clf = make_pipeline(ExtractNames(), DictVectorizer(), LogisticRegressionCV())
    clf.fit(df_train['name'], df_train['gender'])
    df_test['predicted'] = clf.predict(df_test['name'])
    

From here, if you wish, you could modify the `ExtractNames` transformer to do
more sophisticated extraction of features (using some of your code from
above), and you end up with a pipeline implementation of your procedure, but
lets you simply call `predict()` on an input list of strings. Hope that helps!

