ValueError: Inconsistent Shapes Error in Scikit Learn Train_Test Split
Question: I'm preparing to run some predictions on a csv document comparing job
descriptions to salary outcomes. I've split the data set into training and
test where features is what I'm working with and target is what I'm
predicting. When I go to print and confirm that these records were separated
properly I get the following error: ValueError: Inconsistent Shapes

My code and the resulting error follow:

    
    
    import csv
    import numpy as np
    
    # create posting & label list
    postList = []
    labelList = []
    filename = '\Users\yantezia.patrick\Downloads\Postings.csv'
    csvFile = csv.reader(open(filename, 'r'), delimiter=",")
    for row in csvFile:
        postList.append(row[2])
        labelList.append(row[10]) #appending specific columns to specific list #these willbe labels
    
    # remove first row
    postList = postList[1:] #clearing out the header rows
    labelList = labelList[1:]
    
    temp = np.array([float(i) for i in labelList])
    med = np.median(temp)
    for i, val in enumerate(labelList):
        if float(val) >= med:
            labelList[i] = 1
        else:
            labelList[i] = 0
    
    # subset list
    postList = postList[:100]
    labelList = labelList[:100]
    print postList[:2]
    
    from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
    import pandas as pd
    
    # create term matrix
    cv = CountVectorizer(lowercase=True, stop_words='english', ngram_range=(1,3), min_df=10)
    tfidf = TfidfVectorizer(lowercase=True, stop_words='english', ngram_range=(1,3), min_df=10)
    
    tf_dm = cv.fit_transform(postList)
    tfidf_dm = tfidf.fit_transform(postList)
    
    pd.DataFrame(tfidf_dm.toarray(),index=postList,columns=tfidf.get_feature_names()).head(10)
    
    tfidf.get_feature_names()
    tm = tm.toarray()
    print tf_dm
    tm = cv.fit(postList)
    print tm.vocabulary_
    print tf_dm.shape
    print tfidf_dm.shape
    
    #add labels to word vector
    from sklearn.cross_validation import train_test_split
    
    features_train1 = train_test_split(tf_dm, labels, test_size=0.33, random_state=42)
    features_test1 = train_test_split(tf_dm, labels, test_size=0.33, random_state=42)
    target_train1 = train_test_split(tf_dm, labels, test_size=0.33, random_state=42)
    target_test1 = train_test_split(tf_dm, labels, test_size=0.33, random_state=42)
    
    features_train2 = train_test_split(tfidf_dm, labels, test_size=0.33, random_state=7)
    features_test2 = train_test_split(tfidf_dm, labels, test_size=0.33, random_state=7)
    target_train2 = train_test_split(tfidf_dm, labels, test_size=0.33, random_state=7)
    target_test2 = train_test_split(tfidf_dm, labels, test_size=0.33, random_state=7)
    
    print np.sum(target_train1)
    print np.sum(target_test1)
    print target_train1
    print target_test1
    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    <ipython-input-82-53ecd8559f48> in <module>()
    ----> 1 print np.sum(target_train1)
          2 print np.sum(target_test1)
          3 print target_train1
          4 print target_test1
    
    C:\Users\yantezia.patrick\AppData\Local\Continuum\Anaconda\lib\site-packages\numpy\core\fromnumeric.pyc in sum(a, axis, dtype, out, keepdims)
       1707         except AttributeError:
       1708             return _methods._sum(a, axis=axis, dtype=dtype,
    -> 1709                                 out=out, keepdims=keepdims)
       1710         # NOTE: Dropping the keepdims parameters here...
       1711         return sum(axis=axis, dtype=dtype, out=out)
    
    C:\Users\yantezia.patrick\AppData\Local\Continuum\Anaconda\lib\site-packages\numpy\core\_methods.pyc in _sum(a, axis, dtype, out, keepdims)
         23 def _sum(a, axis=None, dtype=None, out=None, keepdims=False):
         24     return um.add.reduce(a, axis=axis, dtype=dtype,
    ---> 25                             out=out, keepdims=keepdims)
         26 
         27 def _prod(a, axis=None, dtype=None, out=None, keepdims=False):
    
    C:\Users\yantezia.patrick\AppData\Local\Continuum\Anaconda\lib\site-packages\scipy\sparse\compressed.pyc in __add__(self, other)
        340         elif isspmatrix(other):
        341             if (other.shape != self.shape):
    --> 342                 raise ValueError("inconsistent shapes")
        343 
        344             return self._binopt(other,'_plus_')
    
    ValueError: inconsistent shapes
    


Answer: Can you please explain what you are trying to do? The definitions of
`features_train1`, `features_test1`, `target_train1` and `target_test` are the
same. As you fixed the random state, they will all have the same content. Have
you tried to look at what any of these are? Please look at [the documentation
of train_test_split](http://scikit-
learn.org/dev/modules/generated/sklearn.cross_validation.train_test_split.html#sklearn.cross_validation.train_test_split).
`features_train1` is a tuple, where the first two elements are a split of
`tf_dm` and the second two are a split of `labels`.

What you most likely meant to do was

    
    
    features_train1, features_test1, labels_train1, labels_test1 = train_test_split(tf_dm, labels, test_size=0.33, random_state=42)
    

