Sklearn tfidf vectorize returns different shape after fit_transform() I'm new to ML and trying out basic samples using sklearn. I have achieved converting text (single dimension) to numbers using TF-IDF and got the predictions correct.

Now I have a different use-case to predict Work Hours based on Marital-Status and Relationship Status so my training data is now two dimension.

I have DataFrame total of 24420 rows * 2 columns of shape --> (24420, 2)

After this DataFrame is been passed to TfidfVectorizer.fit_transform() function it turns to shape --> (2, 3) Not sure why!

And while training mnb.fit(x_train_tf, y_train) it returns an error


  ValueError: Found input variables with inconsistent numbers of samples: [2, 24420]


You could see the initial DataFrame shape of 24420*2 it transposed to 2*24420

All code is up here.

Questions:


Why after fit_transform() the shape changes to (2,3) what happened to (24420,2) data? What does this denotes?
Why does the ValueError: Found input variables with inconsistent numbers of samples [2, 24420] occurs?
Why initial DataFrame shape changed to 2*24420?


What is wrong with the implementation?
What is wrong with your implementation is that you are passing a dataframe directly to tfidf vectorizer. If you check your data, it would look like this -

&gt;&gt;&gt; x_train_tf.toarray()
array([[ 0.70710678,  0.        ,  0.70710678],
       [ 0.        ,  1.        ,  0.        ]])


If you check what features you are getting, you will see -

&gt;&gt;&gt; tfidf.get_feature_names()
[u'marital', u'relationship', u'status']


What's happening is, while passing dataframe, the TfidfVectorizer is only taking the column names and converting them into numeric form.

I don't think you need to use tfidf here. As far as I understand, your data is categorical text, so use pandas.get_dummies() instead of tfidf. This will convert your categorical data to numeric form which you can use for modelling.

&gt;&gt;&gt; pd.get_dummies(x_train.iloc[:10,])
       marital-status_Divorced  marital-status_Married-civ-spouse  marital-status_Never-married  marital-status_Widowed  relationship_Husband  relationship_Not-in-family  relationship_Own-child  relationship_Unmarried 
1521                   1                          0                              0                        0                       0                            1                         0                        0 
2274                   0                          1                              0                        0                       1                            0                         0                        0 
20209                  0                          1                              0                        0                       1                            0                         0                        0 
5529                   0                          1                              0                        0                       1                            0                         0                        0 
27639                  0                          1                              0                        0                       1                            0                         0                        0 
26670                  0                          0                              1                        0                       0                            1                         0                        0 
16635                  0                          0                              0                        1                       0                            0                         0                        1 
30824                  0                          0                              1                        0                       0                            1                         0                        0 
15181                  0                          0                              1                        0                       0                            0                         1                        0 
9119                   1                          0                              0                        0                       0                            0                         1                        0 


Here, for demo purpose, I have taken only 10 rows.