title: FeatureUnion , pipeline categorical features with tfidf features throwing error
body: <p>I am trying to concat features from tfidf and other categorical features to perform classification on the resultant dataset. From various blogs I understand that FeatureUnion can be used to concat the features and then pipeline the same to algorithm (in my case Naive bayes). </p>
  
  <p>I have followed the code from this link - <a href="http://zacstewart.com/2014/08/05/pipelines-of-featureunions-of-pipelines.html" rel="nofollow noreferrer">http://zacstewart.com/2014/08/05/pipelines-of-featureunions-of-pipelines.html</a></p>
  
  <p>When I try to execute the code it is giving error </p>
  
  <pre><code>TypeError: no supported conversion for types: (dtype('O'),)
  </code></pre>
  
  <p>Below is the code which I am trying to execute: </p>
  
  <pre><code>class textdata():
      def transform(self, X, Y):
          return X[desc]
  
      def fit(self, X, Y):
          return self
  
  class one_hot_trans():
      def transform(self, X, Y):
          X = pd.get_dummies(X, columns=obj_cols)
          return X
      def fit(self, X, Y):
          return self
  
  pipeline = Pipeline([
                    ('features', FeatureUnion([
  
                            ('ngram_tf_idf', Pipeline([
                                                        ('text', textdata()),
                                                        ('tf_idf', TfidfTransformer())
                                                      ])), 
                            ('one_hot', one_hot_trans())
                          ])),
                    ('classifier', MultinomialNB())
              ])
  
  
  d_train, d_test, y_train, y_test = train_test_split(data, data[target], test_size=0.2, random_state = 2018)
  pipeline.fit(d_train, y_train)
  </code></pre>
  
  <p>Can anyone help me in resolving this error. </p>
  
  <p><strong>Note:</strong> data has total 9 columns with 1 target variable (categorical) and 1 text column (on which I want to perform tfidf) and rest are categorical (obj_cols in above code). </p>
  
  <p><strong>Edit:</strong>
  Thanks Vivek. I did not notice that. It was by mistake i have put transformer instead of Vectorizer. Even after replacing I am getting below error.</p>
  
  <pre><code>~\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\pipeline.py in _fit_transform_one(transformer, weight, X, y, **fit_params)
      579                        **fit_params):
      580     if hasattr(transformer, 'fit_transform'):
  --&gt; 581         res = transformer.fit_transform(X, y, **fit_params)
      582     else:
      583         res = transformer.fit(X, y, **fit_params).transform(X)
  
  ~\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\pipeline.py in fit_transform(self, X, y, **fit_params)
      745         self._update_transformer_list(transformers)
      746         if any(sparse.issparse(f) for f in Xs):
  --&gt; 747             Xs = sparse.hstack(Xs).tocsr()
      748         else:
      749             Xs = np.hstack(Xs)
  
  ~\AppData\Local\Continuum\anaconda3\lib\site-packages\scipy\sparse\construct.py in hstack(blocks, format, dtype)
      462 
      463     """
  --&gt; 464     return bmat([blocks], format=format, dtype=dtype)
      465 
      466 
  
  ~\AppData\Local\Continuum\anaconda3\lib\site-packages\scipy\sparse\construct.py in bmat(blocks, format, dtype)
      598     if dtype is None:
      599         all_dtypes = [blk.dtype for blk in blocks[block_mask]]
  --&gt; 600         dtype = upcast(*all_dtypes) if all_dtypes else None
      601 
      602     row_offsets = np.append(0, np.cumsum(brow_lengths))
  
  ~\AppData\Local\Continuum\anaconda3\lib\site-packages\scipy\sparse\sputils.py in upcast(*args)
      50             return t
      51 
  ---&gt; 52     raise TypeError('no supported conversion for types: %r' % (args,))
      53 
      54 
  
  TypeError: no supported conversion for types: (dtype('float64'), dtype('O'))    
  </code></pre>
  
  <p><strong>Edit::</strong> </p>
  
  <p>I have checked for the unique values in all the categorical variables except for description column and I found none of the values appearing in test data which are not there in train. Am I doing something wrong. </p>
  
  <pre><code>for col in d_train.columns.drop(desc):
      ext = set(d_test[col].unique().tolist()) - set(d_train[col].unique().tolist())
      if ext: print ("extra columns: \n\n", ext)
  </code></pre>
  
  <p><strong>Edit2::</strong> 
  Additional info - details of the d_train, d_test features mentioned. Can anyone help I am still getting "dimension mismatch" error on predict method. </p>
  
  <pre><code>obj cols::  ['priority', 'ticket_type', 'created_group', 'Classification', 'Component', 'ATR_OWNER_PLANT', 'created_day']
  d_train cols::  Index(['priority', 'ticket_type', 'created_group', 'Description_ticket', 'Classification', 'Component', 'ATR_OWNER_PLANT', 'created_day'], dtype='object')
  d_test cols::  Index(['priority', 'ticket_type', 'created_group', 'Description_ticket','Classification', 'Component', 'ATR_OWNER_PLANT', 'created_day'], dtype='object')
  
  d_train shape::  (95080, 8)
  d_test shape::  (23770, 8)
  desc::  Description_ticket
  </code></pre>