Scikit-Learn's Pipeline: A sparse matrix was passed, but dense data is required
Question: I'm finding it difficult to understand how to fix a Pipeline I created (read:
largely pasted from a tutorial). It's python 3.4.2:

    
    
    df = pd.DataFrame
    df = DataFrame.from_records(train)
    
    test = [blah1, blah2, blah3]
    
    pipeline = Pipeline([('vectorizer', CountVectorizer()), ('classifier', RandomForestClassifier())])
    
    pipeline.fit(numpy.asarray(df[0]), numpy.asarray(df[1]))
    predicted = pipeline.predict(test)
    

When I run it, I get:

    
    
    TypeError: A sparse matrix was passed, but dense data is required. Use X.toarray() to convert to a dense numpy array.
    

This is for the line `pipeline.fit(numpy.asarray(df[0]),
numpy.asarray(df[1]))`.

I've experimented a lot with solutions through numpy, scipy, and so forth, but
I still don't know how to fix it. And yes, similar questions have come up
before, but not inside a pipeline. Where is it that I have to apply `toarray`
or `todense`?


Answer: Unfortunately those two are incompatible. A `CountVectorizer` produces a
sparse matrix and the RandomForestClassifier requires a dense matrix. It is
possible to convert using `X.todense()`. Doing this will substantially
increase your memory footprint.

Below is sample code to do this that I found at
<http://zacstewart.com/2014/08/05/pipelines-of-featureunions-of-
pipelines.html> which allows you to call .todense() in a pipeline stage.

    
    
    class DenseTransformer(TransformerMixin):
    
        def transform(self, X, y=None, **fit_params):
            return X.todense()
    
        def fit_transform(self, X, y=None, **fit_params):
            self.fit(X, y, **fit_params)
            return self.transform(X)
    
        def fit(self, X, y=None, **fit_params):
            return self
    

Once you have your `DenseTransformer`, you are able to add it as a pipeline
step.

    
    
    pipeline = Pipeline([
         ('vectorizer', CountVectorizer()), 
         ('to_dense', DenseTransformer()), 
         ('classifier', RandomForestClassifier())
    ])
    

Another option would be to use a classifier meant for sparse data like
`LinearSVC`.

    
    
    from sklearn.svm import LinearSVC
    pipeline = Pipeline([('vectorizer', CountVectorizer()), ('classifier', LinearSVC())])
    

