Title: Error with fitting a pipeline for clustering both text and numeric data
Tags: python;scikit-learn;sklearn-pandas
Question: I'm having some difficulties with fitting a pipeline on a data set for the purpose of K-means clustering using scikit-learn. For illustration purposes let's say i have a certain DataFrame like the following (only much larger):
df = pd.DataFrame({'Text': ['the traveler', 'alien', 'titanic', 'real leather', 'prestige'], 'Numeric': [22, 41, 2, 78, 3]})
Now i have written a FunctionTransformer for obtaining both text and numeric features from the dataframe:
get_text_data = FunctionTransformer(lambda x: x['Text'], validate=False)
get_numeric_data = FunctionTransformer(lambda x: x['Numeric'], validate=False)
after that i wrote the following pipeline using sklearn.pipeline:
pl = Pipeline([('union', FeatureUnion(transformation_list = [('numeric_features', Pipeline([('selector', get_numeric_data), ('imputer', Imputer())])), ('text_features', Pipeline([('selector', get_text_data), 'vectorizer', TfidfVectorizer(token_pattern=TOKENS_ALPHANUMERIC))]))])), ('kmeans', KMeans(n_clusters=4))])
Now i have tried to fit the pipeline like this:
pl.fit(df)
And i get the following error:
ValueError: blocks[0,:] has incompatible row dimensions
I'm a bit lost and will appreciate any help in the matter from you guys
Thanks.
Comment: Can you post the full error message, is it the `FeatureUnion`, `Pipeline`, `Imputer`, `TfidfVectorizer` or `KMeans` that's producing that error? Also does `sklearn` accept `pandas` data frames as input, I thought it only takes numpy arrays and scipy sparse matrices?

Comment: @MattiLyra Yes, it does take Pandas dataframe. It will internally convert it to appropriate one. To the OP, unless you post the data to duplicate this behaviour, and the complete stack trace of error, we are unable to help

Comment: Well i've tried other techniques rather then k-means, including classifiers (although it makes no sense in this context) and i keep getting the same error. I have also tried to fit the pipeline this way: pl.fit(df.values). but then i get another error: IndexError: only integers, slices (':'), ellipsis ('...'), numpy.newaxis('None') and integer or boolean arrays are valid indices

Comment: Could you add line breaks to your code? It's hard to read right now.



{'source': 'StackExchangeDataset', 'subset': 'stackoverflow.com', 'metadata': {'AnswerCount': 0, 'CommentCount': 4, 'ContentLicense': 'CC BY-SA 3.0', 'CreationDate': '2017:00:05/07/17', 'FavoriteCount': 1.0, 'PostTypeId': 1, 'Tags': '<python><scikit-learn><sklearn-pandas>', 'Title': 'Error with fitting a pipeline for clustering both text and numeric data', 'ViewCount': 202, 'id': 43830270, 'user_AboutMe': None, 'user_AccountId': 10846499.0, 'user_DisplayName': 'Uri T', 'user_Reputation': 9.0, 'user_Views': 3.0, 'user_id': 7975783.0}}