title: Sklearn Pipeline: How to build for kmeans, clustering text?
body: <p>I have text as shown :</p>
  
  <pre><code> list1 = ["My name is xyz", "My name is pqr", "I work in abc"]
  </code></pre>
  
  <p>The above will be training set for clustering text using kmeans.</p>
  
  <pre><code>list2 = ["My name is xyz", "I work in abc"]
  </code></pre>
  
  <p>The above is my test set.</p>
  
  <p>I have built a vectorizer and the model as shown below:</p>
  
  <pre><code>vectorizer = TfidfVectorizer(min_df = 0, max_df=0.5, stop_words = "english", charset_error = "ignore", ngram_range = (1,3))
  vectorized = vectorizer.fit_transform(list1)
  km=KMeans(n_clusters=2, init='k-means++', n_init=10, max_iter=1000, tol=0.0001, precompute_distances=True, verbose=0, random_state=None, copy_x=True, n_jobs=1)
  km.fit(vectorized)
  </code></pre>
  
  <p>If I try to predict the cluster for my test set of "list2":</p>
  
  <pre><code>km.predict(list2)
  </code></pre>
  
  <p>I get the error below:</p>
  
  <pre><code>ValueError: Incorrect number of features. Got 2 features, expected 5
  </code></pre>
  
  <p>I was told to use <code>Pipeline</code> to solve this issue. So I wrote the code below:</p>
  
  <pre><code>pipe = Pipeline([('vect', vectorizer), ('vectorized', vectorized), ('kmeans',km )])
  </code></pre>
  
  <p>But I get the error:</p>
  
  <pre><code>TypeError                                 Traceback (most recent call last)
  /mnt/folder/Text_Mining/&lt;ipython-input-14-321cabc3bf35&gt; in &lt;module&gt;()
  ----&gt; 1 pipe = Pipeline([('vect', vectorizer), ('vectorized', vectorized), ('kmeans',km )])
  /usr/local/lib/python2.7/dist-packages/scikit_learn-0.13-py2.7-linux-x86_64.egg/sklearn/pipeline.pyc in __init__(self, steps)
       87                 raise TypeError("All intermediate steps a the chain should "
       88                                 "be transforms and implement fit and transform"
  ---&gt; 89                                 "'%s' (type %s) doesn't)" % (t, type(t)))
       90
       91         if not hasattr(estimator, "fit"):
  TypeError: All intermediate steps a the chain should be transforms and implement fit and transform'  (0, 2)     1.0
  (1, 4)        0.57735026919
  (1, 3)        0.57735026919
  (1, 1)        0.57735026919
  (2, 0)        1.0' (type &lt;class 'scipy.sparse.csr.csr_matrix'&gt;) doesn't)
  </code></pre>
  
  <p>I think that maybe the output of <code>vectorized</code> does not  implement a fit and transform, but how do I do that in this particular case? I'm new to Machine Learning. 
  Also, how to get the labels from the kmeans model? When I run kmeans, I can access the cluster labels by using <code>km.labels_</code>. How to do something similar in Pipeline?</p>