title: Filter categories in data frame before generating dummy columns for them
body: <p>I have a dataset with categorical values in some columns (one row may contain multiple categories, separated by <code>,</code>). Example:</p>
  
  <pre><code>  user hashtags
  0   u1      a,b
  1   u2      a,c
  2   u3        c
  </code></pre>
  
  <p>I want to make dummy columns for these categories. I'm also not interested in categories that have very few occurrences in the dataset. Currently, I'm generating the dummy columns and then dropping the ones with few occurrences, like this (<code>chunk</code> is the original data frame):</p>
  
  <pre><code>dummies_hashtags = chunk['hashtags'].str.get_dummies(sep=',')
  dummies_hashtags.columns = dummies_hashtags.columns.map(lambda c: 'hashtag_' + c)
  
  # get rid of dummy columns with usage below 10
  usage = dummies_hashtags.sum(0)
  high_usage = dummies_hashtags[np.where(usage &gt;= 10)[0]]
  low_usage = dummies_hashtags[np.where(usage &lt; 10)[0]]
  dummies_hashtags = high_usage
  dummies_hashtags['other_hashtags'] = low_usage.sum(1)
  </code></pre>
  
  <p>Notice I'm also adding a column for the number of categories with a low occurrence.</p>
  
  <p>This approach works but is very slow. My idea about how to improve it is to first get all unique categories and their counts, then delete categories with low counts, <strong>before</strong> generating the dummy columns.</p>
  
  <p>I would like to ask you this: would this approach actually improve anything? How would it be implemented? (<code>np.unique</code> with <code>return_counts=True</code> comes to mind). Also, is there a better approach to this problem?</p>
  
  <p>(Note: The dataset is a <code>SparseDataFrame</code> already).</p>