title: Counting frequency of word and number of distinct ids associated with the word
body: <p>In addition to counting the frequency of the words in a document, I'd like to count the number of distinct ids the word is associated with. It's easier to explain with an example:  </p>
  
  <pre><code>from pandas import *
  from collections import defaultdict
  d = {'ID' : Series(['a', 'a', 'b', 'c', 'c', 'c']),
    'words' : Series(["apple banana apple strawberry banana lemon",
    "apple", "banana", "banana lemon", "kiwi", "kiwi lemon"])}
  df = DataFrame(d)
  
  &gt;&gt;&gt; df
    ID                                       words
  0  a  apple banana apple strawberry banana lemon
  1  a                                       apple
  2  b                                      banana
  3  c                                banana lemon
  4  c                                        kiwi
  5  c                                  kiwi lemon
  
  # count frequency of words using defaultdict
  wc = defaultdict(int)
  for line in df.words:
  linesplit = line.split()
  for word in linesplit:
    wc[word] += 1
  # defaultdict(&lt;type 'int'&gt;, {'kiwi': 2, 'strawberry': 1, 'lemon': 3, 'apple': 3, 'banana': 4})
  # turn in to a DataFrame
  dwc = {"word": Series(wc.keys()),
        "count": Series(wc.values())}
  dfwc =  DataFrame(dwc)
  &gt;&gt;&gt; dfwc
     count        word
  0      2        kiwi
  1      1  strawberry
  2      3       lemon
  3      3       apple
  4      4      banana
  </code></pre>
  
  <p>The counting the frequency of words part is simple, as shown above. What I'd like to do is to obtain the output as following, which gives the number of distinct ids associated with each word:  </p>
  
  <pre><code>   count        word  ids
  0      2        kiwi    1
  1      1  strawberry    1
  2      3       lemon    2
  3      3       apple    1
  4      4      banana    3  
  </code></pre>
  
  <p>Ideally I'd like it to be at the same time as counting the word frequency.. but I'm not sure how I can integrate it. </p>
  
  <p>Any pointer would be much appreciated!</p>