title: TfidfVectorizer Returning 0 for Ngrams in Pandas DF with Duplicate IDs
body: <p>I have a grouped df:</p>
  
  <pre><code>id    text
  100   he loves ice cream
  100   she loves ice
  100   i hate avocado
  </code></pre>
  
  <p>I am extracting bigrams, frequencies, and tfidf scores with this function:</p>
  
  <pre><code>def extractFeatures(groupedDF, textCol):
      features = pd.DataFrame()
      for id, group in tqdm(groupedDF):
             freq = cv.fit_transform(group[textCol])
             tfidf = tv.fit_transform(group[textCol])
             freq = sum(freq).toarray()[0]
             tfidf.todense()
             tfidf = tfidf.toarray()[0]
             freq = pd.DataFrame(freq, columns=['frequency'])
             tfidf = pd.DataFrame(tfidf, columns=['tfidf'])
             dfinner = pd.DataFrame(cv.get_feature_names(), columns=['ngram'])
             dfinner['id'] = id
             dfinner = dfinner.join(freq)
             results = dfinner.join(tfidf)
             features = features.append(results)
      return features
  </code></pre>
  
  <p>This results in the following df:</p>
  
  <pre><code>id    ngram         frequency    tfidf
  100   hate avocado  1            0
  100   he loves      1            .3
  100   i hate        1            0
  100   ice cream     1            .3
  100   loves ice     2            .6 
  100   she loves     1            0 
  </code></pre>
  
  <p>The tfidf scores are contrived. So, the function is correctly finding the frequencies. It is then finding tfidf scores for the first row of the grouped df (including for the bigram that appears in multiple rows. Finally, it is not finding the tfidf scores for bigrams unique to the second and third rows.</p>
  
  <p>In addition, while the tfidf scores are contrived, it is true that they are identical for any bigram that has the same frequency within a particular document. So any bigram with a frequency of 1 in the first row will have the .3 tfidf score. Any bigram with a frequency of 1 in another row might have a tfidf score of .24. This is weird because the term frequency of each bigram is certainly different.</p>
  
  <p><strong>Two Questions:</strong></p>
  
  <ol>
  <li>Why are the tfidf scores for the second and third rows not found?</li>
  <li>Why are the tfidf scores identical for a particular bigram appearing at the same frequency in a particular document?</li>
  </ol>
  
  <p>Thank you for any insight you all might have!</p>