title: TF-IDF score from TfIdfTransformer in sklearn on same word in two sentences with same frequency
body: <p>If I have two sentences containing the same word, and this word appears with the same counts (frequency) in both sentences, why is the Tf-Idf score I get different for them?</p>
  
  <p>Consider this list of texts:</p>
  
  <pre><code>data = [
      'Jumper in knit.',
      'A sweater in knit, black sweater.',
  ]
  </code></pre>
  
  <p>and consider that I fit and transform a <code>CountVectorizer</code> and a <code>TfIdfTransformer</code> on it, as in </p>
  
  <pre><code>count_vec = CountVectorizer(stop_words='english')
  tf_transformer = TfidfTransformer(use_idf=True)
  
  X_counts = count_vec.fit_transform(x_data_manual)
  X_tfidf = tf_transformer.fit_transform(X_counts)
  </code></pre>
  
  <p>Then I print the features with their IDFs scores:</p>
  
  <pre><code>print zip(count_vec.get_feature_names(), tf_transformer.idf_)
  </code></pre>
  
  <p>obtaining</p>
  
  <blockquote>
    <p>[(u'black', 1.4054651081081644), (u'jumper', 1.4054651081081644), (u'knit', 1.0), (u'sweater', 1.4054651081081644)]</p>
  </blockquote>
  
  <p>so we can see that all tokens have the same IDF except 'knit', all legit.</p>
  
  <p>If I now ask for printing both the counts matrix and the TF-IDFs matrix, </p>
  
  <pre><code>print X_counts.todense()
  print X_tfidf.todense()
  </code></pre>
  
  <p>what I get is, respectively </p>
  
  <blockquote>
    <p>[[0 1 1 0]<br>
     [1 0 1 2]]</p>
  </blockquote>
  
  <p>(which is legit), and</p>
  
  <blockquote>
    <p>[[ 0.     0.815  0.58   0.   ] </p>
    
    <p>[ 0.426  0.     0.303  0.852]]</p>
  </blockquote>
  
  <p>Now, I thought the TF-IDF score would be a multiplication of the term frequency (however calculated from the count/raw frequency) and the IDF, but I'm seeing 'knit' having a different score in the two sentences despite the frequencies are the same. </p>
  
  <p>So what I'm missing/misunderstanding?</p>