title: How to calculate TF*IDF for a single new document to be classified?
body: <p>I am using document-term vectors to represent a collection of document. I use <a href="http://en.wikipedia.org/wiki/Tf%E2%80%93idf" rel="noreferrer">TF*IDF</a> to calculate the term weight for each document vector. Then I could use this matrix to train a model for document classification.</p>
  
  <p>I am looking forward to classify new document in future. But in order to classify it, I need to turn the document into a document-term vector first, and the vector should be composed of TF*IDF values, too.</p>
  
  <p><strong>My question is, how could I calculate the TF*IDF with just a single document?</strong></p>
  
  <p>As far as I understand, TF can be calculated based on a single document itself, but the IDF can only be calculated with a collection of document. In my current experiment, I actually calculate the TF*IDF value for the <strong>whole</strong> collection of documents. And then I use <strong>some</strong> documents as training set and <strong>the others</strong> as test set.</p>
  
  <p>I just suddenly realized that this seems not so applicable to real life.</p>
  
  <h2>ADD 1</h2>
  
  <p>So there are actually 2 subtly different scenarios for classification:</p>
  
  <ol>
  <li>to classify some documents whose content are known but label are not
  known.</li>
  <li>to classify some totally unseen document.</li>
  </ol>
  
  <p>For 1, we can combine <strong>all</strong> the documents, both with and without labels. And get the TF*IDF over all of them. <strong>This way, even we only use the documents <em>with labels</em> for training, the training result will still contain the influence of the documents <em>without labels</em>.</strong></p>
  
  <p>But my scenario is 2.</p>
  
  <p>Suppose I have the following information for term <strong><em>T</em></strong> from the summary of the <strong>training</strong> set corpus:</p>
  
  <ul>
  <li>document count for T in the training set is <strong><em>n</em></strong></li>
  <li>total number of training documents is <strong><em>N</em></strong></li>
  </ul>
  
  <p>Should I calculate the IDF of t for a <strong>unseen</strong> document D as below?</p>
  
  <p><strong>IDF(t, D)= log((N+1)/(n+1))</strong> </p>
  
  <h2>ADD 2</h2>
  
  <p>And what if I encounter a term in the new document <strong>which didn't show up in the training corpus before</strong>?
  How should I calculate the weight for it in the doc-term vector?</p>