tf-idf using data on unigram frequency from Google
Question: I'm trying to identify important terms in a set of government documents.
Generating the term frequencies is no problem.

For document frequency, I was hoping to use the [handy Python scripts and
accompanying data](http://norvig.com/ngrams/) that Peter Norvig posted for his
chapter in "Beautiful Data", which include the frequencies of unigrams in a
huge corpus of data from the Web.

My understanding of tf-idf, however, is that "document frequency" refers to
the number of documents containing a term, not the number of total words that
_are_ this term, which is what we get from the Norvig script. Can I still use
this data for a crude tf-idf operation?

Here's some sample data:

    
    
    word    tf       global frequency
    china   1684     0.000121447
    the     352385   0.022573582
    economy 6602     0.0000451130774123
    and     160794   0.012681757
    iran    2779     0.0000231482902018
    romney  1159     0.000000678497795593 
    

Simply dividing tf by gf gives "the" a higher score than "economy," which
can't be right. Is there some basic math I'm missing, perhaps?


Answer: As I understand, Global Frequency is equal "inverse total term frequency"
mentioned here
[Robertson](http://www.soi.city.ac.uk/~ser/idfpapers/Robertson_idf_JDoc.pdf).
From this Robertson's paper:

    
    
    One possible way to get away from this problem would be to make a fairly radical re-
    placement for IDF (that is, radical in principle, although it may be not so radical 
    in terms of its practical effects). ....
    the probability from the event space of documents to the event space of term positions 
    in the concatenated text of all the documents in the collection. 
    Then we have a new measure, called here 
    inverse total term frequency:
    ...
    On the whole, experiments with inverse total term frequency weights have tended to show
    that they are not as effective as IDF weights
    

According to this text, you can use inverse global frequency as IDF term,
albeit more crude than standard one.

Also you are missing [stop words](https://en.wikipedia.org/wiki/Stop_words)
removal. Words such as the are used in almost all documents, therefore they do
not give any information. Before tf-idf , you should remove such stop words.

