Discarding rare words when comparing texts - per text, per comparison, or per codex? I'm trying to compare texts (read: books) using KL divergence of N-gram usage frequency. 

first I have to calculate the frequency of N-grams, and I see (perhaps unsurprisingly) that many of the words used are "rare" (for example, they appear  &lt;10 times in the text). I wish to replace these rare words with tokens ("UNK", for unknown word).

Let's say I am comparing 10 texts. At what point should I count the rare words to discover which can be considered rare?
I can think of three options:


Count words for each text separately. If "Boat" is a word is rare in the first text and frequent in second one, it is replaced with UNK only in the first text, and not in the second one.
Count words for each comparison. For 10 texts I'm carrying out 45 (two-sided) comparisons. If I choose this option, I will not replace the word "Boat" in either text when comparing the first and second texts. I will however replace it with UNK when comparing the first and third texts (assume that "Boat" is rare in the third text as well).
Count words for the entire collection. Using this option, "Boat" is never replaced in any of the texts, due to it being a non-rare word in one of the texts.


What option would serve me better in finding meaningful N-gram representations of texts? 

Many thanks.
Tf-idf (text frequency - Inverness document frequency) will highlight your interests.

https://en.m.wikipedia.org/wiki/Tf–idf