'The Corpus helps with tasks involving integer representations of
words. This object is used to filter, subsample, and convert loose
word indices to compact word indices.
\'Loose\' word arrays are word indices given by a tokenizer. The word
index is not necessarily representative of word\'s frequency rank, and
so loose arrays tend to have \'gaps\' of unused indices, which can make
models less memory efficient. As a result, this class helps convert
a loose array to a \'compact\' one where the most common words have low
indices, and the most infrequent have high indices.
Corpus maintains a count of how many of each word it has seen so
that it can later selectively filter frequent or rare words. However,
since word popularity rank could change with incoming data the word
index count must be updated fully and `self.finalize()` must be called
before any filtering and subsampling operations can happen.
Arguments
out_of_vocabulary : int, default=-1
Token index to replace whenever we encounter a rare or unseen word.
Instead of skipping the token, we mark as an out of vocabulary
word.
skip : int, default=-2
Token index to replace whenever we want to skip the current frame.
Particularly useful when subsampling words or when padding a
sentence.
Examples
>>> corpus = Corpus()
>>> words_raw = np.random.randint(100, size=25)
>>> corpus.update_word_count(words_raw)
>>> corpus.finalize()
>>> words_compact = corpus.to_compact(words_raw)
>>> words_pruned = corpus.filter_count(words_compact, min_count=2)
>>> # words_sub = corpus.subsample_frequent(words_pruned, thresh=1e-5)
>>> words_loose = corpus.to_loose(words_pruned)
>>> not_oov = words_loose > -1
>>> np.all(words_loose[not_oov] == words_raw[not_oov])
True'

def __init__(self, out_of_vocabulary=(-1), skip=(-2)):

    self.counts_loose = defaultdict(int)
    self._finalized = False
    self.specials = dict(out_of_vocabulary=out_of_vocabulary, skip=skip)
