text mining and text analytics term frequency--inverse document frequency (tf-idfa well-known formula to fill up the document-term matrix is tf-idf or term frequency multiplied by inverse document frequency binary bag of words assigns true or false (term is there or not)while simple frequencies count the number of times the term occurred tf-idf is bit more complicated and takes into account how many times term occurred in the document (tftf can be simple term counta binary count (true or false)or logarithmically scaled term count it depends on what works best for you in case tf is term frequencythe formula of tf is the followingtf ft, tf is the frequency (fof the term (tin the document (dbut tf-idf also takes into account all the other documents because of the inverse document frequency idf gives an idea of how common the word is in the entire corpusthe higher the document frequency the more commonand more common words are less informative for example the words "aor "thearen' likely to provide specific information on text the formula of idf with logarithmic scaling is the most commonly used form of idfidf log( /|{ : }|with being the total number of documents in the corpusand the |{ : }being the number of documents (din which the term (tappears the tf-idf score says this about termhow important is this word to distinguish this document from the others in the corpusthe formula of tf-idf is thus ff log : , idf we won' use tf-idfbut when setting your next steps in text miningthis should be one of the first things you'll encounter tf-idf is also what was used by elasticsearch behind the scenes in it' good way to go if you want to use tf-idf for text analyticsleave the text mining to specialized software such as solr or elasticsearch and take the document/term matrix for text analytics from there before getting to the actual bag of wordsmany other data manipulation steps take placetokenization--the text is cut into pieces called "tokensor "terms these tokens are the most basic unit of information you'll use for your model the terms are often words but this isn' necessity entire sentences can be used for analysis we'll use unigramsterms consisting of one word oftenhoweverit' useful to include bigrams (two words per tokenor trigrams (three words per tokento capture extra meaning and increase the performance of your models