text mining techniques this does come at costthoughbecause you're building bigger term-vectors by including bigrams and/or trigrams in the equation stop word filtering--every language comes with words that have little value in text analytics because they're used so often nltk comes with short list of english stop words we can filter if the text is tokenized into wordsit often makes sense to rid the word vector of these low-information stop words lowercasing--words with capital letters appear at the beginning of sentenceothers because they're proper nouns or adjectives we gain no added value making that distinction in our term matrixso all terms will be set to lowercase another data preparation technique is stemming this one requires more elaboration stemming and lemmatization stemming is the process of bringing words back to their root formthis way you end up with less variance in the data this makes sense if words have similar meanings but are written differently becausefor exampleone is in its plural form stemming attempts to unify by cutting off parts of the word for example "planesand "planeboth become "plane another techniquecalled lemmatizationhas this same goal but does so in more grammatically sensitive way for examplewhile both stemming and lemmatization would reduce "carsto "car,lemmatization can also bring back conjugated verbs to their unconjugated forms such as "areto "be which one you use depends on your caseand lemmatization profits heavily from pos tagging (part of speech taggingpos tagging is the process of attributing grammatical label to every part of sentence you probably did this manually in school as language exercise take the sentence "game of thrones is television series if we apply pos tagging on it we get ({"game":"nn"},{"of":"in},{"thrones":"nns},{"is":"vbz},{" ":"dt},{"television":"nn}{"series":"nn}nn is nounin is prepositionnns is noun in its plural formvbz is third-person singular verband dt is determiner table has the full list table list of all pos tags tag meaning tag meaning cc coordinating conjunction cd cardinal number dt determiner ex existential fw foreign word in preposition or subordinating conjunction jj adjective jjr adjectivecomparative jjs adjectivesuperlative ls list item marker md modal nn nounsingular or mass