advanced tokenizationstemmingand lemmatization as mentioned previouslythe feature extraction in the countvectorizer and tfidf vectorizer is relatively simpleand much more elaborate methods are possible one particular step that is often improved in more sophisticated text-processing applications is the first step in the bag-of-words modeltokenization this step defines what constitutes word for the purpose of feature extraction we saw earlier that the vocabulary often contains singular and plural versions of some wordsas in "drawbackand "drawbacks""drawerand "drawers"and "drawingand "drawingsfor the purposes of bag-of-words modelthe semantics of "drawbackand "drawbacksare so close that distinguishing them will only increase overfittingand not allow the model to fully exploit the training data similarlywe found the vocabulary includes words like "replace""replaced""replace ment""replaces"and "replacing"which are different verb forms and noun relating to the verb "to replace similarly to having singular and plural forms of nountreating different verb forms and related words as distinct tokens is disadvantageous for building model that generalizes well this problem can be overcome by representing each word using its word stemwhich involves identifying (or conflatingall the words that have the same word stem if this is done by using rule-based heuristiclike dropping common suffixesit is usually referred to as stemming if instead dictionary of known word forms is used (an explicit and human-verified system)and the role of the word in the sentence is taken into accountthe process is referred to as lemmatization and the standardized form of the word is referred to as the lemma both processing methodslemmatization and stemmingare forms of normalization that try to extract some normal form of word another interesting case of normalization is spelling correctionwhich can be helpful in practice but is outside of the scope of this book working with text data