installing other necessary packages for building natural language processing applications by using nltkwe need to install the necessary packages the packages are as followsgensim it is robust semantic modeling library that is useful for many applications we can install it by executing the following commandpip install gensim pattern it is used to make gensim package work properly we can install it by executing the following commandpip install pattern concept of tokenizationstemmingand lemmatization in this sectionwe will understand what is tokenizationstemmingand lemmatization tokenization it may be defined as the process of breaking the given text the character sequence into smaller units called tokens the tokens may be the wordsnumbers or punctuation marks it is also called word segmentation following is simple example of tokenizationinputmangobananapineapple and apple all are fruits outputmango banana pineapple and apple all are fruits the process of breaking the given text can be done with the help of locating the word boundaries the ending of word and the beginning of new word are called word boundaries the writing system and the typographical structure of the words influence the boundaries in the python nltk modulewe have different packages related to tokenization which we can use to divide the text into tokens as per our requirements some of the packages are as followssent_tokenize package as the name suggestthis package will divide the input text into sentences we can import this package with the help of the following python codefrom nltk tokenize import sent_tokenize