word_tokenize package this package divides the input text into words we can import this package with the help of the following python codefrom nltk tokenize import word_tokenize wordpuncttokenizer package this package divides the input text into words as well as the punctuation marks we can import this package with the help of the following python codefrom nltk tokenize import wordpuncttokenizer stemming while working with wordswe come across lot of variations due to grammatical reasons the concept of variations here means that we have to deal with different forms of the same words like democracydemocraticand democratization it is very necessary for machines to understand that these different words have the same base form in this wayit would be useful to extract the base forms of the words while we are analyzing the text we can achieve this by stemming in this waywe can say that stemming is the heuristic process of extracting the base forms of the words by chopping off the ends of words in the python nltk modulewe have different packages related to stemming these packages can be used to get the base forms of word these packages use algorithms some of the packages are as followsporterstemmer package this python package uses the porter' algorithm to extract the base form we can import this package with the help of the following python codefrom nltk stem porter import porterstemmer for exampleif we will give the word 'writingas the input to this stemmer them we will get the word 'writeafter stemming lancasterstemmer package this python package will use the lancaster' algorithm to extract the base form we can import this package with the help of the following python codefrom nltk stem lancaster import lancasterstemmer for exampleif we will give the word 'writingas the input to this stemmer them we will get the word 'writafter stemming