you can also choose to download all necessary datasets without the gui by using the following command from the ipython or python shell nltk download('all'halt_on_error=falseonce the download is finished we will be able to use all the necessary functionalities and the bundled data of the nltk package we will now take look at the major modules of nltk library and introduce the functionality that each of them provides corpora the starting point of any text analytics process is the process of collecting the documents of interest in single dataset this dataset is central to the next steps of processing and analysis this collection of documents is generally called corpus multiple corpus datasets are called corpora the nltk module nltk corpus provides necessary functions that can be used to read corpus files in variety of formats it supports the reading of corpora from the datasets bundled in nltk package as well as external corpora tokenization tokenization is one of the core steps in text pre-processing and normalization each text document has several components like paragraphssentencesand words that together make up the document the process of tokenization is used to break down the document into these smaller components this tokenization can be into sentenceswordsclausesand so on the most popular way to tokenize any document is by using sentence tokenization and\or word tokenization the nltk tokenize module of the nltk library provides functionality that enables efficient tokenization of any textual data tagging text document is constructed based on various grammatical rules and constructs the grammar depends on the language of the text document each language' grammar will contain different entities and parts of speech like nounspronounsadjectivesadverbsand so on the process of tagging will involve getting text corpustokenizing the text and assigning metadata information like tags to each word in the corpora the nltk tag module contains implementation of different algorithms that can be used for such tagging and other related activities stemming and lemmatization word can have several different forms based on what part of speech it is representing consider the word flyit can be present in various forms in the same textlike flyingfliesflyerand so on the process of stemming is used to convert all the different forms of word in to the base formwhich is known as the root step lemmatization is similar to stemming but the base form is known as the root word and it' always semantically and lexicographically correct word this conversion is crucialas lot of times the core word contains more information about the documentwhich can be diluted by these different forms the nltk module nltk stem contains different techniques that can be used for stemming and lemmatizing corpus