title: Should you Stem and lemmatize?
body: <p>I am currently working with python NLTK to preprocess text data for <a href="https://www.kaggle.com/uciml/sms-spam-collection-dataset/data" rel="nofollow noreferrer">Kaggle SMS Spam Classification Dataset</a>. I have completed the following steps during preprocessing:</p>
  <ol>
  <li>Removed any extra spaces</li>
  <li>Removed punctuation and special characters</li>
  <li>Converted the text to lower case</li>
  <li>Replaced abbreviations such as lol,brb etc with their meaning or full form.</li>
  <li>Removed stop words</li>
  <li>Tokenized the data</li>
  </ol>
  <p>Now I plan to perform lemmatization and stemming separately on the tokenized data followed by TF-IDF done separately on lemmatized data and stemmed data.</p>
  <p><strong>Questions are as follows:</strong></p>
  <ul>
  <li>Is there a practical use case to perform lemmatization on the tokenized data and then stem that lemmatized data or vice versa</li>
  <li>Does the idea of stemming the lemmatized data or vice versa make any sense theoretically, or is it completely incorrect.</li>
  </ul>
  <p><strong>Context:</strong> I am relatively new to NLP and hence I am trying to understand as much as I can about these concepts. The main idea behind this question is to understand whether lemmatization or stemming together make any sense theoretically/practically or whether these should be done separately.</p>
  <p><strong>Questions Referenced:</strong></p>
  <ul>
  <li><a href="https://stackoverflow.com/questions/49354665/should-i-perform-both-lemmatization-and-stemming">Should I perform both lemmatization and stemming?</a>: The answer to this question was inconclusive and not accepted, it never discussed why you should or should not do it in the first place.</li>
  <li><a href="https://stackoverflow.com/questions/1787110/what-is-the-difference-between-lemmatization-vs-stemming/1787121#1787121">What is the difference between lemmatization vs stemming?</a>: Provides the ideas behind stemming and lemmatization but I was unable to conclude the answers to my questions based on this</li>
  <li><a href="https://stackoverflow.com/questions/17317418/stemmers-vs-lemmatizers?noredirect=1&amp;lq=1">Stemmers vs Lemmatizers</a>: Explains the pros and cons, as well as the context in which stemming and lemmatization, might help</li>
  <li><a href="https://stackoverflow.com/questions/62651893/nlp-stemming-and-lemmatization-using-regular-expression-tokenization">NLP Stemming and Lemmatization using Regular expression tokenization</a>: The question discusses the different preprocessing steps and does stemming and lemmatization separately</li>
  </ul>