title: NLP Stemming and Lemmatization using Regular expression tokenization
body: <p>Define a function called <code>performStemAndLemma</code>, which takes a parameter. The first parameter, <code>textcontent</code>, is a string. The function definition code stub is given in the editor. Perform the following specified tasks:</p>
  <p>1.Tokenize all the words given in <code>textcontent</code>. The word should contain alphabets or numbers or underscore. Store the tokenized list of words in <code>tokenizedwords</code>. (Hint: Use regexp_tokenize)</p>
  <ol start="2">
  <li><p>Convert all the words into lowercase. Store the result into the variable <code>tokenizedwords</code>.</p>
  </li>
  <li><p>Remove all the stop words from the unique set of <code>tokenizedwords</code>. Store the result into the variable <code>filteredwords</code>. (Hint: Use stopwords corpora)</p>
  </li>
  <li><p>Stem each word present in <code>filteredwords</code> with PorterStemmer, and store the result in the list <code>porterstemmedwords</code>.</p>
  </li>
  <li><p>Stem each word present in <code>filteredwords</code> with LancasterStemmer, and store the result in the list <code>lancasterstemmedwords</code>.</p>
  </li>
  <li><p>Lemmatize each word present in <code>filteredwords</code> with WordNetLemmatizer, and store the result in the list <code>lemmatizedwords</code>.</p>
  </li>
  </ol>
  <p>Return <code>porterstemmedwords</code>, <code>lancasterstemmedwords</code>, <code>lemmatizedwords</code> variables from the function.</p>
  <p>My code:</p>
  <pre><code>from nltk.corpus import stopwords
  def performStemAndLemma(textcontent):
      # Write your code here
      #Step 1
      tokenizedword = nltk.tokenize.regexp_tokenize(textcontent, pattern = '\w*', gaps = False)
      #Step 2
      tokenizedwords = [x.lower() for x in tokenizedword if x != '']
      #Step 3
      unique_tokenizedwords = set(tokenizedwords)
      stop_words = set(stopwords.words('english')) 
      filteredwords = []
      for x in unique_tokenizedwords:
          if x not in stop_words:
              filteredwords.append(x)
      #Steps 4, 5 , 6
      ps = nltk.stem.PorterStemmer()
      ls = nltk.stem.LancasterStemmer()
      wnl = nltk.stem.WordNetLemmatizer()
      porterstemmedwords =[]
      lancasterstemmedwords = []
      lemmatizedwords = []
      for x in filteredwords:
          porterstemmedwords.append(ps.stem(x))
          lancasterstemmedwords.append(ls.stem(x))
          lemmatizedwords.append(wnl.lemmatize(x))
      return porterstemmedwords, lancasterstemmedwords, lemmatizedwords
  </code></pre>
  <p>Still the program is not working fine. Not passing the 2 test cases. Highlight the mistake in above code and provide alternate solution for the same.</p>