title: User Warning: Your stop_words may be inconsistent with your preprocessing
body: <p>I am following <a href="http://brandonrose.org/clustering" rel="nofollow noreferrer">this</a> document clustering tutorial. As an input I give a txt file which can be downloaded <a href="https://gofile.io/?c=LutAVj" rel="nofollow noreferrer">here</a>. It's a combined file of 3 other txt files divided with a use of \n. After creating a tf-idf matrix I received this warning:</p>
  <pre><code>,,UserWarning: Your stop_words may be inconsistent with your preprocessing. 
  Tokenizing the stop words generated tokens ['abov', 'afterward', 'alon', 'alreadi', 'alway', 'ani', 'anoth', 'anyon', 'anyth', 'anywher', 'becam', 'becaus', 'becom', 'befor', 'besid', 'cri', 'describ', 'dure', 'els', 'elsewher', 'empti', 'everi', 'everyon', 'everyth', 'everywher', 'fifti', 'forti', 'henc', 'hereaft', 'herebi', 'howev', 'hundr', 'inde', 'mani', 'meanwhil', 'moreov', 'nobodi', 'noon', 'noth', 'nowher', 'onc', 'onli', 'otherwis', 'ourselv', 'perhap', 'pleas', 'sever', 'sinc', 'sincer', 'sixti', 'someon', 'someth', 'sometim', 'somewher', 'themselv', 'thenc', 'thereaft', 'therebi', 'therefor', 'togeth', 'twelv', 'twenti', 'veri', 'whatev', 'whenc', 'whenev', 'wherea', 'whereaft', 'wherebi', 'wherev', 'whi', 'yourselv'] not in stop_words.
    'stop_words.' % sorted(inconsistent))&quot;. 
  </code></pre>
  <p>I guess it has something to do with the order of lemmatization and stop words removal, but as this is my first project in txt processing, I am a bit lost and I don't know how to fix this...</p>
  <pre><code>import pandas as pd
  import nltk
  from nltk.corpus import stopwords
  import re
  import os
  import codecs
  from sklearn import feature_extraction
  import mpld3
  from nltk.stem.snowball import SnowballStemmer
  from sklearn.feature_extraction.text import TfidfVectorizer
  
  
  stopwords = stopwords.words('english')
  stemmer = SnowballStemmer(&quot;english&quot;)
  
  def tokenize_and_stem(text):
      # first tokenize by sentence, then by word to ensure that punctuation is caught as it's own token
      tokens = [word for sent in nltk.sent_tokenize(text) for word in nltk.word_tokenize(sent)]
      filtered_tokens = []
      # filter out any tokens not containing letters (e.g., numeric tokens, raw punctuation)
      for token in tokens:
          if re.search('[a-zA-Z]', token):
              filtered_tokens.append(token)
      stems = [stemmer.stem(t) for t in filtered_tokens]
      return stems
  
  
  def tokenize_only(text):
      # first tokenize by sentence, then by word to ensure that punctuation is caught as it's own token
      tokens = [word.lower() for sent in nltk.sent_tokenize(text) for word in nltk.word_tokenize(sent)]
      filtered_tokens = []
      # filter out any tokens not containing letters (e.g., numeric tokens, raw punctuation)
      for token in tokens:
          if re.search('[a-zA-Z]', token):
              filtered_tokens.append(token)
      return filtered_tokens
  
  
  totalvocab_stemmed = []
  totalvocab_tokenized = []
  with open('shortResultList.txt', encoding=&quot;utf8&quot;) as synopses:
      for i in synopses:
          allwords_stemmed = tokenize_and_stem(i)  # for each item in 'synopses', tokenize/stem
          totalvocab_stemmed.extend(allwords_stemmed)  # extend the 'totalvocab_stemmed' list
          allwords_tokenized = tokenize_only(i)
          totalvocab_tokenized.extend(allwords_tokenized)
  
  vocab_frame = pd.DataFrame({'words': totalvocab_tokenized}, index = totalvocab_stemmed)
  print ('there are ' + str(vocab_frame.shape[0]) + ' items in vocab_frame')
  print (vocab_frame.head())
  
  #define vectorizer parameters
  tfidf_vectorizer = TfidfVectorizer(max_df=0.8, max_features=200000,
                                   min_df=0.2, stop_words='english',
                                   use_idf=True, tokenizer=tokenize_and_stem, ngram_range=(1,3))
  
  with open('shortResultList.txt', encoding=&quot;utf8&quot;) as synopses:
      tfidf_matrix = tfidf_vectorizer.fit_transform(synopses) #fit the vectorizer to synopses
  
  print(tfidf_matrix.shape)
  </code></pre>