title: removing stop words using spacy
body: <p>I am cleaning a column in my <code>data frame</code>, Sumcription, and am trying to do 3 things:</p>
  
  <ol>
  <li>Tokenize</li>
  <li>Lemmantize</li>
  <li><p>Remove stop words </p>
  
  <pre><code>import spacy        
  nlp = spacy.load('en_core_web_sm', parser=False, entity=False)        
  df['Tokens'] = df.Sumcription.apply(lambda x: nlp(x))    
  spacy_stopwords = spacy.lang.en.stop_words.STOP_WORDS        
  spacy_stopwords.add('attach')
  df['Lema_Token']  = df.Tokens.apply(lambda x: " ".join([token.lemma_ for token in x if token not in spacy_stopwords]))
  </code></pre></li>
  </ol>
  
  <p>However, when I print for example: </p>
  
  <pre><code>df.Lema_Token.iloc[8]
  </code></pre>
  
  <p>The output still has the word attach in it:
  <code>attach poster on the wall because it is cool</code></p>
  
  <p>Why does it not remove the stop word?</p>
  
  <p>I also tried this:</p>
  
  <pre><code>df['Lema_Token_Test']  = df.Tokens.apply(lambda x: [token.lemma_ for token in x if token not in spacy_stopwords])
  </code></pre>
  
  <p>But the str <code>attach</code> still appears.</p>