title: python fit_transform index error for large dataset
body: <p>I am doing text classification on sentiment analysis, where I have a large training set and test set</p>
  
  <pre><code>test_data_df.shape (46346, 2)  train_data_df.shape  (69518, 2)
  </code></pre>
  
  <p>The first column of train_data_df is label, which has value 1 if email is a personal attack and 0 if not. The second column of train_data_df is comment, which is the email content. However, when I try to use </p>
  
  <pre><code>fit_transform
  </code></pre>
  
  <p>to transform corpus into feature vectors, I got</p>
  
  <pre><code> corpus_data_features = vectorizer.fit_transform(train.comment.tolist() + 
   test.comment.tolist())
   File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site- 
   packages/sklearn/feature_extraction/text.py", line 839, in fit_transform
   self.fixed_vocabulary_)
   File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-
   packages/sklearn/feature_extraction/text.py", line 762, in _count_vocab
   for feature in analyze(doc):
   File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-
   packages/sklearn/feature_extraction/text.py", line 241, in &lt;lambda&gt;
   tokenize(preprocess(self.decode(doc))), stop_words)
   File "assignment3.py", line 23, in tokenize
   stems = stem_tokens(tokens, stemmer)
   File "assignment3.py", line 14, in stem_tokens
   stemmed.append(stemmer.stem(item))
   File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site- 
   packages/nltk/stem/porter.py", line 665, in stem
   stem = self._step1b(stem)
   File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-
   packages/nltk/stem/porter.py", line 376, in _step1b
   lambda stem: (self._measure(stem) == 1 and
   File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site- 
   packages/nltk/stem/porter.py", line 258, in _apply_rule_list
   if suffix == '*d' and self._ends_double_consonant(word):
   File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-
   packages/nltk/stem/porter.py", line 214, in _ends_double_consonant
   word[-1] == word[-2] and
   IndexError: string index out of range
  </code></pre>
  
  <p>Below is my code, which I learnt from <a href="https://www.codementor.io/jadianes/data-science-python-r-sentiment-classification-machine-learning-du107otfg" rel="nofollow noreferrer">https://www.codementor.io/jadianes/data-science-python-r-sentiment-classification-machine-learning-du107otfg</a>. Could anyone help me to point out where I might possibly went wrong? Thank you very much!</p>
  
  <pre><code>import numpy as np 
  import re, nltk
  from sklearn.feature_extraction.text import CountVectorizer        
  from nltk.stem.porter import PorterStemmer
  test_data_df = pd.read_csv('test.tsv', delimiter="\t", quoting=3)
  train_data_df = pd.read_csv('train.tsv', delimiter="\t", quoting=3)
  stemmer = PorterStemmer()
  def stem_tokens(tokens, stemmer):
      stemmed = []
      for item in tokens:
          stemmed.append(stemmer.stem(item))
      return stemmed
  
  def tokenize(comment):
      comment = re.sub("[^a-zA-Z]", " ", comment)
      tokens = nltk.word_tokenize(comment)
      stems = stem_tokens(tokens, stemmer)
      return stems
  
  vectorizer = CountVectorizer(
      analyzer = 'word',
      tokenizer = tokenize,
      lowercase = True,
      stop_words = 'english',
      max_features = 200
  )
  corpus_data_features = 
  vectorizer.fit_transform(train_data_df.comment.tolist() +  test_data_df.comment.tolist())
  </code></pre>
  
  <p>The last line of code is where problem comes from. Thank you!</p>