title: Error in Python while doing text preprocessing
body: <p>I have written a couple of functions to work on text documents and convert them into bag of words. Before that I am cleaning the text by removing the stop words, tokenization etc and storing the cleaned text docs as a list which I intend to pass as an argument to another function which would create bag of words features from it. </p>
  
  <p>Here is the functions: </p>
  
  <pre><code>def cleaningDocs(doc,stem):  # 'S' for Stemming, 'L' for Lemmatization
      """This function cleans each doc string by doing the following: 
      i)   Removing punctuation and other non alphabetical characters
      ii)  Convert to Lower case and split string into words (tokenization)
      ii)  Removes stop words (most frequent words)
      iii) Doing Stemming and Lemmatization
      """
  
      # Removing punctuations and other non alphabetic characters
      import re
      alphabets_only=re.sub(r'[^a-zA-Z]'," ",doc)
  
      # Converting to lower case and splitting the words(tokenization)
      words_lower=alphabets_only.lower().split()
  
      # Removing stop words (Words like 'a', 'an', 'is','the' which doesn't contribute anything
      from nltk.corpus import stopwords
      useful_words = [w for w in words_lower if not w in set(stopwords.words("english"))] 
  
      # Doing Stemming or Lemmatization (Normalising the text)
      from nltk.stem import PorterStemmer, WordNetLemmatizer
      if (stem=='S'):  # Choosing between Stemming ('S') and Lemmatization ('L')
          stemmer=PorterStemmer()
          final_words=[stemmer.stem(x) for x in useful_words]
      else: 
          lemma=WordNetLemmatizer()
          final_words=[lemma.lemmatize(x) for x in useful_words]
  
      return(str(" ".join(final_words)))    
  </code></pre>
  
  <p>Now here is a list of document strings .This is a pandas Series object. </p>
  
  <pre><code>type(docs)
  Out[53]:
  pandas.core.series.Series
  </code></pre>
  
  <p>Each element within this document is a string. Basically each element is a text document and I want to pre-process each text document(get rid of stop words, lemmatization etc) and saving that as a new processed list. </p>
  
  <pre><code>type(docs[0])
  Out[55]:
  str
  </code></pre>
  
  <p>Ideally I want to do something like this: </p>
  
  <pre><code>doc=[]
  for x in docs:
      doc.append(cleaningDocs(x,"L"))
  </code></pre>
  
  <p>So that for each string document within docs series we get rid of it with stop words and other things and save it back to a list of documents. </p>
  
  <p>The above code is giving me this error: </p>
  
  <pre><code>---------------------------------------------------------------------------
  TypeError                                 Traceback (most recent call last)
  &lt;ipython-input-56-61345bb4d581&gt; in &lt;module&gt;()
        1 doc=[]
        2 for x in docs:
  ----&gt; 3     doc.append(cleaningDocs(x,"L"))
        4 
  
  &lt;ipython-input-42-6e1c58274c3d&gt; in cleaningDocs(doc, stem)
       13     # Removing punctuations and other non alphabetic characters
       14     import re
  ---&gt; 15     alphabets_only=re.sub(r'[^a-zA-Z]'," ",doc)
       16 
       17     # Converting to lower case and splitting the words(tokenization)
  
  /Users/mtripathi/anaconda/lib/python2.7/re.pyc in sub(pattern, repl, string, count, flags)
      153     a callable, it's passed the match object and must return
      154     a replacement string to be used."""
  --&gt; 155     return _compile(pattern, flags).sub(repl, string, count)
      156 
      157 def subn(pattern, repl, string, count=0, flags=0):
  
  TypeError: expected string or buffer
  </code></pre>
  
  <p>Whereas if I de-bug it by just passing the first element of docs (first document)  by adding break to above for loop; it works totally fine. </p>
  
  <pre><code>doc=[]
  for x in docs:
      doc.append(cleaningDocs(x,"L"))
      break
  
  doc
  </code></pre>
  
  <p>if you see the function does as required by removing the stop words from the original document and lemmatization it and then saving it back to a new processed list of documents. But it is working if I am sending only one document at a time. While sending all documents in a for loop it is throwing an error why would that be?. </p>
  
  <p>EDIT: </p>
  
  <p>Ok I just checked the type of each element in docs and see there is some issue happening that there is some elements getting casted as float there. See below: </p>
  
  <pre><code>for x in docs: 
      print(type(x))
  &lt;type 'str'&gt;
  &lt;type 'str'&gt;
  &lt;type 'str'&gt;
  &lt;type 'float'&gt;
  &lt;type 'str'&gt;
  &lt;type 'str'&gt;
  &lt;type 'str'&gt;
  &lt;type 'str'&gt;
  &lt;type 'str'&gt;
  &lt;type 'str'&gt;
  &lt;type 'str'&gt;
  &lt;type 'str'&gt;
  &lt;type 'str'&gt;
  &lt;type 'str'&gt;
  &lt;type 'str'&gt;
  &lt;type 'str'&gt;
  &lt;type 'str'&gt;
  &lt;type 'str'&gt;
  &lt;type 'str'&gt;
  </code></pre>
  
  <p>So couple of things: </p>
  
  <p>1) I want to see those elements which are float in this doc. 
  2). Second if I want to convert all the elements to string I guess we can do .astype()?</p>