title: np.nan is an invalid document, expected byte or unicode string in CountVectorizer
body: <p>I am trying to create dependency colums for each non numeric attribute and eliminate those non-numeric attributes in adult data set from UCI. I am using CountVectorizer from sklearn.feature_extraction.text lib. But I got stuck where my program says, np.nan is an invalid document, expected byte or unicode string."</p>
  
  <p>I just want to understand why am I getting that error. Can anyone help me out, Thankyou.</p>
  
  <p>here goes my code,</p>
  
  <pre><code>import pandas as pd
  from sklearn.cross_validation import train_test_split
  from sklearn.feature_extraction.text import CountVectorizer
  
  def check(ex):
      try:
          int(ex)
          return False
      except ValueError:
          return True
  
  feature_cols = ['age', 'workclass', 'fnlwgt', 'education', 'education-num', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'capital-gain', 'capital-loss', 'hours-per-week', 'native-country', 'Target']
  
  data = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data', header=None, names = feature_cols)
  
  feature_cols.remove('Target')
  X = data[feature_cols]
  y = data['Target']
  
  X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = 1)
  
  columns = X.columns
  
  vect = CountVectorizer()
  
  for each in columns:
      if check(X[each][1]):
          temp = X[each]
          X_dtm = pd.DataFrame(vect.fit_transform(temp).toarray(), columns = vect.get_feature_names())
          X = pd.merge(X, X_dtm, how='outer')
          X = X.drop(each, 1)
  
  print X.columns
  </code></pre>
  
  <p>Error is like this</p>
  
  <p>Traceback (most recent call last):
    File "/home/amey/prog/pd.py", line 41, in 
      X_dtm = pd.DataFrame(vect.fit_transform(temp).toarray(), columns = vect.get_feature_names())</p>
  
  <p>File "/usr/lib/python2.7/dist-packages/sklearn/feature_extraction/text.py", line 817, in fit_transform
      self.fixed_vocabulary_)</p>
  
  <p>File "/usr/lib/python2.7/dist-packages/sklearn/feature_extraction/text.py", line 752, in _count_vocab
      for feature in analyze(doc):</p>
  
  <p>File "/usr/lib/python2.7/dist-packages/sklearn/feature_extraction/text.py", line 238, in 
      tokenize(preprocess(self.decode(doc))), stop_words)</p>
  
  <p>File "/usr/lib/python2.7/dist-packages/sklearn/feature_extraction/text.py", line 118, in decode</p>
  
  <pre><code>raise ValueError("np.nan is an invalid document, expected byte or "
  </code></pre>
  
  <p>ValueError: np.nan is an invalid document, expected byte or unicode string.</p>
  
  <p>[Finished in 3.3s with exit code 1]</p>