title: UnicodeDecodeError in Python Classification Arabic Datasets
body: <p>I have Arabic datasets for classification using Python; two directories (negative and positive) in a Twitter directory.</p>
  
  <p>I want to use Python classes to classify the data. When I run the attached code, this error occurs:</p>
  
  <p>>
    File "C:\Users\DEV2016\Anaconda2\lib\encodings\utf_8.py", line 16, in decode
      return codecs.utf_8_decode(input, errors, True)</p>
  
  <blockquote>
    <p>UnicodeDecodeError: 'utf8' codec can't decode byte 0xc7 in position 0: invalid continuation byte</p>
  </blockquote>
  
  <pre><code>import sklearn.datasets
  import sklearn.metrics
  import sklearn.cross_validation
  import sklearn.svm
  import sklearn.naive_bayes
  import sklearn.neighbors
  
  dir_path = "E:\Twitter\Twitter"
  
  # Loading files into memory
  files = sklearn.datasets.load_files(dir_path)
  
  # Calculating BOW
  count_vector = sklearn.feature_extraction.text.CountVectorizer()
  word_counts=count_vector.fit_transform(files.data)
  
  # Calculating TFIDF
  tf_transformer = sklearn.feature_extraction.text.TfidfTransformer(use_idf=True).fit(word_counts)
  X = tf_transformer.transform(word_counts)
  
  # Create classifier
  # clf = sklearn.naive_bayes.MultinomialNB()
  # clf = sklearn.svm.LinearSVC()
  n_neighbors = 11
  weights = 'distance'
  clf = sklearn.neighbors.KNeighborsClassifier(n_neighbors, weights=weights)
  
  # Test the classifier
  # Train-test split
  test_size=0.4
  X_train, X_test, y_train, y_test = sklearn.cross_validation.train_test_split(X, files.target, test_size=test_size)
  
  # Test classifier
  clf.fit(X_train, y_train)
  y_predicted = clf.predict(X_test)
  print (sklearn.metrics.classification_report(y_test, y_predicted,
  target_names=files.target_names))
  print ('Confusion Matrix:')
  print (sklearn.metrics.confusion_matrix(y_test, y_predicted))
  </code></pre>
  
  <p><strong>Traceback</strong></p>
  
  <pre><code>File "&lt;ipython-input-19-8ea269fd9c3d&gt;", line 1, in &lt;module&gt;
  runfile('C:/Users/DEV2016/.spyder/clf.py', wdir='C:/Users/DEV2016/.spyder')
  
  File "C:\Users\DEV2016\Anaconda2\lib\site-
  packages\spyder\utils\site\sitecustomize.py", line 705, in runfile
  execfile(filename, namespace)
  
  File "C:\Users\DEV2016\Anaconda2\lib\site-
  packages\spyder\utils\site\sitecustomize.py", line 87, in execfile
  exec(compile(scripttext, filename, 'exec'), glob, loc)
  
  File "C:/Users/DEV2016/.spyder/clf.py", line 18, in &lt;module&gt;
  word_counts=count_vector.fit_transform(files.data)
  
  File "C:\Users\DEV2016\Anaconda2\lib\site-
  packages\sklearn\feature_extraction\text.py", line 869, in fit_transform
  self.fixed_vocabulary_)
  
  File "C:\Users\DEV2016\Anaconda2\lib\site-
  packages\sklearn\feature_extraction\text.py", line 792, in _count_vocab
  for feature in analyze(doc):
  
  File "C:\Users\DEV2016\Anaconda2\lib\site-
  packages\sklearn\feature_extraction\text.py", line 266, in &lt;lambda&gt;
  tokenize(preprocess(self.decode(doc))), stop_words)
  
  File "C:\Users\DEV2016\Anaconda2\lib\site-
  packages\sklearn\feature_extraction\text.py", line 116, in decode
  doc = doc.decode(self.encoding, self.decode_error)
  
  File "C:\Users\DEV2016\Anaconda2\lib\encodings\utf_8.py", line 16, in decode
  return codecs.utf_8_decode(input, errors, True)
  
  UnicodeDecodeError: 'utf8' codec can't decode byte 0xc7 in position 0:
  invalid continuation byte
  </code></pre>