title: UnicodeDecodeError, ascii processing for Snowball stemming algorithm in python
body: <p>I am having some trouble getting general files to read into a program that I have made. The problem I am currently having is that pdfs are based in some kind of mutated utf-8 including a BOM that throws a wrench into my entire operation. Within my application I am using the Snowball stemming algorithm which requires ascii input. There are a number of topics involving getting errors to resolve towards utf-8, however none of them involve sending them into the Snowball algorithm, or consider the fact that ascii is what I want the end result to be. Currently the file I am using is a notepad file using a standard ANSI encoding. The specific error message I get is this: </p>
  
  <pre><code>File "C:\Users\svictoroff\Desktop\Alleyoop\Python_Scripts\Keywords.py", line 38, in Map_Sentence_To_Keywords
      Word = Word.encode('ascii', 'ignore')
  UnicodeDecodeError: 'ascii' codec can't decode byte 0x96 in position 0: ordinal not in range(128)
  </code></pre>
  
  <p>My understanding was that within python, including the ignore argument would simply pass over any non-ascii characters encountered and in this way I would bypass any BOM or special characters, but clearly this is not the case. The actual code called is here:</p>
  
  <pre><code>def Map_Sentence_To_Keywords(Sentence, Keywords):
      '''Takes in a sentence and a list of Keywords, returns a tuple where the
      first element is the sentence, and the second element is a set of
      all keywords appearing in the sentence. Uses Snowball algorithm'''
      Equivalence = stem.SnowballStemmer('english')
      Found = []
      Sentence = re.sub(r'^(\W*?)(.*)(\n?)$', r'\2', Sentence)
      Words = Sentence.split()
      for Word in Words:
          Word = Word.lower().strip()
          Word = Word.encode('ascii', 'ignore')
          Word = Equivalence.stem(Word)
          Found.append(Word)
      return (Sentence, Found)
  </code></pre>
  
  <p>By including the general non-greedy non-character regex removal to the front of the string I also hoped that trouble characters would be removed, but again this is not the case. I have attempted a number of other encodings besides ascii, and a strict base64 encoding works, but is highly non-ideal for my applications. Any ideas on how to fix this in an automated way?</p>
  
  <p>Initial decoding of Element is failing, but returns a unicode error when actually passed to the encoder.</p>
  
  <pre><code>for Element in Curriculum_Elements:
          try:
              Element = Element.decode('utf-8-sig')
          except:
              print Element 
          Curriculum_Tuples.append(Map_Sentence_To_Keywords(Element, Keywords))
  
  def scraping(File):
      '''Takes in txt file of curriculum, removes all newlines and returns that occur \
      after a lowercase character, then splits at all remaining newlines'''
      Curriculum_Elements = []
      Document = open(File, 'rb').read()
      Document = re.sub(r'(?&lt;=[a-zA-Z,])\r?\n', ' ', Document)
      Curriculum_Elements = Document.split('\r\n')
      return Curriculum_Elements
  </code></pre>
  
  <p>Shown code generates the curriculum elements seen. </p>
  
  <pre><code> for Element in Curriculum_Elements:
          try:
              Element = unicode(Element, 'utf-8-sig', 'ignore')
          except:
              print Element 
  </code></pre>
  
  <p>This typecasting hackaround actually works, but then the conversion back to ascii is a bit wonky. Returns this error:</p>
  
  <pre><code>Warning (from warnings module):
    File "C:\Python27\lib\encodings\utf_8_sig.py", line 19
      if input[:3] == codecs.BOM_UTF8:
  UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  </code></pre>