title: Encoding Error with Beautiful Soup: Character Maps to Undefined (Python)
body: <p>I've written a script that is supposed to retrieve html pages off a site and update their contents.  The following function looks for a certain file on my system, then attempts to open it and edit it:</p>
  
  <pre><code>def update_sn(files_to_update, sn, table, title):
      paths = files_to_update['files']
      print('updating the sn')
      try:
          sn_htm = [s for s in paths if re.search('^((?!(Default|Notes|Latest_Addings)).)*htm$', s)][0]
          notes_htm = [s for s in paths if re.search('_Notes\.htm$', s)][0]
  
      except Exception:
          print('no sns were found')
          pass
  
      new_path_name = new_path(sn_htm, files_to_update['predecessor'], files_to_update['original'])
      new_sn_number = sn
  
      htm_text = open(sn_htm, 'rb').read().decode('cp1252')
      content = re.findall(r'(&lt;table&gt;.*?&lt;\/table&gt;.*)(?:&lt;\/html&gt;)', htm_text, re.I | re.S) 
      minus_content = htm_text.replace(content[0], '')
      table_soup = BeautifulSoup(table, 'html.parser')
      new_soup = BeautifulSoup(minus_content, 'html.parser')
      head_title = new_soup.title.string.replace_with(new_sn_number)
      new_soup.link.insert_after(table_soup.div.next)
  
      with open(new_path_name, "w+") as file:
          result = str(new_soup)
          try:
              file.write(result)
          except Exception:
              print('Met exception.  Changing encoding to cp1252')
              try:
                  file.write(result('cp1252'))
              except Exception:
                  print('cp1252 did\'nt work.  Changing encoding to utf-8')
                  file.write(result.encode('utf8'))
                  try:
                      print('utf8 did\'nt work.  Changing encoding to utf-16')
                      file.write(result.encode('utf16'))
                  except Exception:
                      pass
  </code></pre>
  
  <p>This works in the majority of cases, but sometimes it fails to write, at which point the exception kicks in and I try every feasible encoding without success:</p>
  
  <pre><code>updating the sn
  Met exception.  Changing encoding to cp1252
  cp1252 did'nt work.  Changing encoding to utf-8
  Traceback (most recent call last):
    File "C:\Users\Joseph\Desktop\SN Script\update_files.py", line 145, in update_sn
      file.write(result)
    File "C:\Users\Joseph\AppData\Local\Programs\Python\Python36\lib\encodings\cp1252.py", line 19, in encode
      return codecs.charmap_encode(input,self.errors,encoding_table)[0]
  UnicodeEncodeError: 'charmap' codec can't encode characters in position 4006-4007: character maps to &lt;undefined&gt;
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "C:\Users\Joseph\Desktop\SN Script\update_files.py", line 149, in update_sn
      file.write(result('cp1252'))
  TypeError: 'str' object is not callable
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "scraper.py", line 79, in &lt;module&gt;
      get_latest(entries[0], int(num), entries[1])
    File "scraper.py", line 56, in get_latest
      update_files.update_sn(files_to_update, data['number'], data['table'], data['title'])
    File "C:\Users\Joseph\Desktop\SN Script\update_files.py", line 152, in update_sn
      file.write(result.encode('utf8'))
  TypeError: write() argument must be str, not bytes
  </code></pre>
  
  <p>Can anyone give me any pointers on how to better handle html data that might have inconsistent encoding?  </p>