title: Struggling with utf-16 encoding/decoding
body: <p>I'm parsing a document that have some UTF-16 encoded string.</p>
  
  <p>I have a byte string that contains the following:</p>
  
  <pre><code>my_var = b'\xc3\xbe\xc3\xbf\x004\x004\x000\x003\x006\x006\x000\x006\x00-\x001\x000\x000\x003\x008\x000\x006\x002\x002\x008\x005'
  </code></pre>
  
  <p>When converting to utf-8, I get the following output:</p>
  
  <pre><code>print(my_var.decode('utf-8'))
  #&gt; þÿ44036606-10038062285
  </code></pre>
  
  <p>The first two chars <strong>þÿ</strong> indicate it's a BOM for UTF-16BE, <a href="https://en.wikipedia.org/wiki/Byte_order_mark#Byte_order_marks_by_encoding" rel="nofollow noreferrer">as indicated on Wikipedia</a></p>
  
  <p>But, what I don't understand is that if I try the UTF16 BOM like this:</p>
  
  <pre><code>if value.startswith(codecs.BOM_UTF16_BE)
  </code></pre>
  
  <p>This returns false. In fact, printing <code>codecs.BOM_UTF16_BE</code> doesn't show the same results:</p>
  
  <pre><code>print(codecs.BOM_UTF16_BE)
  #&gt; b'\xfe\xff'
  </code></pre>
  
  <p><strong>Why is that?</strong> I'm suspecting some enconding issues on the higher end but not sure how to fix that one.</p>
  
  <p>There are already a few mentions of how to decode UTF-16 on Stackoverflow (like <a href="https://stackoverflow.com/questions/8827419/converting-utf-16-utf-8-and-remove-bom">this one</a>), and they all say one thing: Decode using <code>utf-16</code> and Python will handle the BOM.</p>
  
  <p>... But that doesn't work for me.</p>
  
  <pre><code>print(my_var.decode('utf-16')
  #&gt; 뻃뿃㐀㐀　㌀㘀㘀　㘀ⴀ㄀　　㌀㠀　㘀㈀㈀㠀㔀
  </code></pre>
  
  <p>But with UTF-16BE:</p>
  
  <pre><code>print(my_var.decode('utf-16be')
  #&gt; 쎾쎿44036606-10038062285
  </code></pre>
  
  <p>(the bom is not removed)</p>
  
  <p>And with UTF-16LE:</p>
  
  <pre><code>print(my_var.decode('utf-16le')
  #&gt; 뻃뿃㐀㐀　㌀㘀㘀　㘀ⴀ㄀　　㌀㠀　㘀㈀㈀㠀㔀
  </code></pre>
  
  <p>So, for a reason I can't explain, using only <code>.decode('UTF-16')</code> doesn't work for me. Why?</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>The original source string isn't the one I mentioned, but this one:</p>
  
  <pre><code>source = '\376\377\0004\0004\0000\0003\0006\0006\0000\0006\000-\0001\0000\0000\0003\0008\0000\0006\0002\0002\0008\0005'
  </code></pre>
  
  <p>I converted it using the following:</p>
  
  <pre><code>def decode_8bit(cls, match):
      value = match.group().replace(b'\\', b'')
      return chr(int(value, base=8)).encode('utf-8')
  
  my_var = re.sub(b'\\\\[0-9]{1,3}', decode_8bit, source)
  </code></pre>
  
  <p>Maybe I did something wrong here?</p>