title: Deserializing UTF-8 encoded XML fails when BOM is present
body: <p>I have a really weird issue here: I'm building an interface to a third-party system which provides XML files (with a UTF-8 encoding) over a SFTP server.</p>
  
  <p>I download those files in my C# code, and then I try to deserialize them into a C# object. For most of the files, this works quite nicely, but for some, it just keeps bombing out....</p>
  
  <p>Imagine a DTO class like this:</p>
  
  <pre><code>public class Person
  {
      public string FirstName { get; set; }
      public string LastName { get; set; }
      public int Age { get; set; }
  }
  </code></pre>
  
  <p>and a XML like this:</p>
  
  <pre><code>&lt;?xml version="1.0" encoding="utf-8"?&gt;
  &lt;Person&gt;
      &lt;FirstName&gt;John&lt;/FirstName&gt;
      &lt;LastName&gt;Doe&lt;/LastName&gt;
      &lt;Age&gt;42&lt;/Age&gt;
  &lt;/Person&gt;
  </code></pre>
  
  <p>What I'm doing on my side in my C# code is:</p>
  
  <ul>
  <li>download the file content as a byte array from the SFTP server</li>
  <li>extract an UTF-8 encoded string from that binary data</li>
  <li>use that string representation for the deserialization process</li>
  </ul>
  
  <p>Something like this:</p>
  
  <pre><code>// get bytes from SFTP server
  byte[] content = _sftpClient.Download(fileName);
  
  // convert content to a UTF-8 string
  string contentAsString = Encoding.UTF8.GetString(content);
  
  try
  {
      // deserialize that string into a "Person" instance
      XmlReaderSettings settings = new XmlReaderSettings();
      settings.IgnoreComments = true;
      settings.IgnoreProcessingInstructions = true;
      settings.IgnoreWhitespace = true;
      settings.CheckCharacters = false;
  
      using (StringReader str = new StringReader(contentAsString))
      using (XmlReader xr = XmlReader.Create(str, settings))
      {
          XmlSerializer ser = new XmlSerializer(typeof(Person));
  
          if (ser.CanDeserialize(xr))
          {
              Person person = ser.Deserialize(xr) as Person;
          }
      }
  }
  catch (Exception exc)
  {
      Console.WriteLine("ERROR: {0} - {1}", exc.GetType().Name, exc.Message);
  }
  </code></pre>
  
  <p>Now I analyzed the files that worked, and those that didn't - and the difference is a three-byte prefix in the binary data (<code>0xEF 0xBB 0xBF</code>) - the "Unicode BOM" (Byte-Order Mark).</p>
  
  <p>I am aware of that BOM, and that's the reason I'm not using the <strong>binary</strong> data fetched from the SFTP Server directly. When I convert those types of files into the XML string <code>contentAsString</code>, this string <em>appears</em> to be identical - at least I can't see any difference at all.</p>
  
  <p>But the files with the 3-byte BOM at the beginning (in the binary data) cause the deserialization to fail on this line</p>
  
  <pre><code>if (ser.CanDeserialize(xr))
  </code></pre>
  
  <p>with an error:</p>
  
  <blockquote>
    <p>SystemException: Data at the root level is invalid. Line 1, position 1.</p>
  </blockquote>
  
  <p>But how on earth does the <strong>string</strong> know / "preserve" that information about the 3-byte BOM? I was expecting that by turning the byte array into a  UTF-8 encoded string, any differences would go away and the BOM should no longer be relevant...</p>
  
  <p>Any ideas on how I can reliably deal with both files <em>with</em> or <em>without</em> the 3-byte BOM? </p>