title: How to correctly determine character encoding of text files?
body: <p>Here is my situation: I need to correctly determine which character encoding is used for given text file. Hopefully, it can correctly return one of the following types:</p>
  
  <pre><code>enum CHARACTER_ENCODING
  {
      ANSI,
      Unicode,
      Unicode_big_endian,
      UTF8_with_BOM,
      UTF8_without_BOM
  };
  </code></pre>
  
  <p>Up to now, I can correctly tell a text file is <code>Unicode</code>, <code>Unicode big endian</code> or <code>UTF-8 with BOM</code> by calling the following function. It also can correctly determine for <code>ANSI</code> if the given text file is not originally a <code>UTF-8 without BOM</code>. <strong>The problem is that when the text file is <code>UTF-8 without BOM</code>, the following function will mistakenly regard it as a <code>ANSI</code> file.</strong></p>
  
  <pre><code>CHARACTER_ENCODING get_text_file_encoding(const char *filename)
  {
      CHARACTER_ENCODING encoding;
  
      unsigned char uniTxt[] = {0xFF, 0xFE};// Unicode file header
      unsigned char endianTxt[] = {0xFE, 0xFF};// Unicode big endian file header
      unsigned char utf8Txt[] = {0xEF, 0xBB};// UTF_8 file header
  
      DWORD dwBytesRead = 0;
      HANDLE hFile = CreateFile(filename, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
      if (hFile == INVALID_HANDLE_VALUE)
      {
          hFile = NULL;
          CloseHandle(hFile);
          throw runtime_error("cannot open file");
      }
      BYTE *lpHeader = new BYTE[2];
      ReadFile(hFile, lpHeader, 2, &amp;dwBytesRead, NULL);
      CloseHandle(hFile);
  
      if (lpHeader[0] == uniTxt[0] &amp;&amp; lpHeader[1] == uniTxt[1])// Unicode file
          encoding = CHARACTER_ENCODING::Unicode;
      else if (lpHeader[0] == endianTxt[0] &amp;&amp; lpHeader[1] == endianTxt[1])//  Unicode big endian file
          encoding = CHARACTER_ENCODING::Unicode_big_endian;
      else if (lpHeader[0] == utf8Txt[0] &amp;&amp; lpHeader[1] == utf8Txt[1])// UTF-8 file
          encoding = CHARACTER_ENCODING::UTF8_with_BOM;
      else
          encoding = CHARACTER_ENCODING::ANSI;   //Ascii
  
      delete []lpHeader;
      return encoding;
  }
  </code></pre>
  
  <p>This problem has blocked me for a long time and I still cannot find a good solution. Any hint will be appreciated.</p>