title: UTF-16BE to UTF-8 using Boost.Locale yields garbage
body: <p>I am working with an API which returns UTF-16BE strings. I need to convert those to UTF-8 for displaying inside a UI (which in turn accepts char* buffers). To that end, I decided to employ <code>boost::locale::conv::utf_to_utf()</code> and write a conversion routine:</p>
  
  <pre><code>// defined by the API
  typedef uint16_t t_wchar_t;
  typedef std::basic_string&lt;t_wchar_t&gt; t_wstring;
  
  char* ToUtf8(const t_wstring &amp;utf16)
  {
      // print out the input buffer, using printfs instead of cout because I have to
      printf("t_wchar_t = %zu, wchar_t = %zu, char = %zu\n", 
              sizeof(t_wchar_t), sizeof(wchar_t), sizeof(char));
      const t_wchar_t *inBuf = utf16.c_str();
      const size_t inSize = utf16.size();
      // buf2str is my debugging function for printing buffers as raw bytes
      printf("UTF16 size: %zu, buf: %s\n", inSize, 
              buf2str(inBuf, inSize).c_str());
  
      // make a copy of the input buffer, prepend a BE BOM 
      // (didn't work without it, does not work with it either)
      t_wchar_t *workBuf = new t_wchar_t[inSize + 1];
      workBuf[0] = 0xfeff;
      std::memcpy(workBuf + 1, inBuf, inSize * sizeof(t_wchar_t));
      printf("Workbuf: %s\n", buf2str(workBuf, inSize + 1).c_str());
  
      // perform conversion, print out the result buffer
      const string utf8Str = boost::locale::conv::utf_to_utf&lt;char&gt;(workBuf, 
              workBuf + inSize + 1);
      const size_t utf8Size = utf8Str.size();
      printf("UTF8 size: %zu, buf: %s\n", utf8Size, 
              buf2str(utf8Str.c_str(), utf8Size).c_str());
  
      // allocate a char buffer, copy the result there and return the pointer
      char *ret = new char[utf8Size + 1];
      std::memcpy(ret, utf8Str.c_str(), (utf8Size + 1)*sizeof(char));
      printf("Return buf[%zu]: &lt;%s&gt;\n", 
              buf2str(ret, utf8Size + 1).c_str());
      delete [] workBuf;
      return ret;
  }
  </code></pre>
  
  <p>This however returns garbage when run on the API strings as well as some test data:</p>
  
  <pre><code>int main()
  {
      // simulate the input, make an example UTF-16BE stream from raw bytes
      const unsigned char test[] ={ '\0', 'H', '\0', 'e', '\0', 'l', '\0', 'l', '\0', 'o', 
              '\0', ',', '\0', ' ', '\0', 'w', '\0', 'o', '\0', 'r', '\0', 'l', 
              '\0', 'd', '\0', '!' };
      // create a t_wstring from the 16bit code sequences directly
      const t_wstring testStr(reinterpret_cast&lt;const t_wchar_t*&gt;(test), 13);
      printf("test data: %s\n", buf2str(testStr.c_str(), testStr.size()).c_str());
  
      char* utf8 = ToUtf8(testStr);
        delete [] utf8;
  
      return 0;
  }
  </code></pre>
  
  <p>Here is some output from the program for the "Hello, world!" string. As you can see, the UTF8 buffer after the convertion contains garbage.</p>
  
  <blockquote>
    <p>test data: [13/26] '' (0) 'H' (72) '' (0) 'e' (101) '' (0) 'l' (108) '' (0) 'l' (108) '' (0) 'o' (111) '' (0) ',' (44)<br>
    '' (0) ' ' (32) '' (0) 'w' (119) '' (0) 'o' (111) '' (0) 'r' (114) '' (0) 'l' (108) '' (0) 'd' (100) '' (0) '!' (33)<br>
    t_wchar_t = 2, wchar_t = 4, char = 1<br>
    UTF16 size: 13, buf: [13/26] '' (0) 'H' (72) '' (0) 'e' (101) '' (0) 'l' (108) '' (0) 'l' (108) '' (0) 'o' (111)<br>
    '' (0) ',' (44) '' (0) ' ' (32) '' (0) 'w' (119) '' (0) 'o' (111) '' (0) 'r' (114) '' (0) 'l' (108) '' (0) 'd' (100)<br>
    '' (0) '!' (33)<br>
    Workbuf: [13/26] '' (0) 'H' (72) '' (0) 'e' (101) '' (0) 'l' (108) '' (0) 'l' (108) '' (0) 'o' (111) '' (0) ',' (44)<br>
    '' (0) ' ' (32) '' (0) 'w' (119) '' (0) 'o' (111) '' (0) 'r' (114) '' (0) 'l' (108) '' (0) 'd' (100) '' (0) '!' (33)<br>
    UTF8 size: 42, buf: [42/42] '' (228) '' (160) '' (128) '' (230) '' (148) '' (128) '' (230) '' (176) '' (128) '' (230)<br>
    '' (176) '' (128) '' (230) '' (188) '' (128) '' (226) '' (176) '' (128) '' (226) '' (128) '' (128) '' (231) '' (156)<br>
    '' (128) '' (230) '' (188) '' (128) '' (231) '' (136) '' (128) '' (230) '' (176) '' (128) '' (230) '' (144) '' (128)<br>
    '' (226) '' (132) '' (128) '' (226) '' (188) '' (179)  </p>
  </blockquote>
  
  <p>What am I doing wrong? Thanks.</p>
  
  <p><b>EDIT:</b>
  Thanks to @TheUndeadFish's comment, I added endianness conversion on my working buffer before conversion and now it works as expected:</p>
  
  <pre><code>for (size_t i = 0; i &lt; inSize; ++i)
  {
      workBuf[i] = be16toh(workBuf[i]);
  }
  </code></pre>