   Converting narrow to wide strings in a cross platform friendly way
  Hey guys,I've quickly discovered what initially seemed like a very simple thing is, in fact, not so much.The issue is very simple. I want two constructors:MyClass::MyClass(char *String)MyClass::MyClass(wchar_t *String)Where the first constructor first converts the narrow string to a wide string, and then calls the second constructor. I have my compiler settings (MSVC) set to using Unicode Character Set.The idea here is I want to be able to instanciate the class either way:MyClass ClassInstance1("SomeString");MyClass ClassInstance2(L"SomeString");internally, the class holds the string as a wchar_t array.I've looked at the MultiByteToWideChar() function, but that appears to be a Windows only kind of solution. I've also checked out mbstowcs(), which, if I understand correctly, is more cross platform friendly. But it gives you no information on how big of a buffer you are going to need.I guess part of what is confusing me is what does it mean to be a wide character? I'd like to store strings internally in UFT16 format (little endian), but I'm not sure what the ramifications of that are with regards to other platforms (Windows, MacOS, PS3, XBOX, Unix).What advice can you guys give me on these issues?   
  First off, wchar_t's size is platform defined.  On Windows it is 16-bit (basically UTF-16).  I believe on a Mac it's 32-bit (UTF-32).  wchar_t in reality is not a very portable concept.So with this said, your not going to find a great platform independent way of doing this as you really need to know what your platform is expecting wchat_t to be.If you want to store your data in UTF-16 format, you just need to store it using an array of 16-bit values.  But you may be required to convert it to the native format, using another library.[edit]Added Wikipedia link   ;  Hmmm, I see. Thanks for letting me know, I didn't realize wchar_t itself was going to cause me problems. That's quite unfortunate.Okay, so I guess the followup questions would be:1. Is there a way to detect the incoming format (UTF8, UTF16, etc.)2. Having detected it, can it be converted to a different format (UTF16)I suspect the answer to #1 is "probably not", at least not directly. I could define it on a per-compiler basis, though, right? If somebody passes inMyClass ClassInstance1("SomeString");using MSVC with the UNICODE definition off, then I know that's a UTF8 string. And so on and so forth for other settings/compilers. Then, it's "just" a matter of converting it to the style I want (UTF16) and storing it.Yeesh, this is complicated.   ;  Have you looked into ICU?  It's pretty much the gold standard for working with Unicode in C, C++ and Java.  YConsider using its UnicodeString as your internal format, and use its conversion functions at the platform boundry.http://site.icu-project.org/;Quote:Original post by RonHiler1. Is there a way to detect the incoming format (UTF8, UTF16, etc.)There are some ways to detect this, assuming it is a form of Unicode.  It usually involves putting a specific series of bytes at the beginning of a file.  Note that this is more for file reading, not handling individual strings.Quote:Original post by RonHilerI could define it on a per-compiler basis, though, right?That's correct.  The nice thing is the native formats are usually defined by the platform, or at least detectable by some platform specific method.Quote:Original post by RonHilerusing MSVC with the UNICODE definition off, then I know that's a UTF8 string.Don't assume too much here, it's basically the same answer as before; see what the platform defines it as.  The MSDN documentation tends to just list it as a multi-byte encoding. Quote:Original post by RonHilerYeesh, this is complicated.Very much so.  Unicode was designed to make things easier, but there currently isn't a lot of native support in languages like C/C++.  There are some constructs coming in C++0x that should make things a little easier.But what Valarie said is correct.  That ICU library tends to be standard on handling different encodings.   ;Quote:Original post by RattrapQuote:Original post by RonHiler1. Is there a way to detect the incoming format (UTF8, UTF16, etc.)There are some ways to detect this, assuming it is a form of Unicode.  It usually involves putting a specific series of bytes at the beginning of a file.  Note that this is more for file reading, not handling individual strings.Slightly off topic but there's a bug that's been spread around the Internet to do with Notepad where if you type "bush hid the facts" in a new file and save it then re-open the file you'll get a bunch of random characters instead of the original text that was saved, this could be related to this series of bytes you mention (I don't know a great deal about unicode)? It's reported to work with most combinations of [4 letter word] [3 letter word] [3 letter word] [5 letter word], however "woah cow eat grass" is one string that does not work :-);  You'd have to look in the Unicode standard.  If I remember correctly, I think the code sequence isn't in the standard itself, but was kind of an amendment/side document.  It's not required to be in a file.   ;  Okay, cool, thanks guys.I'll look at the ICU code. I'm not interested in using them as a string class, but I could certainly use their conversion tables :)If I can get the compiler specific encodings for string literals for each compiler/setting, and use the ICU tables to convert to what I want to use (UTF16), then that gives me what I need.   ;Quote:Original post by RonHilerI guess part of what is confusing me is what does it mean to be a wide character? Not a lot. About all you can say is that a wide character will be at least as big as a regular character.The size of the character says nothing about the encoding. However, when you see wide character strings on Windows, one can typically assume UTF16 unless told otherwise.Quote:I'd like to store strings internally in UFT16 format (little endian), but I'm not sure what the ramifications of that are with regards to other platforms (Windows, MacOS, PS3, XBOX, Unix).Many non-windows platforms tend to use UTF8, which means you don't need anything other than fopen() to open a file with a 'foreign' name; the same old C and C++ APIs that take char*s carry on working. And on Windows, it's only a couple of lines to convert from UTF16 to UTF8. For this reason, I tend to standardize around UTF8 internally.Problems may arise if you need to do lots text manipulation e.g. find the kth character, or whatever, as character retrieval is no longer O(1). But UTF16 and UTF32 don't help in this regard either (unless you stay inside the basic multilingual plane).   ;Quote:Original post by RonHilerI'll look at the ICU code. I'm not interested in using them as a string class, but I could certainly use their conversion tables :)I would strongly advise you not to. ICU is huge and complicated; unless you have somewhat complex needs when it comes to string handling, it's an overkill.I think the best approach is to actually roll your own class, since std :: string is useless when it comes to Unicode. Here's a grossly simplified version of what I'm currently using: (My string class has a lot more goodies, including copy-on-write, reference counting, java-style iterators, private implementation and lots of utility methods.)/* This is missing *a lot* functionality; it's just to demonstrate the concept. */typedef unsigned short u16;typedef unsigned int u32;class string_t{    public:    string_t( const char * string )    {   type = utf_8;   pointer8 = strdup( string );    }    string_t( const u16 * string )    {   type = utf_16;   pointer16 = strdup16( string );    }    string_t( const u32 * string )    {   type = utf_32;   pointer32 = strdup32( string );    }    string_t( const wchar_t * string )    {   // This will be optimized away by the compiler anyway.   if( sizeof( wchar_t ) == sizeof( u16 ) )   {  type = utf_16;  pointer16 = strdup16( string );   }   else   {  type = utf_32;  pointer32 = strdup32( string );   }    }    const char * c_str( ) const    {   convert_to_utf8( );   return pointer8;    }    const u16 * utf16( ) const    {   convert_to_utf16( );   return pointer16;    }    const u32 * utf32( ) const    {   convert_to_utf32( );   return pointer32;    }    const wchar_t * wc_str( ) const    {   (...)    }    (...append, find, substr, a crapload of other useful methods...)    private:    union    {   char * pointer8;   u16 * pointer16;   u32 * pointer32;    };   enum type_t { utf_8, utf_16, utf_32 };    type_t type;}This allows you to use whatever encoding you need to use without worrying about it's specifics. Of course, such a class requires quiet a bit of code, and has lots of opportunities for optimization (for example, an append method would probably need to do some heuristics when it gets a string of different encoding - reencoding which one will be more optimal), however I think it's worth it, *if* you can do it right. (A class like this is useful in virtually every project.)   