   char/wchar_t/UTF-8
  Hi,Browsing the web I found this heated discussion: http://trac.wxwidgets.org/ticket/9672about chars, wchar_t's and UTF-8 strings.I thought I knew almos everything I needed and then, reading on, I found this:Quote:>The justification for this is the C++ standard. What better>    justification can there be? String literals for unicode must>    begin with L, so wxString foo(L"foo") will compile. wxT()>    is just a more complicated way of saying L"".Sorry, but that's just completely wrong, string literals are string literals and I've never seen "L" being used to start them, no matter whether I was using Unicode or not.Maybe because all the libraries I've seen so far (with the exception of wx) are usingUTF-8 to encode Unicode stuff.>    If you use --enable-unicode, you are telling wxWidgets that>    you want to use wchar_t instead of char. Consequently you>    must say L"" instead of "". If you don't like this, then use>    --disable-unicode. Where is the problem?Sorry, but whether or not I'm using unicode should ideally be completely orthogonal to whether or not I'm using wide characters. If I use --enable-unicode, I'm telling wxWidgets that I want to use Unicode (E.g. UTF-8 encoded), the use of wchar_t is needlessly forced on me in this case, and I'll be quite happy if that won't be happening any more, because it's really annyoing if you continuously have to cast to/from UTF-8 encoded strings to that totally insane Unicode format used by wx-2.8 and earlier. Could someone please elaborate? If my locale defaults to UTF-8 and I am using chars, will I have the UTF-8 charset, but be limited to one byte chars?Thanks in advance.   
  Oh dear. I'm not even sure where to start on how convoluted that whole thing is.Quote:Sorry, but that's just completely wrong, string literals are string literals and I've never seen "L" being used to start them, no matter whether I was using Unicode or not.Maybe because all the libraries I've seen so far (with the exception of wx) are usingUTF-8 to encode Unicode stuff.C++ has string literals and wide string literals. It does not, and never has had, any comprehension of UTF-8 or any other full unicode encoding. It has normal strings, and wide strings.Wide characters are basically just a simple Unicode encoding (UCS-2/UTF-16, to be exact), that suffices for most common uses.[Edit] Reading more of the thread, it appears that the moron you quoted has the basic problem that he thinks Unicode is UTF-8. That's a very, very misguided view. UTF-8 is a very popular encoding for Unicode characters, but it's certainly not the only one. Unfortunately the current state of affairs is such that the storage type is frequently mixed up with the encoding.   ;  Frankly: C++ has no native mechanism for dealing with unicode in any manner. wchar_t, char, etc. are all pretty much hacks when it comes to dealing with any sort of localization or string operations that are non-trivial in any sense.Part of the problem is that wchar_t has no defined size in the C++ standard, and is implementation sized. This prevents one from saying "Well, C++ supports UTF-16 because wchar_t is a UTF-16 code unit." Furthermore, the size of char isn't even guaranteed to be 8 bits (although in practice it is), but is simply guaranteed to be the size of a single addressable unit such that sizeof(char) = 1 and that is the size of a "C++" byte.One can construct UTF-8 strings using char as char's have no inherit quality to them that make them individual nor unique. Thus you can use them, provided you write the appropriate support code (or obtain the appropriate support code), to indicate individual code-units of a UTF-8 string, forming the code-points of the UTF-8 characters you are interested in displaying.   ;  I assume when you say "if my locale defaults to UTF-8" you're talking about Linux, because Windows does not support UTF-8 locales. The answer to your is most probably "no", but I don't know the full context of what you're thinking about to be able to make a firm declaration :-)That discussion is specific to wxWidgets which uses UTF-16 for encoding Unicode (and particularly wxWidgets on Linux).The main point is the difference between how Unicode is handled on Windows and how it is handled on Linux. Windows uses UTF-16 internally to represent Unicode, while Linux uses UTF-8 internally. So wxWidgets have just taken the "Windows way" of handling Unicode, which makes the API look kind of funny to someone used to the "Linux way".Personally, from an API user's point of view, I don't think there's really all that much difference either way. UTF-16 makes the implementation of certain things easier, while UTF-8 makes backwards compatibility somewhat easier. It's just a trade-off.;Quote:Original post by PromitWide characters are basically just a simple Unicode encoding (UCS-2/UTF-16, to be exact), that suffices for most common uses.On Windows compilers anyways. The *nix platforms I've dealt with (including OS X) use 32-bit wchar_t.   ;Quote:Original post by SiCraneQuote:Original post by PromitWide characters are basically just a simple Unicode encoding (UCS-2/UTF-16, to be exact), that suffices for most common uses.On Windows compilers anyways. The *nix platforms I've dealt with (including OS X) use 32-bit wchar_t.Fair enough -- I should have been more specific. The standard does not say anything about the issue, like Washu said.   ;  Well, I already knew most of what has been answered 'till now. I'm quite experienced with wxWidgets -I've even contributed a very very little patch :). I have a moderate understanding of how unicode works: The different encodings, the different size of a char even within the same encoding, the differences between UTF-8/16/32, and so on. I can say that my understanding of unicode is far deeper than that of any of the co-workers I've had(which is not so much, you see). So my question was more towards how all this applies to C++. (Quite interesting, BTW, the fact that MacOS has 32 bit wchar_t. I guess that would be nice. In some aspects, at least.Codeka: Yes, I was referring to Linux. IIRC I can change my default charset. So what do you not know and need to know in order to make a firm declaration?Reading on the thread, I have seen statements about using wchar_t being something wrong (So what the heck are asians supposed to do?) and insinuations about the char type going to change with c++0x to handle unicode, but I've been doing an effort to follow the new standard for a couple of years and had never read anything alike. So I guess, if I want to support encodings greater than 8 bits without writing a full unicode conversion library, I must stick to wchar_t. Right?Quote:    Changed 11 months ago by vadz    * cc vaclavslavik added    * status changed from new to confirmedReplying to hajokirchhoff:>The justification for this is the C++ standard.FWIW I would be highly surprised if C++0x didn't support UTF-8 strings. I.e. C++ standard is going to change in this direction as well (or if it doesn't, pity for C++).;  Actually, it depends on which non-8 bit encodings you want to support and what platform you're coding on. On Windows if you want to use UTF-16, then wchar_t is perfectly reasonable. On OS X, you'll generally use short rather than wchar_t. Similarly, if you want to use UTF-32, on Windows you'd typedef a 32 bit type for the code unit and on OS X you could use wchar_t. If you want your code to work with both, then you'll need typedefs for the different platforms.Of course, you could just use a third party Unicode library like ICU to deal with your Unicode needs, and let it worry about the typedefs.   ;Quote:Original post by ravengangrelReading on the thread, I have seen statements about using wchar_t being something wrong (So what the heck are asians supposed to do?) and insinuations about the char type going to change with c++0x to handle unicode, but I've been doing an effort to follow the new standard for a couple of years and had never read anything alike. So I guess, if I want to support encodings greater than 8 bits without writing a full unicode conversion library, I must stick to wchar_t. Right?The current wchar_t type in C++ is pretty much useless.The C++0X introduces two new char types (16 and 32 bit).  If you can't fix a problem, you can at least make it worse.  It also introduces Unicode literals (using the 'U' prefix) but I'm not sure about detail on the type of those data.  It's pretty unusual for a language standard to constrain data like that.Your best bet when handling i18n of text is to use Unicode, and specifically the UTF-8 implementation of Unicode, unless you need to parse text.  If you're parsing text, all bets are off since you need a locale-specific parser anyways.  UTF-8, unlike Microsoft's UNICODE, supports all of Unicode.  If you need to widen UTF-8 into a wide character, you need at least 24 bits.So I would recommend sticking to plain ol' char and use UTF-8 as your encoding and widen as needed.   ;  Thank you guys!!   