   STL Wide Character Classes and Unicode
  Forgive the rather long post. The questions, I think, are related.I'm starting to wonder if I completely misunderstood the use of wstring, w(i/o)fstream, wcout, &c.If I have the following (Visual C++ 2005 EE):#include <iostream>#include <fstream>int main() {  std::wofstream fout( L"out.txt" );  wchar_t gu = 0x3050;  std::wcout << gu << std::endl;  fout << gu << std::endl;  return 0;}I get exactly nothing in both stdout and out.txt. In fact, if I pass in a string with a wide character to operator<<, everything following that character will be omitted, and the stream will enter a bad() state. I would mind less if I were getting garbage. Is this a locale thing (If i remove gu from the stream insertion operations, I get what appears to be an ASCII file with one CRLF line ending)? Or am I doing something completely wrong (and if so, what would be a good resource to inform myself)?Now, here's another sample:#include <fstream>#include <string>int main() {  std::wifstream fin( L"in.txt" );  std::wstring str;  std::getline( fin, str );  return 0;}in.txt is a Unicode (Codepage 1200) file with CRLF line endings, as advertised by MC++ 2005 EE. The problem is that getline leaves a wide CR (L'\r') at the end of str. I understand that getline will stop at fin.widen( '\n' ); (L'\n'), but why isn't the behavior the same as a normal std::ifstream (which, in my experience, will remove the offending CR)? Putting a fin.widen( '\r' ); in the third parameter of getline seems unethical.I've tried P.J. Plauger's Simple_codecvt class in both samples, but results were the same.Finally, I actually took the time *gasp* to read the Unicode Standard's section 5.8 ("Newline Guidelines"). I found that Unicode has it's own line separation characters, Paragraph Separator (PS) and Line Separator (LS). I could pass in LS or PS into getline, but I have found that text editors on Windows seem to have a hard time with these characters; and since I need input files to be easily written by Human Beings, this is not really a viable option -- unless someone knows of a way to get notepad(2) to recognize them.Before anyone asks me if I *need* to use Unicode, I ask: how does one handle Latin characters and Hiragana/Katakana with ASCII?Any and all help would be greatly appreciated.jfl.   
  well first, std::wofstream needs a const char * to open a file (in MSVC 2003 anyway), but a compiler would have caught that.My best bet is that the character 0x3050 doesn't represent anything in whatever the default character set is? If I modify your test app to look like the following...#include <iostream>#include <fstream>int main(){  std::wofstream fout("c:/out.txt");  std::wcout << L"Hi" << std::endl;  fout << L"Whaddup" << std::endl;  return 0;}This works fine. Perhaps there is some kind of environment setting that needs to be set to recognize such a character. This is my best bet, but I'm not sure. hth.   ;  This bit me a couple months ago.  By default, wcout and wofstream will narrow the text when they output. So you give them a double byte character, they spit out a single byte.  Below is some code i found on a newsgroup (i've lost the link to it) that'll prevent the narrowing of wide characters.#include <iostream>#include <fstream>#include <locale>typedef std::codecvt<wchar_t, char, mbstate_t> Mybase; // CLASS Simple_codecvt class Simple_codecvt : public Mybase { public:typedef wchar_t  _E;typedef char  _To;typedef mbstate_t _St;explicit Simple_codecvt(size_t _R = 0)    : Mybase(_R) {} protected:virtual result do_in(_St& _State, const _To *_F1, const _To *_L1, const _To *&_Mid1, _E *F2, _E *_L2, _E *&_Mid2) const {return (noconv);}virtual result do_out(_St& _State, const _E *_F1, const _E *_L1, const _E *&_Mid1, _To *F2, _E *_L2, _To *&_Mid2) const {return (noconv);}virtual result do_unshift(_St& _State, _To *_F2, _To *_L2, _To *&_Mid2) const {return (noconv);}virtual int do_length(_St& _State, const _To *_F1, const _To *_L1, size_t _N2) const _THROW0() {return (_N2 < (size_t)(_L1 - _F1) ? _N2 : _L1 - _F1); }virtual bool do_always_noconv() const _THROW0() {return (true);}virtual int do_max_length() const _THROW0() {return (2);} virtual int do_encoding() const _THROW0() {return (2);} };int _tmain(int argc, _TCHAR* argv[]){std::wofstream stream;std::locale loc = std::locale(stream.getloc(), new Simple_codecvt);stream.imbue(loc);stream.open("out.txt", std::ios_base::out|std::ios_base::binary);stream << "hello world";stream << static_cast<wchar_t>(0x03C8);std::wcout.imbue(loc);std::wcout << L"unicode:";std::wcout << static_cast<wchar_t>(0x03C8);if(std::wcout.fail()){std::wcout.clear(std::ios_base::goodbit);std::wcout << L"fail" << std::endl;}return 0;}I'm sure someone else here can better explain how all the locale stuff works, but this should get you headed in the right direction.   ;  oops, just read that you tried P.J. Plauger's Simple_codecvt(the link i was looking for).  I never had any trouble using it to output unicode before.  Although if i recall correctly, trying to get any unicode character set to show up in windows console is next to impossible.   ;  Another thing to keep in mind with unicode file io is that you must open the file in binary mode to correctly read or write text.The following code worked fine for me.std::wstring str;std::wifstream stream;std::locale loc = std::locale(stream.getloc(), new Simple_codecvt); // from post abovestream.imbue(loc);stream.open("C:\\unicode_doc.txt", std::ios_base::in|std::ios_base::binary);stream >> str; // str now has some unicode text;  OK, sorry for the late reply. It's 6-9 hours later here. I'm going to write separate posts for each response.Quote:well first, std::wofstream needs a const char * to open a file (in MSVC 2003 anyway), but a compiler would have caught that.Indeed. MSVC++2005 has a wchar_t * parameter as well as char *Quote:My best bet is that the character 0x3050 doesn't represent anything in whatever the default character set is?Well, 0x3050 represents 'gu' in Hiragana. My understanding was that using Unicode was precisely so that one could represent these characters without a change of character set.Quote:If I modify your test app to look like the following...#include <iostream>#include <fstream>int main(){  std::wofstream fout("c:/out.txt");  std::wcout << L"Hi" << std::endl;  fout << L"Whaddup" << std::endl;  return 0;}This works fine.Which is fine in my application too. It's once I've used a character outside of the "ASCII range" that things go a little crazy. See below.Quote:Perhaps there is some kind of environment setting that needs to be set to recognize such a character. This is my best bet, but I'm not sure.You may be right. At least I found that Lucida Console does not contain any Japanese characters in the Character Map, which might explain the lack of support in the console. Notepad seems to be temporarily changing the fonts to display Hiragana characters with Lucida Console.jfl.[Edited by - jflanglois on December 8, 2005 6:40:52 AM];Quote:Original post by telmarcAnother thing to keep in mind with unicode file io is that you must open the file in binary mode to correctly read or write text.The following code worked fine for me.std::wstring str;std::wifstream stream;std::locale loc = std::locale(stream.getloc(), new Simple_codecvt); // from post abovestream.imbue(loc);stream.open("C:\\unicode_doc.txt", std::ios_base::in|std::ios_base::binary);stream >> str; // str now has some unicode textWell, this does change one thing, and that is that the wstring was using four bytes per "character" (I assume it was doing so incorrectly, since the first four bytes were 0x00FF and 0x00FE ). The line-ending problem remains. Does anyone know of a robust method for checking line endings (including the "traditional" CRLF, CR, LF, and the Unicode LS and PS)?Is there any detailed information about using Unicode with the STL on-line, or even in published material?I'm currently downloading IBM's ICU to see how it works. But in the meantime, does anyone know of any other Unicode support libraries? I hear that boost has something in the works, but nothing specific.jfl.   