<p>Almost 5 years ago Joel Spolsky wrote this article, <a href="http://www.joelonsoftware.com/articles/Unicode.html" rel="nofollow noreferrer">"The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)"</a>.</p>

<p>Like many, I read it carefully, realizing it was high-time I got to grips with this "replacement for ASCII".  Unfortunately, 5 years later I feel I have slipped back into a few bad habits in this area.  Have you?</p>

<p>I don't write many specifically international applications, however I have helped build many ASP.NET internet facing websites, so I guess that's not an excuse.  </p>

<p>So for my benefit (and I believe many others) can I get some input from people on the following:</p>

<ul>
<li>How to "get over" ASCII once and for all</li>
<li>Fundamental guidance when working with Unicode.</li>
<li>Recommended (recent) books and websites on Unicode (for developers). </li>
<li>Current state of Unicode (5 years after Joels' article) </li>
<li>Future directions.</li>
</ul>

<p>I must admit I have a .NET background and so would also be happy for information on Unicode in the .NET framework.  Of course this shouldn't stop anyone with a differing background from commenting though.</p>

<p>Update: See <a href="https://stackoverflow.com/questions/898/internationalization-in-your-projects">this related question</a> also asked on StackOverflow previously.</p>

[{'answer_id': 59125, 'author': 'Matt Sheppard', 'author_id': 797, 'author_profile': 'https://Stackoverflow.com/users/797', 'pm_score': 2, 'selected': False, 'text': "<p>I spent a while working with search engine software - You wouldn't believe how many web sites serve up content with HTTP headers or meta tags which lie about the encoding of the pages. Often, you'll even get a document which contains both ISO-8859 characters and UTF-8 characters.</p>\n\n<p>Once you've battled through a few of those sorts of issues, you start taking the proper character encoding of data you produce really seriously.</p>\n"}, {'answer_id': 59150, 'author': 'fijter', 'author_id': 3215, 'author_profile': 'https://Stackoverflow.com/users/3215', 'pm_score': 4, 'selected': True, 'text': '<p>Since I read the Joel article and some other I18n articles I always kept a close eye to my character encoding; And it actually works if you do it consistantly. If you work in a company where it is standard to use UTF-8 and everybody knows this / does this it will work.</p>\n\n<p>Here some interesting articles (besides Joel\'s article) on the subject:</p>\n\n<ul>\n<li><a href="http://www.tbray.org/ongoing/When/200x/2003/04/06/Unicode" rel="noreferrer">http://www.tbray.org/ongoing/When/200x/2003/04/06/Unicode</a></li>\n<li><a href="http://www.tbray.org/ongoing/When/200x/2003/04/26/UTF" rel="noreferrer">http://www.tbray.org/ongoing/When/200x/2003/04/26/UTF</a></li>\n</ul>\n\n<p>A quote from the first article; Tips for using Unicode:</p>\n\n<ul>\n<li>Embrace Unicode, don\'t fight it; it\'s probably the right thing to do, and if it weren\'t you\'d probably have to anyhow.</li>\n<li>Inside your software, store text as UTF-8 or UTF-16; that is to say, pick one of the two and stick with it.</li>\n<li>Interchange data with the outside world using XML whenever possible; this makes a whole bunch of potential problems go away.</li>\n<li>Try to make your application browser-based rather than write your own client; the browsers are getting really quite good at dealing with the texts of the world.</li>\n<li>If you\'re using someone else\'s library code (and of course you are), assume its Unicode handling is broken until proved to be correct.</li>\n<li>If you\'re doing search, try to hand the linguistic and character-handling problems off to someone who understands them.</li>\n<li>Go off to Amazon or somewhere and buy the latest revision of the printed Unicode standard; it contains pretty well everything you need to know.</li>\n<li>Spend some time poking around the Unicode web site and learning how the code charts work.</li>\n<li>If you\'re going to have to do any serious work with Asian languages, go buy the O\'Reilly book on the subject by Ken Lunde.</li>\n<li>If you have a Macintosh, run out and grab Lord Pixel\'s Unicode Font Inspection tool. Totally cool.</li>\n<li>If you\'re really going to have to get down and dirty with the data, go attend one of the twice-a-year Unicode conferences. All the experts go and if you don\'t know what you need to know, you\'ll be able to find someone there who knows.</li>\n</ul>\n'}, {'answer_id': 59185, 'author': 'Jim Puls', 'author_id': 6010, 'author_profile': 'https://Stackoverflow.com/users/6010', 'pm_score': 2, 'selected': False, 'text': '<p>Rule of thumb: if you never munge or look inside a string and instead treat it strictly as a blob of data, you\'ll be much better off.</p>\n\n<p>Even doing something as simple as splitting words or lowercasing strings becomes tough if you want to do it "the Unicode way".</p>\n\n<p>And if you want to do it "the Unicode way", you\'ll need an awfully good library. This stuff is incredibly complex.</p>\n'}, {'answer_id': 59225, 'author': 'OwenP', 'author_id': 2547, 'author_profile': 'https://Stackoverflow.com/users/2547', 'pm_score': 2, 'selected': False, 'text': '<p>The .NET Framework uses Windows default encoding for storing strings, which turns out to be UTF-16.  If you don\'t specify an encoding when you use most text I/O classes, you will write UTF-8 with no BOM and read by first checking for a BOM then assuming UTF-8 (I know for sure <code>StreamReader</code> and <code>StreamWriter</code> behave this way.)  This is pretty safe for "dumb" text editors that won\'t understand a BOM but kind of cruddy for smarter ones that could display UTF-8 or the situation where you\'re actually writing characters outside the standard ASCII range.</p>\n\n<p>Normally this is invisible, but it can rear its head in interesting ways.  Yesterday I was working with someone who was using XML serialization to serialize an object to a string using a <code>StringWriter</code>, and he couldn\'t figure out why the encoding was always UTF-16.  Since a string in memory is going to be UTF-16 and that is enforced by .NET, that\'s the only thing the XML serialization framework could do.  </p>\n\n<p>So, when I\'m writing something that isn\'t just a throwaway tool, I specify a UTF-8 encoding with a BOM.  Technically in .NET you will always be accidentally Unicode aware, but only if your user knows to detect your encoding as UTF-8.</p>\n\n<p>It makes me cry a little every time I see someone ask, "How do I get the bytes of a string?" and the suggested solution uses <code>Encoding.ASCII.GetBytes()</code> :(</p>\n'}]
['https://Stackoverflow.com/questions/59105', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/5023/']