What would happen if all languages began doing strings in UTF-8?
Question: Unicode is awesome. There aren't too many people who disagree with this.

Apart from Python 3 ([which did it
wrong](http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/)), what
would be the negative impact (if any) of the next major version of all
programming languages defaulting to using Unicode/UTF-8 strings?

I'm talking specifically about the many cases which require workarounds to get
UTF-8. For example, running a Java program:

    
    
    java ... -Dfile.encoding=UTF-8
    

Or working with strings in Python 2:

    
    
    # -*- coding: utf8 -*-
    ...
    unicode_string = u"This is Unicode Text"
    

Certain MySQL databases default to a different character encoding by default:

    
    
    [server]
    collation_server=utf8_unicode_ci
    character_set_server=utf8
    

etc. etc.

Why don't we all just default to using Unicode/UTF-8 and allow users to use
the workarounds if they need support for other character encodings? What would
be the problems with doing this?


Answer: UTF-8 is a variable-length encoding, which is slower to parse than fixed-
length encodings. Example: the 7th character of an ASCII string is always the
7th byte. We don't know exactly where the 7th character of a UTF-8 string is
in memory without starting from the beginning of the string and parsing the
whole thing. For long strings this can be expensive.

So for string operations where finding specific substrings based on
character/byte position is important (SQL databases are a great example of
this) other encodings can often be preferable.

Additionally, UTF-8 encodes non-english text (outside the ASCII range) as two
or more bytes, while a lot of character encodings (koi8-r for Russian, as an
example) encode all of the commonly used characters of other languages in a
single byte, which is handy for mediums such as email where all the data must
be sent over the network.

GB2312 is the primary Chinese character set, which encodes the entire Chinese
alphabet in two-byte characters, while all of these characters would be 3
bytes in UTF-8 (50% increase)

UTF-8 is amazing for compatibility, but in terms of how it represents
characters in memory, other encodings outcompete it in a lot of scenarios.

