Are the 4-byte UTF-8 Kanji rare enough that I can ignore them?


I'm writing a programming algorithm which converts [code
point](http://en.wikipedia.org/wiki/Code_point)s of Kanji characters to their
respective [UTF-8](http://en.wikipedia.org/wiki/UTF-8)
[octet](http://en.wikipedia.org/wiki/Octet_%28computing%29)s.

My problem is, if I don't include 4-octet characters, and only deal with
3-octet ones, I will store each code point in 2 bytes for every character in
the text. If I also include 4-octet Kanji, I need to store each code point in
4 bytes (actually they theoretically consume 3 bytes, but it is not practical
to store 3 byte variables for optimization concerns); that makes two times
more memory consumption.

[code]

    Normal Kanji ==> UTF-8 octets: 3 bytes ==> UTF-8 code point: up to 0xFFFF (2 bytes)
    Rare Kanji   ==> UTF-8 octets: 4 bytes ==> UTF-8 code point: above 0xFFFF (4 bytes)
    
[/code]

I read [another question about
this](https://stackoverflow.com/a/3681395/245376). It is said that only **some
very rarely-used characters in the "CJK Unified Ideographs Extension B" and
"CJK Compatibility Ideographs Supplement" blocks** are encoded with 3 octets.

What does _rarely used_ mean in this case? How often are they used? Can I
completely ignore them? What happens if my algorithm converts these _rarely
used_ characters into some kind of "unidentified character" code?

(Please don't move this question to Stack Overflow. I'm more concerned with
linguistic usage of these characters. My question is not related to
programming.)




[I doubt that this is on topic, but per your request I will not move it.
Others may decide otherwise.]

In theory, the often repeated answer is "not very often". However, in
practice, actual usage is surprisingly higher than expected. With the
continued adoption and usage of Unicode, more and more characters which were
technically difficult or impossible to input are increasingly being used. Just
the other day I needed to fix a bug in our software which did not handle this
correctly. (The character in question was , hokke, U+29E3D.)

Also note that there are some regular JIS (X 0213:2004) characters that are
mapped to Unicode characters > U+FFFF. So these characters are expected to be
available and will be used.

Personally speaking, I heavily rely on many characters above U+FFFF. At the
moment, I am waiting for the next extension set to be passed because the
current sets do not include several characters that I need. (Until then, I map
them to the PUA, but this is not a long-term solution.)

> What happens if my algorithm converts these rarely used characters into some
> kind of "unidentified character" code?

Then part of your text stream will be unreadable. This will naturally hurt
human readability as well.

There is no escape: you must support 4-byte characters in UTF-8. Although I
would strongly recommend UTF-16 for the best balance. Most of your text will
be a single character in length, and those "few, rare" ones above U+FFFF will
consist of two surrogates, which are easy enough to process.




[I doubt that this is on topic, but per your request I will not move it.
Others may decide otherwise.]

In theory, the often repeated answer is "not very often". However, in
practice, actual usage is surprisingly higher than expected. With the
continued adoption and usage of Unicode, more and more characters which were
technically difficult or impossible to input are increasingly being used. Just
the other day I needed to fix a bug in our software which did not handle this
correctly. (The character in question was , hokke, U+29E3D.)

Also note that there are some regular JIS (X 0213:2004) characters that are
mapped to Unicode characters > U+FFFF. So these characters are expected to be
available and will be used.

Personally speaking, I heavily rely on many characters above U+FFFF. At the
moment, I am waiting for the next extension set to be passed because the
current sets do not include several characters that I need. (Until then, I map
them to the PUA, but this is not a long-term solution.)

> What happens if my algorithm converts these rarely used characters into some
> kind of "unidentified character" code?

Then part of your text stream will be unreadable. This will naturally hurt
human readability as well.

There is no escape: you must support 4-byte characters in UTF-8. Although I
would strongly recommend UTF-16 for the best balance. Most of your text will
be a single character in length, and those "few, rare" ones above U+FFFF will
consist of two surrogates, which are easy enough to process.

