The UTF-8-Everywhere Manifesto - <AUTHOR> http://www.utf8everywhere.org/ <comment><USER2> Really good article. You'll get nothing from me but heartfelt agreement. I especially liked that the article was giving numbers about how inefficient UTF8 would be to store Asian text (not really apparently).

Also insightful, but obvious in hindsight: Not even in utf-32 you can index specific character in constant time due to the various digraphs.

The one property I really love about UTF8 is that you get a free consistency check as not every arbitrary byte sequence is a valid UTF8 string.

This is a really good help for detecting encoding errors very early (still to this day, applications are known to lie about the encoding of their output).

And of course, there's no endianness issue, removing the need for a BOM which makes it possible for tools that operate at byte levels to still do the right job.

If only it had better support outside of Unix.

For example, try opening a UTF8 encoded CSV file (using characters outside of ASCII of course) in Mac Excel (latest versions. Up until that, it didn't know UTF8 at all) for a WTF experience somewhere between comical and painful.

If there is one thing I could criticize about UTF8 then that would be its similarity to ASCII (which is also its greatest strength) causing many applications and APIs to boldly declare UTF8 compatibility when all they really can do is ASCII compatibility and emitting a mess (or blowing up) once they have to deal with code points outside that range.

I'm jokingly calling this US-UTF8 when I encounter it (all too often unfortunately), but maybe the proliferation of "cool" characters like what we recently got with Emoji is likely going to help with this over time.

<comment><USER3> "The one property I really love about UTF8 is that you get a free consistency check as not every arbitrary byte sequence is a valid UTF8 string."

You don't get this at all using UTF-8. You only get it if you attempt to decode the string which even something like strlen doesn't do. Strlen will happily give you wrong answers about how many characters are in a UTF-8 string all day long and never ever attempt to check the validity of the string. Take your valid UTF-8 and change one of the characters to null, now it doesn't work in many circumstances with 'UTF-8' code.

Also, should the free consistency check ever actually work you're in a bigger pickle as you now have to figure out whether the string is wrongly encoded UTF-8 or someone sent you extended ASCII.

I did a lot of work with unicode apps. I used to have a series of about 5 strings that I could paste into a 'UNICODE' application and have it invariably break.

One was an extended ASCII string that happend to be valid UTF-8 sans BOM :)

One was a UTF-8 string with BOM and has 0x00 inside :) (I call this string how to tell if it was written with C)

One was a UTF-8 string with a BOM :)

One UTF-8 string with a some common latin characters, a couple japanese, and a character outside the BMP.

Two UTF-16 strings in LE/BE with and sans BOM.

<comment><USER4> > You only get it if you attempt to decode the string which even something > like strlen doesn't do.

Because strlen() is a count of chars in a null-terminated char[], not a decoder. Ever. It's character set agnostic.

> Strlen will happily give you wrong answers about how many characters are in > a UTF-8 string all day long and never ever attempt to check the validity of > the string.

Because, again, strlen() counts chars in a null-terminated char[]. It is giving you the _right_ answer, you are asking it the _wrong question_.

> Take your valid UTF-8 and change one of the characters to null, now it > doesn't work in many circumstances with 'UTF-8' code.

Which means it's not a valid UTF-8 decoder, but is instead treating the buffer as Modified UTF-8[1].

> that I could paste into a 'UNICODE' application

Clipboards or pasteboards in many operating systems butcher character set when copying and pasting text. Generally, the clipboard cannot be trusted to do the right thing in every circumstance. On Windows, in particular, character set can get transposed to the system character set or something rather arbitrary when text is copied.

> One was a UTF-8 string with BOM and has 0x00 inside :) (I call this string > how to tell if it was written with C)

> One was a UTF-8 string with a BOM :)

Don't use the BOM[2] in UTF-8. It's recommended against.

So really, your point is that some implementations are bad, and you have a bag of tricks for breaking implementations that don't handle all corner cases? That's pretty universal even in the non-Unicode world; there's bad implementations of everything. Windows is an especially bad implementation of most things Unicode.

A valid decoder will, indeed, consistency-check an arbitrary string of bytes as UTF-8. The OP is correct, and your corner cases don't refute his point.

[1]: <http://en.wikipedia.org/wiki/UTF-8#Modified_UTF-8>

[2]: <http://en.wikipedia.org/wiki/UTF-8#Byte_order_mark>

<comment><USER3> A name like strlen suggests that it's designed to take the length of a string, if it was called count_null_ter_char_array then I'd tend to believe you. It's not character set agonistic, it's monotheistic at the shrine of ASCII, it's all over the coding style.

Null is valid UTF-8, it just doesn't work with C 'strings'. I can get null out of a UTF-8 encoder with no problem.

My point is that UTF-8 is nowhere near the panacea being described and if you have to touch the strings themselves that it's far better to use UTF-16 in the vast majority of cases. The only time you ever really want to use UTF-8 is if you're dealing with legacy codebases, it's a massive hack.

<comment><USER5> I do not understand how UTF-16 could be better for this reason. wcslen works exactly like strlen but on wide chars instead of chars.

</comment></comment></comment></comment></comment><comment><USER6> Ok, let me be the first approving top level comment: This document is correct. The author of this document is smart. You should follow this document.

As jwz said about backups: "Shut up. I know things. You will listen to me. Do it anyway."

</comment><comment><USER7> Yes! I have been meaning to write something like this for years.

There is only one thing I would add: Never add a BOM to an UTF-8 file!! It is redundant, useless and breaks all kinds of things by attaching garbage to the start of your files.

Edit: Here is the interesting story of how Ken Thompson invented UTF-8: <http://doc.cat-v.org/bell_labs/utf-8_history>

<comment><USER8> The mark isn't useless; it clearly identifies files as UTF-8 so they can be processed as such immediately. Otherwise a program has to "sniff" several bytes to see if the encoding could be something different, and it may not guess correctly.

Also, how can "all kinds of things" break with this mark? If something is reading UTF-8 _correctly_ then it'll be fine with the mark; and if it's not reading UTF-8 correctly then it will screw up a lot more than the mark at the beginning of the file.

<comment><USER9> This argument is silly. Why not prefix every UTF-8 string with a BOM then? It's wasteful and unnecessary, because UTF-8's clean structure already makes it trivial to detect, and false positives are all but impossible for real- world text. There's a paper out there that proves this.

The UTF-8 BOM was a Microsoft invention. Nobody else uses it, and it breaks tons of things. Two examples off the top of my head: Unix hashbang scripts (i.e. #!/bin/bash), and PHP scripts (the BOM will trigger HTTP header finalization before any code is run).

<comment><USER8> You wouldn't prefix every string with it because presumably your API or program's state has already determined the string's encoding. I am not suggesting that every fragment of text has to be explicit (I agree that would be ridiculous). I am only stating facts: there is nothing incorrect about having the mark, a conformant reader must be able to handle the mark, and the mark has _some_ value as a short-cut for avoiding elaborate decoding tricks.

<comment><USER9> The thing is, the BOM is metadata, it doesn't belong in content. It violates the contract of .txt files, which is: the entire file is a single string of content.

Recognizing it at the edges of your program and stripping it out is not the end of the world, but it's annoying and no other (8-bit) encoding works that way. In fact, I find it hard to believe UTF-8 BOMs in MS programs were anything more than a programmer error. Once such files were out in the wild, everyone else had to deal with them.

<comment><USER8> There are already plenty of cases that valid UTF-8 readers have to deal with (unused ranges of code points, invalid byte combinations, etc.). Ignoring a BOM is trivial by comparison. A UTF-8 reader honestly _doesn't care_ about the "stringness" of a .txt file because of all the other crap that can be in a byte stream.

Older programs do care, but as I've said elsewhere in the thread an ASCII file can remain ASCII (no BOM). There's no reason to BOM-ify an old ASCII file if it really is ASCII and only ASCII-expecting programs will ever use it.

Over time these old programs will either be upgraded or go away and it will finally be safe to say that inputs must be UTF-8. At _that_ time, the BOM has no reason to exist.

</comment></comment></comment></comment></comment></comment><comment><USER10> Sadly, the pervasiveness of JavaScript means that UTF-16 interoperability will be needed as least as long as the Web is alive. JavaScript strings are fundamentally UTF-16. This is why we've tentatively decided to go with UTF-16 in Servo (the experimental browser engine) -- converting to UTF-8 every time text needed to go through the layout engine would kill us in benchmarks.

For new APIs in which legacy interoperability isn't needed, I completely approve of this document.

<comment><USER11> Yeah, it's really sad the number of legacy APIs which have standardized on UTF-16.

The Windows API calles UTF-16 "Unicode". Most Mac OS X APIs use UTF-16. JavaScript and Java both use UTF-16. ICU uses UTF-16. So while UTF-8 is technically superior in almost every way, it's going to be an uphill battle to standardize on it.

I appreciate that new languages like Rust and Go made the choice of UTF-8 as their native text encoding. But there's a lot of inertia for UTF-16, and I'm not sure it'll be easy to ever get free of it.

<comment><USER12> OS X APIs generally use NSString/CFString, which hide the actual encoding of the string; they can be any encoding at all internally.

<comment><USER11> While they could, in theory, hide the actual encoding, the APIs all refer to UTF-16 code units as "characters;" so while they could use UTF-8 as the internal encoding, you need to use and understand UTF-16 in order to interact with them properly. When you ask for the "length" of a string, you are told the number of UTF-16 code units. When you get a character at an index, you get the UTF-16 code unit. That's what I mean when I say the APIs use UTF-16; everything in the API that deals with individual "chracters" is actually referring to UTF-16 code units.

The same is true of JavaScript; while you could technically implement the strings however you want, the APIs are all oriented around UTF-16 code units. And the Windows API as well, is all built around UTF-16 code units.

The problem with all of these APIs is that they make the mistake of conflating characters and code units. They all make the assumption that a character consists of a single, fixed width integer, of some given size (16 bits in the case of UTF-16). It is better to distinguish between indexing in code units (such as bytes in UTF-8 or 16 bit integers in UTF-16) and indexing in code points, or glyphs, or whatever higher level concept you are talking about. Really, for anything higher than the code unit level, you should be dealing with variable-length strings, and not try to force that into fixed length units. With UTF-8, there's no temptation to treat a single code unit as being an independently meaningful entity, as that assumption breaks down as soon as you get past the ASCII range; while with UTF-16, it's easy to make that mistake, since it holds true for everything in the Basic Multilingual Plane, which contains most characters you're likely to encounter on a day to day basis.

</comment></comment></comment></comment><comment><USER13> Personally, I prefer UTF-8 as well. However, I think this whole debate about choice of encoding gets blown out of proportion.

Consider the following diagram:

                                   [user-perceived characters] <-+
                                                ^                |
                                                |                |
                                                v                |
                      [characters] <-> [grapheme clusters]       |
                           ^                    ^                |
                           |                    |                |
                           v                    v                |
          [bytes] <-> [codepoints]           [glyphs] <</comment><comment>----+

Choice of encoding only affects the conversion from bytes to codepoints, which is pretty straight-forward: The subtleties lie elsewhere...

</comment><comment><USER14> Disagree

"UTF-16 is the worst of both worlds—variable length and too wide"

Really, the author tries to convince the reader, but it's not that clean cut.

One of the advantages of UTF-16 is knowing right away it's UTF-16 as opposed to deciding if it's UTF-8/ASCII/other encoding. Sure, for transmission it's a waste of space (still, text for today's computer capabilities is a non issue even if using UTF-32)

"It's not fixed width" But for most text, it is. Sure, you can do UTF-32 and it may not be a bad idea (today)

Yes, Windows has to deal with several complications and with backwards compatibility, so it's a bag of hurt. Still, they went the right way (internally, it's unicode, period.)

"in plain Windows edit control (until Vista), it takes two backspaces to delete a character which takes 4 bytes in UTF-16"

If I'm not mistaken this is by design. The 4 byte characters is usually typed as a combination of characters, so if you want to change the last part of the combination you jut type one backspace.

<comment><USER11> > One of the advantages of UTF-16 is knowing right away it's UTF-16 as opposed > to deciding if it's UTF-8/ASCII/other encoding. Sure, for transmission it's > a waste of space (still, text for today's computer capabilities is a non > issue even if using UTF-32)

First of all, if you don't know the encoding, then you don't know the encoding, and you will need to figure out if it's UTF-8, UTF-16, ISO-8859-1, etc. If you happen to know that it's UTF-16, you still need to figure out if it's UTF-16BE or LE.

> "It's not fixed width" But for most text, it is.

This is a dangerous way of thinking. One of the big problems with UTF-16 is that for most text, it is fixed width; so many people make that assumption, and you never notice the problem until someone tries to use an obscure script or an emoji character. This means that bugs can easily be lurking under the surface; while with UTF-8, anything besides straight ASCII will break if you assume fixed width, making it much more obvious.

> Sure, you can do UTF-32 and it may not be a bad idea (today)

UTF-32 isn't really meaningfully fixed width either. Sure, each code point is represented in a fixed number of bytes, but code points are not necessarily the interesting unit you want to index by. A glyph could be composed of several code points. Most of the time, you actually want to deal with text in longer units such as words or tokens, which are going to be variable width anyhow. The actual width of individual code points is only really of interest to low-level text processing libraries, not most applications.

<comment><USER14> "First of all, if you don't know the encoding, then you don't know the encoding"

True. But as you said, you have to know if it's BE or LE on UTF16. And there are ways to determine that automatically. (or it's on the same platform so it doesn't matter). With "ASCII compatible" codes, you can't.

I guess the main issue to me is that UTF-16 is not "ASCII compatible" so you know it's a different beast altogether.

And don't worry, I'm not assuming UTF-16 is fixed width. One should use the libraries and not try to solve this 'manually'.

About UTF-32 think: CPU registers and operations. Working with bytes is inefficient (even with the benefit of smaller size).

<comment><USER7> > True. But as you said, you have to know if it's BE or LE on UTF16.

Yes, with UTF-16 you need to know not just the encoding, but also the endianness. That makes UTF-16 _worse_ , not better.

UTF-16 is really the worst of all possible worlds: tons of wasted space with all the complexities of variable encoding without fixed endianness.

</comment></comment></comment></comment><comment><USER8> Markus Kuhn's web page has a lot of useful UTF-8 info and valuable links (e.g. samples of UTF-8 corner cases that people often miss).

<http://www.cl.cam.ac.uk/~mgk25/unicode.html>

<comment><USER15> This is a great resource; it was extremely useful when I was writing a UTF-8 library myself. I found the UTF-8 stress test file is particularly useful to run tests against: <http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt>

<comment><USER16> Hmm, TextMate has problems with "5.2 Paired UTF-16 surrogates" in that stress test file.

(Yes, I interpreted the file as UTF-8 in TextMate).

</comment></comment></comment><comment><USER17> Totally agree re: UTF-8 vs other Unicode encodings.

But are there still still hold-outs who don't like Unicode? Last I heard some CJK users were unhappy about Han Unification: <http://en.wikipedia.org/wiki/Han_unification>

<comment><USER18> The main problem is that it means sort-by-unicode-codepoint puts things in a ridiculous order in japanese/korean. I kind of wish UTF-8 had the latin alphabet in a silly order, so that western programmers would realise they need to use locale-aware sort when sorting strings for display.

<comment><USER19> I spoke with several Japanese people who said that some valid characters are not representable in Unicode.

That means that it's not just a technical problem (expensive sort routines or inefficient encodings) -- it's a _semantic_ problem.

<comment><USER20> The way I've heard it explained, there are some historical alternate versions of some characters (A Latin-alphabet equivalent might be the way we sometimes draw "a" with an extra curl across the top, and sometimes without) that have the exact same semantic meaning, and so they were 'unified' to a single code- point. Unfortunately. some people spell their names exclusively with one variant or the other, and Han unification makes that impossible in Unicode.

</comment></comment></comment></comment><comment><USER21> For those who don't know it, UTF8-CPP[1] is a good lightweight header-only library for UTF conversions, mostly STL-compatible.

[1] <http://utfcpp.sourceforge.net/>

</comment><comment><USER22> That collection of best practices can hardly be considered as "UTF-8 Everywhere Manifesto" as it focuses on Windows and C++. It's good, but I'd rather see more manifesto like document for all cases on a domain like that.

<comment><USER23> I suspect this is mainly because Windows C++ programmers are the largest group that they feel need convincing. Which isn't totally their fault, Microsoft haven't done well by them by not offering good support for UTF-8; you can convert to/from it using WideCharToMultiByte but that's pretty low level, and higher-level APIs like CString will cheerfully munge UTF-8 strings for you. They also tend to conflate Unicode and UTF-16 which again doesn't help less experienced programmers realise that there might be alternatives.

I've been through the Windows Unicode stuff at a previous job, which ended up using mostly UTF-16 with some UTF-8 for interfacing to third party libraries and for files which needed to be backward compatible to ASCII (plus significant space savings, which I fought hard for). I think I prefer that approach though, since after the (difficult) conversion you didn't need to worry about encodings in 99% of the code. By their rules you'd gain significant complexity by transforming all over the place in any non-trivial GUI code.

</comment></comment><comment><USER24> Text is maddening, the modern Tower of Babel.

Is there a definitive reference, or small handful of references, to learn all that's worth knowing about text, from ASCII to UTF-∞ and beyond?

<comment><USER25> Joel Spolsky's 'The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)' is a good start: <http://www.joelonsoftware.com/articles/Unicode.html>

Like a few other specialised fields (cryptography comes to mind) the key takeaway is to use a library and rely on the work of people who know it better than you do and have handled all the subtleties already :)

</comment></comment><comment><USER26> I use UTF-8 for transmitted data and disk I/O, and I use UCS-4 (wchar_t on Linux/FreeBSD) for internal representation of strings in my software.

I generally agree with this article, but I disagree with it on the point that UTF-8 is the only appropriate encoding for strings stored in memory, and also I disagree on the point wchar_t should be removed from C++ standard or made sizeof 1, as in Android NDK.

Let me explain why.

In UTF-8 single Unicode character may be encoded in multiple ways. For example NUL (U+0000) can be encoded as 00 or as C0 80. The second encoding is illegal because it's longer than necessary and forbidden by standard, but naive parser may extract NUL out of it. If UTF-8 input was not properly sanitized, or there is a bug in charset converter, this may result in exploit like SQL injection or arbitrary filesystem access or something like that: malicious party can encode not only NUL, but ", /, \ etc this way.

Also UTF-8 string can't be cut at arbitrary position. Byte groups (UTF-8 runes) must be processed as a whole, so appear either on left side or on the right side of cut.

Reversing of UTF-8 string is tricky, especially when illegal character sequences are present in input string and corresponding code points (U+FFFD) must be preserved in output string.

I think UTF-8 for network transmitted data and disk I/O is inevitable, but our software should keep all in-memory strings in UCS-4 only, and take adequate security precautions in all places where conversion between UTF-8 and UCS-4 happens.

And sizeof(wchar_t)==4 in GCC ABI is not a design defect, wchar_t exists for a good reason. I admit that sizeof(wchar_t)==2 on Windows is utterly broken.

<comment><USER27> Concerning "cut at an arbitrary position" actually utf-8 is the only codec that can deterministically continue a broken stream because bytes that start a character are special.

</comment></comment><comment><USER16> The strangest thing about Unicode (any flavor) is that NULL, aka \0, aka "all zeros" is a valid character.

If you claim to support Unicode, you have to support NULL characters; otherwise, you support a subset.

I find most OS utilities that "accept" Unicode fail to accept the NULL character.

FWIW, UTF-8 has a few invalid characters (characters that can never appear in a valid UTF-8 string). Any one of them could be used as an "end of string" terminator if so desired, for situations where the string length is not known up front.

We could even standardize which one (hint hint). I suggest -1 (all 1s).

UPDATE: I meant "strange" as in "surprising", especially for those coming from a C background, like me.

<comment><USER15> No. NUL is backwards-compatible with ASCII, and is used everywhere. Choosing some arbitrary invalid UTF-8 byte for use as a terminator would be a terrible decision. If you want to handle NUL, simply use length-annotated slices instead of C-style NUL-terminated strings. Anything else is completely wrong.

<comment><USER16> Did you even read my comment? NULL is a valid UTF-8 character.

If specific languages and their standard libraries _choose_ to treat it as a string terminator (C, I'm looking at you), well, then fine.

But it's still valid Unicode. If you claim to support Unicode strings, but don't support the NULL character in those strings, you don't support Unicode strings in their entirety.

Going further, there's nothing in the ASCII spec that requires NULL to only appear at the end of a valid string. That's a C language convention, AFAIK (maybe it started earlier...).

<comment><USER28> I think what you are trying to say is:

"because UTF-8 has invalid character sequences, we could potentially use one of them to represent end-of-string, which would allow us the flexibility of a null-terminated string (not keeping track of the length) without the restriction of no-nulls-allowed."

You're right! Great. But you are not revealing a "strange thing" about Unicode. You are instead making a general comment about null-terminated strings. So why use such inflammatory and misleading language like "If you claim to support Unicode, you have to support NULL characters"?

Update: I don't object to your idea at all, it's a neat trick! It's just that the way it's phrased, it sounds like Unicode's design _contributed_ to this NULL-terminal problem, when in fact even NULL-terminated ASCII strings cannot 'handle' a null character in this sense.

To augment your idea, though, how about you use '0xFF 0x00' as a terminator? This way, backward-compatibility is preserved in all cases except UTF-8 => ASCII with NULLs, and in this case the string will be truncated rather than a buffer overflow (i.e. "fail closed").

</comment></comment></comment></comment><comment><USER29> Can someone explain to me how UTF-8 is endianness independent? I don't mean that I am arguing the fact, I just don't understand how it is possible. Don't you have to know which order to interpret the bits in each byte? And isn't that endianness?

<comment><USER30> It's endianness independent in the sense that the order in which you interpret the _bytes_ in each _character_ does not depend on the processor architecture, unlike UTF-16.

If your processor interprets the _bits_ in each _byte_ in a different order, that might be a problem, but it's not what we're talking about when we usually talk about the endianness of character encodings.

<http://en.wikipedia.org/wiki/Endianness>

<comment><USER29> Thank you. That is very good to learn and I looked over the wikipedia article. But as far as _byte_ order, how is that architecture independent? Is it just that utf-8 dictates that the order of the bytes always be the same, so whatever system you're on, you ignore its norm, and interpret bytes in the order utf-8 tells you to?

<comment><USER31> utf-8 is a single byte encoding. Reversing the order of a sequence that's one byte long just gives back that one byte.

<comment><USER30> No it isn't. Any letter with an accent will take up two bytes. Most non-Latin characters take up three bytes, sometimes even four.

<comment><USER31> Poorly phrased. It can take multiple bytes to fully define one codepoint, but the encoding is defined in terms of a stream of single bytes. In other words, each unit is one byte, hence flipping each unit gives back the same unit.

This is not the case for UTF-16 and UTF-32.

<comment><USER32> AFAIK the correct term is "byte oriented".

</comment></comment></comment></comment></comment></comment></comment><comment><USER33> can not agree more! it will be a much better world if we all use utf8 for external string presentation. i don't care about what your app use internally, but if it generates output, please use utf8.

</comment><comment><USER34> I there a simple set of rules for people who currently have code which use ASCII, to check for UTF-8 cleanness?

In particular, what should I watch out for to make an ASCII parser UTF-8 clean?

<comment><USER8> If you're reading something in pieces, like a buffer that fills 256 bytes at a time, you have to be careful. UTF-8 is a multi-byte encoding so the last byte in your buffer may not completely finish a code point. Unlike older code that can just read a bunch of bytes and use them, with multi-byte encodings you have to have a way to deal with "left-overs" until new bytes show up.

Fortunately the UTF-8 encoding (e.g. see the Wikipedia page) makes it clear when a byte is the beginning of a new point and it tells you how many intermediate bytes should follow.

</comment></comment><comment><USER35> How could we avoid acronyms like 'utf-8'?

We can do better than that. Unicode8?

<comment><USER36> Just use the term "string" to refer to utf-8, and the term "data in nonstandard encoding X" to refer to other encodings.

In the article he puts in in terms of std::string, but more generally I think this is what he means.

<comment><USER37> You're confusing things. Strings cannot be utf-8 any more than you can be your signature.

"strings" are abstract data structures. They are lists of characters. Not bytes, not integers, but characters. Often, we use the Unicode character set as the set of allowable characters. There are other character sets.

Internally, strings often represent characters as integers. When using the Unicode character set, strings then use the Unicode encoding to integers (a table mapping characters to unique numbers). Sometimes we use other character sets and encodings.

Unfortunately, integers are abstract. You can't store them in a file or transmit them over a network until you pick a concrete representation as bytes. How many bits per integer? Big or little endian? Etc. That's where UTF-8 comes into play.

UTF-8 is a merely a compressed data format used to represent a sequence of integers as a sequence of bytes - a way that happens to have some properties that make it convenient for representing strings.

UTF-8 is not Unicode.

UTF-8 can also be used for other types of numerical data. As a silly example, suppose you had a list of ages of houses. Many houses are less than 100 years old. A few are more than 300 years old. An efficient serialization of that data would be to represent the ages as integers and then utf-8 encode your list of integers.

Some true statements: A character set is a set of characters. Characters are not integers or bytes. A mapping from characters to integers is an encoding. Unicode is a standard that defines a character set and an encoding to integers. Mapping integers to bytes is confusingly also called encoding. UTF-8 is an encoding from integers to bytes. Unicode defines a set of characters and an encoding of characters to integers. UTF-8 is an encoding of integers to bytes. UTF-8 is not Unicode.

<comment><USER36> I'm just saying that if we standardize on encoding, we dont need to talk about encoding. Which is my interpretation of the original document.

Separate point: There is no such thing as an abstract string or integer in a computer, no matter what language you are using. Every string in a computer has an encoding - you have to store it as ones and zeros.

If we standardize on UTF-8 as an encoding, we just dont need to use the awkward phrase "UTF-8" in ordinary conversation.

</comment></comment></comment></comment><comment><USER38> That page is misleading when it comes to Japanese text: UTF-8 sucks for Japanese text. UTF-8 and UTF-16 aren't the only two choices within the whole world, which is demonstrated in their choice of encoding Shift-JIS.

<comment><USER39> Can you elaborate on that? Why does Unicode suck for Japanese text?

<comment><USER40> Not only kanji, but also hiragana and katakana (syllabic alphabets) encode to three bytes per character. Shift-JIS can encode all three to two bytes, as well as half-width katakana to one byte per character.

However, if size is such a concern (eg for web transmission), text compression neutralizes the perceived benefit of region-specific encodings.

Shift-JIS' continued popularity has much more to do with change aversion than it does technical merit.

<comment><USER19> As I said above, I spoke with several Japanese people who said that some valid characters are not representable in Unicode.

Some details can be found here: <http://en.wikipedia.org/wiki/Han_unification>

</comment></comment></comment></comment><comment><USER3> tl:dr; Use UTF-8 when you need to use unicode with legacy APIs, never anywhere else.

UNIX isn't UTF-8 because UTF-8 is better, UNIX is UTF-8 because you can pass UTF-8 strings to functions that expect ASCII and it kinda works. This is really the only thing you need to know about UTF-8 and why it's better.

There are few pieces of software that don't have to talk to legacy APIs that store strings natively in UTF-8.

C# and Java are probably the best examples of software that was engineered from the ground up and thus uses UTF-16 internally because it's much less likely to run into issues like String.length returning 32 yet only containing 31 characters. If you use UTF-8 expect this result anytime a string contains a real genuine apostrophe.

"UTF-8 and UTF-32 result the same order when sorted lexicographically. UTF-16 does not."

This is complete and utter bullshit, to sort a string lexicographically you need to decode it, if you've decoded the string into UNICODE then they sort the exact same way.

There are lots of gotchas for sorting UNICODE strings including normalization because you can write the semantically equivalent strings in unicode multiple ways. eg. ligatures.

If you're sorting bit strings that happen to contain UTF-8/32 then you're not sorting lexicographically and your results will be screwed up anyway.

<comment><USER6> > decoded the string into UNICODE

I think you are quite confused.

1) Unicode is not an acroynm. 2) You cannot "decode into Unicode". I think you mean "decode into codepoints". 3) If that is what you mean, then you are wrong about sorting: Sorting UTF-8 and UTF-32 bytestrings will indeed sort them lexicographically by code point, which was the author's point. No, that will not generally be the sort you _want_; but no amount of 'decoding' will give you the sort you want. For that you need to first normalize, and then follow the collation rules, which don't sort by raw code points at all.

</comment></comment><comment><USER41> ASCII and UTF-8 are too US centric. That's why adoption in places like China is so low.

Also, if there's variable length encoding why can't we just do a proper way and improve size for the same computational cost?

<comment><USER11> Did you read the article, including the part about Asian text? Like it or not, most text these days is embedded in markup languages like XML or HTML, in which all of the markup is within the ASCII range. This, coupled with the fact that UTF-8 gives you a factor of 2 savings over UTF-16 for the ASCII range, while only a factor of 1.5 increase over UTF-16 for CJK characters, means that for much text (such as anything on the Web), UTF-8 is actually smaller than UTF-16 even for CJK text.

Yes, ASCII is obviously too US centric; you can't encode any writing systems other than the Roman alphabet in ASCII. However, that's not at question here. The question is, which Unicode encoding should you use, so you can represent all writing systems in a single encoding. And the major contenders are UTF-8 and UTF-16. The point of this article is, for that purpose, UTF-8 is a far better choice.

> Also, if there's variable length encoding why can't we just do a proper way > and improve size for the same computational cost?

What do you mean by a "proper way"? If size is what you care about, just compress your data. Compression will do a lot better for a much wider range of data than some clever encoding will. UTF-8 is a carefully constructed encoding designed to meet several design criteria. For instance, you could get better size for a wider range of character sets by having a single byte to represent switching between character sets; so you could use that byte, and then a whole bunch of 2 byte CJK characters. But that would defeat one of the design goals of UTF-8, which is to be self synchronizing. That means that if you get a partial sequence (such as a sequence that has been truncated), you can start decoding the characters after a fixed number of bytes. In the case of UTF-8, you will never have to go more than 3 bytes before you can start decoding again. In my hypothetical scheme where certain symbols were used to switch between character sets, you would not be able to interpret anything until you found the next such symbol. This makes UTF-8 more robust in the face of errors.

Another design goal of UTF-8 was to be backwards-compatible with ASCII. Like it or not, ASCII has been the standard encoding for decades, and there is a lot of text in ASCII and a lot of software that uses ASCII delimiters and the like.

So, while it would be possible, in theory, to define a character encoding that is more "fair" than UTF-8, that ignores many of the other goals of the design of UTF-8. And UTF-8 is widely supported and used (it is the most popular encoding on the Web, even in places like Japan, and a close second in China), while a new encoding would require another large, global, and painful transition process to introduce.

<comment><USER42> The author compares UTF-8 to UTF-16, while there are a myriad better encodings than both for different Asian languages.

For instance: EUC-JP in Japan, BIG5 in Taiwan, GB in China, etc. Different EUC encodings are variable-length and are a lot more efficient since they put common subsets in lower parts of the table for each language, close to each other so they also compress better, while allowing tricks for text matching and searches (not really necessary for web sites, but it's nice to use the same encoding throughout applications sometimes). Russian and Greek are also basically multiplied by two in size.

There are a lot of other considerations.

If you think a 30%+ saving in size (and latency) is not a big deal, then you're a lot more likely to lose to local competitors. Note that gzipped or otherwise compressed text makes differences even worse at least in the case of Japanese - where UTF-8 text gets de-aligned all over the place to odd byte sizes and compresses worse. Add to that the fact that Asians browse the net A LOT from the phone and have done so for much longer than westerners and in a bigger percentage, and you have your problem exacerbated even further.

There is a lot more to consider and like it or not it's not as simple as "UTF-8 for everything and everybody, ever!"

<comment><USER30> Have you ever worked with a system that needs to deal with more than one language at the same time? What if your users want to mix Japanese with Russian in the same sentence? Or Japanese and simplified Chinese? (Yes, people do that.)

In the global Internet, UTF-16 and UTF-8 are the only games in town.

<comment><USER42> All the damn time I'm using several languages.

Then UTF (and EUC's) are the way to go.

It's not like you have to use the same encoding all the time.

<comment><USER43> > It's not like you have to use the same encoding all the time.

Then you are going to feed someone garbage. Why feed people garbage?

<comment><USER42> ??

Not if you know what you're doing. Not any more than using utf8 exclusively all the time and for all purposes.

<comment><USER43> > Not if you know what you're doing.

This is nice in theory. In practice, people make mistakes. Make it easy on yourself.

> Not any more than using utf8 exclusively all the time and for all purposes.

Maybe I was unclear: Feeding me Chinese text in UTF-8 is _not_ garbage. Feeding me _anything_ in one of the GB encodings _is_ garbage.

Garbage, to me, is text in an encoding I can't handle. If you only use UTF-8, that cannot possibly happen.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER44> Strings (NSString) on Apple platforms are UTF-16. The Apple platforms are not exactly lagging behind in either multilingual, or text processing. I wonder what this team of three people knows that Apple doesn't? Or is it the other way around, that Apple knows something they don't, and when it comes to shipping products that work in the real world, Apple has figured out how to do it?

<comment><USER45> NSStrings are opaque--you always call accessor functions and never have access to the low level backing store. The reason they are good is that you can't get data into or out of them without specifying an encoding, which leaves the actual encoding of the backing store as an implementation detail.

The fact is, I don't even _know_ (or see documented) that the backing _is_ UTF-16--Apple is free to change that at their whim and no user programs would break.

<comment><USER12> It's not documented (presumably) for that very reason.

In fact, the opposite is implied by initWithBytesNoCopy:length:encoding:freeWhenDone: - it should be possible _right now_ to have NSStrings with arbitrary internal representations, even if most other creation methods currently convert to UTF16.</comment></comment></comment>


Ask HN: Why is this shady actor building a JavaScript botnet? - <AUTHOR> I run ChooseJarvis.com with my partner and our team. It's a social media automation tool. That's not relevant at all, it's just my job to tell you that.
Our referrer logs show tons of hits from:
http://cookie-law-enforcement-ii.xyz/
This is classic referrer spam.
The above site links to
http://cookie-consent.org/?lang=en
Which is a mini marketing project of
http://front.to/
Which according to ahrefs and google, nobody has ever heard of.
Looks to me like they're building a massive Javascript botnet.
Why?
There's a lot of things you can do, right? Keylog all website visitors (?), replace all adsense units with your own ads, use all the clients/visitors as a DDOS botnet, click fraud (?), serve downloads, serve exploits, etc...
The developers/websites have a sort of effortless polish that makes me more curious than I would otherwise be.
If this is indeed shady, and it looks like it, this is scary. I imagine a LOT of webmasters might fall for this. <comment><USER2> So front.to doesn't see to be changing the JS you upload (yet) but the js file at cookie-consent.org is interesting ( //cdn.front.to/libs/cookieconsent.min.js ) got google analytics in there, and of course could be changed to run whatever at any time... Maybe someone could use that to identify more about this shady actor?

un-minified js: [https://gist.github.com/benmcnelly/8e56aa308bef72c7aa007b7c1...](https://gist.github.com/benmcnelly/8e56aa308bef72c7aa007b7c1c6d24a9)

<comment><USER3> Here's the original [https://github.com/silktide/cookieconsent2/blob/1.0.9/cookie...](https://github.com/silktide/cookieconsent2/blob/1.0.9/cookieconsent.js) It seems they are actually injecting that google analytics part.</comment></comment>


The Matthew Effect - <AUTHOR> https://en.wikipedia.org/wiki/Matthew_effect <comment><USER2> As with anything secular, this parable has been misinterpreted to suit worldly matters. The original parable refers to resources used to do good, thus the growing investments is not only about material wealth but as well as any other kind of resources (wisdom, knowledge, time, love, etc) to do the most good for people. Refer to Luke 12:48 and the parable of the shrewd manager for a better context [https://www.desiringgod.org/interviews/does-jesus-commend- di...](https://www.desiringgod.org/interviews/does-jesus-commend-dishonesty- in-luke-16)

</comment><comment><USER3> So true, the riches families will accumulate wealth at a rate that will make them the riches families forever at an exponential rate.

By the way, this also applies to personal knowledge the more you learn the easier it will be for you to accumulate more knowledge in relation to others.

It's nuts how much the Matthew Effect applies to the accumulation of X.</comment>


Writing and Programming, Crafts Worth Honing - <AUTHOR> http://www.martin-brennan.com/writing-programming-crafts-worth-honing/

<comment><USER2> How do you change yourself to love writing if you are currently someone for whom it feels futile and frustrating?

<comment><AUTHOR> I found it hard to get into at first and I think that it's mostly the way you go about it. Write about subjects you love, try different ways of writing (e.g. I just started writing stories longform with pad and paper).

Most of all, I think you should write for yourself, not anyone else's approval/recognition. If you can see at least small improvements in your writing over time, then you have success instead of futility. There's a great quote from Kurt Vonnegut about this:

"Find a subject you care about and which you in your heart feel others should care about. It is this genuine caring, and not your games with language, which will be the most compelling and seductive element in your style.

I am not urging you to write a novel, by the way — although I would not be sorry if you wrote one, provided you genuinely cared about something. A petition to the mayor about a pothole in front of your house or a love letter to the girl next door will do."</comment></comment>


Samsung, Nokia say they don't know how to track a powered-down phone - <AUTHOR> http://arstechnica.com/security/2013/11/samsung-nokia-say-they-dont-know-how-to-track-a-powered-down-phone/ <comment><USER2> They are probably not even lying that the baseband processor is shut down by the application processor, as it controls the power and system clocks.

But the dirty little secret is that the baseband processor is still a completely uncontrolled subsystem, loaded with some propietary binary blob by trustworthy companies like Qualcomm. GPS and even the microphones are usually integrated into the baseband, not part of the application processor that runs your Android. So you have a perfectly capable ARM processor running a propietary RTOS system, written completely in C (or C++ occasionally) with access to all the vital periphery and a gigantic attack surface in dealing with all the mobile communication protocols. The only reason there hasn't been a complete breakdown yet is that it's difficult for amateur researchers to exploit, you need expensive RF hardware and the mobile communication protocols are huge bodies of closed committee designed standards. But it is without a doubt in the reach of the NSA, and they are probably actively exploiting baseband processors already.

(Interestingly, since baseband processor have grown in complexity, most smartphones can now update the firmware on them, so there are lots of firmware images floating around. I highly recommend just even running strings on them, its quite enlightening. Some examples from a Nexus 4 radio:

        Failed do spoof USB cable disconnection
        Assertion os_mutex_pool_ptr[mutex_index_in_pool].is_available == 0 failed
        hsu_al_ser_open: hsu_al_ser_base_open for port NMEA (%d) returned failure
        Conversion to UTF-16 failed! Returned %d, expected %d
        Unexpected IP family %d - assuming IPv4
        inflate 1.2.3 Copyright 1995-2005 Mark Adler
        Received ARP Request
        CxM - Received WLAN Early Grant Release

(Yes, these are format strings! And this device has all the good stuff: classic 2005 zlib, a homebrew network stack, homebrew character conversion routines, homebrew operating system, homebrew USB stack...)

<comment><USER3> The only proper way to turn off a phone is to remove the battery, in lieu of nuking it from orbit (just to be sure).

<comment><USER4> A Faraday cage should also ensure that no one can track you.

<comment><USER5> As seen on KS:

[http://offpocket.com/](http://offpocket.com/)

<comment><USER6> Seems like the price is a bit steep for a little bit of fabric.

<comment><USER7> Indeed, a metal cocktail shaker works as a faraday device as well.

</comment></comment></comment></comment></comment></comment><comment><USER8> _" Back in July 2013, The Washington Post reported that nearly a decade ago, the National Security Agency developed a new technique that allowed spooks to "find cellphones even when they were turned off."_

Since this technique pre-dates smartphones, it is unlikely to involve installing software on the phone. At best, the NSA might have found that a given model of phone didn't properly power-down its radio when the phone was powered-down. Given access to the cellular network it might be possible to ping the phone and make it disclose its position via triangulation.

Very hard to see how this could be anything other than deliberate disinformation by the NSA though.

<comment><USER9> A remote command capability was revealed in the trial of a mob boss. The FBI used a non-smartphone as a room bug. A reasonable surmise is that the phone can be commanded to periodically power on and listen for commands. Those commands could include turning on the speakerphone, jacking up the agc, and auto-answering without indicating the phone is on.

Turning on periodically so it can be tracked would be one part of what it takes to implement a room bug.

</comment></comment><comment><USER10> There's tracking a phone while it's off, and then there's realtime tracking of a phone while it's off. Two very different things, and two very different attacks.

"a new NSA technique enabled the agency to find cellphones even when they were turned off"

The current administration is very careful with choosing their words. I haven't seen WP's source, but I wonder if this is more about the phone blipping its receivers to record some local MAC addresses and scrambling codes and then uploading the data the next time the phone's powered on.

You know when the word "collect" doesn't mean what you think it does, I wouldn't bet on nailing the word "find". :-)

</comment><comment><USER11> Personally I've never understood why with every single Laptop and Mobile I've ever owned (where I could remove the battery) if I turned off the device, with full battery, eventually the battery dies...

Yet if I power off the same device, take the battery out and leave it for the same amount of time, then put it back and power it on, its a full battery...

On the same note, my iPhone 4, 15" Mac Book Pro w/ Retina, and Lumia 925, all when turned off completely, eventually the batteries die...

Just slower than if they were turned on...

<comment><USER12> Just guessing, but a battery inside a computer/phone, even if turned off is probably still powering at the very least a few wires. Even if they don't go anywhere (i.e. they are not connected anywhere else) they probably are slowly eating away the battery. On the other hand, I don't know how modern computers/phones turn off, but I doubt they involve a full circuit cut (i.e. there is no big red button making a click sound and closing all wires coming out of the battery)

<comment><USER13> From what i found with my phone, I would like to confirm that even if phone is turned off, power is still supplied to few wires. When i remove battery from my phone and then put it back and turn it on, it would ask me to configure date settings. But if I just turn the phone off and then turn it on, it doesn't prompt me to configure date settings. so I can tell that at least the clock on my phone uses battery even when the phone is turned off.

</comment></comment></comment><comment><USER14> If I were tasked to implement this, I would arrange for the phone to appear to be powered down when in fact is is not, and for malware to do this when shutting down from inside the normal OS. If you "powered off" from software (by shutting down from a menu or holding the power down until the screen goes blank, but not the many seconds it takes to trigger a more hardware-level hard power off), then I would make the screen go blank and make all other inputs unresponsive, except for the normal power on input.

For bonus points, I would arrange for the baseband to transmit only very minimally as necessary, so it isn't noisily detectable from RF pickups such as nearby speakers.

The technical details would get simplified, and management would hear that I can track a "powered-down" phone.

<comment><USER15> That's exactly what FinFisher police spyware does: pretends to shut off your phone but the mic is still recording

</comment></comment><comment><USER16> Thanks to Snowden we know the NSA backdoor crypto code and have leverage over the telcos to get the help they need. We also know that solutions are plausibly deniable as far as the big name companies that we know about. So the same thng could go on here.

If I had to put this in place I would get something that worked even if there were no cellphone masts in the area. Get the radio to listen to something entirely different, broadcast from some box that could be put in a car or in one of those electronic listening planes the military have. Have it work at the radio level on the phone so the cpu does not need to be used. The reply could be an entirely different identifier to the IMEA or SIM identifier with it being a simple database 'select' to get these codes.

</comment><comment><USER17> If there's any sort of oscillator still running in the phone, or any other circuit switching with a predictable pattern of rising/falling edges, you'd think it might be possible to pick up EM radiation with a sensitive enough receiver/antenna.

A bit more "out there", maybe it is possible to pick up a powered down antenna? Think that an antenna is a (typically passive) conductor, designed to resonate at a particular frequency. If the antenna is irradiated with that frequency, wouldn't the antenna couple to the field and disturb it is some way? If those disturbances can be measured, then the antenna (and consequently the phone) can be detected.

<comment><USER18> Sounds like you're describing The Thing[1]. It might work -- but then how could you tell cellphones apart?

[1]: [https://en.wikipedia.org/wiki/Thing_%28listening_device%29](https://en.wikipedia.org/wiki/Thing_%28listening_device%29)

</comment></comment><comment><USER19> My Sony Ericsson K310i used to discharge in a week when turned off with a SIM- card present and only lose 5% in a month when turned off with SIM-card removed.

At the time I thought it was due to poor power management, but now it really makes me wonder.

</comment><comment><USER20> I wonder how a hypothetical 'bugged' phone would do the data transmissions? I used to have a phone on a contract that had excessively high data charges. As a result, I only used it for voice / SMS. If a bug had been planted on it, and it used normal IP networking, it would be obvious when I got my bill.

I wonder if there are ways that a pwned phone could transmit to an attacker without hitting the billing system? Non-billed SMS? Or are there other techniques on GSM? (e.g. network operator updates get pushed to phones and they aren't billed; there must be some other low-level two-way messaging capabilities)

</comment><comment><USER21> The tracking might refer to listening to conversations that takes place near a shut down phone. The technique might be similar to the one used in laser microphones ([http://en.wikipedia.org/wiki/Laser_microphone](http://en.wikipedia.org/wiki/Laser_microphone)), but instead of measuring vibrations, to measure the electro magnetic field variance of phone's microphone.

</comment><comment><USER22> Are NFC enabled 'phones both NFC devices and NFC readers or just NFC readers?

If they start adding RFID tags to 'phones, the only safe way to not be tracked will be not to carry the 'phone.

<comment><USER23> In some devices the NFC transceiver is part of the battery, assuming it has it's own CPU and firmware it could be easily exploited to be trackable from relatively large distances.

Other "near field" devices have, such as payment cards and passpoets, have been successfulyy communicated with or exploited using directional antennas from further away than you might think.

</comment></comment><comment><USER24> Why is it that when flying, we are sometimes requested to put phones into flight mode before switching them off?

<comment><USER25> At a guess it might be so that if you turn it back on during flight it's already in airplane mode?

<comment><USER24> Good thinking.

</comment></comment></comment><comment><USER26> one of the ways that coiuld be used to track feature phoned while they were switched on however was sending specially crafted messages sent to the phone that would force a location update which you can use to grab the cell id, and basically determine movement once you have a few of them.

</comment><comment><USER27> As long as we have non free software running on their phones, it would be hard to believe the claim as is.

<comment><USER28> Why?

It's quite possible to check if a phone is transmitting, without even opening it.

Additionally, it is usually possible to see what parts of a device are consuming power (or at least have current).

<comment><USER27> Though, it might not be transmitting all the time when off. Imagine a schedule when a switched off phone transmits once a day, at a random time. Although it is not impossible to record such transmissions, it isn't quite simple.

</comment></comment></comment><comment><USER29> Never go outside without a large hat. Switch vehicles at underpasses or car parks. Don't forget the milk.</comment>


What a non-neutral 'Net looks like, UK-style - <AUTHOR> http://arstechnica.com/tech-policy/news/2009/06/what-a-non-neutral-net-looks-like.ars <comment><USER2> Some truth here but mostly FUD. This is not really non-net neutrality in the typical sense.

 _Throttling of "heavy users," even those on unlimited monthly data plans? Check._

This is not inherently anti net neutrality. DSL providers have a certain amount of bandwidth and this has to be distributed out via users. There is no such thing as an "unlimited monthly data plan." They can _call_ it that, but if everyone attempted to download a terabyte of data at once, everything would crawl to a halt. That's not a threat to net neutrality but basic bandwidth math!

<comment><USER3> True, although in actuality if everyone just went online and tried to download a terabyte of data. TCP back off would be the reason that things came to a crawl, which is a user side thing.

</comment></comment><comment><USER4> There's an easy market solution for people who want traffic that this ISP is throttling - change your damn ISP. Notice that this solution involves no posturing politicians, no complex legislation, no costly enforcement, no restrictions on business. It's simple and effective. bt will quickly get the reputation as being the ISP for n00bs and will see themselves lose business, others will have an example in the flesh to disuade them from trying it on.

<comment><USER5> I'm not sure what its like in the UK, but where I'm at in the US there are two ISPs that service my area, and both offer very similar plans. "Change your damn ISP" only works if there is another ISP that offers something different in your area.

<comment><USER6> Where I live in London I have access to close to 100 ISPs that will take my cash. Where my parents live in the sticks there are two. One is BT who have to provide them a service but it's severely restricted (in bandwidth terms) and the other is Demon who are respectable enough.

It's not the open market place the governments market models predict.

<comment><USER7> In the sticks everything goes through the expensive BT Ipstream product anyway.

</comment></comment></comment></comment><comment><USER3> Is the BBC considering using end to end encryption for its iPlayer service? It seems to me that they could really stop a lot of this throttling in its tracks by preventing deep packet inspection.</comment>


Continuous urbanization in Japan - <AUTHOR> http://devonzuegel.com/post/continuous-urbanization-in-japan <comment><USER3> I believe a number of factors tell the tale:

* As noted, the zoning laws and general policies surrounding real estate have made it so that residential development keeps with demand. In every developed area, the buildings consistently climb to 8 stories high. I can't speak for Europe but this is something the U.S. simply cannot do within its existing policy regime, now that the automobile suburbs are built out. It's stuck with an escalating price bubble for the foreseeable future.

* The shinkansen itself cannot be underestimated either. Other cities tend to be like "suburbs of Tokyo" in the Japanese model, taking pressure off the city to go supertall.

* Although you can find neighborhoods with large office parks and highways(e.g. Akasaka), the zoning laws permit using your home to do business, which reduces average commutes and requirements for dedicated workspace, and goes a long way to explain the high degree of similarity between any two random urban streets in the country.

* The rail lines are also real estate companies, which creates a virtuous cycle around every transit stop to pile on amenities and maximize the attractiveness of each neighborhood. You have to go quite far out to find "sleepy" train stations.

* The keiretsu business model that built contemporary Japan is likewise intertwined with the transit and development pattern. The economics of the conglomerate dictate that business units should complement and support each other which allows for a certain degree of robustness, further encouraging neighborhoods that have many similar patterns. Even in Sapporo, with it's 19th-century gridded blocks designed by an American, the overall development is distinctly Japanese. U.S. style patterns tend to favor superlative extremes of optimization - the biggest, the fastest, the most profitable - and not average case outcomes. Manhattan may be exceptional simply because we've assumed every city should be exceptional.

<comment><USER4> You're missing two large factors:

* Tokyo is in a very active earthquake zone. Building earthquake-proof supertalls is even more expensive than usual, and not practical in the large portions of the city built on reclaimed land.

* Sunshine laws make it very difficult to build tall buildings even when zoning regulations otherwise allow it. This is why you often see weirdly shaped (triangular etc) buildings in improbable places.

Shinkansen commuting is also a fairly marginal phenomenon, the cost is prohibitive and I recall reading that the number of active Shinkansen commuter passes is measured in the thousands (for a megalopolis of 30 million plus).

<comment><USER5> TBF I find their sunshine laws make sense. Tokyo to me feels way more liveable than any Western megacity I know.

<comment><USER6> I agree. Tokyo is my favourite city, for many reasons. Livability and comfort being two of the top. I've already lived mid-term in Tokyo, and I see myself returning in the future.

<comment><USER5> right. it looks a bit ugly at first if you're not into it, but living there after a while it all really makes sense. great people too generally - <USER2> and welcoming like in a big village, yet leaving you alone when you need it like in a big city. perfect for my tastes.

</comment></comment></comment></comment></comment><comment><USER7> It's pretty depressing that Tokyo is kinda the high point of urbanism and even it has so many issues. The bar is set really low for nice cities

Tokyo has many city centers, but I think it fails pretty badly when it comes to office space distribution. There's still concentration of large companies in a couple central areas, meaning everyone is going into a couple spots every morning, and leaving every evening.

There's a big status thing involved where companies want to be in the good neighborhoods so their address tells people that the company's successful. It means that huge swaths of the city are empty during the day. This compounds into lack of as good public services in certain areas, basically creating dreadful suburb communities where there's nothing but housing and convenience stores for miles, and local businesses can't survive on the paltry foot traffic

Meanwhile everyone and their dog is in the same 5 neighborhoods

Tokyo is lucky in that there's permissive zoning, so theoretically this can be corrected. But there's almost no incentive to do so because there's still a lot of prime construction to be had in other parts of the city

The city is going to end up with a bunch of holes in its urban planning because theres not much glory in developing the less popular places

The dark side of the transit network being in place is that there's not as much incentive to ensure local growth. So everything is a 40 minute train ride away.

(To a greater extent the Shinkansen did this for neighboring cities. You can now get to Tokyo in an hour, so there is less pressure to have stuff in your town. People end up getting used to it and moving to Tokyo)

</comment><comment><USER8> The development permissiveness in Tokyo is amazing. In the US, space under overpasses is wasted, and often becomes desolate and quite scary for pedestrians to traverse. In Tokyo, you'll see retail built under the overpass.

<comment><USER9> Interestingly, there is a mall underneath the Manhattan bridge overpass in Chinatown NYC. It's a pretty cool place:

[http://bedfordandbowery.com/2016/12/your-next-favorite- book-...](http://bedfordandbowery.com/2016/12/your-next-favorite-book-or- album-is-hiding-in-a-mall-under-the-manhattan-bridge/)

[http://gothamist.com/2014/11/18/dim_sum_dance_party_you_guys...](http://gothamist.com/2014/11/18/dim_sum_dance_party_you_guys.php)

<comment><USER8> That's neat. I can't think of anything like that in DC. These are all over the place in Tokyo.

<comment><USER9> After thinking a bit more, I also remembered Pershing Square Cafe underneath Park Avenue across from Grand Central Station. Kinda the same thing on a smaller scale.

[https://firstname.lastname@example.com,-73.9777846,3a,75y,1...](https://email@example.com,-73.9777846,3a,75y,190.77h,100.71t/data=!3m6!1e1!3m4!1sucyEbqJyUWSYt_BUtqLQSQ!2e0!7i16384!8i8192)

</comment></comment></comment></comment><comment><USER10> One factor not mentioned is that Tokyo was almost completely destroyed by bombs and especially by firestorms during World War 2. This required rebuilding much of the city after the war. Cities like New York have never been destroyed. San Francisco was destroyed by earthquake much further back (in 1906 before modern skyscrapers were common). So some of what we see here may be rethinking following destruction as well as other factors.

<comment><USER11> On the other hand, the cities in the US that were built out postwar tend to be sprawly and not very dense. San Francisco arguably got lucky to be (re-)built before detached single family homes were in vogue.

<comment><USER12> Even the US cities that have grown post war generally had their city centers long before the war, and their tallest buildings tend to be there. This is about tall building density, not population density.

</comment></comment></comment><comment><USER13> I've noticed similar in some European cities in terms of an apparent higher average density. I've always attributed it to cities that have existed longer, and that were already significantly more established by the time technology (bicycles, trains, cars) helped people commute further.

In the absence of this technology, I've always assumed people tended to stay more local, which meant that cities ended up a cluster of local communities, rather than one large central district with a large number of commuters.

That said, this seems to conflict with the example of Tokyo, which I understand was significantly rebuilt after the second world war.

<comment><USER14> There's also the fact that, not only was Tokyo rebuilt after the war, but that it is continually being rebuilt in a way that U.S. cities aren't. My understanding is that there's a strong preference in Japan for living in newly constructed residences, which means that older housing stock is being replaced at a much higher rate than elsewhere.

<comment><USER15> There is a strong preference for new houses, add in their superstition where if someone dies in a house it might as well be considered unmarketable and you get a market where you have to always keep building.

</comment></comment></comment><comment><USER16> This blogpost lays out a pretty convincing case that difference between Japanese and (say) American zoning laws are the main reason for this type of "continuous urbanization" in Japan.

[http://urbankchoze.blogspot.com/2014/04/japanese- zoning.html](http://urbankchoze.blogspot.com/2014/04/japanese-zoning.html)

<comment><USER17> It's not unique to Japan: many other Asian cities exhibit dynamic, multi-polar cities and extraordinary continuous density.

> 1- Zoning is a national law, not a municipal by-law

This is pretty much it.The fact that strong central governments can ensure good density and growth means that Asian cities are growing as needed to serve the national economic agenda. In the West, particularly Europe and the US, cities are ruled by local municipal powers who all have strong incentives to stop growth and increase prices/rents. There's really no mystery here.

<comment><USER18> Interesting policy in urbanization/national economic agenda in China.

[https://www.economist.com/china/2018/06/23/china-is- trying-t...](https://www.economist.com/china/2018/06/23/china-is-trying-to- turn-itself-into-a-country-of-19-super-regions)

</comment></comment></comment><comment><USER19> London is similar to Tokyo. After the Second World War, Blitzed-out land was bought cheaply by local councils and high-rise, high-density (but generally good quality) housing was built on those sites, hence an uneven spread of high-rise buildings throughout the city. Historically though the tallest buildings had to be church spires and the first true "skyscraper" in London was probably the Post Office Tower (now BT Tower) in Fitzrovia in the 1960s. It now stands pretty isolated as most of the tallest buildings in London are concentrated around the City and Canary Wharf, in the east and east docklands areas.

</comment><comment><USER20> My understanding was that the actual _ground_ in NYC roughly determined building height. Manhattan was originally a swampy island, so you need to find something hard underneath (bedrock) to build on if you want 50 stories.

<comment><USER21> > _you need to find something hard underneath (bedrock) to build on if you > want 50 stories_

This is actually something of a myth [1].

[1] [https://www.google.com/amp/observer.com/2012/01/uncanny- vall...](https://www.google.com/amp/observer.com/2012/01/uncanny-valley-the- real-reason-there-are-no-skyscrapers-in-the-middle-of-manhattan/amp/)

</comment></comment><comment><USER22> > I'm still at a loss as to what caused this dynamic. My best guess is that > the city developed alongside a mature, well-supported rail system that could > whisk people from one part of the city to another, so the advantage of being > in the middle of it all was lower. Manhattan's subway system could be > described in much the same way though, so this answer is not satisfying.

I think the author is correct in saying that it's due to the rail system. The difference between Tokyo and New York is that the Yamanote line runs in a circle, whereas Manhattan's subway lines don't.

The Yamanote line not only connects most of Tokyo's major districts (Shinjuku, Shibuya, Ginza, Ikebukuro, etc.), but it's also the line that connects other train lines together. It's kind of like the beltway/orbital/ring road of a typical American city, just that it's a railway not a freeway. My understanding is that after WW2 when the city was being rebuilt, it was a conscious decision not to define a central neighborhood, and instead have all the lines connect to any station on the Yamanote line instead.

Note that this is not a Japanese thing - Osaka's tall buildings are clustered around Umeda, and Nagoya's are in the area around Shin-Nagoya station. Those cities also don't have a central circle line like the one Tokyo has.

<comment><USER23> Osaka has a central circle line, Osaka Loop Line. [https://en.wikipedia.org/wiki/Osaka_Loop_Line](https://en.wikipedia.org/wiki/Osaka_Loop_Line)

The main stations along this line are probably Kyobashi and Tennoji, the latter of which has at least the tallest building in Japan.

I would mainly divide Osaka in two centers, with Umeda being the CBD and Namba/Shinsaibashi area to be the center for people going out.

<comment><USER22> Ah you're right, I don't know how I missed that! I guess it just doesn't follow the city's main neighborhoods like Tokyo's one does. There goes my theory...

</comment></comment></comment><comment><USER6> Looking out over Tokyo from one of the city centres (Tokyo is really a prefecture, and a collection of smaller cities), you can see that the other cities look quite built up, and the areas between are the smaller buildings the author is talking about.

I don't think the image provided does it justice, as it just showed a small subsection of Tokyo, and not what the city actually looks like.

</comment><comment><USER24> No mention to earthquake or typhoons?

I always thought the urbanization of Japan was mostly limited by natural conditions.

Indeed, there is no point in making tall buildings if they are more likely to fall and cost more in repairs and anti-seismic technology

<comment><USER12> San Fransisco was one of the main cities Tokyo was compared to, which faces very similar natural conditions.

</comment></comment><comment><USER25> How can the author claim the lack of density in a city such as Tokyo when it has districts like Shibuya or Akiba which you really can't go any more dense without adding stories to the existing building.

</comment><comment><USER26> Léon Krier is worth mentioning here. He's an architect associated with New Urbanism who also creates illustrations which demonstrate his points well.

For height limits you have the architectural stutter [https://www.architectural- review.com/Pictures/web/p/y/l/spee...](https://www.architectural- review.com/Pictures/web/p/y/l/speech-stutte_380.jpg)

San Francisco vs Japan seems very much like monocentric vs polycentric development [https://architecturehereandthere.files.wordpress.com/2017/01...](https://architecturehereandthere.files.wordpress.com/2017/01/screen- shot-2017-01-10-at-1-50-08-pm.png)

\-- and organic expansion through duplication [https://3.bp.blogspot.com/-5JgoOuUmeDg/WMhJTUWeNWI/AAAAAAAAr...](https://3.bp.blogspot.com/-5JgoOuUmeDg/WMhJTUWeNWI/AAAAAAAArrE/V1Fvtt37HacOF5b76ly6UndVpvPpdgdlwCLcB/s1600/Leon%2BKrier%2Bon%2BOur%2BFuture%2BCities.jpg)

In this talk here he talks about the classical vs vernacular layout styles [https://youtu.be/kFiYL8AvvnY?t=1150](https://youtu.be/kFiYL8AvvnY?t=1150)</comment>