Unicode Is Awesome - <AUTHOR> https://wisdom.engineering/awesome-unicode/ <comment><USER3> Unicode is pretty amazing.

People REALLY like to complain about unicode, but where it's complicated, it's because the _problem space_ is complicated. Which it is. People are actually complaining that they wish handling global text wasn't so complicated, like, that humans had been a lot simpler and more limited with their inventions of alphabets and how they were used in typesetting and printing and what not, and that legacy digital text encodings historically had happened differently than they did, they're not actually complaining about unicode at all, which had to deal with the hand of cards it was dealt.

That unicode worked out as nice a solution as it is to storing global text is pretty amazing, there were some really smart and competent people working on it. When you dig into the details, you will be continually amazed how nice it is. And how well-documented.

One real testament to this is how _well adopted_ Unicode is. There is no actual guarantee that just because you make a standard anyone will use it. Nobody forced anyone to move from whatever they did to Unicode. (and in fact most eg internet standards don't force Unicode and are technically agnostic as to text encoding). That it has become so universal is because it was so well- designed, it solved real problems, with a feasible migration path for developers that had a cost justified by it's benefits. (When people complain about aspects of UTF-8 required by it's multi-facetted compatibility with ascii, they are missing that this is what led to unicode actually winning).

The OP, despite the title, doesn't actually serve as a great argument/explanation for how Unicode is awesome. But I'd read some of the Unicode "annex" docs -- they are also great docs!

<comment><USER4> If we could go back in time to Unicode's beginning and start over but with all that we know today... Unicode would still look a lot like what it looks like today, except that:

      - UTF-8 would have been specified first
      - we'd not have had UCS-2, nor UTF-16
      - we'd have more than 21 bits of codespace
      - CJK unification would not have been attempted
      - we might or might not have pre-composed codepoints[0]
      - a few character-specific mistakes would have gone unmade

which is to say, again, that Unicode would mostly come out the same as it is today.

Everything to do with normalization, graphemes, and all the things that make Unicode complex and painful would still have to be there because they are necessary and not mistakes. Unicode's complexity derives from the complexity of human scripts.

[0] Going back further to create Unicode before there was a crushing need for it would be impossible -- try convincing computer scientists in the 60s to use Unicode... Or IBM in the 30s. For this reason, pre-composed codepoints would still have proven to be very useful, so we'd probably still have them if we started over, and we'd still end up with NFC/NFKC being closed to new additions, which would leave NFD as the better NF just as it is today.

<comment><USER5> > Or IBM in the 30s

Love an interesting sci-fi scenario. UTF-8 was a really neat technical trick, and a lot of the early UTF-8 technical documentation was already on IBM letterhead. I think if you showed up with the right documents at various points in history IBM would have been ecstatic to have an idea like UTF-8, at least. UTF-8 would have sidestepped a lot of mistakes with code pages and CCSIDs (IBM's attempts at 16-bit characters, encoding both code page and character), and IBM developers likely would have enjoyed that. Also, they might have been delightfully confused about how the memo was coming from inside the house by coworkers not currently on payroll.

Possibly that even extends as far back as the 1930s and formation of the company, because even then IBM aspired to be a truly International company, given the I in its own name.

I'm not sure how much of the rest of Unicode you could have convinced them of, but it's funny imagining explaining say Emoji to IBM suits at various points in history.

<comment><USER4> I.. agree. After all, even before ASCII people already used the English character set and punctuation to type out many non-English Latin characters on typewriters (did typesetters do that with movable type, ever? I dunno, but I imagine so). Just as ASCII was intended to work with overstrike for internationalization, i can imagine combining codepoints having been a thing even earlier.

OTOH, it wouldn't have been UTF-8 -- it would have been an EBCDIC-8 thing, and probably not good :)

<comment><USER5> There actually is a rarely used, but standardized because it needed to be, UTF-16 variant UTF-EBCDIC, if you needed nightmares about it. [0]

In some small decisions EBCDIC makes as much or more sense than ASCII; the decades of problems have been that ASCII and EBCDIC coexisted from basically the beginning. (IBM could have delayed the System/360 while ASCII was standardized and likely have saved decades of programmer grief.) The reasons that UTF-EBCDIC is so bad (such as that it is always a 16-bit encoding) could likely have been avoided had IBM awareness of UTF-8 ahead of time.

Maybe if IBM had something like UTF-8 as far back as the 1930s, AT&T needing backward compatibility with their teleprinters might not have been as big of a deal and ASCII might have been more IBM dominated. Or since this is a sci-fi scenario, you just impress on IBM that they need to build a telegraph compatible teleprinter model or three in addition to all their focus on punch cards, and maybe they'd get all of it interoperable themselves ahead of ASCII.

Though that starts to ask about the scenario what happens if you give UTF-8 to early Baudot code developers in the telegraph world. You might have a hard time to convince them they need more than 5-bits, but if you could accomplish that, imagine where telegraphy could have gone. Winking face emoji full stop

[0] [https://en.wikipedia.org/wiki/UTF- EBCDIC](https://en.wikipedia.org/wiki/UTF-EBCDIC)

<comment><USER3> > Maybe if IBM had something like UTF-8 as far back as the 1930s, AT&T needing > backward compatibility with their teleprinters might not have been as big of > a deal

I think this points to why the science fiction scenario really is a science fiction scenario -- I think decoding and interpreting UTF-8, using it to control, say, a teleprompter, is probably significantly enough more expensive than ASCII that it would have been a no go, too hard/expensive or entirely implausible to implement in a teleprompter using even 1960s digital technology.

<comment><USER5> Yeah, and I was thinking about it this morning the reliance of FF-FD for surrogate pairs would shred punch cards (too many holes) and probably be a big reason for IBM to dismiss it when they were hugely punch card dependent and hadn't made advances like the smaller square hole punches that could pack holes more densely and with better surrounding integrity.

"Sigh, another swiss cheese card jammed the reader from all these emojis."

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> > Unicode is simply a 16-bit code - Some people are under the misconception > that Unicode is simply a 16-bit code where each character takes 16 bits and > therefore there are 65,536 possible characters. This is not, actually, > correct. It is the single most common myth about Unicode, so if you thought > that, don't feel bad.

Verity Stob has a great column

[https://www.theregister.co.uk/2013/10/04/verity_stob_unicode...](https://www.theregister.co.uk/2013/10/04/verity_stob_unicode/)

where she says that it is wrong to call this a myth, since that was how it was originally designed. It is better characterized as being obsolete, rather than a myth.

<comment><USER7> It's a myth about the current version of Unicode.

Whether it's true about some obsolete version is hair-splitting at this point.

</comment></comment><comment><USER8> IMHO the article should mention that UTF-16 was (more or less) a hack to fix Windows and some other systems which didn't see the light and use UTF-8 from the start. UTF-16 has all the disadvantes of UTF-8 (variable length) and UTF-32 (endianess), but none of the advantages (encoding as endian-agnostic, 7-bit ASCII compatible byte stream like UTF-8, or a fixed-width encoding like UTF-32). UTF-16 should really be considered a hack to talk to (mainly) Windows APIs.

Also, obligatory link to: [https://utf8everywhere.org/](https://utf8everywhere.org/)

<comment><USER9> Windows and many other operating systems and languages (Java) got on board with Unicode back when the character set would fit in 16bits. The character set originally used was UCS-2 (not UTF-16). UTF-16 came next to extend the Unicode character set beyond 65536 code points.

UTF-8 wasn't even invented until well after all these operating systems and languages deployed Unicode.

They didn't see the light of day to use UTF-8 because they didn't have a time machine to make that possible.

<comment><USER8> I actually checked a while ago when UTF-8 was created, and it was just around the same time when Windows NT was developed with 16-bit "early" Unicode support. UTF-8 was created in September 1992 [1], and Windows NT came out mid 1993, but I guess it was too late for Windows to change to UTF-8 (and I guess the advantages of UTF-8 haven't been as clear back then).

But IMHO there's no excuse to not use UTF-8 after around 1995 ;)

[1] [https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt](https://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt)

<comment><USER10> Also, UTF-16 was only published in July 1996 (although the need for more than 16 bits was probably apparent a bit earlier). So before that, Unicode was only a 16-bit encoding, and UCS-2 was enough. UTF-8 was initially just a nice trick to keep using ASCII characters for things like directory separators (/) and single-byte NUL terminators. By 1995 its superiority certainly wasn't apparent yet.

Also, Windows internals were completely 16-bit-character based, including e.g. the NTFS disk format, so by 1992 that was already quite hard to change.

That said, it is crazy that NT didn't have full UTF-8 support, including in console windows, by about 2000.

</comment></comment></comment></comment><comment><USER11> I mean, awesome for whom? It might be awesome for end users as you can type in or copy/paste things without caring which language you're using. But for programmers, Unicode is a bloated monstrosity and a source of endless nightmare. Eventually, it's not going to be awesome for end users either because it will be plagued by a lot of (subtle) inconsistencies. Unicode looks a lot like a leaky abstraction to me (because of poor foresight), and it's getting worse each year.

<comment><USER12> If you think Unicode is a "bloated monstrosity and a source of endless nightmare," what would you remove from Unicode?

And if you're going to respond "emoji", I'll point out that removing emoji doesn't actually remove anything that makes text processing with Unicode difficult, just makes it more likely that people will assume that what works for English works for everybody.

(Side note: it is not possible to accurately represent modern English text solely with ASCII, as English does contain several words with accented characters, such as façade and résumé).

<comment><USER13> 'Remove' is too strong, since Unicode is entrenched. But there are things that should have been done differently. For instance, combining characters and operators should have been placed before the base character rather than after, so that (a) it would be possible to know when you've reached the end of a character^W glyph^W grapheme cluster without reading ahead, and (b) dead keys would be identical to the corresponding characters.

> _façade and résumé_

ASCII (1967) allowed for them: c BS , _or_ , BS c ↦ ç and e BS ' _or_ ' BS e ↦ é. Encoding ç as 63 CC A7 is not manifestly better than encoding it as 63 08 2C.

<comment><USER12> > ASCII (1967) allowed for them: c BS , or , BS c ↦ ç and e BS ' or ' BS e ↦ > é. Encoding ç as 63 CC A7 is not manifestly better than encoding it as 63 08 > 2C.

Doesn't work for ñ, since the ASCII ~ is often typeset in the middle of the box instead of in a position to appear above an 'n' character. " is a pretty poor substitute for ◌̈ though, especially when you're trying to write ï as in naïve. And then there's the æ of archæology, which doesn't work with overwriting.

I'll also point out that ç is U+00E7 in Unicode and C3 A7 in UTF-8, not 63 CC A6, since it's a precomposed character (and NFC form is usually understood to be the preferred way to normalize Unicode unless there's a reason to do something else).

<comment><USER13> Tilde exists in ASCII _because_ of its use as an accent. (In 1967 the non- diacritic interpretation was an overline.) The use in programming languages, and lowering to fit other mathematical operators, came later.

There was never any requirement that 'n BS ~' have the same appearance as 'n' overprinted with '~', although terminals capable of making the distinction didn't appear until the 70s.

Precomposed characters aren't relevant to illustrating composition mechanisms.

</comment></comment></comment></comment></comment><comment><USER14> I swear there should be some rule or law about how Unicode articles will inevitably muddle code units/points / grapheme clusters / bytes together.

> _String length is typically determined by counting codepoints._

> _This means that surrogate pairs would count as two characters._

If you were counting code points, a surrogate pair would be 1. If it's two, you're counting code units.

> _Combining multiple diacritics may be stacked over the same character. a + ̈ > == ̈a, increasing length, while only producing a single character._

Not if you're counting code points _or_ code units, which would both produce an answer of "2", and that's a great example of why you shouldn't count with either.

The dark blue on black in tables is next to invisible. And then to put _that_ on white on the alternate rows is just eyeball murder.

> _Since there are over 1.1 million UTF-8 glphys_ (sic)

UTF-8 glyphs _twitch_ ; aside from that, I'm really curious how they got that number. In some ways, a font has it easy; my understanding is that modern font formats can do one glyph for acute accent, one glyph for all the vowels/letters, and then compose the glyphs into arrangements for having them combined. (IDK if those are also "glyphs" to the font or not.) But it's less drawing, at least. OTOH, some characters have >1 appearance/"image", AIUI.

<comment><USER15> Also a law that the author is thinking of one and only one programming language.

> String length is typically determined by counting codepoints.

That depends _entirely_ on what "strings" you are talking about.

In C/Go/Rust/Ruby, char*/string/std::string::String/String is bytes.

In Java/JavaScript, java.lang.String/String is UTF-16 code units.

In Python 3, str is code points.

In Swift, String is extended grapheme clusters.

In Haskell, there are various different "string" types in common use.

And in C++, std::basic_string is a generic container for whatever element type you want. (std::string specialization being for bytes.)

EDIT: Clarified that I don't disagree with parent comment; merely pointing out additional less-than-precise language.

<comment><USER16> I'm in love with Swift's approach, where the default representation is a well defined thing that both users and developers think of as "characters", but all the other representations are trivially accessible.

<comment><USER17> I disagree. Grapheme clusters are locale-dependent, much like string collation is locale-dependent. What Unicode gives you by default, the (extended) grapheme cluster, is as useful as the DUCET (Default Unicode Collation Element Table); while you can live with them, you would be unsatisfied. In fact there are tons of Unicode bugs that can't be corrected due to the compatibility reason, and can only be fixed via tailored locale-dependent schemes.

I would like to avoid locales in the language core. It would be great to have locale stuffs in the standard library, but without locale information you can't treat strings as (human) texts.

<comment><USER18> Can you give examples of locale-dependent things, or issues with extended grapheme clusters?

<comment><USER17> Hangul normalization and collation is broken in Unicode, albeit for slightly different reasons. The Unicode Collation Algorithm explictly devotes two sections related to Hangul; the first section, for "trailing weights" [1], is recommended for the detailed explanation.

The Unicode Text Segmentation standard [2] explicitly mentions that Indic _aksaras_ [3] require the tailoring to grapheme clusters. Depending on the view, you can also consider orthographic digraphs as examples (Dutch "ij" is sometimes considered a single character for example).

[1] [https://www.unicode.org/reports/tr10/#Trailing_Weights](https://www.unicode.org/reports/tr10/#Trailing_Weights)

[2] [https://unicode.org/reports/tr29/](https://unicode.org/reports/tr29/)

[3] [https://en.wikipedia.org/wiki/Aksara#Grammatical_tradition](https://en.wikipedia.org/wiki/Aksara#Grammatical_tradition)

</comment></comment></comment></comment></comment></comment><comment><USER19> If you'd like to explore Unicode characters, you can use the Unicode Character finder, a web app I built some years ago: [https://www.mclean.net.nz/ucf/](https://www.mclean.net.nz/ucf/)

The app allows you to paste in a character to find out more about it, or to search the database of character descriptions to find what you're after.

You can link to a specific character to share with your friends and family: [https://www.mclean.net.nz/ucf/?c=U+130BA](https://www.mclean.net.nz/ucf/?c=U+130BA)

<comment><AUTHOR> For exploration, additionally I'd recommend [http://shapecatcher.com/](http://shapecatcher.com/) It allows you to draw the shape you are looking for, and with some form of ML, sorts by similarity. It has come in handy a few times for finding the characters I'm unable to describe.

</comment></comment><comment><USER20> Page doesn't render without JS enabled. Enabling JS causes questionable CSP requests.

<comment><AUTHOR> Just a quick and scrappy "Ghost" blog running on a $5 Digital Ocean droplet with the usual analytics.

</comment></comment><comment><USER21> > data to be transmitted in a byte, word or double word oriented format (i.e. > in 8, 16 or 32-bits per code unit)

I don't think I've heard "word" mean "16 bits" since the 1980's ... and apparently neither has Wikipedia: [https://en.wikipedia.org/wiki/Word_(computer_architecture)#T...](https://en.wikipedia.org/wiki/Word_\(computer_architecture\)#Table_of_word_sizes)

<comment><USER22> In Windows world DWORD is 32 bits large: [https://docs.microsoft.com/en- us/openspecs/windows_protocols...](https://docs.microsoft.com/en- us/openspecs/windows_protocols/ms-dtyp/262627d8-3418-4627-9218-4ffe110850b2)

<comment><USER23> And WORD 16 bits: [https://docs.microsoft.com/en- us/openspecs/windows_protocols...](https://docs.microsoft.com/en- us/openspecs/windows_protocols/ms-dtyp/f8573df3-a44a-4a50-b070-ac4c3aa78e3c)

(QWORD 64 bits)

</comment></comment></comment><comment><USER24> Unicode is great, emoji are a (technically impressive) monstrosity.

<comment><USER5> Emoji are an almost critical Unicode democratizing need. They aren't doing anything that other languages encoded with Unicode don't already do (and haven't already done since the beginning of Unicode). This article itself points out several key existing relatives, such as how Arabic, one of the most common and important written languages in the world, or the very important CJK family of written languages, used ZWJ and ZWNJ well before Emoji made it "cool" to other parts of the world, most especially the English-writing contingent that has long thought of Unicode as simply "ASCII plus a bunch of other stuff I might never use". Suddenly a lot of English documents have embedded emoji that deeply matters to the writers, and there are fewer excuses to treat Unicode as "ASCII+" and more cases where doing so is not only wrong (broken surrogate pairs, incorrect codepoint analysis for ZWJ, etc), but very visibly wrong in a way that users care and complain about it.

</comment></comment><comment><USER25> Unicode has two really great features.

* It names and defines things and sets standard. This seems trivial but is incredibly useful.

* Unicode encodings, mainly UTF-8 are good storage format for text (as a data structure for editing text, not so much if you want to be universal).

Unicode has one really horrible failing.

The 'user-perceived character' (Unicode terminology) is arguably the most important unit in text. Unicode approximates user-perceived characters using set of general rules to define grapheme clusters. A Grapheme cluster is a sequence of adjacent code points that should be treated as a unit by applications. Unfortunately the ruleset and definition is inadequate. Sometimes you need two grapheme clusters to define one unit.

If you get UTF-8 encoded and normalized string from somewhere from some unspecified time and era, don't know what application wrote it, using what version of UNICODE standard and what was the locale, you may lose some information.

Unicode should have added explicit encoding for user-perceived character boundaries (either fixed grapheme cluster eoncoding or completely different encoding). Let the writing software define it explicitly. It would have been future-proof (new software in the future can understand old strings) and past- proof (ancient software can understand and edit strings written in the future).

</comment><comment><USER26> Unicode definitely has flaws but that doesn't mean we should throw the baby out with the bathwater and go back to "ASCII and other character sets." There's a reason we moved on from that world. However, I bet we will see another encoding coming up eventually (within 30 years) which solves the problems Unicode currently has and introduces a new set of problems as well. I saw this comment [0] about how that encoding should get started.

> Greek, for example, has a lot of special-casing in Unicode. Korean is > devilishly hard to render correctly the way Unicode handles it. And once you > get into the right-to-left scripts, scripts that sort-of-sometimes omit > vowels, or Devanagari (the script used to write a bunch of widely-spoken > languages in India), you start needing very different capabilities than > what's involved in Western European writing. _The better approach probably > would have been to start with those, and work back to the European scripts_

[0] [https://www.reddit.com/r/programming/comments/b09c0j/when_zo...](https://www.reddit.com/r/programming/comments/b09c0j/when_zo%C3%AB_zo%C3%AB_or_why_you_need_to_normalize_unicode/eiekg1x/)

Funnily enough, URLs still can't do actual Unicode.

<comment><USER27> Unicode URL has serious security problems.

The canonical example is google.com vs gооgle.com.

<comment><USER28> That was solved years ago by IDN/Punycode (implemented by any browser worth their salt).

</comment></comment></comment><comment><USER17> Years ago I've posted support material [1] for Hangul filler mentioned in the article, reproduced below:

\---

U+3164 HANGUL FILLER is one of the stupidest choices made by character sets. Hangul is noted for its algorithmic construction and Hangul charsets should ideally be following that. Unfortunately, the predominant method for multibyte encoding was ISO 2022 and EUC and both required a rather small repetoire of 94 × 94 = 8,836 characters [0] which are much less than required 19 × 21 × 28 = 11,172 modern syllables.

The initial KS X 1001 charset, therefore, only contained 2,350 frequent syllables (plus 4,888 Chinese characters with some duplicates, themselves becoming another Unicode headache). Notwithstanding the fact that remaining syllables are not supported, this resulted in a significant complexity burden for _every_ Hangul-supporting software and there were confusion and contention between KS X 1001 and less interoperable "compositional" (johab) encodings before Unicode. The standardization committee has later acknowledged the charset's shortcoming, but only by adding four-letter (thus eight-byte) ad-hoc combinations for all remaining syllables! The Hangul filler is a designator for such combinations, e.g. `<fliler>ㄱㅏ<filler>` denotes `가` and `<filler>ㅂㅞㄺ` denotes `뷁` (not in KS X 1001 per se).

Hangul filler was too late in the scene that it had virtually no support from software industry. Sadly, the filler was there and Unicode had to accept it; technically it can be used to designate a letter (even though Unicode does not support the combinations) so the filler itself should be considered as a letter as well. What, the, hell.

[0] It is technically possible to use 94 × 94 × 94 = 830,584 characters with three-byte encoding, but as far as I know there is no known example of such charset designed (thus no real support too).

\---

I should also mention that early Mozilla (and thus Firefox) had once supported ad-hoc combinations for KS X 1001, got interoperability problems and dropped the support later. Nowadays we treat KS X 1001 as an alias of Windows code page 949 for the sake of compatibility [2].

[1] [https://github.com/Wisdom/Awesome- Unicode/issues/4](https://github.com/Wisdom/Awesome-Unicode/issues/4)

[2] [https://encoding.spec.whatwg.org/#index-euc- kr](https://encoding.spec.whatwg.org/#index-euc-kr)

</comment><comment><USER29> Unicode is an inspirational standard. We started with so many different character encodings and wound up pretty universally using Unicode. I wouldn't be surprised to see browsers start to drop support for other encodings - <USER2> even uses them at this point?

Are there any scenarios where you _wouldn 't_ use Unicode, other than an every-byte-matters embedded system?

<comment><USER30> There's a not-insignificant number of Japanese websites that can only correctly display using EUC-JP or ShiftJIS.

It seems very Latin/ASCII centric to push for disabling non-UTF-8 encodings, especially since the only reason UTF-8 works so well on ASCII websites is due to its backwards compatibility.

If it were the reverse, and UTF-8 were backwards compatible with EUC-JP/CJK, but not ASCII, I doubt you'd be pushing for eschewing other formats since it would break so many english websites.

<comment><USER12> There is no character in EUC-JP or Shift-JIS that is not in Unicode--the explicit goal of Unicode in its original formulation was to be able to losslessly round-trip any other charset through Unicode, and the initial version of Unicode incorporated the source kanji lists for the EUC-JP/Shift- JIS charsets in their entirety.

<comment><USER30> That's true, but you misunderstood what I meant.

The parent comment seemed to be implying that we should drop support for non- utf8 charsets.

To me, that rings like saying a website with 'charset=EUC-JP' (such as [http://www.os2.jp/](http://www.os2.jp/)) should be broken, as in browsers should error out or display a large quantity of black boxes due to it using a non-utf-8 encoding.

I'm claiming the only reason the author thinks that's really viable is because in our western-centric world, we see mostly ascii and utf8. Things that, if you flip to only utf-8, both still look fine.

CJK websites, on the other hand, that are using the equivalent of ASCII will have to be manually upgraded to display correctly if browsers drop their support.

Sure, all their characters can be represented in utf-8, but there's large swathes of websites that will never be updated to a new charset, and it's only a western-centric view that can so blithely suggest breaking them all.

<comment><USER12> Windows-1252/ISO-8859-1 (the two charsets are so commonly conflated that it's often best to treat them as one) was the dominant [non-ASCII] charset of the web until around 2007 or 2008, and their prevalence more recently is only about 5%.

A collection of Usenet messages gathered in 2014 (see [http://quetzalcoatal.blogspot.com/2014/03/understanding- emai...](http://quetzalcoatal.blogspot.com/2014/03/understanding-email- charsets.html) for full details) showed that out of 1,000,000 messages, about 530,000 were actually ASCII; 270,000 were ISO-8859-1 or Windows-1252; and only 75,000 were UTF-8. More modern numbers would probably show higher UTF-8 counts, although Usenet is notoriously conservative in terms of technology.

What I'm trying to elucidate here is that the rise of UTF-8 isn't because most text is ASCII, but because there's been a rather more concerted effort to default content generation to UTF-8 and treat other charsets only as legacy inputs. Well, with the exception of the Japanese, who tend to be strongly averse to UTF-8. (I've been told that Japanese email users would rather have their text get silently mangled than silently converted to UTF-8 because you're quoting an email with a smart quote [not present in any of the 3 Japanese charsets], whereas every other locale was happy changing the default charset for writing to UTF-8).

</comment></comment></comment></comment></comment><comment><USER31> What's the code point for uppercase superscript Z?

<comment><USER13> There isn't one. Unicode considers superscripting a matter of presentation, which Unicode doesn't cover, except when it does.

<comment><USER32> More particularly: Presentation variant is not a justification for inclusion in Unicode BUT prior encoding in another character set is.

Unicode sets a high priority on roundtripping. The idea is that if you take some data in any one character set X and convert it to Unicode, you should preserve all the meaning by doing this, such that you could losslessly convert it back to encoding X.

It's like the wordprocessor problem where users say they only want 10% of the features of a popular wordprocessor but it turns out each user wants a different 10% and so the only way to deliver what they all want is to deliver 100% of the features. Likewise, Unicode has all the weird features of every legacy character set which was embraced BUT it doesn't arbitrarily add new weird features, although you could argue that some of the work done for Unicode has that effect e.g. the way flags work or the Fitzpatrick modifiers.

If Unicode had insisted upon never encoding anything that might be a presentation feature, it'd be a long forgotten academic project that never went anywhere and we'd all be using some (probably Microsoft designed) 16-bit ASCII superset today.

<comment><USER31> Is there a realitvely easy way to find the character set that was included for uppercase superscript W? (ᵂ)

<comment><USER32> In the case of U+1D42 Modifier Letter Capital W I was wrong about the cause, it was in fact specifically added on the rationale that for this purpose (phonetics) the presentation was semantic in nature, and so the plain text (thus Unicode) needed to preserve these symbols which could otherwise be handled by a presentation layer.

U+1D42 Modifier Letter Capital W was added in Unicode 4.0 as part of the Phonetic Extensions and Wikipedia provides a long list of Unicode committee paperwork regarding this: [https://en.wikipedia.org/wiki/Phonetic_Extensions](https://en.wikipedia.org/wiki/Phonetic_Extensions)

You can see that initially it would have been numbered differently and then over the course of several drafts the proposal evolved until it was assigned U+1D42

<comment><USER31> Does that mean there is hope to complete the [a-Z] sub/super set?

</comment></comment></comment></comment></comment></comment><comment><USER33> [https://youtu.be/dMnPM6z6z40](https://youtu.be/dMnPM6z6z40)

</comment><comment><AUTHOR> The Emoji emoji modifiers are pretty cool. \- Skin color modifiers \- Character combiners: \- man [ZWJ] woman [ZWJ] boy [ZWJ] girl === family of 4

<comment><USER34> "How to shrink a family using MySQL" or: "Unicode Emojis, Code Points, and Grepheme Clusters"

[https://twitter.com/johannescode/status/1183716981612208128?...](https://twitter.com/johannescode/status/1183716981612208128?s=19)

</comment></comment><comment><AUTHOR> Just posted another Unicode article to HN. "Hacking GitHub with Unicode"

[https://news.ycombinator.com/item?id=21693550](https://news.ycombinator.com/item?id=21693550)

</comment><comment><AUTHOR> Unicode reverse character:

'hello \u{202e} world'; 'hello dlrow' // Visual equivalent</comment>


Ask HN: Walking Desk? - <AUTHOR> I have a standing desk and I'm considering buying an under-desk treadmill to "convert" it to a walking desk.<p>I am curious to know if anyone here uses a walking desk, and how it affects their productivity.  Is it something you find yourself using often? Are you able to remain productive while walking? <comment><USER2> I've used a treadmill desk (LifeSpan) for a few years off-and-on and I love it. Personally, I do not like standing at a tall desk. The feeling of "just standing around" never quite goes away. I slouch, fidget, shift my weight from one foot to the other, and even look around as though I'm stuck waiting in line for something. For some reason the act of walking (very, very slowly mind you) doesn't bother me at all, and I feel engaged with my work the entire time. I have not done any testing to gauge my productivity compared with sitting down in a chair for hours. Even if I learned there was no performance improvement while walking, I still prefer to walk than sit. Depending on your priorities, the most obvious difference will be aesthetic, it does wonders for your...rear view.

A more portable, less expensive option worth looking into is a "balance board," and unstable platform the side of a skateboard that keeps you physically engaged by tilting in a surfing-like stance.

</comment><comment><USER3> I'm pretty sure "No such thing as a fish" mentioned that working on a treadmill doesn't do you any good.I think they said the research showed that only one thing was improved (energy levels I believe) but they users were failing at everything else like focus and memory.

<comment><USER4> Can confirm, just listened to this episode. Treadmill desk users were less productive.

</comment></comment><comment><USER5> Doesnt Linus Torvalds use one? Im sure Ive seen a few photos of him with one?

<comment><AUTHOR> Not sure about Linus... I know Wolfram does/did ....</comment></comment>


Eve: How I miraculously escaped a gate camp in a Retriever and earned 100M ISK - <AUTHOR> http://ghost.pilgrimbreak.com/how-i-miraculously-escaped-a-gate-camp-in-a-retriever-and-earned-100m-isk/

<comment><USER2> Good story. I really like EVE and was ready to convert to a paying customer after my 15 day trial, but two things kept me from doing it.

First was cost, retail is 20$ a month, with a discount you can get it down to 10$ a month for the first year if you prepay, but after that I think the biggest discount is 15$. Don't quote me I am going off of memory.

Personally I am willing to pay 10$ a month. Prepaying is fine. I am not really cost sensitive and could be a whale (heck I am for Star Citizen).

The last issue was skill training. It's a bullshit mechanic to force you to subscribe for years on end because it is time based. You can choose which skills are trained, but there is no gameplay element you can use to progress beyond a lack of skill training. You can't even pay.

I honestly would have converted were it not for a mechanic so obviously designed to keep you pumping in a monthly fee by preventing you from "finishing" the game at your own pace. I like to play games, and then be done with them and move on.

I know my decision making process is different then most. I play zero games on my phone and tablet. I feel like this is similar to the IAP model where you have to fork over the megabucks to get anywhere.

<comment><AUTHOR> I understand your concern about the price. I played the beta 11 years ago and did not subscribe. Yet, I followed the development of the game as a non player, always wanting to come back to the game to fully understand it. Of course the many stories I heard about the game helped keeping me motivated: [https://truestories.eveonline.com/truestories/index.html](https://truestories.eveonline.com/truestories/index.html)

As for the training process, I tried to adapt my type of gaming with what skills I had at the moment from the beginning. I spent the first 2 months figuring out how to shape a balanced pilot: not buying crazy stuff you can loose (or you can't even equip because you don't fully understand the requirements mechanics), train skills that seem to matter, not focus on a certain game mode etc. And it has been rewarding because at no point so far have I felt enslaved by the skills system.

But maybe that is because I am a fanboy and lack objectivity ;)</comment></comment>


Beautiful men, women are more likely to be smarter with high IQs, too: study - <AUTHOR> http://www.nydailynews.com/lifestyle/2011/01/16/2011-01-16_beautiful_men_women_have_high_iqs_too_study.html <comment><USER2> Actual article is at <http://personal.lse.ac.uk/KANAZAWA/pdfs/I2011.pdf>.

They used two populations with different measurements of attractiveness and of intelligence, and analysed them separately.

In the first population they got results that suggest on the order of 11-14 IQ points per standard deviation in attractiveness.

In the second, 2 IQ points per standard deviation.

The first population was of children in the UK. Their measure of "attractiveness": the children were assessed at ages 7 and 11, by their teachers, as one of "attractive", "unattractive", "underfed or undernourished", "abnormal feature", and "scruffy or slovenly & dirty". I kid you not. You may, if your own intelligence happens to be reasonable, be able to think of one or two problems with assessing "attractiveness" in this way when you're looking for correlations with intelligence.

(One more detail: attractiveness was treated as a binary variable in Kanazawa's study, with a child being reckoned "attractive" if put into the "attractive" category both at age 7 and at age 11. More than 60% of children were reckoned "attractive" by this criterion. Draw your own conclusions.)

The second population was of US high school pupils. They were reassessed about 7-8 years later, at age 18-28. (How were any of them 28 at that point? I don't know.)

The NY Daily News story says:

> The study indicated attractive men have IQs that are 13.6 points above the > average, while beautiful women are 11.4 points higher than average.

which is simply rubbish: those figures are from the first population only (and the second gives much, much weaker results), they concern children of age 7 or 11 and not "men" or "women", and they're really measuring all kinds of things other than attractiveness.</comment>


The Tiger Oil Memos - <AUTHOR> http://www.lettersofnote.com/2010/08/tiger-oil-memos.html <comment><USER2> Or how to fail as a boss and a human being.</comment>


Farewell to Tinderbox, the world's 1st? 2nd? Continuous Integration server - <AUTHOR> http://oduinn.com/blog/2014/06/04/farewell-to-tinderbox/

<comment><AUTHOR> The history lesson was nice, but I actually thought this was more interesting as a description of the challenges that Mozilla faced with their original CI server and the replacement that they designed.</comment>


Defining Success For a Bootstrapper - <AUTHOR> http://www.skmurphy.com/blog/2007/11/23/success-for-a-boostrapper/

<comment><AUTHOR> I guess I wrote this to counter Silicon Valley's "social media" culture, that requires success to be financial and buzzworthy.

"The work of an unknown good man is like a vein of water flowing hidden in the underground, secretly making the ground greener." Thomas Carlyle

"The mature man lives quietly, does good privately, assumes personal responsibility for his actions, treats others with friendliness and courtesy, finds mischief boring and keeps out of it. Without this hidden conspiracy of good will, society would not endure an hour." Kenneth Rexroth</comment>


Mac Apps With Beautiful Interfaces - <AUTHOR> http://designshack.co.uk/articles/software/10-fresh-mac-apps-with-beautiful-interfaces-and-what-we-can-learn-from-them

<comment><USER2> 1\. wunderlist. How is this beautiful? They crammed a wood-styled UI into OS X's standard window chrome. The toggle switches at the bottom are not the OS- provided ones and look like they're intended to match the iPhone app more than the OS they're running on. Things is a far better example here.

2\. Reeder. See [http://danielkennett.org/blog/2010/12/analysing-a-touch- to-d...](http://danielkennett.org/blog/2010/12/analysing-a-touch-to-desktop- ui-port-using-fitts-law-reeder/)

3\. Sparrow. These are not OS X toolbar icons -- they're UIKit icons! They look as out-of-place on OS X as OS X ones would on an iOS device.

5\. DaisyDisk. The UI here is very unique and usable. The mouse-over support on the file graph is actually very, very useful for determining where storage space is being used.

6\. Transmit 4. I personally found Transmit 3's favorites interface and syncing interface to be more usable. The animations they've added slow down use and some buttons (e.g., disconnect) are far harder to reach than in T3.

7\. Courier. While the interface is novel and interesting at first glance, I have to question how useful it is for repetitive tasks. I have not used the app, so I can't comment there, but more often than not fancy graphics get in the way of speed.

8\. 1Password. This one I love. It's useful and very usable. I've used multiple versions of this program and the UI changes they've made with the current one far outshine the previous versions both in terms of appearance and usability.

My only gripe still would be with the browser plugin. I've tried setting up other less computer-savvy people to use it, and the menu does not make it immediately obvious to them which item to select to log in -- the top several options are too crammed together visually.

<comment><USER3> 4\. It's not that unique, is it? I used KDE's FileLight[1][2] some years ago and it had that same visualization.

[1]: <http://www.methylblue.com/filelight/>

[2]: <http://kde-apps.org/content/show.php?content=9887>

<comment><USER4> Exactly, and FileLight was hardly the first to have it either. This sort of disk usage visualization is quite old.

<comment><USER5> Every time I see the wood interface, I think of Radio Shack and Tandy circa 1970's.

</comment></comment></comment></comment><comment><USER6> The performance of some of these apps is appalling. The Sparrow GMail client idles in the background on my MacBook Air around 20 or 30% CPU usage on one core. If you try to scroll the list of mail, resize the window, or breathe on it wrong, it will chug, the graphics will stutter, and it will peg itself at 100% cpu for several seconds. If I have it running, it cuts my estimated remaining battery life from 4 hours to 1 hour.

Kiwi 2, the twitter client, uses 100mb of ram for one twitter account. What?

I know most people aren't going to check these numbers or care, but when I open an email client and feel my laptop getting warm underneath my hands, _something is wrong._

<comment><USER7> Sparrow may not be _that_ terrible; on first launch, it fetches, caches, and indexes the entirety of your Gmail account. I'm hoping that the CPU usage will subside once it finishes that process, but I still have a few thousand messages to go...

Edit: After finishing caching everything, Sparrow's CPU usage dropped to ~0% when idle. Scrolling, however, causes it to rocket up to ~70-80% on a 2.66 GHz Core i7 MacBook Pro.

<comment><USER8> It also uses about twice as much memory as Mail.app.

Anyway the interface is so much better that I'm willing to make the tradeoff!

<comment><USER9> The app has some serious spikes, but just like you said, the interface blows away Mail.app. Considering how infrequent most people should have their email open, it is well worth the cpu usage.

</comment></comment></comment></comment><comment><USER10> I agree that Transmit and 1Password are exemplary Mac OS X user interfaces. The other apps mentioned: not so much. To me, they look like iOS ports. Lots of chrome, wasted screen real estate.

My personal favorites:

Versions - <http://versionsapp.com/>

Kaleidoscope - <http://www.kaleidoscopeapp.com/>

Coda - <http://panic.com/coda/>

Espresso - <http://macrabbit.com/espresso/>

PixelMator - <http://www.pixelmator.com/>

BoinxTV - <http://www.boinx.com/boinxtv/overview/>

LittleSnapper - <http://www.realmacsoftware.com/littlesnapper/>

OmniFocus - <http://www.omnigroup.com/products/omnifocus/>

<comment><USER11> Kaleidoscope is very pretty (as is its website), but the lack of ability to do any editing/merging once you view the diff is jarring. I wish they'd spend their time on that.

</comment></comment><comment><USER12> I'll probably get downvoted for this, but it looks like the UI for these Mac apps is wildly inconsistent and each app has a very idiosyncratic idea of how it should look, which many times doesn't seem to take usability into account.

I don't think it looks prettier, but to me most modern GTK apps on Ubuntu seem much more consistent and strict as far as respecting the platform UI guidelines.

<comment><USER13> > "it looks like the UI for these Mac apps is wildly inconsistent and each app > has a very idiosyncratic idea of how it should look"

Have you used an iDevice recently? Their apps are designed around the fact that they can have the entire screen to themselves, and this sort of reasoning is bleeding out into computer application development. Some of the designs there could well be iPad apps, if it weren't for the window borders.

It's strange that when I used OS X, I was told that the proper way to do things was to use floating windows instead of having everything maximised, and now Lion is encouraging full-screen interfaces.

</comment></comment><comment><USER14> My least-favorite trend in Mac app design: using Helvetica instead of Lucida Grande, the system font, often because Helvetica is the default on iOS (cf. Reeder, a Mac port of an iOS app).

Why is this wrong? 1) Lucida Grande was optimized for legibility on low- and medium-resolution screens [1], and Helvetica is a 50-year-old print font. 2) Lucida Grande is the system font, and other fonts stick out like a sore-thumb, contrary to the consistency designer's strive for [2].

The problem, of course, is that Apple keeps using Helvetica in their own Mac apps, most recently across vast swaths of iPhoto, seemingly for the sole benefit of making it more like an iOS app (especially when full screen), disregarding much of what makes a Mac app unique.

[1] <http://www.tug.org/store/lucida/designnotes.html>

[2] <http://blog.cocoia.com/2008/swiss-interface-syndrome/>

</comment><comment><USER15> Far too many physical metaphors, I thought we got over that in the early 90's with MS-Bob.

Designers, please, if you are thinking of using wood paneling or shelves in your design, _stop_ , move on to the next idea.

</comment><comment><USER16> The problem with a lot of Mac apps is that:

1\. They have glossy websites that talk about how awesome they are. 2\. They have copy that claims they have awesome UI. 3\. You download them, try them, and realize they do almost nothing except look pretty.

It's pretty hard to find exceptions to this.

</comment><comment><USER17> Daisy Disk isn't a new idea... the Windows Freeware 'Scanner' by Steffen Gerlach was the first I saw to implement this disk view 8-9 years ago. Still find that utility priceless to this day.

<http://www.steffengerlach.de/freeware/>

<comment><USER18> Filelight has also been doing this in KDE since about 2003. I still think it's slightly easier to use than DaisyDisk, since the labels actually point to the sunburst chart. But DaisyDisk does look much spiffier.

</comment></comment><comment><USER19> Hate to say this, but it's been a while since there is any beautiful UI software on Windows.

</comment><comment><USER20> I would put as number one the new and still-in-beta "Tower" git client for Mac: <http://www.git-tower.com/>

</comment><comment><USER21> DaisyDisk is really awesome, I use this app a lot on my Macbook Air because disk space is still relatively scarce compared to my other machines. I like it a lot and recommend it.

</comment><comment><USER22> ah yes, let's praise inconsistent user interfaces...

I only like the SweetFM. wich is nothing clever, they just copied the first version of the official lastFM desktop application (at least how it looked in linux). Probably the guys at lastFM did some usability research and ditched that old version for something more itunesque.</comment>


China Is Harvesting the DNA of Its People. Is This the Future of Policing? - <AUTHOR> https://www.nytimes.com/2020/07/24/opinion/china-dna-police.html <comment><USER2> Hey, look at this guy, Mr Hong - DNA says he will murder three people next year - curious, he has the same DNA antigenicity as the President of the Politbureau - what do you suggest. Terminate, send the balls and heart to the Presidents Doctor, the rest in parts storage for the President and children...

</comment><comment><USER3> It is not the future it is the now, and it is such a scam.</comment>


Angels Out of America (WSJ house editorial) - <AUTHOR> http://www.google.com/search?q=%22interference+from+regulators.+The+law+requires+only%22+site%3Awsj.com Click on the results of the search for the full text if you're not a subscriber; real URL is http://online.wsj.com/article/SB10001424052748704671904575194483171910348.html
Various commentators on the issue will be pleased to see that the wealth threshold part of the proposal is dead last in the list of bad effects.
It's good <comment><AUTHOR> Click on the results of the search for the full text if you're not a subscriber; real URL is [http://online.wsj.com/article/SB1000142405274870467190457519...](http://online.wsj.com/article/SB10001424052748704671904575194483171910348.html)

Various commentators on the issue will be pleased to see that the wealth threshold part of the proposal is dead last in the list of bad effects.

It's good to see this get laser focused attention from _The Wall Street Journal_.</comment>


Bitrot and atomic COWs: Inside "next-gen" filesystems - <AUTHOR> http://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/ <comment><USER2> Bit worrying not to see a single mention of ECC memory when discussing protection from bitrot, especially with filesystems which depend on correctly functioning memory to provide the protections people expect from them.

People sneer at me for being a stickler for it, but between 50GB of memory and 26TB of ZFS-protected storage, I see ECC corrections about as often as I see disk checksum errors - maybe half a dozen of each in the past year or two. Frankly I think it's idiotic it's not more common and better supported.

<comment><USER3> The article was, sadly, missing a lot of stuff. When it said _" (most arrays don't check parity by default on every read)"_ I knew the author was not up to the task of writing this article. FWIW, a RAID system _has_ to calculate parity every time it reads a stripe so that it will know how to change it if something in the stripe changes. There are of course file system errors that are invisible to RAID, if your FS _writes_ corrupted data (as it might with ECC failures) the RAID subsystem will happily compute the correct parity for the stripe.

Granted I spent nearly 10 years immersed in storage systems (5 at NetApp, 4 at Google) but still there is some easily checked stuff missing from this article and so its point (which is hardening in the filesystems is good) is lost.

<comment><USER4> _> a RAID system has to calculate parity every time it reads a stripe so that it will know how to change it if something in the stripe changes_

Nope. On a non-degraded array with parity (R5, 6, ...) it will only read the block it needs to from the drive it is on. For an array that mirrors without parity (R1, ...) it will read the block from one of the drives. There is no need to bother another drive with the read operation unless it needs to check parity (because it has been told to on each ready as some controllers can be so told), you actually reduce the performance benefit of the striping if you do as you may be moving the heads of the other drive(s) "unnecessarily" and potentially away from another block they were about to be asked to read.

With an un-degraded array unless explicitly told to check parity on read the controller will not touch the parity blocks until a write happens at which point it will read the other relevant data blocks in order to regenerate the parity block. This is why the RAID 5 write penalty exists but there is no read penalty (in fact there is a read bonus due to striping over multiple devices).

Parity blocks will, unless checking parity on every read, only ever get read if the array is in a degraded state, in which case you can only derive some data blocks by reading the other blocks in that stripe (data and parity) and working out from them what the missing block should be.

<comment><USER3> I will take your word for it that some systems make this "optimization" but any such systems is fundamentally broken at the design level. Disk drives can, and do, return corrupted sectors without any indication of failure, the RAID system you describe would not catch those failures and would thus 'fail' in terms of providing any sort of data reliabilty.

<comment><USER5> So let's say you read a full RAID-5 stripe including parity and the parity does not match because of a silent error. How do you know which block contains the error? Classic RAID does not have any checksums. AFAICT classic RAIDs are screwed in the face of silent errors, so they might as well implement as many optimizations as they can within the bounds of their (unrealistic) failure model.

<comment><USER3> Fiber Channel drives used the DIF part of the sector, reading 526 bytes per sector rather than 512. SATA based RAID systems will often use a separate sector in the group to hold individual sector checksums. So 16 sectors, where 15 are 'data' and the 16th is crc data.

<comment><USER6> Could you tell us what RAID controllers actually does this? Because I've never seen one do checksumming at all.

Anyone that is actually available for "normal" people (which is relevant for the discussion of the article), i.e. not enterprise SAN?

<comment><USER3> All fiber channel HBAs can generate errors when the DIF don't correlate, its part of the FC spec. I don't have access to the LSI controller firmware source so I couldn't say one way or another if they do this with SATA drives. It should be possible to test though.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> As a Solaris admin, it's a little strange to hear ZFS called a "next-gen" filesystem. (It's been around for at least 7-8 years!) But it's good to see these ideas getting usable in other operating systems, especially with the current state of Solaris licensing.

<comment><USER8> I'm using HFS+. It's a last-last gen filesystem if ever there was one.

It's the only FS I've ever had silent corruption on, and it used to happen (10.4? 10.5? 10.6?) _all the time_.

I'd kill to have Apple just buy a license to NTFS.

<comment><USER9> Uh, you might want to look at NTFS performance for your given use case. If, say, you're a C++ programmer and your use case is reading lots of small files quickly and closing them again so you can compile, NTFS is a horror cabinet.

HFS beats that quite handily (and gets _obliterated_ by ExtFS2 performance.)

<comment><USER10> So having slow compiles on NTFS is worse than having silent corruption on HFS?

<comment><USER9> Depends on the use case, no?

Would I want to store a lot of important write-once data on HFS? Reluctantly, if at all. High-throughput database? Yuck, no.

Would I trust it with a source-tree that is stored in a DVCS and immediately restorable, while also giving me much faster compiles? Yep.

Would I choose either FS for a server? Nope. That was the point - there is no single "best" file system, or the world would've long ago settled on it. Look at what you need it for, and make your decision accordingly.

</comment></comment></comment></comment></comment><comment><USER11> Is bitrot something the average hard drive user even needs to worry about? I know the hard drives themselves at the hardware level implement checksums. Is it really necessary to also have it at the filesystem level?

I am legitimately asking because I have a good 2TB of family photos on hard drives and spooky stories about random bits flipping freak me out.

<comment><USER12> Running ZFS for many years has shown, yes, this happens, and will continue to get worse as density goes up.

<comment><USER13> I've been running ZFS on several servers with tenfolds of TB's with data. I see checksum errors every month, from a single bit to several megabytes.

<comment><USER14> That sounds like you have something else wrong. With ECC memory on server hardware, we've seen 0 checksum errors in the last 6 months and I've seen only 2 ever. A typical server has 136TB of raw hdd space and we get about 71TiB usable. It's about 80% full.

</comment></comment></comment></comment><comment><USER15> I think the next generation file system has CoW, are based on erasure codes and are internet distributed. For example your home NAS have InternetFS, now you can reach all your files where ever you are, all files in the home directory are always in sync. Failure of a hard drive does not matter. If your house burns down, your friends or someone else on the internet has the pieces you need to reconstruct your data. The next gen filesystem also have built in versioning via the CoW so you can always revert to an earlier version of an file.

InternetFS(Encrypted, distributed, always in sync, cheap snapshots, p2p)

Want to share Photos or movies with your friends and family? No problem just right click on the file, select the friend from a list, they see the file and they can view it on their computer.

This next gen file system will be incredibly easy to use, cross platform. As simple to use as Facebook, Skype, Email.

<comment><USER16> Hey! It sounds like you're describing ori [1]. Be sure to read the paper [2] because the website is a bit sparse on info. It's still young but the syncing, cheap snapshots, auto backups, etc. have been really nice. (it's only encrypted over transport though, I wish it were encrypted locally too).

1: [http://ori.scs.stanford.edu/](http://ori.scs.stanford.edu/) 2: [http://dl.acm.org/ft_gateway.cfm?id=2522721&ftid=1403940&dwn...](http://dl.acm.org/ft_gateway.cfm?id=2522721&ftid=1403940&dwn=1)

<comment><USER17> Great project.

Needs encryption, access controls and some kind of public namespace for publishing.

</comment></comment></comment><comment><USER18> I have a small NAS4Free box at home with ZFS and automatic zpool scrubbing. Barring a physical disaster (fire, flood, etc), I expect my data to be safe indefinitely. It was pretty easy to set it up on an HP Microserver, I recommend it.

<comment><USER19> I have a Drobo in my Amazon cart, was about to pull the trigger until I read this article.

Do you know if your NAS4Free solution a viable solution for a household full of Macs that need a shared Time Machine destination? I'm tempted: it's cheaper, and NAS4Free will keep evolving.

<comment><USER20> Anything that can run Samba server can be a target for Time Machine backup, google DIY Time Machine. I considered FreeNAS but figured it would be too much time to ste up and went with the Synology Intel box instead, could not be happier with it. Synology has step-by-step instructions for setting it up as a Time Machine backup target.

<comment><USER21> Be warned that network-based TM is _extremely_ unreliable. (Just google time machine synology failure).

<comment><USER22> I dumped time machine entirely for an rsync based solution. Take a look @ `--link-dest`

</comment></comment></comment></comment></comment><comment><USER23> Listen to Belt and Suspenders [1] and Computational Skeuomorphism [2] episodes of Hypercritical for an excellent discussion of filesystems: what the hell do they do and how do they compare.

[1]: [http://5by5.tv/hypercritical/56](http://5by5.tv/hypercritical/56) [2]: [http://5by5.tv/hypercritical/57](http://5by5.tv/hypercritical/57)

<comment><USER24> +1 on Hypercritical's discussion on widely used file systems and their problems. When I saw this article on Ars I was surprised that John Siracusa _wasn 't_ the author. Somewhere he must be smiling.

<comment><USER23> My thoughts exactly. Listening to John, he somehow manages to refer to how sucky HFS+ is literally in every other episode of whatever podcast he happens to be on.

</comment></comment></comment><comment><USER25> ZFS is also wonderfully usable. It's a delight to admin. See [http://rudd-o.com/linux-and-free-software/ways-in-which- zfs-...](http://rudd-o.com/linux-and-free-software/ways-in-which-zfs-is- better-than-btrfs)

<comment><AUTHOR> That seems pretty out of date. I only follow btrfs lightly but at least the RAIDZ and send/receive points are wrong. Current btrfs supports those.

</comment></comment><comment><USER26> I want F2FS for my phones. It seems to make storage 50-100 percent faster compared to ext4 from the benchmarks I've seen. Motorola started using it in Moto X and Moto G, but I hope Google makes it the default for Android in the next Android version.

<comment><AUTHOR> F2FS sounds like a bit of a hack. It's basically a way to get performance out of the storage-provided flash translation layer that pretends the underlying storage isn't flash. It also doesn't have the fancy ZFS/btrfs features. I wonder if the COW that these do can be tuned to work well with flash devices.

</comment></comment><comment><USER27> Anyone know how the latest versions of NTFS stand up against these?

<comment><USER28> Microsoft has introduced ReFS [1] as a potential successor to NTFS in Windows Server 2012. It has a few of the 'next-gen' filesystem features that the article mentions, such as integrity checking, but no copy-on-write and it's not feature-equivalent to NTFS yet. Also, you can't boot from it yet.

[1] [http://en.wikipedia.org/wiki/ReFS](http://en.wikipedia.org/wiki/ReFS)

<comment><USER29> It seems to do COW and checksumming of metadata, but not of data. According to [http://blogs.msdn.com/b/b8/archive/2012/01/16/building- the-n...](http://blogs.msdn.com/b/b8/archive/2012/01/16/building-the-next- generation-file-system-for-windows-refs.aspx) there's a feature "integrity stream" which is opt-in per-file (or per-subtree and inherited by all files) checksumming. It doesn't seem to do COW, but can be paired with "storage spaces"[0] from which bitrotted files can be recovered.

[0] [http://blogs.msdn.com/b/b8/archive/2012/01/05/virtualizing-s...](http://blogs.msdn.com/b/b8/archive/2012/01/05/virtualizing- storage-for-scale-resiliency-and-efficiency.aspx)

<comment><USER2> Integrity streams are enabled by default on mirrored pools, as per your first link:

> By default, when the /i switch is not specified, the behavior that the > system chooses depends on whether the volume resides on a mirrored space. On > a mirrored space, integrity is enabled because we expect the benefits to > significantly outweigh the costs.

Plus:

> When this option, known as "integrity streams," is enabled, ReFS always > writes the file changes to a location different from the original one. This > allocate-on-write technique ensures that pre-existing data is not lost due > to the new write

</comment></comment></comment></comment><comment><USER30> If you are really concerned about bitrot, and raid is apparently "not the solution", generate your own parity files for important stuff: [http://parchive.sourceforge.net/#clients](http://parchive.sourceforge.net/#clients)

<comment><USER31> That's very fiddly compared to using ZFS and getting it "for free".

<comment><USER32> Unless ZFS isn't easily available on the OS you are using.

<comment><USER33> Linux has an easy to install port ([http://zfsonlinux.org/](http://zfsonlinux.org/)). I remember OSX as having to use it via FUSE. I guess that leaves only Windows.

OS/2 had installable filesystems. Does Windows have something like it?

<comment><USER2> OS X has a native port - [http://open-zfs.org/wiki/Distributions#ZFS- OSX](http://open-zfs.org/wiki/Distributions#ZFS-OSX)

Windows has ReFS - [http://en.wikipedia.org/wiki/ReFS](http://en.wikipedia.org/wiki/ReFS)

<comment><USER33> I didn't mean like ZFS, but like installable filesystems. It should be possible, with some effort, make ZFS run on Windows.

ReFS is beta-quality and lack several features ZFS has since its first production-grade release. It's not really an apples to apples comparison.

</comment></comment></comment></comment></comment></comment><comment><USER34> In our days, it can take 10+ hours to stream all the data off a multi TB disk. A single disk can contain an amazing quantity of data, thus, it can be very valuable and sensitive.

I'd like a file system that duplicates sensitive data on the same drive. The file system data should be duplicated too, and marked in some way as to be able to reconstruct a disk after failure.

I don't need to save on space, I only need safety. And keep in mind that it takes many hours to dump a single time the contents of a disk - so, it's practically inaccessible as a whole over short periods of time.

For now, I get by with DropBox and TimeMachine but it's far from perfect. My photo collection alone is 1TB, so, no luck in backing it up in the cloud.

<comment><USER25> Use ZFS. Create a filesystem in your pool with copies=n to get n copies on the same disc.

But still use mirrors or raidz to protect against failure of a drive.

</comment></comment><comment><USER35> Stupid question: even if we use checksums and parity files and such to verify the integrity of our data, how do we verify that these integrity measures don't themselves become corrupted? Is it just "that's very unlikely to happen"?

<comment><USER3> Its actually a reasonable question, for parity it was always possible for two, complementary, bit errors to result in a successful parity computation in the presence of corrupted data. The CRC function can detect multi-bit errors because it encodes not only bit state but also bit sequence in the error check. There is an excellent discussion of the tradeoffs in the book with Richard Feynman's lectures on computation.

Generally when you design such a system you can often say what would have to be true for you to "miss" that something was corrupted. In our parity example, an even number of bits would have to change state. In the CRC example bit changes would need to be correlated across a longer string of bits. Once you have ways that you know you would not be able to detect errors, then you start breaking the system apart to change up detection and correction. So for example at NetApp a block (which was 4K bytes at the time I was there) on disk was 8 sectors, then there was an additional sector that included information about both a CRC calculation for the available bytes, as well as information about which block it was and what 'generation' it was (monotonically increasing number indicating file generation). The host bus adapter (HBA) would do its own CRC check on the data that came from the drive, passed through it, and landed in memory. That would detect most bit flips that occurred on the channel (SATA or Fibre Channel port) as data went through it. ECC on memory would detect if memory written had its bits flipped. Software would recompute the block parameters and compare them to the data in the check sector.

So if data on the disk was bad, that check sector would not work, if the data had been written correctly initially and gone bad, the RAID parity check would catch it, if the data was corrupted crossing the disk/memory channel the HBA would catch it, if the data got to memory but memory corrupted it, the ECC would catch it, if the memory some how didn't see the corruption the check against the block check sector would catch it. All layers of interlocking checks and re-checks in order to decrease the likelyhood that something corrupted yoru data without knowing it.

<comment><USER35> This is fascinating! Thanks everyone for answering.

</comment></comment></comment><comment><USER36> Huge kudos to ZFS and btrfs, but I am very disappointed with one detail of next generations file systems: Why, oh why, do we still not have a file-type metadata field? We are still using silly file name extensions and magic mime- type detection. In the Age of Types (OOP and FP strong type systems) it only makes sense for the file system to do the same.

<comment><USER37> I personally think the biggest problem with current file systems is lack of ACID semantics. It is a tragedy that you can't rollback a set of file system changes if a shell script fails halfway. Why should I have to bring out SQLite if I want transactions, but don't need relations?

(Also I agree--yes MIME types in filesystem please)

<comment><USER38> Even if such a thing were to exist, it would probably by necessity be built on some lower-level system that doesn't provide such guarantees… like a filesystem.

<comment><USER5> IIRC in the old Tandem OS transactions were a basic service, even lower level than the file system. So their file system could use the transaction service to perform arbitrary transactions without much complexity. These days all the world's a VAX though.

</comment></comment></comment></comment><comment><USER39> Because the author does seem to care about defining what COW means, because "atomic cow" doesn't readily google to its proper meaning, and because it doesn't seem to be otherwise mentioned here:

COW: Copy-On-Write

[http://en.wikipedia.org/wiki/Copy-on- write](http://en.wikipedia.org/wiki/Copy-on-write)

</comment><comment><USER40> Given that hard disks already do a CRC check when reading data (as far as I am aware), I don't see how this adds anything. The author is assuming that there is no hardware-level checksum or CRC, which I believe is incorrect.

</comment><comment><USER41> just backup your data to blueray (25 GB) or use PaperBack 1.0 for a 1 or more MB per A4 depending on compression. [http://www.ollydbg.de/Paperbak/index.html](http://www.ollydbg.de/Paperbak/index.html) My girlfriend is already printing out the family pictures as she doesn't trust me keeping the backups of the jpegs.

<comment><USER29> There's no guarantee that the data will reach your BR disk intact. Or that a writable BR will survive for years (let alone decades)

<comment><USER42> It's pretty trivial to check that a write succeeded. Unless you mean getting corrupted before putting the data into the redundant/backup system, which can happen just as easily with ZFS or almost any setup.

</comment></comment></comment><comment><USER43> So, Btrfs+linux or ZFS+BSD for a home server ? I thought the former was not production ready. I am confused.

</comment><comment><USER44> Nothing about Hammer? Pity.</comment>


Facebook is broken - <AUTHOR> https://techcrunch.com/2017/06/04/when-you-look-into-the-news-feed-the-news-feed-looks-into-you/ <comment><AUTHOR> > because "engagement" is the metric, Facebook inevitably selects for the > shocking and the outrageous.

Anecdotally, I've noticed this myself. Anything I post that's related to politics get lots of likes and comments, while it's crickets for everything else.

</comment><comment><USER2> The medium is the message</comment>


What's the best way to build a multi-tenant system like Hashnode? - <AUTHOR> My team and I are currently build an platform in which each user can add their custom domains. I'm just curious. How do companies like Hashnode do that? <comment><USER2> Build an app server that understands hosts names. For example a middleware that parses host names, extracts domains / subdomains and passes that to route handlers.

That's all there is to it.</comment>


Memory Safe C/C++: Time to Flip the Switch - <AUTHOR> http://blog.regehr.org/archives/939

<comment><USER2> C/C++ are often used for things like drivers and operating systems. That code can't have elaborate runtime baggage.

So if the memory safety were part of the std lib heap for instance, go for it!

But if it means making one instruction (store into array) into 50 (checkthebounds, the store if safe else throw an exception) then it needs to be optional.

<comment><USER3> Yes. Good thing about C/C++ (particularly code with statically allocated memory and zero or few virtual calls) is that the resulting code behavior is very predictable. You can almost always expect it to be memory bound, and by counting your memory accesses and cache misses guesstimate the performance really well.

I guess, that adding memory safety layer may destroy that predictability. If it does, than no - it needs to be optional.

</comment></comment><comment><USER4> Memory Safe C/C++... Sounds like Go :)- <http://golang.org/>

The future is now!

<comment><USER5> Rust I think has more potential to eventually replace C/C++. Besides having modern language features, it still allows the programmer to get down into the nuts and bolts of how memory is being used. (Not that I feel Rust is perfect either...)

Go has some cool features... but I have yet to find a really compelling reason to use it over anything else. I also find it to be kinda gross looking. (But maybe I am just weird in that respect?)

<comment><USER5> From an InfoQ interview:

"InfoQ: How does it compare with Google Go?

GH: Go's a good language. It's less complex than Rust, but also less ambitious. So you can pick your poison. Go's memory model, for example, has no concept of isolating memory between co-routines or threads, nor of controlling mutability. Everything can always race on the same global mutable heap. Similarly it has only one kind of pointer, which can always be null, and all pointers in all co-routines are subject to a single, global garbage collector. Rust statically differentiates all these cases, divides memory and pointers up into different types, which means we can control safety and performance much better, but at the cost of the programmer having to think more. Rust also provides a few extra "modern bits" that Go lacks: generic types, destructors, disjoint unions, that sort of thing. But to its credit, Go gets a lot of mileage out of the stuff it included, and I'm glad it's making headway."

<http://www.infoq.com/news/2012/08/Interview-Rust>

<comment><USER4> The counter argument is Rust is going to end up an even more bloated ugly language than C++. Go is going to add some of those features, but they are committed to a slow "do it right approach" as opposed to "bolt it on".

>>which means we can control safety and performance much better, but at the cost of the programmer having to think more

Outside of systems software, there is no room for this. If the top 5% of programmers wrote all the software in the world Rust would be great, but that is just not the case.

<comment><USER5> I'm guessing you have not really looked into Rust. The idea is to also take a "do it right" approach rather then "à la carte". Keep in mind, the focus in in a different domain. Not really sure what basis you have for the "even more bloated" comment.

There has been a lot of movement this past year around features and syntax in Rust... but the language is very much still in development.

The main difference with Rust is the focus is on trying to make memory management safe yet applicable to lower level and performance critical programming. (A domain a lot of people are currently using (and getting bitten by) C++ for, hence why Rust is aimed at replacing it.)

In regards to 'thinking more', this is in comparison to higher level languages. Things are _way better_ compared to C++ on the other hand. Don't forget that replacing C++ is the target market for the language, not Java/Go/Etc.

Anyhow, no reason to have a flame war on the subject. Rust isn't a hammer and neither is Go. :)

<comment><USER4> >>I'm guessing you have not really looked into Rust. The idea is to also take a "do it right" approach rather then "à la carte". Keep in mind, the focus in in a different domain. Not really sure what basis you have for the "even more bloated" comment.

C++ is too complex of a language IMHO. Complexity is bad for new people as it takes a long time to _know_ the entire language. And its bad for existing people as it provides too many ways to do the same thing. Rusts features look to be a superset of C++... You could substitute "too complex" with "too big".

>>Anyhow, no reason to have a flame war on the subject. Rust isn't a hammer and neither is Go. :)

I thought this was an interesting and friendly conversion, not a flame war. I do think everyone involved in this conversion probably agrees with each other in general, just maybe not particulars.

And yes everyone has their favorite hammer, that doesn't mean its not worth discussing which hammers are best for which tasks.

<comment><USER5> >> Rusts features look to be a superset of C++... You could substitute "too complex" with "too big".

Feature wise I'd say Rust has more in common with a functional language.

      * Consistent type system. (base types like int are objects)
      * Emphasis on type safety.
      * First class functions and Lambdas.
      * Pattern matching.
      * All memory is somehow managed or owned. (#1)
      * Isolated concurrent memory. (No need for semaphores/mutexes)
      * 'variables' are immutable by default.
      * Types with traits instead of a "class" system. (#2)
      * No exception catching (#3)
      * Generics (#4)
      * No direct null/nil (Options are used instead.)
      * Concurrency is built in.
      * Can allocate on the stack, or via a managed 'box' on the heap. (#5)
      * Has the ability to link to and call C code.
      * Has the ability to break out into inline assembly.
      * Macros. (#6)

The manual is reasonably decent: <http://static.rust-lang.org/doc/rust.html>

#1) Technically you can break out into unsafe memory operations or assembly. You have to specifically mark your code as 'unsafe' when doing so however.

#2) Similar to Haskell type classes and way simpler then C++ classes.

#3) There is a kind of error/exception but they are way simpler and not meant to be managed like in C++/Java

#4) Don't confuse generics with C++ templates. (C++ templates can be used for the same purpose, but is WAY more complex and interesting.)

#5) Perhaps the most complex part of the language... Boxes are a bit like smart pointers but with way less code involved.

#6) I have not gotten into this very deeply yet but it looks more like Lisp macros instead of C/C++ macros/preprocessing.

For perspective... C++ has about 84 keywords. Go has 25 keywords plus several predeclared identifiers (true and false for example). Rust currently has 32 keywords. (Some of which are just syntactic sugar).

>> I thought this was an interesting and friendly conversion, not a flame war.

( _Was trying to show that she saw the subject light heartedly_.) It was meant more as a tease then an accusation :)

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> Garbage collection. That's what all of these decades of research have led to, and that's what you should embrace. It's time. It's been time.</comment>


It's SysAdmin Appreciation Day - <AUTHOR> http://sysadminday.com/

<comment><USER2> Show your appreciation by sending them an email and CC'ing everyone you work with. They'll love that.</comment>


Eric 'ESR' Raymond Philly JUG presentation - <AUTHOR> http://youtu.be/1b17ggwkR60

<comment><USER2> I'm honestly not sure how I feel about his argument that the GPL is no longer needed. While I'm more sympathetic with BSD licensing now than I was 10 years ago, I do think that the GPL still makes a lot of sense for infrastructure- type projects.

My gut feeling is that his position that the existence of the internet makes concerns about the whims of any given legal jurisdiction irrelevant is more than a bit naive.

<comment><USER3> I think his problem with the GPL is that it allows "open source" project owners to constructively restrict commercial use (by offering a separatel- licensed (non-GPL) version.)

I also openly question the truth of his statement that he was an early reader of the JVM specification.

<comment><USER4> I have no idea , but one of ESR's favorite topics of conversation seems to be bigging up how important he thinks he is.

<comment><AUTHOR> I assume you watched the video? (I recorded and posted it) Personally, I actually thought he was pretty modest overall. He did mention his academic background and 'hacker credentials' when he was discussing functional programming languages and Haskell in particular, but other than that I didn't think he came across like he was promoting himself. I've been running this users' group for 12 years and I've seen much bigger egos from people who didn't have ESR's credentials.</comment></comment></comment></comment>


The Dictator's Daughter: Tweets from Gulnara Karimova - <AUTHOR> http://www.newyorker.com/online/blogs/newsdesk/2013/01/tweets-from-gulnara-the-dictators-daughter.html?mbid=gnep&google_editors_picks=true <comment><USER2> If Kim Jong Un had started a Twitter account while his father was still in power, would we have a similar attitude towards him?

Totalitarianism is not funny, and its perpetrators and perpetuators should be held in disgust.</comment>


Ask YC: How to debug javascript in IE?? - <AUTHOR>

I can't stand Internet Explorer - it is so frustrating trying to debug javascript in IE when it works perfectly fine in Firefox.<p>Is there a better way to debug javascript in IE other than narrowing down your code with alerts?  Any help would be greatly appreciated - thank you! <comment><USER2> There's also Firebug Lite (<http://getfirebug.com/lite.html>) and MS Visual Web Developer 2008 Express (free) (<http://www.microsoft.com/express/vwd/>)

<comment><AUTHOR> awesome - thanks for firebug lite. That's exactly what I was looking for.

</comment></comment><comment><USER3> <http://www.billyreisinger.com/jash/>

<comment><AUTHOR> thank you very much - I am going to give this a try now

</comment></comment><comment><USER4> I haven't had to deal much with IE in the last couple of years, but when I did, the only sane way I found to debug JS code was to have Visual Studio installed, and to use the VS debugger to hook into JS code.

</comment><comment><USER5> just to let you know of good refs for IE 7 ( i refuse to acknowledge IE 6

<http://www.naterkane.com/blog/2007/04/02/ie7-quirks/>

[http://weblog.openlaszlo.org/archives/2007/03/how-we-deal- wi...](http://weblog.openlaszlo.org/archives/2007/03/how-we-deal-with-browser- quirks-with-a-compendium-of-ie-7-issues/)

[http://www.sitepoint.com/blogs/2007/11/30/internet- explorer-...](http://www.sitepoint.com/blogs/2007/11/30/internet-explorer- doesnt-just-suck-it-also-blows/)

<http://www.gtalbot.org/BrowserBugsSection/>

(yes, not the most appealing design, but verrry time-saving)

TO track IE 7 and 8, check "browsershots.org" "ies4osx" and

<http://tredosoft.com/IE7_standalone></comment>


Ask HN: Which marketplace category misses an high quality app - <AUTHOR>

I struggled to come up with a title that really sums up my question in one sentence, so here it is in more detail:
HN has both, people who use a lot of apps on a daily basis, and people who make a living with developing apps.
I thought we could collect everybodies opinion on what (kind of) apps have room for improvement. What apps are either completely missing, or you wish there would be a better alternative?
People who want to develop mobile apps, but don't have an idea for one, could pick a up-voted app suggestion and work on a more decent version of it. Win win for everybody.
This works for every platform, so I propose we format comments like this: [Platform, App Category/Or name of a specific app] <comment><AUTHOR> [Android, Fitness App] I wish there was a really decent app to easily create work-outs. I like the approach of gainfitness.com, but unfortunately they only have an iphone version. Overall I think the fitness category on android has room for a couple of more apps who don't suck.</comment>


Rep. Jesse Jackson, Jr. calls out iPad as a job-killer - <AUTHOR> http://i.tuaw.com/2011/04/16/rep-jesse-jackson-jr-calls-out-ipad-as-a-job-killer/

<comment><USER2> Reminds me of a story I heard a long time ago.

While touring the Soviet Union, a foreign dignitary saw a team of a hundred workers digging an irrigation canal with shovels. The foreign dignitary commented that, with adequate machinery, a single worker could create the canal in a couple hours. The Soviet representative replied: "We prefer using workers with shovels so we create more jobs for our comrades". To that, the foreign dignitary replied "Then, why don't you use a thousand workers with spoons instead of a hundred with shovels?"</comment>


PPP just got another $250B – do not wait. Apply immediately - <AUTHOR> https://accountalent.com/tuesday-april-21st-ppp-2-0-great-comments-from-ppp-facebook-group/ <comment><USER2> Any info copy and pasted for those not on FB?</comment>


A course on Peer-to-Peer Applications - <AUTHOR> http://cel.archives-ouvertes.fr/cel-00544132/en/ <comment><USER2> very interesting ... Thanks for sharing :)</comment>


Microsoft Surface UnBoxing Photos - <AUTHOR> http://www.teehanlax.com/blog/?p=990 <comment><USER2> Because computers should be something that require four grown men to move between rooms.

<comment><USER3> Bigger and smaller computers are a good thing, opening up new types of interactions that are not possible with a 17-22" inch monitor, a 95-105 key keyboard, and a mouse.

</comment></comment><comment><USER4> Looks like a huge iPhone.

<comment><USER5> I saw one of these being demoed in the valley recently and that was the thing that struck me. "Meh, it's just a big iPhone." Apple have stolen Microsoft's thunder by putting multi-touch in the hands of consumers way before MS did.

That isn't to say these devices aren't fun. They just don't have quite the same wow factor as they did pre-iPhone.

</comment></comment><comment><USER6> The future is here and it's not an iPhone... ...it's a big ass table! take that Apple <http://www.youtube.com/watch?v=CZrr7AZ9nCY>

</comment><comment><USER7> Looks awesome. I wish they would have taken photos that were less 'artistic' looking and more interesting. The photos are mostly of them moving it, then using it at odd angles..

</comment><comment><USER8> They should really start pushing surface units in different sizes. As amazing as they are (they have one at my uni that i've had a chance to check out), they are pretty big. Right now the size that they are coming in are good for people sitting around the unit touching it and checking it out, but I would like to see what other form factors surface technology can be in.

They also may want to work on bringing the price on these things down too.

<comment><USER9> They should make them bigger. In Detroit. My floor isn't lighting up rings around my feet and telling me to buy shit when I walk on it. Why not?

</comment></comment><comment><USER10> They had a big product placement in The Day the Earth Stood Still. Looks like MS is trying to get the "cool" buzz going again with Surface.

Everything that I've read about it looks really cool.</comment>


What Happened to the Megafauna? - <AUTHOR> https://daily.jstor.org/what-really-happened-to-the-megafauna/ <comment><USER2> Imagine if, instead of spending your childhood, teenage years and early adulthood in schools, you just spent it learning to hunt, then continued to learn and practice hunting for the rest of your life. Imagine if everyone in your family, and everyone else you knew had also spent their entire lives learning to hunt and hunting. Imagine if every genius born into every generation devoted their entire intelligence to the problem of hunting, from their first moment to their last. Imagine if this had been going on for generation after generation, with knowledge and expertise refine and passed down through mentorship and oral tradition.

Collectively, it's a level of intellectual effort that humanity has likely never brought to bear on any other problem besides, perhaps, farming. As such, it's perhaps no surprise that early humans were spectacularly successful at hunting so many of even the largest and most fearsome animals to extinction.

<comment><USER3> > Collectively, it's a level of intellectual effort that humanity has likely > never brought to bear on any other problem besides, perhaps, farming.

I mean, if you are measuring in terms of brain cycles, perhaps. But huge amounts of these brain cycles were wasted because people lived in small tribes and information decayed quickly in space and time, requiring everyone to continually re-invent things. And the detailed knowledge for hunting one type of animal in one region was not particularly applicable to other animals in other regions, so these are to a large extent just different problems.

<comment><USER4> IIRC, in the absence of writing, humans had great oral memory. Not only that but they used other mental tricks. EX: Aborigines connecting myths to places in terms like "... And this is where angry bear slept for three days. The water here is still poisonous today from his angry urine..." and so on.

I would say it's more precarious than "modern knowledge" but we haven't really tested the longevity of tech civilization. There are still tribes in Siberia who tell legends of the hunter of the sky tracking the great deer. The "problem" is that they're talking about a set of constellations that were last visible 20000+ years ago. If disaster strikes, how many mediums will be rendered useless (floppies, CDs, lithograph, tapes, etc.)?

</comment></comment></comment><comment><USER5> When we go to a museum and see skeletons or life-size models of Pleistocene megafauna, we tend to assume these animals behaved just like the wild animals of today: wary, elusive, and ferocious if cornered. Then we consider the size- difference and shudder.

However, extinct megafauna may have behaved differently. The wild animals of today are the product of tens of thousands of years of intense selection pressure from humans. It's possible that a Mammoth behaved less like an African elephant and more like a Dodo. i.e. Humans migrating into areas where humans were previously unknown may have encountered animals that didn't view them as threats and which did not evade pursuit or defend themselves as modern wild game does.

</comment><comment><USER6> "Mammals went extinct as humans expanded, and they went extinct in areas where the climate was stable but humans were new."

That squares with something I think I read in Guns, Germs, and, Steel, which was that the reason African megafauna like elephants, lions, etc. survived to this day is that they co-evolved alongside humans to the point that they knew to be afraid of the puny little buggers. Meanwhile a 2-ton giant marsupial in Australia had no reason to fear and evade the new bipedal neighbors, or so it thought...

<comment><USER7> There were also a lot of 'germs' in Africa that kept human numbers in check until very recently.

</comment></comment><comment><USER8> I don't buy this. If the article explains it correctly, the argument is that they can show that in place where the climate was stable, and humans were new, the megafouna went extinct. But that implies that they believe that there were places on the planet were the climate was stable. If you look at the greenland Ice core data, you can see extreme peaks and falls around the last quarterly extinction. So sounds like BS to me.

<comment><USER9> Well then, let's try "relatively stable" rather than "stable".

These are scientists who probably have some idea of how to measure variance. You're some rando on the internet shooting from the hip. ¯\\_(ツ)_/¯

Here's the paper, go wild: [https://www.jstor.org/stable/43601643](https://www.jstor.org/stable/43601643)

<comment><USER10> I think you are being downvoted because you are using the appeal to authority: 'trust them, they have credentials that say they should be trusted on this very subject'.

This repurposed quote from the raiders of the lost ark might also work:

"I assure you ... We have top men working on it!"

"Who?"

"Top. Men."

<comment><USER11> I think an appeal to authority is only fallacious when it's an appeal to an authority that has no relevant expertise. Something like "my dentist says that anthropogenic climate change isn't real, you don't think you're smarter than a doctor do you?"

In contrast, when it comes down to John Q. Internet vs a scientist with relevant expertise, I think it's a reasonable heuristic to think the scientist is more likely to be correct.

IMO the downvotes were more related to the delivery than the content.

<comment><USER12> Interestingly, there seems to be an equality bias[1] where the experts opinions weight no more than any other person's opinion. So apparently the reasonable heuristic isn't always applied in how we perceive and judge information.

[1][http://www.pnas.org/content/112/12/3835](http://www.pnas.org/content/112/12/3835)

</comment></comment></comment></comment></comment><comment><USER13> Perhaps the komodo dragon was the only megafauna to turn the tables on the new humans.

<comment><USER14> I suspect they just taste bad.

<comment><USER15> IIRC, their hunting method consists of having saliva that is so infested with pathological bacteria that they simply bit their prey and wait for it to succumb to fevers.

Somehow, I doubt eating their meat is a healthy thing to do, even when cooked.

EDIT: turns out that the BBC documentary I got that from has outdated information:

> _Although previous studies proposed that Komodo dragon saliva contains a > variety of highly septic bacteria that would help to bring down prey, > research in 2013 suggested that the bacteria in the mouths of Komodo dragons > are ordinary and similar to those found in other carnivores._

[https://en.wikipedia.org/wiki/Komodo_dragon#Saliva](https://en.wikipedia.org/wiki/Komodo_dragon#Saliva)

<comment><USER13> On the other hand komodo dragons actively hunt humans which is pretty impolite. I would expect that bad habit would lead to their extinction on a small island but it hasn't.

</comment></comment></comment></comment><comment><USER16> Interesting, but it raises the question of why the global extinction didn't happen before. Was there a sudden improvement in human hunting techniques? Or did they stay the same, but the added factor of climate change made it possible for the two of them together to do it, or what?

<comment><USER17> the last ice age peaked about 21k years ago and ended about 11.5k years ago. Humans have been around for about 200k years, so it probably took some time 'getting on its feet' and reaching critical mass with the ending of the ice age to cause a global impact.

i myself do think we played a larger role. Woolly mammoths actually survived till about 4000 years ago by being in the most remote islands possible. To me, its like something you would see if there was some sort of zombie apocalypse and the last humans only survived in some remote island, far far away from land. Eventually, humans did reach that island, to which the mammoth immediately died off. Though climate change probably played a role, and could have been the primary one, I don't feel it was.

</comment></comment><comment><USER18> We ate them. Now we are the megafauna. Will we eat ourselves?

<comment><USER19> Zombies are terrifying for a reason. It's our strengths turned against us. They never stop pursuing.

</comment></comment><comment><USER20> TLDR: we ate them

<comment><USER7> Or got annoyed with them. Modern elephants, lions and tigers tend to get problems if they do in crops or villagers.</comment></comment>


Generate Static html cache files automatically in django - <AUTHOR> http://superjared.com/entry/announcing-staticgenerator-django/ Have a django site you want to be able to weather just about anything? Superjared put together a simple way to tweak your django site to auto generate static html pages on database updates which, when combined with nginx configurations, defaults to these html pages. Cool stuff, and a great way to maximize your server's potential with little effort. <comment><USER2> Is this seriously the first implementation of this in Django?

I seriously hate gloating, but Rails' page caching has been around for two years. I must be mistaken.

edit: After browsing the Django documentation, it appears that I'm right. While you can store the output of a view in the filesystem (or memcached, or whatever), cached hits still need to go through the framework to get dispatched. This appears to be the first way to skip the framework (aside from Squid or similar), which is what Rails' page caching does.

Page caching creates the cached page on the first request, and dishes out the cached output until it is swept (after a new create/update/delete). This seems like the "no duh" approach. The Django "standard" way of setting a timed expiration seems silly and inflexible.

<comment><AUTHOR> the difference here is that the html files are served by nginx, before django is event touched. Removing Django from the equation (and fastcgi or modpython or whatever nginx is proxying too) is where the speedup comes from. This isn't "caching" as much as it is fairly smoothly making certain areas of the site static html.

<comment><USER2> Yes, that's exactly my point. Is this the first approach that does this within the Django framework?

That is exactly how page caching in Rails works, and has been implemented for two years. It seems like Django's idea of "page caching" is equivalent to Rails' "action caching".

<comment><AUTHOR> i see your point. i would assume someone else has done this before in django, but regardless, i thought it was interesting as i haven't seen something like this as easily accessible for us django people.

</comment></comment></comment></comment><comment><USER3> Generating is never the hard part, it's keeping the cache current, pruning it when it gets too big, etc. that's hard. (Ok, I didn't read the article, yet.)

<comment><USER4> Hopefully what I created is elegant enough for people to want to use.

<comment><USER3> yeah it seems simple, that's good. And bypasses the framework, that's very good.

</comment></comment></comment><comment><USER4> woot, i got news'd</comment>