Galactic Tick Day – A celebration of our progress around the milky way - <AUTHOR> http://galactictick.com/ <comment><USER2> Interesting idea to use a unit that is universal to our galaxy. However, using deci-arcseconds (1/360/360/1000) seems arbitrary - it depends on the 360 degrees of a full circle and the decimal system.

Perhaps dividing the time period of 225million years by a power of two would be more universal.

For our life span, 2^27 looks good:

225.000.000 years / 2^27 = 1.67638 years, 611 days

Or perhaps 2^28? That'd be 305.94 days

<comment><USER3> I agree that divisions of 2^27 or 2^28 are in a sense less arbitrary than deci-arcseconds – but while the base (2) seems non-arbitrary, the same can't be said for the exponent (27 or 28). Wouldn't 2^32 be even better? A bit over 19 days.

Or I would argue 2^16 is even less arbitrary than 2^32. 2^(2^(2^2)) feels less arbitrary to me than 2^(2^5). A bit over 3,433 years.

This gives me an idea – write dates/times in hexadecimal. The first four hexits refer to units of 2^16 (roughly 3,433 years). The next four hexits refer to units of 2^32 (roughly 19 days). Then come units of 2^48 (roughly 25 seconds). Then units of 2^64 (about 385 microseconds), etc. However, you need to pick an epoch (zero point), and any choice of epoch is going to feel much more arbitrary than the system as a whole.

<comment><USER4> _" However, you need to pick an epoch (zero point), and any choice of epoch is going to feel much more arbitrary than the system as a whole."_

How about the best estimation for the 13.8 billion years old Big Bang event? Expressed through the number of galactic revolutions, of course.

<comment><USER2> Sure, we will not pick the epoch as someone else, but at least we celebrate it in the same interval.

<comment><USER4> You must be a very optimistic person to think beyond our own galaxy! Taking this path, should we think how shall we deal with the stability of our galactic movement reference if the Andromeda collision will affect it?

</comment></comment></comment></comment></comment><comment><USER5> The definition is very arbitrary: a centi-arcsecond. Why not celebrate a galactic quarter second every 704 days? Just as arbitrary, but much more relatable. As starting point, just take the year 0, so we're living at 4 galactic minutes and 21.25 seconds now.

<comment><USER6> The year 0?

<comment><USER7> this year minus 2016

<comment><USER8> Maybe thaumasiotes was pointing out that there was no year zero. Before the year 1 was the year 1 BC...

<comment><USER3> The traditional AD/BC system (aka the CE/BCE system) doesn't have a year zero. But astronomical year numbering does. (0 = 1 BC, -1 = 2 BC, etc.) Given the astronomical flavour of this discussion, the assumption that BetaCygni was using astronomical year numbering is reasonable.

[https://en.wikipedia.org/wiki/Astronomical_year_numbering](https://en.wikipedia.org/wiki/Astronomical_year_numbering)

<comment><USER5> I'm just a developer who by habit starts counting from 0. I meant to start at January 1st AD 1 0:00 AM.

</comment></comment></comment></comment></comment></comment><comment><USER9> What if our sun is not an exception?

Pluto turns around the Sun, but from Pluto the Sun is very small. Idem, the Sun could turn around a very big and very far star looking very small from the Earth.

We could be in a wide binary where currently the stars are at a maximum opposites, then the binary companion would be a very bright star at night, and not visible during the day.

This could explain the nemesis star theory for mass extinctions [https://en.wikipedia.org/wiki/Nemesis_(hypothetical_star)](https://en.wikipedia.org/wiki/Nemesis_\(hypothetical_star\)) and also the recently discovered anomaly in the planets orbit [https://www.caltech.edu/news/caltech-researchers-find- eviden...](https://www.caltech.edu/news/caltech-researchers-find-evidence- real-ninth-planet-49523) or why the Kuiper Belt ends suddenly or the Pioneer anomaly or maybe even the Oort Cloud.

There are some still unproved theories that state this star could be Sirius binary and we are in fact in a triple star system.

</comment><comment><USER4> It would be interesting to relate the galactic revolution subdivisions with other neutral units like multiples of Planck time. Like others mentioned here, the "tick" is very cultural and therefore not assured to be accepted in the long term by future generations (of our more rational off-springs).

<comment><USER10> Interesting thought. Tough to get to anything useful from Planck time though. 2^^8 (as in 2^2^2^2^2^2^2^2) Planck times would be a bit over 30 microseconds, but then I'm not sure how you'd go from there to a useful time scale.

</comment></comment><comment><USER11> Well you can check your Galactic Tick birthdays [http://jankor.github.io/galacticbday/](http://jankor.github.io/galacticbday/)

If you feel too Sun-centric.

</comment><comment><USER12> Just in time to play Galactic Escape - escape black hole :)

[https://itunes.apple.com/app/apple- store/id1076368667?pt=132...](https://itunes.apple.com/app/apple- store/id1076368667?pt=132536&ct=hn&mt=8)

Sorry for the plug :)

</comment><comment><USER13> Already? Seems like the time between them keeps getting shorter every year! I haven't even started my shopping yet.

</comment><comment><USER14> Oh no, galactic ticks are going to drink all of our blood!!!!

SCNR

</comment><comment><USER15> Um, as the Earth to the Sun, what's the length of the Sun's day with respect to the galactic center.

If my knowledge of mechanics is correct, given enough time, all days tend to infinity? (Like the moon's day with respect to Earth?)

<comment><USER2> Not sure what you mean, but according to [https://en.wikipedia.org/wiki/Solar_rotation](https://en.wikipedia.org/wiki/Solar_rotation) "at the equator the solar rotation period is 24.47 days and almost 38 days at the poles".

<comment><USER15> Ahh, thanks for that. Should have realized that the sun doesn't have a "fixed" surface. It seems that the center, however, has a rotational frequency of about 432nHz which results in a rotation period of 26.8 days.

All of these numbers are way faster than I imagined, which means that the length of the "solar year" is still incredibly long in "solar days".

<comment><USER16> Star core rotation is actually very interesting physics (it affects oscillations in the star in some very fun ways).</comment></comment></comment></comment>


How Tmux Starts Up: An Adventure with Linux Tools (2014) - <AUTHOR> http://blog.chaselambda.com/2014/11/25/how-tmux-starts-up-an-adventure-with-linux-tools.html <comment><USER2> > I got the tmux source running (I suggest using vagrant and compiling it in a > virtual machine).

What's with this need of using virtual machines for almost everything? Why would I need a virtual machine to compile tmux?

<comment><USER3> Doesn't want to dirty his home at all ? (Can use a chroot for this?. Maybe source is unstable and can crash machine ?

<comment> yjftsjthsd-h If an unprivileged userland program (like tmux) can bring down the whole machine, we have bigger problems than an unstable user program.

</comment></comment></comment><comment> ur-whale The double fork technique is not exactly news, to say the least ... or is the article about techniques to figure out what a unix binary does?

If the latter, what is the point since the source of every code path is available, down to the libc code?

I am not seeing the point of this article.

</comment><comment><USER4> tldr: instead of looking at the source to see how tmux daemonizes (spoiler: the same way everything else does), someone tries a bunch of convoluted methods to try to log or intercept fork(), and then looks at the source to see that it daemonizes the way everything else does.

This is slightly informative and clearly the author is learning the basics, but there's not a lot of there there.

</comment><comment><USER5> This is quite old as init is pretty much replaced by the kitchen sink nowadays (aka systemd).

FWIW, a double fork so parent exists is the typical way that daemons were started in the old days. Assume still the like nowadays.

<comment><USER6> I think it's important to distinguish between the concept of the init process vs specific implementations of the init process.

The article talks about the init process in general (the process that runs as PID 1 and picks orphaned processes). There are different implementations of init (such as SysV-style, systemd, Upstart etc) but it doesn't make a difference for this article.</comment></comment>


Are 'manspreading' arrests just police abusing the broken windows theory [video] - <AUTHOR> https://www.youtube.com/watch?v=vO01arvoM_I

<comment><USER2> No, it's discrimination against men. Where is all of the outrage?</comment>


Cascadia Code – OSS Monospaced Font with Ligatures - <AUTHOR> https://github.com/microsoft/cascadia-code <comment><AUTHOR> Disclosure: I work at Microsoft but not on this team (though I'm a huge fan of their work), I'm just a giant monospaced font nerd.

Created for Windows Terminal [1], this looks like a great font -- playing around with it on my Mac now and it looks great in iTerm.

[1]: [https://devblogs.microsoft.com/commandline/cascadia- code/](https://devblogs.microsoft.com/commandline/cascadia-code/)

</comment><comment><USER2> Some examples of the font in use would be cool; if its got ligatures for programming, then lets see em!

<comment><USER3> You may have seen it already but it was showcased on their blog: [https://devblogs.microsoft.com/commandline/cascadia- code/](https://devblogs.microsoft.com/commandline/cascadia-code/)</comment></comment>


ACM-style programming competition, but with support for Lisp, Haskell and others - <AUTHOR> http://www.spoj.pl/

<comment><USER2> Cool, thank you for posting here!

BTW: SPOJ allows you to create your own contests with your own rules and ranking system.</comment>


Pole suddenly popped up in my local park, anyone know what it is? - <AUTHOR> https://www.reddit.com/r/whatisthisthing/comments/dty9nd/this_pole_suddenly_popped_up_in_my_local_park/ <comment><USER2> For anyone who is curious: this device which is apparently LIDAR suddenly appeared in a small park in Groningen, The Netherlands. Unsure for what purpose it was placed there.

<comment><USER3> And - from the supposed manufacturer site - here:

[https://ouster.com/products/os1-lidar- sensor](https://ouster.com/products/os1-lidar-sensor)

It is not exactly a "cheap" piece of hardware:

>OS1-16 Available now for $3,500

>OS1-64 Available now for $12,000 or $8,000 for non-profit research

>OS1-128 Available now for $18,000 or $12,000 for non-profit research

hence - probably - the _need_ for the collar with the spikes.

Maybe I am a pessimist, but set aside specifically Groningen (NL), which may well be an exception, I would estimate the durability of such a thing (I mean before being stolen or vandalized) in an "average" park in two-three days (or nights).

<comment><USER4> I might call that optimism given the nature of the device.

</comment></comment></comment><comment><USER5> Looks very similar to the equipment Lucky Palmer's new company is selling to the US gov for virtual border fencing.

</comment><comment><USER6> As a one-off construction, it appears more likely to be a research project than surveillance.</comment>


Why 'Predator' Is a Subversive Masterpiece - <AUTHOR> https://screenhub.blog/2018/07/23/why-predator-is-a-subversive-masterpiece-screenhub-entertainment/ <comment><USER2> [spoiler alert] One of may favorite quotes: "If it bleeds, we can kill it."</comment>


Ask HN: Looking for feedback on my rate (iOS/Android) - <AUTHOR>

I'm currently a junior studying computer science but for the past year and a half or so I've been doing a decent amount of short-term mobile contract work.
Here is some of my previous work:
* Venmo for Android - https://market.android.com/details?id=com.venmo
* Wolfgang's Vault for Android - https://market.android.com/details?id=com.wolfgangsvault.concertvault
* Battlestar Galactica Bobblehead App - https://market.android.com/details?id=com.laan.BobbleFanSyFy
* Philadelphia Concert Hub for iPhone - http://itunes.apple.com/us/app/philadelphia-concert-hub/id315859665?mt=8
I'm wondering what I should charge per hour.
I'm thinking of raising my rates for new clients but thought I should check here to see if anyone has any input.
Sorry for such a vague question and for not listing my current rate, just want to see what answers I'll get to see if they match what I'm thinking. <comment><AUTHOR> Clickable links:

* Venmo for Android - <https://market.android.com/details?id=com.venmo>

* Wolfgang's Vault for Android - [https://market.android.com/details?id=com.wolfgangsvault.con...](https://market.android.com/details?id=com.wolfgangsvault.concertvault)

* Battlestar Galactica Bobblehead App - <https://market.android.com/details?id=com.laan.BobbleFanSyFy>

* Philadelphia Concert Hub for iPhone - [http://itunes.apple.com/us/app/philadelphia-concert-hub/id31...](http://itunes.apple.com/us/app/philadelphia-concert-hub/id315859665?mt=8)

</comment><comment><USER3> Well, aside from the bobble-head app having no use, the other three seem quite useful and have gotten a few good ratings.

Depending on the client and their app I see iOS/Android dev houses and individuals charging anywhere between $50 and $100 per hour.

Now, I think that both of these numbers are on the high-end. the $50 being for a dev house that has all their staff in india aside from some mgmt here in SF. The $100 being a dev in NYC who was a good dev, but not a designer.

Depending on your situation, interests and what you want to/need to make - <USER2> would go on the lower end to get more business (assuming you have need of more)

Also, I cant tell what exactly was required of you as a developer in creating these projects (I didn't look at them to see the architecture - but clearly you're connecting to services that have the data, and in the case of the music streamer - streaming the content)

How difficult were these for you to make? How many hours did these apps take you? What is your strength and weakness?

I think they look great.

<comment><AUTHOR> Thanks for the feedback.

More details:

Venmo: handled everything but design (but worked alongside their designer)

Wolfgang's Vault: handled everything for the app, based design off of existing iOS app

BSG Bobble: ported from iOS (also agree with it being pointless, fun project though)

Philly Concert Hub: everything (even the web app that gets listings, <http://phillyconcerthub.com>)

None of these were overly difficult, nearly everything is just a native client that connects to some web API.

I can't recall the hours for each, sorry.

Strengths: I think I can build a decent product even though I'm not a designer Good understanding of both iOS and Android

Weaknesses: I wish I did more testing</comment></comment>


An open approach for routing, switching, and transport - <AUTHOR> https://code.facebook.com/posts/1977308282496021/an-open-approach-for-switching-routing-and-transport/ <comment><USER2> Could somebody comment on the significance of this?</comment>


Does Silicon Valley have a racism problem? - <AUTHOR> http://www.vanityfair.com/news/2016/06/does-silicon-valley-have-a-racism-problem <comment><USER2> Yes, next question.

<comment><USER3> but let's blame it on a "pipeline problem"</comment></comment>


Ask HN: What's a small problem you have that might make a great side project? - <AUTHOR> <comment><USER2> The question gets asked so often in slight variation, the last one is still on the first page of [https://news.ycombinator.com/ask](https://news.ycombinator.com/ask) \-- [https://news.ycombinator.com/item?id=15002080](https://news.ycombinator.com/item?id=15002080)</comment>


My Favorite Features in Froyo - <AUTHOR> http://smarterware.org/6085/android-2-2-screenshot-tour-my-favorite-features-in-froyo

<comment><USER2> The issue I have is that we've yet to see whether the tethering/WiFi hotspot features will actually be allowed by the carriers (for free). Sprint has already announced plans that WiFi hotspot service will be $30/mo with their Android 2.1 EVO. It is unreasonable to believe that Sprint is going to let Android 2.2 devices tether for free and charge $30/mo for older devices. In the end, this feature will likely end up similar to iPhone tethering. If you're willing to "jailbrake" the phone, it's there. Of course, it's likely that carriers will be able to detect tethering (it really shouldn't be too hard) and call up the user to tell them that they have to stop or a $XX charge will be added to their account monthly for tethering usage. They might even add the charge without alerting the user - how many people who don't have a data plan get charged without being alerted when they accidentally hit the web button on their phone? I don't think it's wholly unreasonable to believe that carriers might do the same for tethering.

As much as we'd all like to think that Google's in the driver's seat fighting for users rights when Apple won't stand up for them, it's still the networks who are the gatekeepers. Froyo has the technological capability to do tethering/hotspot service. It's a great capability to add and it's great that Google has done so. That doesn't mean it's going to be free.

<comment><USER3> I've been occasionally tethering on Sprint (with PDANet) for almost 3 years and haven't been "caught" yet. I try to VPN so it isn't completely trivial to see that my traffic is from a tethered laptop. But I'd be ok paying $30/month for tethering if I had to. I just wasn't ok with $60/month.

<comment><USER4> That you haven't been caught yet doesn't imply that they won't start trying in the future.

With the vast uptake in smartphones and those smartphones now coming equipped with tethering, carriers just might find it's worthwhile to crack down.

</comment></comment></comment><comment><USER5> Oddly enough, the best feature in Froyo for me has been the ability to update all applications with only a single button press, rather than having to go through each one individually. Being able to elect to have some applications auto-update is just gravy on top.

It used to be I'd have to spend on average 2 minutes every day just updating applications.

<comment><USER6> Just curious, how many apps do you have installed? I spend maybe 5 minutes every other week doing the updates.

<comment><USER5> 70 at the moment.

</comment></comment></comment><comment><USER7> Didn't know that Flash had an "On-demand" feature. I think it's time to accidentally drop my Blackberry in the toilet.

<comment><USER5> I've found it more reliable to turn off the plugin service, as I had a couple of flash blobs sneak through on-demand.

</comment></comment><comment><USER8> The Gallery app is now tied in to Picasa: previously, I could only see my local picture files. Now it shows me my Picasa albums and pictures.

The down-side is that I first discovered this on an iffy Edge connection and it struggled to download the thumbnail pictures for the albums. Once I got home (wifi), it popped up the albums, thumbnails, and pictures nicely.</comment>


Epic commit message - <AUTHOR> https://github.com/chkoreff/Fexl/commit/23f32badb0ab3772a4eddeb8e45d85fe5bb287f2 <comment><USER2> I wrote a few similar commit messages - mainly upon merges, but sometimes when commiting small, but important changes too. The result: I was ridiculed and laughed at... I even seem to remember that someone said I was "unfulfilled writer" (believe me, in my language that was much more unpleasant...).

I can't help but think that majority of programmers are just illiterate and they feel threatened when someone actually has the will (as you can probably see from this post my skill isn't all that great either) to write...

<comment><USER3> > The result: I was ridiculed and laughed at...

Well fuck these people, I for one am sick and tired of looking for _why_ a code section was changed in ways I don't understand in the log or blame and getting "changed stuff" back. I'd much rather people explained clearly (and verbosely if needed) what the bloody hell they're committing this change for.

</comment></comment><comment><USER4> It's a little bit cheesy to submit something that you did yourself and describe it as epic (no snark intended).

Source: contact in whois for domain listed in OP's profile is the same as the author of commit in git log.

Related: My favourite pull request ever <http://github.com/ericf/yui3-gallery/pull/21>

<comment><USER5> The submitter name is also the github repo's name ...

</comment></comment><comment><USER6> If I had to choose, I'd pick this commit message over "bug fixes" or other similarly uninformative commit messages.

<comment><USER7> Actually, I think this is a really good commit message, period. Sure, you don't want to detail _every_ change to this extent, but important architectural fixes are fine.

(That said, speculating about the future in a medium where you don't necessarily record "well, that didn't work" and cannot go back to add something may not be optimal.)

</comment></comment><comment><USER8> Would have been hilarious if at the end of all that we see a single line change in the diff and it's just a deletion.

</comment><comment><USER9> Well, technically, it's not.

It's just a long commit message. Nothing's really 'epic' about it, beyond being rather informative.

<comment><USER10> ep·ic [ep-ik] adjective

1\. noting or pertaining to a long poetic composition, usually centered upon a hero, in which a series of great achievements or events is narrated in elevated style: Homer's Iliad is an epic poem.

2\. resembling or suggesting such poetry: an epic novel on the founding of the country.

<comment><AUTHOR> I was using "epic" with a Jeff Spicoli accent, not in the classical sense. But thanks for the comments!

</comment></comment></comment><comment><USER11> It would be good if you could categorise your source code comments somehow and have your IDE help manage them.

By default you would browse code with important summary comments visible, but if you're interested in any one area, you could then drill down into much more detailed and text dense comments right within your editor to read about design decisions and the bigger picture in plain English.

Comments could be further categorised into, say, business logic related comments or those related to design patterns, or page flow or test cases or internationalization or whatever, giving you more options to learn about the code and drill into the specific area you care about.

This readily accessible living documentation would be a much better way of learning about code bases in plain English without consulting out-of-date technical specs.

<comment><AUTHOR> I like what you're hinting at here. In my case, the "IDE" is vim running in a terminal window, with another terminal window to the right where I run git, grep, and tests. Nevertheless, even in this austere environment I could imagine a vim plugin which would let me wrap some sort of bracketed enclosures around piece of text, possibly nested, with the ability to expand and contract with the press of single key. Without even searching, I'll estimate the probability of that already existing as 85%.

Problem is, anyone viewing the code in a normal text display, e.g. on github or plain text editor, would see all the excruciating detail, all the time. But it's an interesting idea.

Perhaps instead of the expand/collapse, I could just use links. Vim users should note that when you're sitting on a word, if you type "gf" it will Go to that File (i.e. open it up right there), and you can use "o" and "i" the way you use "Back" and "Forward" in a browser.

<comment><AUTHOR> Sorry, I meant you can use "Ctrl-o" and "Ctrl-i" that way.

</comment></comment></comment><comment><USER12> I feel like I've seen this recently on HN

Eh, cannot find it - I guess we just talk a lot about commit messages.

[http://www.hnsearch.com/search#request/all&q=commit+mess...](http://www.hnsearch.com/search#request/all&q=commit+messages&sortby=points+desc)

[https://www.google.com/search?q=site%3Aycombinator.com+commi...](https://www.google.com/search?q=site%3Aycombinator.com+commit+messages&tbo=1&prmdo=1&gbv=1)

Maybe it was this one, but I swear it was more recent:

[http://andrewvos.com/2011/02/21/amount-of-profanity-in- git-c...](http://andrewvos.com/2011/02/21/amount-of-profanity-in-git-commit- messages-per-programming-language/)

[http://geeksta.net/geeklog/exploring-expressions-emotions- gi...](http://geeksta.net/geeklog/exploring-expressions-emotions-github- commit-messages/)

</comment><comment><USER13> This is called: "programming".</comment>


The Tragedy of UCS-2 - <AUTHOR> https://unascribed.com/b/2019-02-08-the-tragedy-of-ucs2.html <comment><USER2> This is a great rundown. I started my career at Microsoft working with and on Win32/COM and saw this play out first hand.

One thing not mentioned here is the history of the "Byte Order Mark" (BOM) in unicode.

(Not an expert here but my understanding having lived it.)

You see, given UCS-2, there are 2 ways to encode any codepoint -- either big endian or little endian. The idea was then to create a codepoint (U+FEFF) that you could put at the start of a text stream that would signify what order the file was encoded in.

Wikipedia page: [https://en.m.wikipedia.org/wiki/Byte_order_mark](https://en.m.wikipedia.org/wiki/Byte_order_mark)

This then got overloaded. When loading a legacy text format often times there is the difficulty of figuring out the code page to use. When applied to HTML, there are a bunch of ways to do it and they don't always work. There are things like charset meta tags (but you have to parse enough HTML to find it and then re-start the decode/parse). But often times even that was wrong. Browsers used to (and still do?) have an "autodetect" mode where it would try divine the codepage based on content. This is all in the name of "be liberal in what you expect".

Enter UTF-8. How can you tell if a doc is US ASCII or UTF-8 if there are no other indications in the content? How does this apply to regular old text files? Well, the answer is to use the BOM. Encode it in UTF-8 and put it at the start of the text file.

But often times people want to treat simple UTF-8 as ASCII and you end up with a high value codepoint in what would otherwise be an ASCII document. And everyone curses it.

Having the BOM littered everywhere doesn't seem to be as much of a problem not as it used to be. I think a lot of programs stopped putting it in and a lot of other programs talk UTF-8 and deal with it silently. Still something to be aware of though.

<comment><USER3> Yeah, the BOM has gotten me a few times.

My most spectacular fail was a program that read UTF-8 or Latin-1 and wrote UTF-16, preserving but not displaying null characters. I believe this was default behavior HyperStudio. Every round-trip would double the size of the file by inserting null bytes every other character. Soon there were giant stretches of null characters between each display character, but the displayed text never appeared to change even though the disk requirements doubled with each launch. That's how I learned about UTF-16!

Speaking of Win32/COM... is there a "tcpdump for COM"? I've got a legacy app that uses COM for IPC and I've been instrumenting each call for lack of one.

<comment><USER4> If there is anything like tcpdump for COM, it would be part of Event Tracing for Windows, but you'd probably prefer to use it via Microsoft Message Analyzer.

</comment></comment></comment><comment><USER5> It always baffled me as a native Chinese speaker that 16-bit was _ever_ considered enough to "be plenty of room to encode every script in modern use, and then some". Common local encodings like Big5 (also 16-bit fixwidth) was already suffering from serious lack of code points at that time. It would be obvious if they consulted literally any experienced programmer from East Asia. Yeah, that was 1990 so it's not an easy thing to do by any means, but you'd expect more from someone aspiring to create such thing.

<comment> Tuna-Fish > It would be obvious if they consulted literally any experienced programmer > from East Asia. Yeah, that was 1990 so it's not an easy thing to do by any > means

They did consult not just experienced East Asian programmers, but Chinese, Korean, Japanese and Vietnamese professors of Linguistics and the relevant local authorities, and had them form the Ideographic Rapporteur Group, which advised them that all the East Asian languages can be encoded in 20940 code points.

Unicode is not centrally run like so many people seem to expect. Rather, the Unicode Consortium itself acts just as the standards body that mates together local standards into a single complete whole. The Consortium makes no decisions about how Chinese is represented, other than allocating them code pages; all the relevant decisions about any language is done by local experts. The problem was that the early 90's were an era of boundless optimism and futurism, which in East Asia resulted in the Han Unification project, with the idea of cutting down the amount of symbols in use, and unifying the representation of all the languages that used a Han-derived alphabet. That... did not go so well.

<comment><USER6> Thanks for that background. I was thinking that the (for lack of a better word) politics of CJKV and Han unification likely had a lot to do with it.

To my eyes, the letter "A" in English (Latin), Greek, and Russian (Cyrillic) looks identical. Why does it need three separate code points? Maybe it gets tricky when the relationship between uppercase and lowercase diverges. In fact, that does appear to be one of the arguments in this technical note:

[https://www.unicode.org/notes/tn26/](https://www.unicode.org/notes/tn26/)

<comment><USER7> I think the most salient point in your link is:

>Even more significantly, from the point of view of the problem of character encoding for digital textual representation in information technology, the preexisting identification of Latin, Greek, and Cyrillic as distinct scripts was carried over into character encoding, from the very earliest instances of such encodings. Once ASCII and EBCDIC were expanded to start incorporating Greek or Cyrillic letters, all significant instances of such encodings included a basic Latin (ASCII or otherwise) set and a full set of letters for Greek or a full set of letters for Cyrillic. Precedent for the purposes of character encoding was clearly established by those early 8-bit charsets.

That's true, although it reminded me of a cool charset: the Russian KOI8-R encoding[1]. This encoding was created so that if you stripped the high bit (presumably on a system unable to handle russian properly) you ended up with semi-legible latinized Russian. Quoting wikipedia:

>For instance, "Русский Текст" in KOI8-R becomes rUSSKIJ tEKST ("Russian Text") if the 8th bit is stripped; attempting to interpret the ASCII string rUSSKIJ tEKST as KOI7 yields "РУССКИЙ ТЕКСТ". KOI8 was based on Russian Morse code, which was created from Latin Morse code based on sound similarities, and which has the same connection to the Latin Morse codes for A-Z as KOI8 has with ASCII.

[1] [https://en.wikipedia.org/wiki/KOI8-R](https://en.wikipedia.org/wiki/KOI8-R)

<comment><USER8> Wow, they made the 8th bit essentially be the case bit, and losing+recreating it mostly just made everything uppercase.

<comment><USER9> No, the case bit is still bit 5. Hiwei, unlike ASCII, KOI7/KOI8's Cyrillic alphabet sets bit 5 to 1 for _uppercase_ characters. This way, even though Cyrillic text remains legible, you also have a clue that the encoding is KOI.

</comment></comment></comment></comment></comment></comment><comment><USER10> It's also worth noting where the terms "UCS-2" and "UCS-4" come from. For a while there was the idea of the Universal Coded Character Set[0] as an _alternative_ to Unicode. This would have been a natively 32-bit encoding. However, it wouldn't have allowed the full 2^32 possible characters, because of its insistence that none of the 4 bytes could be any of the C0 or C1 controls. Still, it would have about 2^29 possible characters as opposed to Unicode's about 2^21.

But the insistence on avoiding C0 and C1 controls would have led to some odd codepoint numbers, and definite incompatibility with ASCII and Unicode; for instance, 'A', instead of 0x41, would have been 0x20202041! Whereas Unicode of course made sure to copy ASCII (indeed, to copy ISO-8859-1) for its first block.

Of course, this didn't come to pass because, well, people were already supporting 16-bit Unicode and didn't want to switch _again_. And I'm kind of glad it didn't -- while it would have meant not having to deal with the compromise that is UTF-16, imagine living in a world where 'A' is not 0x41, but rather 0x20202041! (Or imagine opening up a text file and finding three spaces between each character...)

[0][https://en.wikipedia.org/wiki/Universal_Coded_Character_Set](https://en.wikipedia.org/wiki/Universal_Coded_Character_Set)

<comment><USER11> _" imagine living in a world where 'A' is not 0x41, but rather 0x20202041! (Or imagine opening up a text file and finding three spaces between each character...)"_

On the other hand, often _" obviously broken"_ is better than _" subtly broken"_. There's a ton of undetected encoding problems in the wild right now.

</comment></comment><comment><USER12> I would go further and say that UCS-4 is also a variable length encoding, mostly because of emoji and its endearing ZWJ sequences, not to mention the encoding of flags as two Regional Indicator Symbol codepoints (both in the astral plane, of course), which makes Unicode not self-synchronizing.

So the idea of ever having a fixed-length encoding for Unicode is basically impossible now. Best to just use UTF-8 for everything and logic to group it in to code points, grapheme clusters, or whatever other granularity is needed.

<comment><USER13> Fun fact: because flags are a political nature, and Unicode did not want to limit the length of country codes, any number of Regional Indicator Symbol code points can be placed back to back and treated as an extended grapheme cluster.

That means that what Unicode today considers "one character" can be infinite in size.

<comment><USER14> And worse, the 'operators' are postfix or infix, so if you're handling text sequentially — and let's say, split across network packets with an arbitrary delay in between — you can't even tell you've read a complete character until you receive the next one.

 _Typing_ in many Latin languages generates diacritics using 'dead keys', which are prefix operators. (Originally, on a physical typewriter, these were keys that simply struck the paper without triggering the mechanism to advance the carriage.) If Unicode had taken this hint, life would be easier.

(ISO/IEC 6937 had prefix diacritics, but it was too late.)

<comment><USER15> Complete recognition before processing. It's not the law, just a good idea, but you should do it anyway. In cryptography this lesson had to keep being learned. "Oh, I received this partial data, and I processed it, and then in the next packet I received the MAC and I realised the bad guys altered the data, oops"

Process whole strings, the meaning of a partial string may not be what you hope, even without fancy writing systems and Unicode encoding.

"Give the money to Steph" \- OK, will do

"...enson" \- Crap, OK, somebody chase Steph and get our money back, meanwhile here's James Stephenson's money

"... once he gives you the key" \- Aargh. Somebody chase down James and get the key off him.

<comment><USER16> Wow so interesting an idea. But so many streaming algorithms etc. people stick megabytes of crap into one framed message.

</comment></comment></comment></comment></comment><comment><USER17> Another aspect of this tragedy is that the creation of UTF-16 has forever limited Unicode to only 17 planes. The original UTF-8 encoding ([https://tools.ietf.org/html/rfc2044](https://tools.ietf.org/html/rfc2044)) used up to six bytes, and could represent a full 31-bit range, which corresponds to 32768 16-bit planes; after UTF-16 became common, UTF-8 was limited to four bytes ([https://tools.ietf.org/html/rfc3629](https://tools.ietf.org/html/rfc3629)).

<comment><USER14> The original pattern extends up to at least 36 bits (which makes the PDP-10 refugees happy).

        1  7  0xxxxxxx
        2 11  110xxxxx 10xxxxxx
        3 16  1110xxxx 10xxxxxx 10xxxxxx
        4 21  11110xxx 10xxxxxx 10xxxxxx 10xxxxxx
        5 26  111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
        6 31  1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
        7 36  11111110 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Then, you can either decide to limit the leader to one byte, giving you 42 bits…

        8 42  11111111 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

Or allow multi-byte leaders and continue forever.

        9 53  11111111 110xxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
       10 58  11111111 1110xxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx
        ⋮

The Intergalactic Confederation will not look kindly on UTF-16.

<comment><USER17> > Or allow multi-byte leaders and continue forever.

That loses one very useful property of UTF-8: you always know, by looking at a single byte, whether it's the first byte of a code point or not. It also loses the property that no valid UTF-8 encoding is a subset of another valid UTF-8 encoding. It's better to stop at 36 bits (which keeps another useful property: you'll never find a 0xFF byte within an UTF-8 string, and you'll only find a 0x00 byte when it's encoding the U+0000 code point).

</comment></comment></comment><comment><USER18> Fun fact: for SMS, UCS-2 is used when a message requires more than 128 characters to be rendered.[1]

If you've ever noticed that entering an emoji or other non-ASCII [2] character seems to dramatically increase the size of your SMS, that's because the message is switching from 1-byte characters to 2-byte characters.

[1] [https://www.twilio.com/docs/glossary/what-is- ucs-2-character...](https://www.twilio.com/docs/glossary/what-is- ucs-2-character-encoding)

[2] It's actually GSM-7, not ASCII, but the principle is the same: [https://www.twilio.com/docs/glossary/what-is- gsm-7-character...](https://www.twilio.com/docs/glossary/what-is- gsm-7-character-encoding)

<comment><USER19> I was a flip-phone holdout for a long time. This was not generally supported in new devices sold up to 2015 (when I acquired my first smartphone, and my experience with good phones ended).

Every flip phone I owned did not play well with non-ASCII text. Receiving a message with an emoji would render the entire message, or if I was lucky just the portion following the emoji, unrenderable. Most phones I had failed to just rectangles, which I assume is their method of mojibake.

This was not a causal factor in my switch to a smartphone. It was annoying to have to explain character encodings to friends who didn't understand why they couldn't send me emoji-containing messages.

<comment><USER20> You sound like a blast.

<comment><USER21> Hey, could you please review the site guidelines and stick to the spirit of this site when posting here? Normally we ban accounts that attack others like this, and I actually banned yours, but decided to reverse that because you posted a more substantive comment earlier today. Basically, we're trying for more thoughtful conversation and better signal/noise than internet default.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

You might also find these links helpful for getting the intended use of HN:

[https://news.ycombinator.com/newswelcome.html](https://news.ycombinator.com/newswelcome.html)

[https://news.ycombinator.com/hackernews.html](https://news.ycombinator.com/hackernews.html)

[http://www.paulgraham.com/trolls.html](http://www.paulgraham.com/trolls.html)

[http://www.paulgraham.com/hackernews.html](http://www.paulgraham.com/hackernews.html)

</comment></comment></comment></comment><comment><USER22> JS, like Java, now has implementations that will also store strings as Latin1 when the implementation believes it is safe to do so. This results in significant memory savings [1] in most programs.

1\. [https://blog.mozilla.org/javascript/2014/07/21/slimmer- and-f...](https://blog.mozilla.org/javascript/2014/07/21/slimmer-and-faster- javascript-strings-in-firefox/)

<comment><USER23> I wonder why Latin-1 and not just UTF-8

<comment><USER24> From the article:

* Gecko is huge and it uses TwoByte strings in most places. Converting all of Gecko to use UTF8 strings is a much bigger project and has its own risks. As described below, we currently inflate Latin1 strings to TwoByte Gecko strings and that was also a potential performance risk, but inflating Latin1 is much faster than inflating UTF8.

* Linear-time indexing: operations like charAt require character indexing to be fast. We discussed solving this by adding a special flag to indicate all characters in the string are ASCII, so that we can still use O(1) indexing in this case. This scheme will only work for ASCII strings, though, so it's a potential performance risk. An alternative is to have such operations inflate the string from UTF8 to TwoByte, but that's also not ideal.

* Converting SpiderMonkey's own string algorithms to work on UTF8 would require a lot more work. This includes changing the irregexp regular expression engine we imported from V8 a few months ago (it already had code to handle Latin1 strings).

<comment><USER25> > Linear-time indexing

This is great for ASCII when you know there's no such thing as combining characters/etc, but I would like to remind everyone reading this that there's no such thing as "linear indexing" of user-perceived characters in Unicode.

User-perceived characters need processing in order to index due to Grapheme clusters potentially using many code points together. ([https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundarie...](https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries))

For instance, (on my machine, a dark-skinned male teacher) is a combination of these characters:

\- U+1F468 Man \- U+1F3FE Medium-Dark Skin Tone \- U+200D Zero Width Joiner \- U+1F3EB School

And knowing what the byte index of the start/end of that character in a string cannot be done by just multiplying an offset by some multiple of number of bytes per character.

<comment><USER26> Isn't ASCII indexing constant time and Unicode grapheme indexing ~linear time?

<comment><USER25> Correct, i seem to have quoted the wrong part of the GP comment... I meant to address this quote:

> adding a special flag to indicate all characters in the string are ASCII, so > that we can still use O(1) indexing in this case

But I got confused and addressed the linear indexing part.

But yeah, ASCII is constant time, but you can't assume even UTF-32 can be constant time due to variable length grapheme clusters.

</comment></comment></comment></comment></comment></comment><comment><USER27> The most practical real-world solution across all operating systems is to use "UTF-8 everywhere", and only convert from and to UTF-16 or UTF-32 "at the last minute" when the string data is coming from or going into APIs that don't (yet) accept UTF-8 directly.

Also see: [http://utf8everywhere.org/](http://utf8everywhere.org/)

<comment><USER28> I've always wondered why Windows doesn't allow setting the legacy code page to utf8. It would've been a nice hack to get old apps to work with Unicode, assuming they were at least somewhat MBCS aware. If the programmer assumed a character would always be one byte it might not have worked, but then that program would also have failed with a Japanese code page.

<comment><USER27> There's a fairly new experimental setting in Win10 where UTF-8 is set as the system-wide codepage, allowing to call the "narrow string" functions with UTF-8 strings.

See [https://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows#U...](https://en.wikipedia.org/wiki/Unicode_in_Microsoft_Windows#UTF-8)

"With insider build 17035 and the April 2018 update (nominal build 17134) for Windows 10, a "Beta: Use Unicode UTF-8 for worldwide language support" checkbox appeared for setting the locale code page to UTF-8.[a] This allows for calling "narrow" functions, including fopen and SetWindowTextA, with UTF-8 strings."

<comment><USER29> From that same link:

 _Another popular work-around is to convert the name to the 8.3 filename equivalent, this is necessary if the fopen is inside a library function that takes a string filename and cannot be altered._

8.3 filenames for the win!

<comment><USER30> Though do note this is only an issue when using particular "narrow" functions. If you consistently use the recommended "wide" functions then none of these issues occur.

</comment></comment></comment></comment></comment><comment><USER31> One way to think about text, is as the only data we use that shrinks over the years. Every other type of data continues to grow.

The way this can be stretched to make sense is by thinking for example how much these have grown over 20 years.

Media (music, video, images) grows staggeringly because of 200p > 4k, 128bit mp3 to flac or whatever and there's more of it all that's gone digital.

Code size continues to grow quite significantly even.

But text? Even the massive wikipedia's size, only a fraction is text content.

Then as memory capacity also grows it's really a back and forth battle between computing resources and all other data, except text.

Hard to think of many places 4 byte chars would matter nowadays, maybe the tiniest devices for a few apps?

<comment><USER7> Conversely as CPUs get more and more powerful the overhead of a non-fixed- width encoding also becomes negligible unless you're processing a huge amount of text. And if you process enough text that it becomes a problem then maybe size would be an issue too.

Furthermore 4-byte chars also have the endianess problem of UTF-16.

I think the rule of thumb of "use UTF-8 unless you really have a good reason not to" still holds true today.

<comment><USER31> I think that's a valid and interesting point that the classic time versus space computer science trade-off needs to be considered in this case for large workloads.

One thing that seems different is that you also hear concern from developers about coding complexity and the desire to always have things to be as easy to reason about as possible. Yes these are things a library should just handle for you but it sometimes useful to know what's happening fundamentally or can sometimes allow for less surprises.

Nevertheless, if one of the cases you mentioned became significant I'm not sure that not many would say coding complexity should take precedence over your considerations.

<comment><USER7> I agree with you there, I think in the vast majority of applications these days when it comes to text processing simplicity and robustness beats other concerns.

It's unclear if UTF-32 beats UTF-8 there however. UTF-8's encoding is a bit trickier to deal with of course, but it's really not that bad. On top of that you have a few advantages such as not having to worry about endianess as mentioned previously but also not having to worry too much about alignment or corrupted input (you can re-synchronize UTF-8 at every codepoint since the leading byte has a unique bit pattern). And of course ASCII/C-string compatibility means that it's unlikely to be mangled by non-unicode-aware applications.

In any case the complexity probably won't be in the encoding, be it UTF-8, 16, 32 or anything else. UTF-32 might be constand-width codepoint but in general you don't really care about codepoints since they can be combined in various ways to create characters or things like country flags or complex characters with diacritics.

<comment><USER32> > [...] when it comes to text processing simplicity and robustness beats other > concerns. It's unclear if UTF-32 beats UTF-8 there however [...]

I always wondered how this affected simplicity and efficiency of different text editor implementation strategies. I'm mainly thinking about the variable byte lengths and how it affects anything related to character indices. I'm guessing if it seriously hinders a preferred strategy one would just convert to a plain codepoint of fixed word size internally at the cost of greater memory usage... which actually is not that bad if you only want to support the BMP without any fancy encoding which would only takes up 2 bytes internally.

</comment></comment></comment></comment></comment><comment><USER33> _"A long time ago, at least in computer time, in the far-flung era of 1989, the Unicode working group was really starting to get going"_

I find it strange to start in 1989, and not in 1980, when Xerox had a precursor to Unicode, or 1988, when employees from Xerox and Apple coined the name "Unicode" ([https://en.wikipedia.org/wiki/Unicode#History](https://en.wikipedia.org/wiki/Unicode#History))

It certainly is a bit unfair to not mention Xerox or Apple at all.

 _"that the JVM is gaining the ability to transparently represent strings in memory as Latin-1, a legacy codepage, if possible."_

"Compact strings" ([https://openjdk.java.net/jeps/254](https://openjdk.java.net/jeps/254)) shipped with JDK 9 in September 2017 ([https://docs.oracle.com/javase/9/whatsnew/toc.htm#JSNEW- GUID...](https://docs.oracle.com/javase/9/whatsnew/toc.htm#JSNEW-GUID- BA9D8AF6-E706-4327-8909-F6747B8F35C5))

<comment><USER34> This year is mentioned on the Wikipedia page and its references as the sort of "genesis date" of all the companies' involvement and is more or less when the UCS-2 tragedy began. (This is a post specifically about UTF-16 and UCS-2, not the entire history of Unicode. That'd be... considerably longer.)

 _I mention in another post_ (Correction: In a draft of a post.) I'm still stuck on JDK 8 for various reasons out of my control, and am generally unaware of changes made in 9 or later. I'll amend the post to reflect that the feature shipped.

</comment></comment><comment><USER35> And the tragedy of early adoption! Microsoft looks antiquated for using UTF-16 today, but it was actually pretty cutting edge when it happened. They adopted a lot of tech pretty early too, like XML. All in all, pretty fascinating.

Windows 10 recently added the ability to use UTF-8 for the "ANSI" legacy codepage, which is awesome! I accomplished similar prior using API hooking but a Microsoft solution can simply do a much better job across the board. Now the only reason to set the encoding to anything else is to avoid mojibake when using non-English software. (Dear Japanese software developers: please consider using wide win32 APIs :( )

</comment><comment><USER36> I was under the assumption, that Linux and Mac OS (size of(wchar_t)==4, ergo UCS-4, because it was clear from the beginning, that 16bit won't do it. A compromise "only" windows and java was willing to make. But they went out all the way through the stack.

Linux goes for byte stream encoding (e.g utf-8), because it isn't (/wasn't) doing any string processing. So, encoding efficiency beats algorithmic complexity of string operations. If you need an algorithmic more efficient encoding, that's up to your application, and ucs-4 was the way to do it.

<comment><USER33> Certainly on Mac OS, _sizeof(wchar_t)_ had nothing to do with the way text strings got stored. The Mac had an elaborate system for handling international text that used either bytes or 16-bit quantities for text storage, where the interpretation of what a byte or byte pair meant was stored separately from those bytes. [https://developer.apple.com/library/archive/documentation/ma...](https://developer.apple.com/library/archive/documentation/mac/pdf/Text.pdf):

 _"The character codes may be 1-byte or 2-byte codes, but there is nothing else in the text stream except for those codes. Using font information that your program stores separately, Script Manager routines can help you determine whether a character is 1 or 2 bytes, and other managers allow you to work with either character size."_

More cumbersome to use than modern Unicode API's? Definitely, but it was superior to what Windows provided for years.

</comment></comment><comment><USER34> Wasn't really expecting this post to leave my circles, sure am glad now I checked my sources.

Look, ma, I'm on Hacker News!

</comment><comment><USER37> Thanks to Microsoft, it is now entrenched in the JavaScript and even the Language Server Protocol. Eventually they should just put UTF-16 on fire, and move to UTF-8 as all smart people did already. Microsoft was always who shambles behind progress with the lack of C99 support until recent times, UCS-2, MAX_PATH (MSBuild still doesn't work with long names, in 2019!), and many other similar problems.

<comment><USER38> Yes the language server protocol is particularly insane because they use UTF-16 word positions (sort of, it's JavaScript so it's not anything standard really), but the actual text data is sent as UTF-8! So for my sane UTF-8 speaking language server the text gets converted from UTF-8 to UTF-16 when loaded from disk, then converted back for transmission via the LSP, then converted back to UTF-16 by my language server so I can handle the positions, and then it does everything in reverse again on the way back!

Absolute madness. I think they will fix it eventually though. The madness is too great to ignore.

</comment></comment><comment><USER39> So the next time you're asked in an interview to implement isPalindrome, be sure to ask if you need to support surrogate pairs and diuretics.

</comment><comment> java-man Mojibake:

[https://en.wikipedia.org/wiki/Mojibake](https://en.wikipedia.org/wiki/Mojibake)

love the word.

</comment><comment><USER40> I remember when the Linux distributions switched to unicode implementations of tools and there was a noticeable speed reduction of things like ls. Fortunately the processor speeds kept on improving.

</comment><comment><USER41> I've been wondering, how much processor overhead does UTF8 have? Not just because you can't jump N bytes into a string to get a character offset, but because each byte has to be inspected to find whether it's the start of a multi-byte segment, how many of the subsequent bytes 1-3 are included.

Does that mean there's processor branch-prediction fails and stalling every few bytes in processing UTF8 strings with many multibyte characters in them?

<comment><USER42> > Not just because you can't jump N bytes into a string to get a character > offset

I keep hearing people refer to this operation (mostly in HN threads), I have yet to see an appropriate use of it. So far the only uses I've seen were (for Unicode) misguided attempts at character iteration, which doesn't need this at all.

<comment><USER43> At work we have to parse various different fixed-width column files. We only need data from a handful of the columns, so we use the substring function to extract the data we need.

Of course these files are so old they predate proper encoding and are implicitly Windows-1252 or similar.

<comment><USER44> That's been my experience, I've done it but never anything that a scanning or tokenizing algorithm couldn't do.

So being able to iterate over characters is important, but constant time access to a specific character seems _very_ rare.

I think the only case where I've wanted a specific character was when I wrote games that loaded maps from text files and those were ASCII regardless.

<comment><USER43> Some of the records are over 1k chars wide, and we need stuff near the end.

Though if one had to decode the line from UTF-8, you're iterating anyway so...

<comment><USER44> I could see a case for it if you were finding line endings in bytes mode, and then casting a buffer to a UTF-8 string. Then you'd benefit from a reverse iterator, which UTF-8 supports just fine.

I imagine the main reason people would use indexing is to scan through strings in lockstep, e.g. to implement comparison, but iterators can cover that case as well, especially if you can copy them.

</comment></comment></comment></comment></comment></comment><comment><USER45> Reading this, I'm thinking if it's possible (and if yes, how), to write java and javascript program in other encodings that they internally use, for example UCS-4. If yes, what is doing the conversation (parser, compiler?).

<comment><USER34> I've personally done this in Java, as mentioned. Generally I use fastutil to create an IntList (which is just an int[] with some conveniences) and use Java's Character class to convert between UTF-16 and UCS-4 on-the-fly.

It's not very fast, but it makes flawless Unicode support easy. :P

</comment></comment><comment><USER46> There's other tragedies too. SCSU lets you reasonably compactly represent strings, but Java doesn't use it so...

</comment><comment><USER47> > _Unix and Linux conveniently sidestepped this whole issue by just shrugging > their shoulders and going "eh, a char has no defined encoding, it's just a > number."._

Wrong: that's a feature of C, which all these systems are based on.

wchar_t is 32 bits on Linux, and, though also just a number, it commonly represents a Unicode code point.

<comment><USER48> I would assume that what author meant by this is that on Unix for the kernel itself names of various entities (filenames and similar things) are simply sequences of bytes that it does not ordinarily try to interpret in any way. (On the other hand SUS specifies that only very limited set of characters is valid in filename, but no "true unix" actually cares about that)

On Windows NT filename is specified as sequence of 16 bit wchar_ts and is supposed to be valid NFC UTF-16. OS X also expects that filenames are valid sequences of unicode codepoints and in this case NFD.

<comment><USER30> > On Windows NT filename is specified as sequence of 16 bit wchar_ts and is > supposed to be valid NFC UTF-16.

I would emphasise that the "supposed to be" is a convention only (similar to the Linux UTF-8 convention). The only filesystem enforced rules are that file/directory names can't include \/:*?"<>| or wchar_t(s) less than 0x20 (aka ascii control characters).

A Win32 path can also have additional rules (such as a file cannot be called "COM") but these aren't always enforced, depending on how the API is called.

</comment></comment></comment><comment><USER49> Unrelated: I love the theme of that blog. Anyone know if it's open source?</comment>


Morris Algorithm (2014) - <AUTHOR> http://page.mi.fu-berlin.de/kashif/algorithm/2014/10/17/morris-algorithm.html

<comment><USER2> I like that this algorithm shows the strength of non-determinism. You simply cannot achieve lower than O(log n) bits in a deterministic way for a counter.

It's also neat that you can trivially extend this to O(log log ... log n) by making the increment probability 2^(2^(... 2^(-X), all the way up to O(log* n).

To the ones questioning the usefulness of this, note that this is essentially equivalent to floating point arithmetic, where X is the exponent (and eps dictates mantissa size). This shows you can do all sorts of unbiased operations just like you would with floating point values. For example, you can add counter A and B e.g. A=10 B=8 by taking C="A+B"=max(A,B)+{1 with prob 2^-(|A-B|+1)}=10+{1 with prob 2^-3}.

</comment><comment><USER3> Honest question: What's the use of compressing the counter for n smaller than log(n) bits? Even if you were counting something on the order of 10^30, log n is only around 100 bits. Wouldn't storing the algorithm take more space than you'd save through this counter?

<comment><USER4> I think it depends on how many counters you want.

For example, I might want an approximate counter of number of reads of each row of a cache. Assume my cache has a lot (millions?) of rows, each of which stores a small amount of data. With the given algorithm, I could store the counter in a single byte per cache row. Without it, I'm probably looking at 4 or 8 bytes. It adds up.

As an aside, the article presents an alternative in which you have multiple simultaneous counters and take the median of the results to get finer bounds. I wonder how this compares with just using a single counter containing more bits and modifying the probability that you "increment" the counter?

[Edit: I realize the answer to my question about modifying the probability of a single counter may be that it's hard to get an unbiased random source for probabilities of 1/3, 1/5, etc.]</comment></comment>


Hillary Clinton Blasts Edward Snowden for Fleeing to Russia and China - <AUTHOR> http://www.motherjones.com/politics/2014/04/hillary-clinton-snowden-nsa-russia-china

<comment><USER2> Well, maybe we should have made the USA more hospitable than Russia and China!

</comment><comment><USER3> The NSA for sure has dirt on Bill.</comment>


Apple giveth, and Apple taketh away – the Escape key - <AUTHOR> http://www.jeffgeerling.com/blog/2016/apple-giveth-and-apple-taketh-away-escape-key-macbook-pro <comment><USER3> Wait! No! Caps lock is for control!

I like my mac, but i guess this is my last one. My hands are cold and sweaty, so they aren't perfectly reliable with touch devices. No escape key makes me sad. It might make Apple sad, but they'll probably just cry themselves to sleep on their giant piles of money.

<comment><USER4> You can make caps lock both escape and control

[http://www.economyofeffort.com/2014/08/11/beyond-ctrl- remap-...](http://www.economyofeffort.com/2014/08/11/beyond-ctrl-remap-make- that-caps-lock-key-useful/)

<comment><USER3> That, that is very clever. Ctrl when pressing chords and esc when used alone.

<comment><USER5> Beware that one of the tools mentioned in that link, Karabiner, does not work on macOS Sierra.

</comment></comment></comment></comment><comment><USER6> I wish Apple would taketh away one of the modifier keys instead.

They have Function, Control, Alt, Option, Command and Shift.

Regularly I forget which one is which so I just start spamming them all.

And I can never remember what ⌥ is supposed to mean.

<comment><USER7> Aren't alt and option the same thing, or have I been missing some nuance for several decades?

BTW, I think ⌥ is supposed to look like scales, representing "option." That's how I've always remembered it, anyway.

<comment><USER8> Actually, now that I look at it afresh doesn't it remind you of a railroad switch. Can't believe I never noticed that before.

<comment><USER9> It's the electricity symbol for a switch.

<comment><USER8> Gotcha, thanks. Seems to be a near variant of a SPDT toggle switch, Single Pole Double Throw. Can't tell if it is technically an electrical symbol or an electronic symbol.

[http://www.rapidtables.com/electric/electrical_symbols.htm](http://www.rapidtables.com/electric/electrical_symbols.htm) (fifth symbol down)

[https://en.wikipedia.org/wiki/Electronic_symbol#Switches](https://en.wikipedia.org/wiki/Electronic_symbol#Switches) (second symbol along)

</comment></comment></comment></comment></comment><comment><USER10> Apple might have made some missteps lately, but they're not stupid.

The point of the "magic" toolbar is to help desktop users finally leverage the useful things you can do with contextually aware keys/controls, which is more in line with how humans approach things, and a massively under-recognized part of the reason that iPads are so much easier for non-computer people to use than PCs or desktops.

See that 'cancel' button in the screenshot? That will _be_ the escape key, under basically all circumstances that you would want an escape key. Here it says "cancel" which is more descriptive and is exactly in line with the language used on all the buttons and dialogues the user will see on the screen. "Cancel" is what it actually does. "Esc" isn't.

Especially if the user is currently in a terminal, or VIM, I would expect the key to show the 'Esc' label and continue as normal. Under bootcamp, or under older macOS applications that don't attempt to interact with the Magic Toolbar API, I would also expect that area to show the 'Esc' key.

I'm looking down while typing this on a Microsoft keyboard and I see 5 giant "My favourites" buttons and some weird "star" button, and a calculator button, all of which I have never used, and which sit above the mouldering function row which again is used fairly infrequently by me, and basically never by my family members. That is not how to design an interface.

</comment><comment><USER11> I just gave up on Apple ever shipping you are MacBooks and received my custom order HP two days before they announced the October 27th event. My (magnesium unibody) ZBook is as slim as my retina MBP, has a higher-PPI display, also comes with a glass trackpad, has user-replaceable battery, 2x M.2 PCIe SSDs, and upgradeable ram. I was able to pay a bit extra and get it with a mobile Xeon CPU (E3-1545m with Intel's top-of-the-line Iris Pro integrated GPU) which is the equivalent of the i7 6920HQ only with more cache and better graphics, meaning I was able to buy 32 GB of ECC RAM for only $200. It has a 4GB nVidia Quadro and still manages to weigh less than my rMBP.

The only thing that sucks is the noise. It's quite even with the fan running at its highest RPM, but the frequency of the resulting noise is very distinguishable and it has a tendency to rev up and down quite suddenly (and often). It doesn't help that there are two fans, one on each side, which turn on and off independently - meaning you can suddenly feel like you've lost hearing in one of your ears until you realize the noise level is imbalanced. I don't know if Apple will introduce Xeon workstations, but even if they did, I'm not sure I'm ready to give up my three USB 3.1 (non-C), three thunderbolt 3 (/USB-C), power, gigabit Ethernet, and 3.5mm ports in exchange for a more- pleasant audio profile.

Did I mention I've been a faithful Mac user for over a decade?

<comment><USER12> Won't you miss macOS? It is not just about the hardware, it is the combination of the macbook with the OS.

<comment><USER11> I used Mac because it gave me a great UX with access to all my favorite command line tools and a posix API I could build against. With Windows 10 _finally_ shipping a real, supported posix target (Ubuntu for Windows - <USER2> an unfortunate name) vs the years of Interix (R) which was discontinued and replaced by "Linux subsystem for Windows" which was discontinued without a replacement until Nadella's "Ubuntu/Bash for Windows", I now no longer have need for OS X.

Plus, Apple doesn't care about macOS. HFS has bitrotted away, the OS itself has taken backseat to mobile, and Apple doesn't know if it cares enough about its enterprise and power users to even release "point releases" of their products - how much engineering tech does it take to replace a Haswell CPU with a Broadwell with no fanfare or ado? Just do it and make it available?

<comment><USER13> > Plus, Apple doesn't care about macOS. HFS has bitrotted away

HFS is already being replaced with APFS which is excellent feature-wise, and is available in Sierra for testing purposes. I expect at this time next year we'll be seeing them use it by default on new macOS and iOS devices.

</comment></comment></comment></comment><comment><USER14> Apple, stop removing things we need and replacing them with BS! Esc is an important key, and I don't need less modifiers on my keyboard (I actually need more - i3 takes up a good bit of S, and Emacs uses CMS: I could do with a Hyper and an Fn to make things less cramped), and while Esc is important, it doesn't deserve the same kind of prime placement as Ctrl does unless your a vim user. I guess swapping it to Ctrl and swapping control to Capslk could work, but Esc really should be bound by default.

</comment><comment><USER15> I was looking forward to buying the new MacBook Pro, but now I'm not so sure. Lenovo replaced the top row on their Carbon X1 laptop with digital keys a couple of years ago, and the results weren't that good. They were hard to use and unpractical. Hopefully Apple's implementation will work better.

<comment><USER16> Honestly bough how often do you hit escape on a mac? So having it as a digital key won't be the end of the world.

<comment><USER17> >50x a day, probably. I practically live in Vim.

<comment><USER18> maybe you mean 50k (not x) because in Vim it's used every 10 seconds, which makes it several thousands times a day

<comment><USER17> yes

</comment></comment></comment></comment></comment><comment><USER19> "We have no escape", said one of my Mac-wielding colleagues.

So true now.

(Full disclosure: I've been writing at taoofmac.com for over 14 years now, and even though I get why Apple is doing this, still find it dumb because I live inside a UNIX terminal most of my time. I'm now considering switching to Linux as soon as I can find a decent mail client, because, like Jamie Zawinsky, I refuse to read my email in a browser, like an animal.)

<comment><USER20> Just curious, what mac-only terminal email client do you use now? I imagine that most terminal email clients work fine on Linux. Or maybe I am reading something incorrectly from your comment?

<comment><USER21> I think he means that Linux email clients are not that good, and Apple Mail is pretty decent.

<comment><USER20> That makes much more sense.

</comment></comment></comment></comment><comment><USER22> I'm a bit more concerned that their isn't a dedicated power button. My Mac has froze a dozen times in the last year, and I'm pretty sure a soft button isn't going to be working.

<comment><USER23> Maybe for a hard reset there will be a new key combination. Like Ctrl+Alt+Del

<comment><USER24> Or a tiny inset reset button you need a paper clip for, like on some tablets/smartphones.

<comment><USER25> I've ruined a laptop microphone that way once, thinking it was a reset button... :P

</comment></comment></comment></comment><comment><USER26> Lenovo released an X1 Carbon in 2014 with a similar touch strip and backtracked a year later. Personally I'm glad they did.

<comment><USER27> This is enough of an example to make one wonder why they would go down this path. It makes me sad as I really do enjoy my MBP, but as a developer, these changes are making the hardware unusable.

<comment><USER28> If it wasn't for the touch bar all we'd be getting is a slightly thinner shell and a slightly faster previous-generation CPU, just as the PC industry starts getting backpack desktops, desktop GPUs in laptops, OLED screens, touch screens, removable screens that become tablets, VR headsets, external GPUs, backlit keyboards you can adjust individual key colors on etc.

PC industry is definitely missing out.

<comment><USER29> > backlit keyboards you can adjust individual key colors on

Just out of interest but what, other than novalty, is the purpose of this?

I've seen it on a Razer gaming laptop. Is it just an entertainment thing or is there some UX using this?

[edit] this is a genuine question BTW and not a weak attempt at snark.

</comment></comment></comment></comment><comment><USER30> I wonder if the reason there is no corresponding Xcode for the iOS 10.1+macOS 10.12.1 release is that they want to hold the official reveal of the magic toolbar. Surely the latest Xcode macOS SDK will have some sort of APIs for configuring the display/graphics/icons on the magic toolbar. Sucks if you're developing for iOS, because unless you want to side-load beta versions of Xcode, you will need to choose between being able to debug on the device or being safe from the remote code execution bug in the JPEG decoder on anything less than iOS 10.1...

<comment><AUTHOR> Not to mention it seems they're withholding a release of the Network Link Conditioner[1] that works with Sierra along with Xcode. There's currently no reliable method for limiting the entire Mac's upload or download bandwidth under Sierra :(

[1] [http://nshipster.com/network-link- conditioner/](http://nshipster.com/network-link-conditioner/)

</comment></comment><comment><USER31> The original Macintosh keyboard didn't have an escape key.[1] Or function keys. Or cursor keys. Or a numeric pad, though you could buy one as an accessory. You're supposed to use the mouse.

[1] [http://www.retrotechnology.com/herbs_stuff/128k2.jpg](http://www.retrotechnology.com/herbs_stuff/128k2.jpg)

<comment><USER19> I used (and refurbished) a bunch of those. But I switched to the Mac full time because of UNIX, not when it ran cooperative multitasking hacks atop a 680x0 CPU.

</comment></comment><comment><USER32> For Excel users, a shortcut that I think few people are aware of. When editing a formula, select part of the formula, and press CAPS LOCK+F9, it will evaluate the value of the selection.

...the CAPS LOCK is not that useless!

</comment><comment><USER33> I just realized that there are still four keys on the bottom left side (next to space).

Until now this used to be: Fn, Ctrl, Option/Alt, Cmd

Since Fn is obsolete now, due to the lack of F-keys, it makes me wonder what the most bottom left key is.

Maybe (hopefully) they moved the Esc there.

<comment><USER34> maybe the function keys accessible with fn+number, thinking about it it makes sense.

</comment></comment><comment><USER35> I have the CAPS LOCK mapped to the escape key, which is great for Vim, etc. However, there's one weird edge case problem. In After Effects the CAPS LOCK is the one quick way to speed up rending of animations, by disabling the render to screen option. It's oddly hard coded to that key specifically and seems difficult to remap to anything else. I know I probably represent 1 of 10 people world-wide who have this problem, but I'm sure there are other odd things hard mapped to the sad ol' CAPS LOCK key.

</comment><comment><USER36> Crazy Vim user here: caps lock is ctrl. jk or kj or fd or something like that is esc.

<comment><USER37> Ctrl-[, otherwise you get Vi Finger from reaching all the way up to Escape all the time. I don't even use they Escape key anymore except to cancel things like GUIs.

<comment><USER38> I like having Escape in the upper left because I can just blindly whack at that corner of the keyboard. Hitting '[' with my pinky is much less reliable.

</comment></comment></comment><comment><USER39> I feel like I'm the only person who uses Caps Lock. When I learned to touch type, I was taught to use the shift key on the side opposite the letter being typed. When typing more than two capital letters in a row, I'd rather hit Caps Lock than paddle my pinkies back and forth. It's a hard habit to break.

</comment><comment><USER40> I use an external keyboard with my mbp mostly, but I use my escape key more regularly than all my F-keys and the fn/command buttons. I really don't think I'd buy a laptop that was missing what I consider to be a key component like that. I'm sure there's a contextual variant, but there's a pretty big difference between a tactile button and whatever haptic OLED nonsense that Apple put up there.

This might be good for their consumer target audience, but I don't think it's good at all for their main proponents - techies and techie students. I'm very surprised that this issue didn't pop up internally. I wonder about their internal culture if this kind of a design went this far.

</comment><comment><USER41> This stuff always brings me back to "there's nothing more simple than one giant button" [https://www.youtube.com/watch?v=9BnLbv6QYcA](https://www.youtube.com/watch?v=9BnLbv6QYcA)

</comment><comment><USER42> Remapping Caps Lock to ESC at the OS level is not a viable proposition for international users.

Source: am French with an AZERTY keyboard, we need Caps Lock to make capitalised accented letters, such as in "À bientôt !"

</comment><comment><USER43> I'm really in need of a new Macbook and don't use Esc that much... but Karabiner/Seil don't support Sierra yet so I can't do Caps Lock -> Esc + Ctrl (please someone correct me if I'm wrong). That said, I have my own mechanical keyboard I use anyway so it's not _that_ bad. I'll be the guinea pig for this new kind of model.

Here in HN we can all say "I use X for Esc!" but this is going to be a problem to the myriads of developers and regular people that don't know about these tricks.

</comment><comment><USER44> "Apple giveth"

Did Apple really invent the escape key?

<comment><AUTHOR> They didn't have it on the original Apple keyboard: [https://en.wikipedia.org/wiki/Apple_Keyboard#/media/File:App...](https://en.wikipedia.org/wiki/Apple_Keyboard#/media/File:Apple_Macintosh_Plus_Keyboard.jpg)

<comment><USER29> I believe that's a Mac keyboard. The original Apple I and II keyboards had an Esc key.

[http://www.hp9845.net/9845/history/comparison/images/closeup...](http://www.hp9845.net/9845/history/comparison/images/closeups/apple2-top- view.jpg)

The modern Mac owes as much to NeXT as it does to the classic Mac. NeXT also had an escape key ...and the ctrl key in the right place! :)

[http://xahlee.info/kbd/i/NeXT_Computer_keyboard_66682.jpg](http://xahlee.info/kbd/i/NeXT_Computer_keyboard_66682.jpg)

</comment></comment></comment><comment><USER45> The position of the shift-lock key is terrible for something as powerful as Escape, for most users, especially those with bear paws like mine. An accidental press can have grave consequences. There's a reason it's at the corner of the keyboard. And it's the one key everybody can find automatically.

</comment><comment><USER46> I'm very dubious of this bar being added to Macbooks. What % of apps are really going to make use of it?

<comment><USER22> The 10.12.1 update would seem to disagree with you. [http://www.macrumors.com/2016/10/25/images-of-new-macbook- pr...](http://www.macrumors.com/2016/10/25/images-of-new-macbook-pro-leaked/)

<comment><USER46> I didn't mean dubious as to whether it exists, just that I'm dubious as to whether it should.

<comment><USER22> I get the feeling that many of us will wish an Apple designer had been dubious as to whether it should.

</comment></comment></comment></comment><comment><USER47> This business about the caps lock key being "useless" has been very puzzling to me. As someone who learned how to property type, I utilize the key quite a bit -- especially when programming.

</comment><comment><USER48> Man. As a emacs user who doesn't use meta key, this is sad!

</comment><comment><USER49> Why wouldn't the bar have a software escape key when showing function keys?</comment>