XSS Twitter in minutes; Why you shouldn't store important data with 37signals - <AUTHOR> http://brian.mastenbrook.net/display/36 <comment><USER2> I'm usually the first person here to jump to 37s' defense. I know some of their people, they're hometown heroes, and I use and like their products.

This is hard to defend, guys.

It is literally the-simplest-thing-not-to-fuck-up. Nobody's asking you not to have security vulnerabilities. In fact: nobody's even asking you to _fix_ vulnerabilities. We just need a reliable way to communicate with you about them.

If you're selling accounts on a web app, you need:

* A security page * With a PGP key * And an email contact * of someone who will write back * who knows what a security vulnerability is * and who will write back quickly.

That's it. Do that, and you're not a punch line. If someone dumps zero-day about you onto Twitter, you're already two steps ahead in the PR war, because you had a reasonable process, and the researcher ignored it.

Bonus points --- things that are trivial to do, but that nobody's even asking you to do:

* You can assign special issue numbers to vulnerabilities, to make the researcher feel like an XSS disclosure isn't the same thing as a bug in your online help.

* You can thank researchers privately, and let them know that you'd really like them to keep disclosing thing to you --- you could even give them (wait for it) a phone number.

* You can do what every vendor with a real security team does, and keep a public web page thanking people who have discreetly disclosed vulnerabilities to you.

<comment><USER3> Actually it's pretty easy to defend.

I never used to have travel insurance but after my first intercontinental flight my suite-case broke and I learned the importance of having one.

In a perfect world your support team would be able to distinguish between someone rambling about bytes and an actual security issue. That's hard without a lot of technical knowledge.

I believe that's why the Rails security team responded rather quickly and 37 Signals support team didn't. I'm sure they will do better in the future.

<comment><USER2> The problem isn't that 37S tech support doesn't know how UTF-8 works. The problem was that security reports were routed to tech support in the first place. Again, the solution to this problem is a single web page with just a couple pieces of information on it.

</comment></comment></comment><comment><USER4> Brian, I'm on the receiving end of email@example.com and @rubyonrails.org. I read your post with great dismay, to put it mildly. You're understandably pissed: we whiffed on our response to you by changing venue to Rails security without keeping you in the loop.

This is my fault. I identified it as a Rails issue and requested that you forward your findings to the Rails security team so we could investigate in concert.

Craig here at 37s narrowed down a root fix with Michael, Rails' security ombudsman, who then enlisted Manfred's help to track down and repair the root cause. What you see today is the end result of those efforts. The security process worked, but you only saw the Rails arm of it. The apparent 37signals arm of it amounted to runaround. Completely not OK.

We now have a security-only email and PGP key at <http://37signals.com/security>. Next time, no runaround.

<comment><USER5> There are still a couple of issues Brian brought up you haven't addressed.

The main one being the hubris of the copy on your security page. Declaring users data to be uncompromisable then justifying this by listing mostly _physical_ restrictions to the datacenter seems to ignore rather the larger security issues for web-based applications. A firewall and latest security patches do not make one immune.

His other peeve seems to be the perceived bullshit of your support team saying they replied to his initial complaint when it appears (at least to him) that they had not, then putting the blame for this on his spamfilters.

<comment><USER6> @tptacek:

Second Google result for "apple security": <http://www.apple.com/support/security/>

<comment><USER2> Yes, and that's a good page. Now tell me how to navigate to it on the OS X site, and note how much security marketing fluff you'll see before you ever find it.

I don't even think Apple is a bad example of the form. I think it's entirely reasonable for them to market security on their main pages, and leave the researchers to find their support page on Google. There are tens of researchers, and millions of customers.

Apple has a lot of really smart people working in security research and software security. Some of them are friends of ours. And some of those people are frustrated with Apple for any number of reasons. But none of them --- in fact, nobody I know that works in software security --- is particularly upset about <http://www.apple.com/macosx/security>. It is what it is.

</comment></comment></comment></comment><comment><USER7> Funny, but I've always thought the 37signals team was bipolar.

On one hand, they're pretty much "our way or the high way" about any support concerns/feature requests.

But, once caught out in public about any issue, they're all over it.

I suggest a very strong dose of self-administered humility for their founders would go a long way. That doesn't mean being un-opinionated; it just means being realistic about human nature and its vagaries as it applies to themselves.

</comment><comment><USER8> For all the horrible, terrible, awful things about Internet Explorer (namingly: standards support and UI) ... they are innovating heavily in security: [http://www.microsoft.com/windows/internet- explorer/features/...](http://www.microsoft.com/windows/internet- explorer/features/safer.aspx)

<comment><USER9> Microsoft is actually decent at security now. The AV they released for free was actually on par with a few commercial products out there (all AV at the moment is pretty bad though, if you're curious). The really difficulty with Microsoft and security is countering their reputation for bad security that they earned over the past several years.

<comment><USER8> That's because Microsoft is good at anything that they throw resources at. Unfortunately, we haven't thrown enough resources at determining what to throw resources at.

<comment><USER9> Well, good enough at least. ;-) Honestly the AV was a good move. Give stuff like that away for free for long enough and the public perception of Microsoft and security may change.

<comment><USER10> On the other hand, over the years people have hated them and filed antitrust claims against them for including free stuff in their releases. Norton used to make a file manager before the Windows one was any good...now their AV product faces a built in competitor.

</comment></comment></comment></comment></comment><comment><USER11> I read through the code, but I don't really know Ruby that well, so I'm not 100% sure of what the exploit was.

Looking at the patch, it appears that their "check for UTF-8" function wasn't perfect. Is this correct? If so, how is this exploitable?

<comment><AUTHOR> As I understand it, Rails' string escaping would treat an invalid byte sequence (eg, 0xFF, 0x1C) as a single multi-byte code point, and thus not filter it, even though 0x1C (which is '<') should have been escaped.

The browser, however, would correctly treat 0xFF as an invalid initial byte, and then interpret the next character point, 0x1C ('<') independently.

So, you could pass arbitrary characters through Rails' string escape functions by prepending an initial invalid byte sequence, and thus cause the browser to interpret arbitrary JS/HTML.

<comment><USER11> Thanks for the response. That does make sense.

</comment></comment></comment><comment><USER12> Can someone help to compile a good security policy and guideline for web apps?

I guess every web app should have a page dedicated to security similar to privacy policy and terms of service.

What are the essential information should go there? Special email dedicated to report security issues? PGP key to encrypt emails? and What else?

<comment><USER13> Don't trust any data that comes from outside your app (urls, querystrings, http request headers, databases, files) and you'll be right.

Perl's Taint Mode enforces this automatically. Don't know if any other languages have it? <http://www.webreference.com/programming/perl/taint/>

<comment><USER14> I don't think taint mode is helpful for this. All data comes from outside of your program, but you somehow still have to display it. As we've seen recently, this is hard to get right. Escaping HTML is one thing, but what if you want to let a user type in a URL? Make sure to exclude javascript: URLs. Even if you whitelist only "<http://...> URLs, how do you know that a browser bug won't allow an attacker to inject JavaScript, compromising any account used by a user of that web browser?

Basically, web browsers need taint mode. The programming language that produces the web page is a whole other issue.

</comment></comment></comment><comment><USER14> I think what we've really learned from this is that the current JavaScript security model is not good for what people are using the web for these days.

We really need something like a "<without-scripts>" block, where anything inside could never use JavaScript (including links that are to javascript: URLs). This would make life a lot easier for web developers.

<comment><AUTHOR> Then XSS attacks would insert </without-scripts> before <script></script> =)

What we need is literal separation of control statements (eg, <script>) from content such that neither can be easily misinterpreted, but that would be a significant departure from existing design.

<comment><USER2> That shouldn't work for the same reason that you can't escape a bound SQL query parameter in a pre-parsed query.

<comment><USER15> A modernized, preferably static version of perl's taint mode might work. You need two types of strings, one for trusted and one for untrusted strings, all your output functions accept only trusted strings, all your input or request parsng function return untrusted strings, and all naive string manipulation functions return untrusted strings if at least one of their arguments are untrusted. Then the possibly vulnerabe code is limited to a few statically identifiable routines that take untrusted strings and return trusted ones. They still may be buggy, but at least you know which parts of your code may induce vulnerabilities and need special attention.

And with modern type systems, these types might even be phantom types incurring no runtime overhead. Athough things like the differences between "safe for passing to a browser" and "safe for passing to my SQL-server" might compliate the architecture.

<comment><USER16> Ruby has a taint mode for String too. I think it's used in Rails, or maybe they rolled their own, but the concept is definitely there.

Problem is, at some point you have to be able to display user-entered data. You indeed mark it as tainted, or equivalent, then escape it as best you can. The issue here was a bug in the escaper. Tainting was working as planned.

</comment></comment></comment></comment></comment><comment><USER17> You found a security exploit, feel special. Finding an exploit isn't a voucher to rant against the people responsible for it. The bottom line is that nobody can be 100% sure that their data is secure after they've put it in the hands of a third party.

<comment><USER18> I think my point was a lot more nuanced than you give it credit for.

<comment><USER19> There's an interesting tangent towards the end -

"Web application security is still an immature field, and many of the layers are sufficiently poorly designed that issues like this will pop up for a good long while. Just like buffer overflows have been a weak spot for C security as long as the Internet has been around, escaping issues will continue to be a weak spot for web security for as long as we're afflicted with this particular architecture."

It seems like a field not only in its infancy but also oddly unglamorous and under-reported. There's no repository (that I know of, at least) of vulnerability reports of major web apps, for instance, yet it's easy to look up an exhaustive history of Flash vulnerabilities down to the seventeenth decimal sub-version. And yet the various XSS/CSRF/etc vulnerabilities are easily as dangerous and as exploitable. Twitter's dreams of a billion users and a new internet were not exposed by a buffer overflow, after all.

<comment><USER2> I think you're probably wrong about that; more security practitioners are familiar with OWASP than with any other security advocacy/advisory group.

<comment><USER19> That's possible especially since I'm not a 'security practitioner' and I'm essentially talking about a subjective personal impression - that it's taken less seriously, is less reported and incidences of specific vulnerabilities or exploits in specific apps are not tracked in the way they are for operating systems and major applications. This may, in part, be because in the case of web apps fixes are immediately available to all users. On the other hand, you can head to the RoR download page right now and click your way to downloading the current vulnerable version of RoR. At no point will you get a suggestion to check for recent security advisories or patches.</comment></comment></comment></comment></comment>


Show HN: Triangulart – Isometric graphic editor, like pixel art with triangles - <AUTHOR> https://maxwellito.github.io/triangulart/ <comment><USER2> This looks like [https://www.marmoset.co/hexels/](https://www.marmoset.co/hexels/) (which is awesome)

<comment><USER3> Yeah, I immediately thought of Hexels too, which already has had a community and lots of artwork done with it for a few years.

I'd suggest having a few examples of artwork and screenshots in this landing page, it's very dry for an art creation tool.

</comment></comment><comment><USER4> Consider mapping right-click to erase or draw with a secondary color. MSPaint had this feature and it was ideal for "pushing around" pixels. I've sorely missed this functionality in most pixel art tools.

<comment><USER5> Aseprite has it :)

</comment></comment><comment><USER6> Doesn't work for me, just a blank screen.

<comment><USER7> Firefox seems to not like my hack on 'location'. There's a PR to fix it but I can't review it atm :-S

</comment></comment><comment><USER8> Is there a way to align the grid vertically? It's much easier to draw buildings etc if you can have vertical lines

<comment><USER7> Click on the 'orientation' label in the creation form ;)

</comment></comment><comment><USER9> This is really nice. Much nicer than the plotter apps I've tried to use recently. Lots of attention given to UX details. I needed something like this with square cels recently. The best web app I could find was kind of terrible. You couldn't save, and it slowed to a crawl after a hundred pixels or so.

</comment><comment><USER6> Why does the dropper tool not let you pick an already used color?

<comment><USER7> So when I was doing stuff with it, I realised I don't wanted to save in my palette all the color I used for the illustration. So add colors to the palette has to be a manual action from the user. But Im open to suggestions, you're not the first and not the last to complain about it :) [some people find it extremely confusing]

</comment></comment><comment><USER10> Similarly, I once wrote a triangular maze generator [https://www.khanacademy.org/computer-programming/triangle- ma...](https://www.khanacademy.org/computer-programming/triangle-maze- generator/1157873034)

</comment><comment><USER11> Would be great if there were a few samples available to load right off the landing page..

<comment> max-m Just placed some "pixels" :)

[https://gist.github.com/max-m/d255237679523b96ca094f885ae2ff...](https://gist.github.com/max-m/d255237679523b96ca094f885ae2ffe8)

</comment></comment><comment><USER12> A minor suggestion: give more buttons in the UI `pointer: cursor` and hover states; it will be easier to see when a click will actually do something (thinking in particular of the black bar of icons at the bottom of the page).

<comment><USER7> Oh yes, that's a silly miss from me :-O Thanks!!

</comment></comment><comment><USER13> Something like this could have a lot of use for the creation of quilt patterns.

</comment><comment><USER14> Cool.. could you make it so command+click deletes the cell?

<comment><USER7> You can select, then press 'backspace' on your keyboard. That should do the trick :)

<comment><USER14> Hmm.. that's not working for me on a mac. I can select a single triangle? option/command + click would be quite convenient vs select each individual cell... since you could drag to erase.

<comment><USER15> XOR-like behavior for the draw tool without any modifiers would work as well.

</comment></comment></comment></comment><comment><USER16> It loads a blank page for me when I start a new canvas.

<comment><USER15> Your ad blocker might need to be disabled. I also had the same problem, but it worked after adding an exception for the page in uBlockO (the extension icon did not indicate that anything was blocked, though).

</comment></comment><comment><USER17> Any examples of interesting drawings drawn with this tool?

<comment><USER7> [https://twitter.com/mxwllt/status/952982195261661184](https://twitter.com/mxwllt/status/952982195261661184) [https://twitter.com/mxwllt/status/901487884871532545](https://twitter.com/mxwllt/status/901487884871532545) [http://maxwellito.com/post/triangulart/](http://maxwellito.com/post/triangulart/)

Not sure it fit the `interesting` criteria ;-)

<comment><USER17> Cool! Thanks!</comment></comment></comment>


Russian Court Bans Telegram App After 18-Minute Hearing - <AUTHOR> https://www.nytimes.com/2018/04/13/world/europe/russia-telegram-encryption.html <comment><AUTHOR> I wonder why Whatsapp is fine, but Telegram is not.</comment>


Details of the Cloudflare outage on July 2, 2019 - <AUTHOR> https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/ <comment><USER2> Appreciate the detail here. It's a great writeup. Wondering what folks think about one of the changes:

      5. Changing the SOP to do staged rollouts of rules in
         the same manner used for other software at Cloudflare
         while retaining the ability to do emergency global
         deployment for active attacks.

One concern I'd have is whether or not I'm exercising the global rollout procedure often enough to be confident it works when it's needed. Of the hundreds of WAF rule changes rolled out every month, how many are global emergencies?

It's a fact of managing process that branches are liability and the hot path is the thing that will have the highest level of reliability. I wonder if anyone there has concerns about diluting the rapid response path (the one having the highest associated risk) by making this process change.

edit: fix verbatim formatting

<comment><USER3> Yep, that's the exact bullet point I was writing a response on. Security and abuse are of course special little snowflakes, with configs that need to be pushed very fast, contrary to all best practices for safe deployments of globally distributed systems. An anti-abuse rule that takes three days to roll out might as well not exist.

The only way this makes sense is if they mean that there'll be a staged rollout of some sort, but it won't be the same process as for the rest of their software. I.e. for this purpose you need much faster staging just due to the problem domain, but even a 10 minute canary should provide meaningful push safety against this kind of catastrophic meltdown. And the emergency process is something you'll use once every five years.

<comment><USER2> Your response highlights a good idea to mitigate the risk I was trying to highlight in mine.

They want to have a rapid response path (little to no delay using staging envs) to respond to emergencies. The old SOP allowed all releases to use the emergency path. By not using it in the SOP anymore, I'd be concerned that it would break silently from some other refactor or change.

Your notion is to maintain the emergency rollout as a relaxation of the new SOP such that the time in staging is reduced to almost nothing. That sounds like a good idea since it avoids maintaining two processes and having greater risk of breakage. So, same logic but using different thresholds versus two independent processes.

<comment><USER3> Right. The emergency path is either something you end up using always, or something you use so rarely that it gets eaten by bit-rot before it gets ever used[0]. So I think we're in full agreement on your original point. This was just an attempt to parse a working policy out of that bullet point.

[0] My favorite example of this had somebody accidentally trigger an ancient emergency config push procedure. It worked, made a (pre-canned) global configuration change that broke everything. Since the change was made via this non-standard and obsolete method, rolling it back took ages. Now, in theory it should have been trivial. But in practice, in the years since the functionality had been written (and never used), somehow all humans had lost the rights to override the emergency system.

<comment><USER4> My personal rule is that any code which doesn't get exercised at least weekly is untrustworthy. I once inherited a codebase with a heavy, custom blue-green deploy system (it made sense for the original authors). While we deployed about once a week, we set up CI to test the deployment every day.

Cold code is dead code.

</comment></comment></comment></comment></comment><comment><USER5> One thing that was interesting to me:

The outage was caused by a regex that ended up doing a lot of backtracking, which caused PCRE, the regex engine, to essentially handle a runaway expression.

This reminded me of a HN post from a couple months back by the author of Google Code Search, and how it worked: [https://swtch.com/~rsc/regexp/regexp4.html](https://swtch.com/~rsc/regexp/regexp4.html) . Interestingly, he wrote his _own_ regex engine, RE2, specifically because PCRE and others did not use real automata and he needed a way to do arbitrary regex search safely.

<comment><USER6> The problem is that a deterministic regex engine (deterministic finite automata or DFA) is strictly less powerful than a non-deterministic one (NFA). DFA's can't backtrack, for example. In addition, DFA's can be quite a bit slower for certain inputs and matches.

<comment><USER7> Actually, it is proven that NFAs and DFAs are equally expressive. See [https://en.wikipedia.org/wiki/Powerset_construction](https://en.wikipedia.org/wiki/Powerset_construction)

<comment><USER6> "You are technically correct. The best kind of correct."

In theory, your statement is perfectly correct. However, quoting that reference:

"However, if the NFA has n states, the resulting DFA may have up to 2^n states, an exponentially larger number, which sometimes makes the construction impractical for large NFAs."

This means that _in practice_ , DFAs are larger, slower, and sometimes can't be run at all if complex enough.

However, this was my mistake. I remembered (vaguely) the 2^n issue and didn't follow up to make sure I was accurate.

And I completely spaced on the fact that neither NFA's nor DFA's handle backreferences without extension.

</comment></comment></comment></comment><comment><USER8> 9\. We had difficulty accessing our own systems because of the outage and the bypass procedure wasn't well trained on.

Suggestion for future, learned from bitter experience: separate your control plane from your data plane. In this case, make sure that the tools you use to manage your infrastructure don't depend on that infrastructure being functional.

That way you won't have to remember how to use a bypass procedure -- it will just be your normal procedure.

<comment><USER9> Yeah, that is true in most cases. However, here is Cloudflare was using Cloudflare on dash.cloudflare.com as well.

This calls for not using Cloudflare for their web dashboard.

<comment><USER10> Then comes the inevitable tweets every few weeks. "cloudflare doesn't event trust cloudflare to run their own control panel"

<comment><USER11> couldn't it be named in order to indicate cloudflare uses cloudflare just in a different way.

</comment></comment></comment></comment><comment><USER12> > The Lua WAF uses PCRE internally and it uses backtracking for matching and > has no mechanism to protect against a runaway expression. More on that and > what we're doing about it below.

We run a WAF based on LuaJIT in resty. Just to be clear, the resty interface to PCRE does provide a DFA mode. Furthermore, Zhang actually ported RE2 (see other comments here) to C as sregex, which is usable from Lua as a c module regardless if it runs in resty or a custom Lua app.

> Switching to either the re2 or Rust regex engine which both have run-time > guarantees. (ETA: July 31)

Not addressed at Cloudflare, since they had a defense in place. But just in case anyone else is running a similar thing in Lua.

And:

> In the longer term we are moving away from the Lua WAF that I wrote years > ago.

Then sregex might be the perfect fit here. Though Rust is technically safer. Depends on what longer term means.

</comment><comment><USER13> > Unfortunately, last Tuesday's update contained a regular expression that > backtracked enormously and exhausted CPU used for HTTP/HTTPS serving.

One of those cases where they had 1 problem, used regular expression and ended up with 2 problems ?

Edit: I really like how much information is given by CloudFlare. 11 points in the "what went wrong analysis" is how every root-cause analysis should be done.

<comment><USER14> Somewhat humorous, as someone [1] (congrats /u/fossuser!) mentioned this failure scenario in the thread about Twitter being down yesterday.

"Pushing bad regex to production, chaos monkey code causing cascading network failure, etc.", in response to a comment from someone who previously worked at Cloudflare.

[1] [https://news.ycombinator.com/item?id=20415608](https://news.ycombinator.com/item?id=20415608)

<comment><USER15> They mentioned it was a regular expression in the original post[0] on the day of the incident, that part isn't news (discussion here[1]).

[0]: [https://blog.cloudflare.com/cloudflare- outage/](https://blog.cloudflare.com/cloudflare-outage/)

[1]: [https://news.ycombinator.com/item?id=20336332](https://news.ycombinator.com/item?id=20336332)

</comment></comment></comment><comment><USER16> > Then we moved on to restoring the WAF functionality. Because of the > sensitivity of the situation we performed both negative tests (asking > ourselves "was it really that particular change that caused the problem?") > and positive tests (verifying the rollback worked) in a single city using a > subset of traffic after removing our paying customers' traffic from that > location.

Haha, so the free customers are crash test dummies for providing test traffic. Nice.

I actually don't mind that much, considering it's basically bulletproof DDoS protection for free. I'd much rather "be the product" in this way than in the way ad companies cause at least.

<comment><USER17> Or you can say all customers were affected but some localized free-tier customers got the fix first.

<comment><USER18> In this case yes, however they also indicate this is how they do their staged rollouts in general. So if they are releasing any other software update that goes through the staged rollout free customers are tested first. If that change broke something, free customers get that first. Which seems fair to me.

<comment><USER19> In my experience it's generally best to roll out changes on testing, staging, and then clients in order of how much they pay, especially if you have SLAs with the highest paying customers.

Impact is generally lower, both to the client, and to your bank account.

<comment><USER20> That sounds strange to me. If you introduce a bug then roll back very quickly, it will only affect high paying customers. If you introduce a bug then roll back a while later, it will impact high paying and low paying customers equally. Why would you want this scenario? If you flip it it seems strictly better to me.

<comment><USER17> The idea is that the fix itself is being tested. If you knew your 'rollback' will work for certain, then you'd just deploy it to everyone asap. But since you don't, you test it and as a potential outcome of your test is no fix or making things worse, you don't test it on your highest-value customers. Imagine what your postmortem would read like if your fix made an even bigger mess.

<comment><USER20> Oh, I misunderstood you originally. I thought you said rollout from highest to lowest. You're actually saying lowest to highest.

<comment><USER17> I'm just describing what I think the sequence in the postmortem is. They were already in the poop and wanted to test their fix in a real but low-impact way.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER21> I love how the post basically concludes with: "This problem has been known since 1968, which is detailed in this paper written by Ken Thompson".

Incredible write up. Really appreciate the detail, and am really impressed by how mature their response coordination seems to be.

</comment><comment><USER22> Note: Golang's stdlib regexp ([https://golang.org/pkg/regexp/](https://golang.org/pkg/regexp/)) is guaranteed to run in time linear to the size of input. Russ Cox has a detailed article [https://swtch.com/~rsc/regexp/regexp1.html](https://swtch.com/~rsc/regexp/regexp1.html)

<comment><USER23> Golang's regexp is derived from RE2, referenced in footnote four of the post.

<comment><USER24> Russ Cox also wrote RE2: [https://swtch.com/~rsc/regexp/](https://swtch.com/~rsc/regexp/)

</comment></comment></comment><comment><USER25> I'd like to imagine that as long as we live, poorly written regular expressions will continue to be the cause of breaking issues.

<comment><USER26> StackOverflow also had an outage a few years ago that was caused by exponential blow-up of a backtracking regular expression.

[https://stackstatus.net/post/147710624694/outage- postmortem-...](https://stackstatus.net/post/147710624694/outage-postmortem- july-20-2016)

<comment><USER27> That blow up is quadratic, not exponential: "This is not classic catastrophic backtracking (talk on backtracking) (performance is O(n²), not exponential, in length), but it was enough."

<comment><USER28> Some years ago I tried to create an exponential time regex in Perl, and only managed a quadratic time one after a bit of experimenting.

But, as you said, quadratic is often already fatal on realistic data.

</comment></comment></comment></comment><comment><USER29> Always appreciate the transparency from you and Cloudflare. :)

My main fright during this outage wasn't really the outage itself, but the fact that I couldn't log into the dashboard and simply click the orange cloud to bypass Cloudflare in the meantime. I'm assuming that this is now covered by this mitigation:

>> 6\. Putting in place an emergency ability to take the Cloudflare Dashboard and API off Cloudflare's edge.

If so, and if this would have prevented the dashboard outage even during the WAF fiasco, this is a huge comfort to me. Just curious, though: how far can you really go in separating Cloudflare "the interface" from Cloudflare "the network?"

And in general, what does everyone on HN think about mission-critical companies using their own infrastructure and being their own customer? Especially when the alternative is using a competitor?

<comment><USER30> I've always been a proponent of separating the monitoring from the infra. Otherwise your insight is binary: the service is either up or down. You don't have an context as to why.

Edit: Additionally, from a competitive standpoint, I don't see a problem with using a third-party platform for a monitoring service.

<comment><USER31> Yes, this is always one of our primary questions we ask when deciding when and how we should dogfood our own services; will we create a circular dependency where our ability to fix an issue on one service is hindered by any chain of dependencies between the service with the issue and the service used to fix it. We always avoid those, or at least have easy alternatives.

</comment></comment></comment><comment><USER32> > In the last few years we have seen a dramatic increase in vulnerabilities in > common applications. This has happened due to the increased availability of > software testing tools, like fuzzing for example (we just posted a new blog > on fuzzing here).

So security/debugging tools _increased_ the number of [discovered/exploited] vulnerabilities, because developers don't use them. Only malware developers and third-party security researchers take the time to test security.

<comment><USER16> Yup, unless you're a seriously security or stability focused company you don't even use basic stuff like static analysis, let alone fuzzing. Tools for these are often either expensive, hard to use or both.

</comment></comment><comment><USER33> Awesome write up as usual John. I'm no expert so I was wondering: "In the initial moments of the outage there was speculation it was an attack of some type we'd never seen before." \- Is there a reason you would go to this first vs checking the last lot of code deploys first/and or at the same time?

<comment><AUTHOR> It was speculation. We were looking at sudden massive CPU use all over the world. Since we do staged deploys that shouldn't happen yet...

</comment></comment><comment><USER34> There really should a prize or something for best postmortems to reward companies for doing this & giving them some PR.

Most of us will (hopefully) never be in a situation like this so "book knowledge" of extremis events is the next best thing available. And that relies on good write-ups.

</comment><comment><USER35> More general questions I would consider asking:

1\. It appears there was a safe path with more safety and scrutiny, and a fast path with less. In this case, over time, the fast path became routine. Are there other places where this pattern could develop or has already developed? Is this tradeoff between speed and scrutiny actually necessary? (ie could you have urgent updates reach production faster but actually receive more scrutiny/more testing, even if that happens after the fact?)

2\. In a similar vein, if the system has a failsafe configuration (eg only changes that have passed the full barnyard, or configurations that have been running safely for more than a certain amount of time), would it be plausible to automatically roll servers back to that configuration if they remain unresponsive for a certain amount of time?

3\. It seems as though there are multiple points (big WAF refactor, credential expiry, internal services dependent on working prod) where a sufficiently cynical engineer would say "I bet there's something here that could, if not bring down the site, at least ruin someone's day". Is there a suitable voice for this kind of cynicism? Eg, a red team or similar? If you were Murphy's Law incarnate, messing with Cloudflare's systems to achieve maximum mischief, where would you start?

4\. I get the sense that there are many reliable and well-tested layers of safety, but is it common to test what happens if they fail anyway? Eg: let's pretend Cloudflare just got knocked out globally by a wizard spell, what do we do? Or let's say our staged rollout system gets completely bypassed because of solar flares, how bad is it? Beyond developing a procedure or training for these kinds of situations, are they actively simulated or practiced?

If anything, I'd guess the root root cause here is a success failure, where the system has been so reliable for so long that the main reactions to it failing are disbelief and unpreparedness. I'm sure it wasn't funny at the time, but it gives me a chuckle to imagine the SREs speculating about Mossad quantum-tunnelling 0days or something because the idea of everything falling over on its own is so unthinkable. Meanwhile, those of us without so many 9s would jump straight to "I probably broke it again."

<comment><USER36> Overall I think these are very thoughtful, and I upvoted your comment.

However, I don't think this question is very fruitful:

> let's pretend Cloudflare just got knocked out globally by a wizard spell, > what do we do?

The way you solve a production issue is you identify its cause and then contain, mitigate, or fix it. I don't think you'd learn anything useful from a drill where there's no specific cause.

Perhaps along similar lines to what you're thinking of, something I could see being useful is to look at components that you've already thought to implement a 'global kill' for, like WAF, for instance. Maybe you could run drills where every machine running WAF starts blackholing packets, or maxing out RAM, or (as happened here) maxing out CPU, the kind of thing where you'd want to execute the 'global kill' in the first place. That way, you can ensure that the 'global kill' switches are actually useful in practice. Something like that seems more grounded to me, making the assumption that something specific is going wrong and not just "magic", while still avoiding too-specific assumptions about what can and can't go wrong.

</comment></comment><comment><USER37> Maybe I'm not the intended audience, but I had to look up the acronym WAF since it came up a lot in this article. I'm assuming it's "web application firewall"?

<comment><USER38> Yup, I had to look it up too. Best reference I found:

[https://www.cloudflare.com/learning/ddos/glossary/web- applic...](https://www.cloudflare.com/learning/ddos/glossary/web-application- firewall-waf/)

</comment></comment><comment><USER39> This is a fantastic postmortem. Thanks jgc!

Can you share any more details about the protection to prevent excessive cpu usage by a regular expression that was accidentally removed?

<comment><USER40> Also interested in this protection. How does it detect this situation and what does it do when something is detected? Did you notice when it was accidentally removed that your monitoring of this condition went to zeros (or was it never happening during normal operations)?

</comment></comment><comment><USER41> Sometimes I wish Cloudflare failed more often so I could read more of these postmortems...

<comment><USER42> Please no.

The last time they had a global problem, everyone scrambled for more than a week. (Cloudbleed)

This 30-minute global outage was pretty nasty, but not anywhere near as awful. Timing helped, as nothing truly critical was affected. (There are some extremely high-volume sporting events which, if affected even just for few minutes, can have a direct impact on the bottom line.)

I do not wish to see more of these. Cloudbleed gave me two weeks of headache and an indigestion problem. This one did basically nothing. If there is a happy middle ground between the two, I am not exactly thrilled at finding out what it is.

<comment><USER43> Agree. Cloudbleed was really, really awful. We should write up all the things we learned from that and all the changes we've made since. Just looking at the number of engineers who are Rust experts since then, for instance.

</comment></comment></comment><comment><USER44> Might be late, but has anyone in CloudFlare tried to switch away from regex to something more efficient and powerful? Tools like re2c can convert 100s of regexs and CFG into a single optimized state machine (which includes no back tracking, as far as I remember). It should easily handle 10s of millions transactions per second per core if the complete state machine fits into the CPU level 3 cache (or lower), with a bit of optimization.

<comment><USER45> There is also Ragel [0], but I think that in this context deploying regexes as strings is safer than generating code and deploying that code (unless Ragel could generate webassembly).

[0]: [http://www.colm.net/open-source/ragel/](http://www.colm.net/open- source/ragel/)

<comment><USER46> Ragel has the advantage that CPU blowups happen at compile time, rather than run-time. Other risks aside, they would have avoided this problem had they been using ragel or something similar to pre-compile their patterns into deterministic machines.

</comment></comment></comment><comment><USER47> For the regex novices here, would anyone mind explaining what that pattern is meant to match? More specifically, what `. _(?:._ =.*)` is meant to do?

<comment><USER48> It's meant to match any number of any characters, then match an equal sign, then match any number of any characters. But it's very badly written. It should instead simply be written

        .*=.*

BTW, your comment got mangled by HN's markdown formatting.

<comment><USER47> Gotcha - I thought I was just missing the point as to why it wasn't simpler since it looked to have been structured that way intentionally.

</comment></comment></comment><comment><USER49> >A protection that would have helped prevent excessive CPU use by a regular expression was removed by mistake during a refactoring of the WAF weeks prior—a refactoring that was part of making the WAF use less CPU.

Faster karma than normal i think.

<comment><USER50> Taking the safeties off to go faster... yes, you will go faster, but it might be right off a cliff.

This is a good lesson on Chesterton's Fence. I've been thinking for a while that we really need the (default behavior) ability to annotate commits after the fact, so that we have a durable commentary that can evolve over time. We should be able to go back and add strongly worded things like "yes this looks broken but it exists due to this bug fix" or "please don't write new code that looks like this. See xyz for a better alternative."

Hell I think I'd be perfectly ok if the code review lived with the code permanently. Regression in the code? Josh warned you it was a bad idea. Maybe we should listen to Josh more?

<comment><USER20> Can't you just add comments to the actual code saying those things? I've seen code comments saying those exact things.

<comment><USER50> There's an art to that, I've seen new code get between the comment and the code, and since the comment is in a separate commit, it's difficult to go back ten refactorings later to answer why. The most interesting bug fixes I do end up exploiting the commit history. Yes it's hidden in plain sight, but it's also more reliable.

These days we treat code as a living breathing thing. No reason we can't do the same to commits.

</comment></comment></comment></comment><comment><USER51> I used to be really into regex and I'm now rusty, but wouldn't the desired representation of .* .* =.* be something closer to [^=]\\* [^=]\\* =[^=]\\* ?

I feel like it could be optimized further but this would be the first step, and wouldn't most experienced regex authors use that from the beginning, nipping the whole backtracking problem in the bud and making the regex _much_ more performant?

<comment><USER48> The original one would match "==", your suggestion would not. To get a clean regex they should switch to

        .*=.*

I don't think they should spend any time contemplating whether a regex will backtrack, because it's hard. Instead they should (and are planning to) simply switch to a better regex library that never backtracks.

</comment></comment><comment><USER52> Well, if I still worked on Hyperscan, this would be my "what am I, a potted plant?" moment. I think Cloudflare is pretty determined to avoid x86-only implementations of anything, though.

It's entertaining to see people making the same mistakes that have been widely known about in network security well before there was Hyperscan, RE2, etc.

<comment><USER53> There are always those potted plant moments. For some reason regular expressions / regex implementations seem to always be in that hole.

</comment></comment><comment><USER54> New idea: scan github for regexes that exhibit unbounded backtracking and generate worst case examples.

<comment><USER12> Do it. I'd be interested.

</comment></comment><comment><USER55> That was a fantastic demonstration of what backtracking meant. Thank you John for your in depth description of what went wrong.

As a follow up, would something like `[^=] _=._ ` be a better capture group regex?

<comment><USER56> Yes. I think HN stole your asterisks. You meant:

      /.*=.*/ becomes /[^=]*=.*/

That is, zero or more 'not-equals-sign-characters', followed by an equals sign.

Where the first regex is 57 steps for x=xxxxxxxxxxxxxxxxxxxxxxxx, the second is just 7.

Avoid using greedy .* for backtracking regex engines! Give your greedy regex engine the hints it needs to do what it does best.

<comment><USER57> Don't use a backtracking regex engine is probably the better lesson. I want a tool that won't unexpectedly poke me with the sharp bit.

</comment></comment></comment><comment><USER58> One thing set of alarm bells in my head from an operational perspective:

> Switching to either the re2 or Rust regex engine which both have run-time > guarantees. (ETA: July 31)

That's short timescales for quite a significant change. I know it's just replacing a piece of automation with one that does the same task, but the guts are all changing and all automation introduces some level of instability, and a bunch of unknowns. Changing the regex engine is just as significant as introducing new automation from an operations perspective, even if it seems like it should be a no-brainer. I'd encourage taking time there (unless this is something they've been working on a lot and are already doing canary testing).

The other steps look excellent, and they should all collectively give ample breathing room to make sure that switching to re2 or Rust's regex engine won't introduce further issues. There's no need to be doing it on a scale of weeks.

Some quick thoughts about Quicksilver: Deploying everywhere super fast is inherently dangerous (for some reason, old school rocketjumping springs to mind. Fine until you get it wrong).

I definitely see the value for customer actions, but for WAF rule rollouts, some kind of (automated) increasing speed rollouts might be good, and might help catch issues even as the deployment steps beyond the bounds of PIG etc. canary fleets. Of course, that's also useless in and of itself unless there is some kind of automated feedback mechanism to retard, stop, or undo changes.

If I can make a reading suggestion: [https://smile.amazon.com/gp/product/0804759464/ref=ppx_yo_dt...](https://smile.amazon.com/gp/product/0804759464/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1) The book is "High Reliability Management: Operating on the Edge (High Reliability and Crisis Management)" (unfortunately not available in electronic form). It's focussed on the energy grid in California, the authors were university researchers specialising in high reliability operations, and they had the good fortune to be present doing a research job at the operations centre right when the California brownouts were occurring in the early 2000s. There's a lot to be gleaned from that book, particularly when it comes to automation, and especially changes to automation.

<comment><USER24> RE2 is bullet proof. It is the defacto re engine used by anyone looking to deploy regexes to the wild (used in the now defunct google code search, for example). It has a track record. Russ Cox, its author, was affiliated with Ken Thompson from very early in his career.

Rust also has a good pedigree for not being faulty. BurntSushi, the author of rust's regex crate also has a good pedigree...

We switched to RE2 for a massive project 2 years ago and haven't looked back. It is a massive improvement in peace of mind.

If anything, I'm surprised that JGC has allowed the use of PCRE in production and on live inputs...

<comment><USER58> I'm absolutely not denying that RE2 is great. Not in the slightest. I even agree with their idea to switch towards it or the Rust one.

Changing anything brings an element of risk, and changing quickly to it, even more so, which is essentially what they're proposing doing. That's where my concern lies.

Their current approach clearly has issues, but it has been running in production for several years now and those issues are fully understood, engineers know how to debug them, and there's a lot of institutional knowledge around covering them. They've put a series of protective measures in place following the incident that takes out one of the more significant risks. That gives them breathing space to evaluate and verify their options, carry out smaller scale experiments, train up engineers across the company around any relevant changes etc. There is no reason to go _fast_

<comment><USER24> i concur with your assessment. i'm sure cloudflare will be cautious and not rush with deployment after the switch. lesson learned?

</comment></comment></comment></comment><comment><USER59> This is such a great retrospective - thanks very much for this, jgrahamc. I really appreciated learning so much about how Cloudflare works internally, and the appendix (with animations!) was delightful.

</comment><comment><USER20> > It might also be obvious that once state 4 was reached (after x= was > matched) the regular expression had matched and the algorithm could > terminate without considering the final x at all.

That's true if you just want a boolean result. But if you want to get the matched string (which it appears the actual code does), then you need to continue, because it's using greedy matching.

</comment><comment><USER60> Amazing postmortem! I have a question: I see a lot of software / process solutions to avoid this to repeat in the future. What about the Human factor?

\- what happened to the engineer(s) responsible for that event? They must feel really bad RN, how do you handle this situation?

\- on a more general point, how do you train individuals to ensure this particular event does not reproduce?

Edit: formatting.

</comment><comment><USER61> Great post! Can't think of many companies that would spend this much time explaining regex.

I was affected by this outage, but I really appreciate Cloudflare taking the time to explain the problem in this much detail. Given their own systems were affected, I'm surprised they mitigated as fast as they did.

</comment><comment><USER62> I'm a relatively novice regex user. Could anyone explain to me why someone might use an expression like `. _(?:._ =.*)` ? What is the meaning of the group if it's boundary could be placed in any number of places? Hope that makes sense.

<comment><USER63> It can be useful depending on how the engine handles the match. The non- capturing group is the important part, the .* is just there so that the 'full match' isn't just an empty string, but contains the whole line the rule is being run against

</comment></comment><comment><USER64> Really appreciate the detail and operational insights in this post-mortem. Great work. Also appreciate the heads up and detailed explanation of the issues with backtracking in regexes. Turns out we use those expensive patterns in a few places.

</comment><comment><USER65> What about: WAF cpu usage wasn't isolated from the ability to serve requests? This would allow requests that don't go throwugh WAF to be able to proceed as usual.

<comment><USER66> A firewall that fails open sounds like a terrible plan.

As far as problems go, an outage is preferable to a breach.

<comment><USER20> I believe WAF is a feature customers enable, not all customers have it enabled. So some customers are already open, and in theory wouldn't need to be affected by a WAF outage.

</comment></comment></comment><comment><USER67> It was too bad WAF took down their entire network. I don't use it but things went down too.

Maybe it's better to separate damage zones for different features.

</comment><comment><USER68> This is such a fantastic blogpost. I love the part about regex backtracking. Great job CF.

</comment><comment><USER69> > At 13:37 TeamCity built the rules

Hmmm...

</comment><comment><USER20> >x=x still takes 555 steps

That's a typo. It should say

>x=xxxxxxxxxxxxxxxxxxxx still takes 555 steps

</comment><comment><USER70> I love how Cloudflare turns an outage into a selling opportunity by explaining exactly why you need their product to a hungry audience waiting to read about it.

</comment><comment><USER71> Incredibly well written, thank you.

</comment><comment><USER67> Regexp::Debugger was a cute one.

</comment><comment><USER72> Talk about a great writeup

</comment><comment><USER73> Obligatory JWZ quote:

Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.

</comment><comment><USER74> TL;DR

Root cause was a bad regex generating excessive backtracking using all CPU on nodes.

The meta-cause is the process workflow:

> But, by design, the WAF doesn't use this process because of the need to > respond rapidly to threats.

The above is in reference to how WAF deployment doesn't use the graduated DOG(fooding)/(guinea)PIG/canary flow.

> We responded quickly to correct the situation and are correcting the process > deficiencies that allowed the outage to occur [...]

Live and learn. Not all WAF deployments are emergency rollouts.

</comment><comment><USER75> Cloudflare lets their customers write their own WAF regex rules right?

And those rules still get run on every box on cloudflares edge network with HTTP requests from strangers on the internet right?

So how come this didn't get triggered by a customer first?

Perhaps it _did_ get triggered by a customer first, but that customer didn't get too much traffic of the URL which triggers the issue, and that box got one thread stuck executing that regex for a few minutes till a health check killed it...? Does this imply that cloudflare runs with random failing health checks across the fleet and there isn't someone looking at core dumps of such failures?

That would align with my experience with seeing occasional "502 bad gateway" errors from cloudflare over the past few years. It also seems likely considering the incident where cloudflare servers leaked sensitive memory contents into HTTP responses which happened so frequently they got cached by google search. Hard to leak arbitrary memory contents without occasional SIGSEGV's...

If the above conjecture is true, it reflects very badly on engineering culture at Cloudflare. The core issue had been seen across the fleet sporadically for a long time, but was ignored, and even during the postmortem process, which should be a very thorough investigation, the telltale pre-warning signs of the issue were _still_ missed.

<comment><USER76> They allow a limited subset of rules, with strict parameters of what logic is allowed. Unless you do something fancy with workers.

Also, the protection for this was removed in a recent update before the incident, so it wouldn't have had an impact if a customer did this until that protect was removed. So maybe a few weeks earlier they might have started seeing some problems. But again, I am pretty sure the logic in the rule that caused the issue isn't available to customers.

</comment></comment><comment><USER77> Here's their _What Went Wrong_ :

      1. An engineer wrote a regular expression that could easily backtrack enormously.
      2. A protection that would have helped prevent excessive CPU use by a regular expression was removed by mistake during a refactoring of the WAF weeks prior—a refactoring that was part of making the WAF use less CPU.
      3. The regular expression engine being used didn't have complexity guarantees.
      4. The test suite didn't have a way of identifying excessive CPU consumption.
      5. The SOP allowed a non-emergency rule change to go globally into production without a staged rollout.
      6. The rollback plan required running the complete WAF build twice taking too long.
      7. The first alert for the global traffic drop took too long to fire.
      8. We didn't update our status page quickly enough.
      9. We had difficulty accessing our own systems because of the outage and the bypass procedure wasn't well trained on.
      10. SREs had lost access to some systems because their credentials had been timed out for security reasons.
      11. Our customers were unable to access the Cloudflare Dashboard or API because they pass through the Cloudflare edge.

Here's my version of what went wrong:

      1. The process for composing complex regular expressions is "engineer tries to shove a lot of symbols into a line" rather than "compile/compose regex programmatically from individual matches"
      2. Production services had no service health watchdog (the kind of thing that makes systemd stop re-running services that repeatedly hang/die)
      3. Performance testing/quality assurance not done before releasing changes (this is not CI/CD)
      4. No gradual rollout
      5. No testing of rollbacks
      6. Lack of emergency response plans / training

All of these things are completely common, by the way, so they're in no way surprising. Budget has to actually be set aside to continuously improve the reliability of a service, or it doesn't get done. These incidents are a good way to get that budget.

(Wrt the regex's, I know they're implementing a new system that avoids a lot of it, but in the new system they can still write regex's which (I think) should be constructed programmatically)

<comment><USER78> I don't see the relevance of how regexes are written to the problem they had. The engineer didn't typo the regex, or have a hard time understanding what it would match.

Instead, they didn't understand the runtime performance of the regex, as it was implemented in their particular system. No amount of syntax can change that.

<comment><USER79> _No amount of syntax can change that_

A framework that allows well-written, "normal" code to parse out what you want, can produce something easier to understand and maintain, surfacing this type of bug in a more obvious way.

Cryptic syntax is the main reason I avoid regexes (particularly complex ones).

Too much obfuscation between the code you write and the steps your program will take. Granted, my concern doesn't apply to master craftsmen who truly understand the nuances of the tool, but in the real world those are few and far between.

ps. I get there was a lot more going on in this postmortem than just one rogue regex.

</comment></comment></comment><comment><USER80> So in response to a catastrophic failure due to testing in prod, they're going to push out a brand new regex engine with an ETA of 2 weeks. Can anyone say testing in prod?

The constant use of 'I' and 'me' (19 occurrences in total) deeply tarnishes this report, and repeatedly singling out a responsible engineer, nameless or not, is a failure in its own right. This was a collective failure, any individual identity is totally irrelevant. We're not looking for an account of your superman-like heroism, sprinting from meeting rooms or otherwise, we want to know whether anything has been learned in the 2 years since Cloudflare leaked heap all across the Internet without noticing, and the answer to that seems fantastically clear.

<comment><AUTHOR> This report is written by me, the CTO of Cloudflare. I say "I" throughout because organizational failings are my responsibilty. If I'd said "we" I imagine you'd be criticizing me for NOT taking responsibility.

If you read the report you'd see I do not blame the engineer responsible at all. Not once. I made that perfectly clear.

<comment><USER17> I wonder if you are able to talk a bit about the development of the Lua-based WAF. I imagine the possible unbounded performance of feeding requests into PCRE must have occurred to you or others at the time - or at least, long before this outage.

I don't mean this as some sort of lame 'lol shoulda known better' dunk - stories about technical organizations' decision-making and tradeoff-handling are just more interesting than the details of how regexes typed in a control panel grow up to become Jira tickets.

<comment><AUTHOR> I did a talk about this years ago: [https://www.youtube.com/watch?v=nlt4XKhucS4](https://www.youtube.com/watch?v=nlt4XKhucS4)

<comment><USER17> It sounds like one of the primary factors was compatibility with existing (or customer-provided) mod_security rules, if I've understood 1.75x speed hyper- you right.</comment></comment></comment></comment></comment>


Swytch.net - diaspora already has a clone - <AUTHOR> http://www.swytch.net

<comment><AUTHOR> The interface looks remarkably similar and they seem to be taking the same approach to privacy

<comment><USER2> hi, I'm the creator of swytch. Actually I don't agree that the interface looks similar - i assume you're referring to the screenshots they released? Besides the colours what else is similar and just how would you expect a news feed to look? What you see on my site is typical of what you might see on any site designed by a programmer. No frills and functional.

As for privacy, we're not the only ones working on addressing the privacy issue, that doesn't make us a clone.

<comment><USER3> Looks nice. Best of luck.</comment></comment></comment>


Reinventing Home Directories – systemd-homed [pdf] - <AUTHOR> https://cfp.all-systems-go.io/media/homed-asg2019.pdf <comment><USER2> I am not comfortable with the feature creep that systemd is slowing bringing in. binfmt mount executing files based on a lookup, similar to windows assoc, intercepting gethostbyname(), this really feels like an attempt to shim old vulnerable windows concepts into linux. I am concerned that linux will not be recognizable in the near future and will be subject to bloat and obfuscation. What would it take to make this stop?

There are some things brought in that I like, but I don't believe systemd is required for them. Cgroups, LXC, to name a couple. Those make functional and operational sense to me.

<comment><USER3> > What would it take to make this stop?

That's a great question, and I have no answers.

I'm personally still trying to decide between moving all of my machines to Slackware or moving away from Linux entirely and to BSD. I'm currently leaning toward the latter.

</comment></comment><comment><USER4> I'm going to pass on this one. How is /etc not 'extensible'? You just add a new file to the directory. Reinventing the windows registry, but this time in JSON, is a lateral move that is only going to complicate maintaining systems.

For user level configuration, you already have too many competing mechanisms for doing this already, and this one requires systemd level integration making it much more coupled to the operating system and system in general.

<comment><USER5> > I'm going to pass on this one. How is /etc not 'extensible'?

Without commenting on systemd-homed, since I haven't watched the talk yet; regular users can't modify /etc. Granted, most applications have user local configuration or a command line option to specify a config file, but not all. That said, I suppose you could always user namespace overlay mount a custom directory over /etc to add custom files to it.

<comment><USER4> The common pattern is to have some system wide file in /etc or /usr/share that is used by default, and allowing the user to override this with a dot-file in their home directory or passing in a config file as a optional argument.

What is being proposed obliterates most of the workflows that people are use to: editing configuration files in the editor of the user's choice and then having the option of using normal backup or using source code repository tools to maintain and version them. Instead, we have the same windows registry mechanism, with some modern flourishes, that most people have ridiculed Microsoft for decades over.

As a user, I can 'strace -e open' and see what locations my program is trying to open. This is very helpful for what configuration data is being read. Everything is a file, and every file operation is a syscall. As a developer, JSON isn't appropriate for every kind of application.

</comment></comment></comment><comment><USER6> talk / context: [https://streaming.media.ccc.de/asg2019/relive/164](https://streaming.media.ccc.de/asg2019/relive/164)

edit: I think the most important part of the talk was an answer to question about why .ssh/authorized_keys won't be possible with this scheme (42:00):

> this is about protecting your data from the system as much as possible, so > that when you are not logged in it's really cut off

This looks like, among other things, an ecryptfs replacement.

</comment><comment><USER7> This seems like a massive boondoggle and a potential security issue.

No thanks.

</comment><comment><USER8> What is with this fetish of putting everything in systemd?

Soon it'll have an X-server and a word processor built in...

</comment><comment><USER9> Some of Lennart's inventions that he presents at conferences seem to lose momentum soon after. A while ago he talked about casync and mkosi. Both of these seemed more interesting to me than this but progress seem slow or stalled. We will see what becomes of this, but I don't think it is a slam dunk and what everyone has always needed but never realized. I think by ruling out remote /server use cases he will limit the interest in it.</comment>


Create a Trending topic on Instagram - <AUTHOR>

I have created Bajanbot.com, The only source of
 trending topics on Instagram.I plan to exploit
my monopoly by allowing any one on the planet to
 create a trending topic on Instagram for free.
To get your topic to become a trending topic  on Instagram , you need to tweet out your topic and
mention @instagram_view in your tweet.<p>You can also submit your Topic in the comments.If
 you follow me on Twitter(@instagram_view)  or
like my website on Facebook, you can increase
your chances of getting in to the exclusive list
 of Trending topics . I will choose 3 different
trending topics and they will be promoted started
 on April 26 ,2011. Maybe this  will prod
Instagram into developing an official list of
Trending Topics.Thank you for submitting a
Trending Topic. <comment><AUTHOR> clickable : <http://bajanbot.com></comment>


America's economy risks the mother of all meltdowns - <AUTHOR> http://news.yahoo.com/s/ft/20080219/bs_ft/fto021920081334359078;_ylt=AozoX8V3CwKFRV6c_RfR1f0E1vAI <comment><USER2> The problem I have with articles like this is they find some fringe figure (either a total bull or bear) and have them talk about the extreme edge of an argument because it makes better press. Because of this motivation, you're not going to get an objective analysis of the issue. The author even said "To answer this question we should ask a true bear".

No one economist is ever going to get this right because economics is more like weather forecasting than anything. You're better looking at "surveys of economists" you can find in many publications to get an idea of what economists on aggregate think. The wisdom of crowds, anyone?

</comment><comment><USER3> Wow, this could become true just like: [http://www.amazon.com/Great- Depression-1990-Ravi-Batra/dp/04...](http://www.amazon.com/Great- Depression-1990-Ravi-Batra/dp/0440201683)

</comment><comment><USER4> Two words: Self-Fulfilling Prophecy.</comment>


Platform and tools best for a social networking website - <AUTHOR> Which is the best platform and tools to create a social networking website <comment><USER2> I'm not sure saying "social networking website" is enough. It's a very, very wide term.

I've tried mentioning a few tools, but there's really no point until you define what exactly your needs are and on what level. Would you like to build it from the scratch, install something that's Open Source or maybe use some SaaS that offers features that you need? What do you need it for, you want to start a new project or organize something within your team?

You really need to be more specific if you want some serious answers.

<comment><AUTHOR> Thanks. We are currently looking at building it from scratch to create a platform for people to connect with each other.so it would be typically creating users, pulling their reference data from other sites like facebook,creating profiles etc.

</comment></comment><comment><AUTHOR> Also, I have read about free open source social networking engines like "Elgg", "Community Engine", how well do these work? Has anyone from this community built anything on these? Are these feasible to scale?</comment>


The High Priest of Heterodoxy - <AUTHOR> https://www.tabletmag.com/jewish-news-and-politics/288293/the-high-priest-of-heterodoxy <comment><USER2> There are some great quotes in this piece. In particular, discussion on the tradeoff between feeling safe and intellectual honesty (which requires some discomfort).

> _Haidt and Lukianoff quote Jeannie Suk Gersen, a Harvard law professor who > says that discussing rape law in the classroom "has become so difficult that > teachers are starting to give up on the subject." She adds that if law > professors stop mentioning sexual assault, that "would be a tremendous > loss—above all to victims of sexual assault."_

Reminds me of this clip from Van Jones re: safe spaces on college campuses: [https://www.youtube.com/watch?v=Zms3EqGbFOk](https://www.youtube.com/watch?v=Zms3EqGbFOk)

</comment><comment><USER3> "A candle flame is easily blown out. A candle flame is fragile. If you have a candle flame you have to protect it with a glass sleeve because any puff of wind will extinguish it. But once it reaches the stage of a roaring fire, the more wind the better. So what Taleb said is that you want to be the fire and wish for the wind" \-- Jonathan Haidt paraphrasing Nassim Taleb

Haidt convincingly expands on this metaphor in terms of how we treat children in this Penn State lecture:

[https://www.youtube.com/watch?v=B5IGyHNvr7E](https://www.youtube.com/watch?v=B5IGyHNvr7E)

This summarizes his overall case neatly: We're increasingly handling our kids like fragile objects, stunting their maturity and reshaping western civilization in their image.

</comment><comment><USER4> > _"The recent spike in students' anxiety and depression is like nothing else > in history," Haidt added, because they grew up exposed to online bullying > and know they could be the next victim._

Every now and then the mental health epidemic comes up on HN and I see ten or twenty different theories about what's causing it. If nothing else, it's galvanized me against believing any one explanation just because someone says it. Has Haidt really cracked the case? It would be nice to see some evidence one way or another.

<comment><USER5> Haidt talks about it in detail in his "The Coddling...". He cites other studies. I can't say that that he's "cracked" it, but if you're interested in why he says that, that's the place to start looking.

</comment></comment><comment><USER6> > Jonathan Haidt's early work attempted to explain the origins of our > political differences. His new Heterodox Academy is looking for ways to move > past them.

(By pushing the same old narrative you've heard from exactly one side)

</comment><comment><USER7> I'm tired of articles that talk about young people today and then only give out examples that happen at small elite liberal arts universities. What that generally tells you is that the author only interacts with a small slice of young people and assumes everyone else in their generation is the same.

<comment><USER8> Haidt's thesis in 'Coddling' is about what is happening at universities and how their influence spreads to the general population. So some examples of what is happening at universities is appropriate.

<comment><USER7> But it's always very specific universities that attract a small and specific subset of students. Where are the examples of this happening at community colleges, or rural land grant schools? As I said it comes off as the only college students the author actually meets are people who go to small elite schools and get an internship in journalism.

</comment></comment></comment><comment><USER9> There's a thoughtful examination of Haidt's claims here ([https://niskanencenter.org/blog/there-is-no-campus-free- spee...](https://niskanencenter.org/blog/there-is-no-campus-free-speech- crisis-a-close-look-at-the-evidence/)), which agrees in part and disagrees in part. Worth reading before passing judgement.

</comment><comment><AUTHOR> I've admired all of the Haidt books I've read, including _Coddling_ : [https://jakeseliger.com/2018/11/27/the-coddling-of-the- ameri...](https://jakeseliger.com/2018/11/27/the-coddling-of-the-american- mind-jonathan-haidt-and-greg-lukianoff/). His earlier book _The Happiness Hypothesis_ is underrated and highly recommended: [https://sivers.org/book/HappinessHypothesis](https://sivers.org/book/HappinessHypothesis)

</comment><comment><USER10> If anyone's still reading this thread, the article cited a tweet from an account that just today wrote:

> When you bring in a population that is less skilled & from nations w/ lower > average IQs, you are "baking in" socioeconomic inequality & fueling > resentment among those less successful groups. This resentment can be > successfully exploited by the identity politics of the left.

The person who wrote this, like Haidt, calls himself a "centrist," which makes me find it hard to take anything these people say about "dialogue" and "bringing people together" at face value.

<comment><USER11> Disqualifying an argument using unstated, inscrutable and ambiguous rules, and then implying guilt by association. Top form there mate.

</comment></comment><comment><USER12> If you already think this is a problem then this guy will reinforce you belief.

It's too pat of an answer and to wide of a subject to think that it explains these conditions.

</comment><comment><USER13> I first saw this as potentially a great possibility. Like, awesome, someone is actually sitting down and researching how to undivided the red/blue divided America?

But then...

> The fledgling academy now occupies a position within America's tumultuous > culture wars as advocates for free speech and the values of the "open > society" and against political correctness and the millennial fervor of > social justice activism.

"Political correctness" is so thoroughly abused as an alt-right general accusation as to be meaningless. All it does here is make me question the institution as just another alt-right organization hiding their beliefs behind the shield of "rationality" and "free speech."

> Of course, the researchers will be labeled white supremacists by those who > fear that more knowledge might interfere with their ideologically pure claim > that American racism is the root cause of every social problem.

Sure, it's annoying to be called a white supremacist by verified Twitter trolls, but we've got equally, if not more so, absurd accusations of anti- Semitism flying all over the fucking place right now, and the correct solution is to leave Twitter to the Twitter trolls.

<comment><USER14> So it turns out that being in favor of allowing multiple points of view to coexist puts you in conflict with people who only want to allow one point of view to be heard. That's... rather unsurprising.

It is my perception that the left is doing better at shutting down opposing speech than the right, and has been since the 90s. If you're going to address the problem that it's happening _at all_ , you have to address the side that's doing it _more and better_.

<comment><USER7> Do you think all view points should be heard? Should Jewish and gay students have to debate Nazi's about whether or not they should be murdered? Should black and women students have to debate if they should count as property?

<comment><USER8> Those are some extreme examples. What about the not so extreme cases? Who would decide which topics are acceptable? A slippery slope.

<comment><USER7> I think extreme examples are useful. The Neo-Nazi's marching in Skokie is an extreme example of Free Speech, and the ACLU fought to allow it because the First Amendment is extreme. On the other hand, I believe that when you are dealing with a quasi-public space like a university we shouldn't be as absolutist with free speech claims. The question then becomes on where we draw the line. If on the other hand you believe that we should be extreme on free speech when it comes to universities and let the Nazi's march at University of Chicago as well as Skokie then clearly you would disagree.

<comment><USER8> You think Nazis shouldn't be allowed to march? What about communists? Why one and not the other? Communists killed far more people in the 20th century.

<comment><USER7> That is not what I said, I think you have to read a bit more carefully.

</comment></comment></comment></comment></comment></comment></comment><comment><USER15> At a glance this looks like just lengthy prose trying to position conservative ideas as the intellectual middle ground.

<comment><USER16> I am not sure why it is downvoted - it is a commonly used tactic of the right dating back at least to "teach the controversy" for creationists and climate change deniers to try to legitimize themselves through a golden mean fallacy.

<comment><USER11> Why is it downvoted?

* because it's a low-effort comment that contains crass generalizations.

* because Haidt et. al. are not right-wing partisan hacks, but bona fide liberals.

<comment><USER15> I don't see where I was crass. Nor do I think it's especially low effort relative to the quality of the content. Much of the post is a recurring tactic of: stating a fairly tame conservative view, extrapolating that since liberals shut down some conservative views that they will shut down even the most nuanced ones, implying that therefore the ones trying to be reasonable are the conservatives.

It's... bad.

I don't know or care who Haidt is it what his rep is</comment></comment></comment></comment>


Why's Beijing So Worried About Western Values Infecting China's Youth? (2017) - <AUTHOR> https://www.chinafile.com/features/whys-beijing-so-worried-about-western-values-infecting-chinas-youth <comment><USER2> They probably wanted to wait a few more years, but now is the perfect opportunity to stop overseas study, it's only going to get harder to control if they don't act soon.

Students in China heed their government's warnings against studying in Australia [https://theconversation.com/students-in-china-heed-their- gov...](https://theconversation.com/students-in-china-heed-their-governments- warnings-against-studying-in-australia-141871)

China Issues Warning to U.S.-Bound Students [https://www.insidehighered.com/news/2019/06/04/chinese- offic...](https://www.insidehighered.com/news/2019/06/04/chinese-officials- warn-students-visa-problems-if-they-come-us)

Chinese citizens are urged not to travel to Canada [https://www.globaltimes.cn/content/1193681.shtml](https://www.globaltimes.cn/content/1193681.shtml)

</comment><comment><USER3> Lots of good citations so the message here seems well informed. Does anyone have good recommendations for Chinese history volumes? Anything that offers a high-level survey of the past 3000 years, to give me a framework for further investigations, would be greatly appreciated.

<comment><USER4> 1) These two single volume histories were good starts for me: a) John Keay's "China: A History" and b) Fairbank/Goldman's "China: A New History".

2) HUP's History of Imperial China six volume series edited by UBC Sinologist Timothy Brook is a deeeep dive.

3) The People's Trilogy by Dikotter is heralded for its presentation of Mao's China. Julia Lovell also has a recent work on Maoism that's been lauded.

4) Finally, something more strategic/contemporaneous and forward looking is Martin Jacques' "When China Rules the World".</comment></comment>


Papers Every Programmer Should Read (At Least Twice) - <AUTHOR> http://blog.objectmentor.com/articles/2009/02/26/10-papers-every-programmer-should-read-at-least-twice <comment><USER2> A paper that would be on my personal list is "What Every Computer Scientist Should Know About Floating-Point Arithmetic" (Goldberg): [http://download.oracle.com/docs/cd/E19957-01/806-3568/ncg_go...](http://download.oracle.com/docs/cd/E19957-01/806-3568/ncg_goldberg.html)

<comment><USER3> Thanks for mentioning it, and saving me the work of typing (or copying) the long title. This paper is great. If I had a dime for every time I've linked to it on Stack Overflow, I would have a lot of foreign change of little practical use.

</comment></comment><comment><USER4> There was an excellent cs theory stack exchange question on this:

[http://cstheory.stackexchange.com/questions/1168/what- papers...](http://cstheory.stackexchange.com/questions/1168/what-papers- should-everyone-read) [pdf]

The top two are nearly tied for: "A mathematical theory of communication" by Claude Shannon

[http://guohanwei.51.net/code/A%20Mathematical%20Theory%20of%...](http://guohanwei.51.net/code/A%20Mathematical%20Theory%20of%20Communication.pdf)

"On Computable Numbers, with an Application to the Entscheidungsproblem" by Alan Turing

[http://l3d.cs.colorado.edu/~ctg/classes/lib/canon/turing- com...](http://l3d.cs.colorado.edu/~ctg/classes/lib/canon/turing-compnum.pdf) [pdf]

I would add that

'The Annotated Turing' by Charles Petzold

<http://www.theannotatedturing.com/>

is an excellent treatment of Turings paper, including much of the relevant additional math and computing history both before and after.

<comment><USER5> More readable versions of the Shannon paper (and a more compressed pdf version[1]) here: <http://cm.bell-labs.com/cm/ms/what/shannonday/paper.html>

[1] [http://cm.bell- labs.com/cm/ms/what/shannonday/shannon1948.pd...](http://cm.bell- labs.com/cm/ms/what/shannonday/shannon1948.pdf)

</comment></comment><comment><USER6> It's not a terrible list, but it's very biased towards the question "how should we structure computer programs?".

This is a good question, but perhaps not the _only_ question, and I'm not sure that a top 10 list would be quite so focused on it, at the expense of algorithms, architecture, concurrency, networks, formal methods, etc.

I also doubt the ranty "Worse is Better" should be on any top 10 list, influential or not. Some of these papers seem better suited to give someone a background to furiously prognosticate here on HN and perhaps LtU than to do anything of consequence.

<comment><USER7> >This is a good question, but perhaps not the _only_ question.

I agree with narrow interpretation of your statement, but disagree with the broad one. Yes, the structure of computer programs isn't the _only_ problem in computer science/software engineering. However, I would argue that it's the most _important_ question. Moreover, it's the only question that translates well across domains. Algorithms, concurrency models, networks, etc. all will vary depending on the exact problem you are trying to solve. The principles behind good program structure, however, remain the same. It doesn't matter if you're making a computational fluid dynamics simulation, nearest neighbor classifier, relational database, enterprise inventory control, or a simple blog engine; taking the time to create a good structure for your program always pays off.

<comment><USER6> Any principle of 'good program structure' that applies across "computational fluid dynamics simulation, nearest neighbor classifier, relational database, enterprise inventory control, or a simple blog engine" is so vague as to be meaningless - or at least, not very interesting.

Methodology weenies are always claiming that they've got abstractions that are _absolutely key_ across very diverse problem areas, usually without any demonstration that said abstractions are useful for solving hard problems in any areas at all.

Compared to this kind of waffly crapulousness, we should compare the classic papers on the design of, say, RISC, TCP/IP, etc. Grinding through how Tomasulo's algorithm worked on the IBM/360, for example, will greatly expand your understanding of actual computer architecture; the only weakness is that it will not especially expand your ability to Issue Pronouncements About What Good Programs Look Like.

<comment><USER6> ... or more concisely, we will still be reading about 'branch and bound', 'parallel prefix', 'greedy algorithms vs divide and conquer', 'exponential backoff', 'multi-level caching', 'speculative execution', etc, long after the Revolution, when the last Object Orientation Guru is strangled with the guts of the last Functional Programming Weenie.

</comment></comment></comment></comment><comment><USER8> I would add "Out of the Tarpit" by Ben Moseley and Peter Marks. I read it at least twice a year.

<http://web.mac.com/ben_moseley/frp/paper-v1_01.pdf>

<comment><USER9> Your link goes to something else about an aikido dojo. Try this link: <http://news.ycombinator.com/item?id=1730320>

<comment><USER8> Fixed. Thanks.

</comment></comment></comment><comment><USER10> Robert Kowalski (1979). "Algorithm = Logic + Control". _Communications of the ACM_ 22(7): 424-436. [http://www.doc.ic.ac.uk/~rak/papers/algorithm%20=%20logic%20...](http://www.doc.ic.ac.uk/~rak/papers/algorithm%20=%20logic%20+%20control.pdf)

(It's the paper that originated Prolog, but is also more broadly interesting for its analysis of, well, algorithms as logic plus control.)

<comment><USER11> Prolog was created around 1972 by Alain Colmerauer.

</comment></comment><comment><USER12> Tony Hoare's ACM Turing Award lecture "The Emperor's Old Clothes":

<http://awards.acm.org/images/awards/140/articles/4622167.pdf>

</comment><comment><USER13> I've always found rpg's Patterns of Software to have some deep insights into the nature of software systems. Particularly the ruminations on habitability.

</comment><comment><USER14> "On the Designing and Deploying Internet-scale Services" by James Hamilton: <http://www.mvdirona.com/jrh/talksAndPapers/JamesRH_Lisa.pdf>

html version: [http://www.usenix.org/event/lisa07/tech/full_papers/hamilton...](http://www.usenix.org/event/lisa07/tech/full_papers/hamilton/hamilton_html/)

</comment><comment><USER15> Butler Lampson's _"Hints for Computer System Design"_ (1983) is a classic paper with some great anecdotes:

[https://research.microsoft.com/en- us/um/people/blampson/33-H...](https://research.microsoft.com/en- us/um/people/blampson/33-Hints/WebPage.html)

</comment><comment><USER16> MapReduce: Simplified Data Processing on Large Clusters by Jeffrey Dean and Sanjay Ghemawat (OSDI'04: Sixth Symposium on Operating System Design and Implementation) <http://labs.google.com/papers/mapreduce.html>

</comment><comment><USER17> Anyone know where I can find an ungated version of "An experimental evaluation of the assumption of independence in multiversion programming"?

<comment><AUTHOR> Simply putting the title into Google gave this as its second hit:

<http://sunnyday.mit.edu/papers/nver-tse.pdf>

<comment><USER17> Thanks. Sorry 'bout that; I only tried Google Scholar.

</comment></comment></comment><comment><USER18> A programmer should not encourage paywalls.

</comment><comment><USER19> It seems the site is down :(

Any mirror link?

<comment><USER8> Google to the rescue?

[http://webcache.googleusercontent.com/search?q=cache:D6mF_SI...](http://webcache.googleusercontent.com/search?q=cache:D6mF_SI2jsYJ:blog.objectmentor.com/articles/2009/02/26/10-papers- every-programmer-should-read-at-least- twice+http://blog.objectmentor.com/articles/2009/02/26/10-papers-every- programmer-should-read-at-least- twice&cd=1&hl=en&ct=clnk&gl=us&source=www.google.com)</comment></comment>


Flying-car venture Terrafugia expands workforce - <AUTHOR> https://www.aopa.org/news-and-media/all-news/2018/april/11/flying-car-venture-terrafugia-expands-workforce <comment><USER2> We've had a "flying car" for generations. It's called the helicopter. It was even able to land on top of Pan Am's headquarters in NYC.

The problem, of course, is the high level of training required to operate such a vehicle, plus the cost of the vehicle itself.

"Flying car" was never a technical impossibility. It was an economic challenge. "Flying cars" are 50-year long hype.

<comment><USER3> High level of training is no longer required. Anyone can use a PS2 joystick to fly ours: [https://youtu.be/b7SjOOuTct0](https://youtu.be/b7SjOOuTct0)

You're right of course, that the economic challenge still exists. This one burns about $100/hr worth of fuel, and there's other maintenance costs on top of that.

<comment><USER4> The real issue is that if you did solve those problems you'd be left with the truth that we don't want thousands of noisy "flying cars" buzzing around, polluting, and occasionally falling on our heads. Flying cars don't scale. If you took a significant percentage of ground traffic into the sky, it would be an absolute nightmare.

</comment></comment></comment><comment><USER5> I saw a presentation by the founder six or seven years ago. At the time, they had working, flying prototypes that were being tested around northern NY and New England ([https://www.youtube.com/watch?v=nnF2yua4KIw](https://www.youtube.com/watch?v=nnF2yua4KIw)). The idea then was the wings would fold up so you could drive it to and from a standard home garage to the airfield, and then start flying.

Clearly there were some major regulatory hurdles they were working through related to licensing and other FAA requirements, but (according to the CEO) they had customers ready to buy and start using them.

The new TF-X model looks a lot slicker, and uses computer-assisted VTOL: [https://www.youtube.com/watch?v=wHJTZ7k0BXU](https://www.youtube.com/watch?v=wHJTZ7k0BXU)

Curious to see how this plays out under new ownership.

<comment><USER6> This class of flying car has never made much sense to me. It has to follow auto and aircraft regulations, with multiple different propulsion systems for different types of travel. Why not just focus on creating a small, autonomous, VTOL aircraft (an enormously hard problem by itself), and then assume the rider will use ride share or public transit for the last mile.

<comment><USER7> Because hangars are pricey.

The whole point of the roadable aircraft is that you don't have to rely on airport infrastructure.

<comment><USER3> Hangers are insignificant compared to the cost of fuel. 1 month of hangar cost is a couple hours of fuel cost.

</comment></comment></comment></comment><comment><USER8> I dislike the confusion between "flying car" as

1) a hybrid vehicle that can both drive on ordinary roads and fly

2) an aircraft that is as easy to fly as a car (is to drive), and possibly doesn't require a runway to operate.

The Terrafugia Transition is an example of the former, but the category has been around for a long time, though without any really convincing entries in my view. Maybe the compromises required are still too great.

Many exciting developments are happening in the latter category, specifically in the subcategory that I like to call "PEVA", namely personal electric VTOL aircraft, for example the Volocopter. That category has arisen because of the improvements in avionics and battery tech. It'll be exciting to watch what happens there.

I wonder whether Terrafugia's TF-2 falls in the second category.

</comment><comment><USER9> Frankly i dont want humans to be flying a car, i want them to be driven automatically using software. A self driving flying Taxi of sort.</comment>


Keep Google fonts updated on your desktop - <AUTHOR> http://googledevelopers.blogspot.co.uk/2013/05/download-google-fonts-to-your-desktop.html

<comment><USER2> What a completely terrible website.

"Would you like to install our font installer?"

No? You want to add a font to a collection to download it?

"How about installing our mac client to help with that???!!?"

....

do not want.

This is even more painful than downloading the fonts manually.

(edit: _this_ is about 100000% more pleasant to use in every way: <https://github.com/w0ng/googlefontdirectory>, for anyone else looking for an easy way to get the fonts that does not involve convoluted websites)

</comment><comment><USER3> Title is really weirdly worded, and it's not necessarily OP's fault; the article itself is poorly worded, and it shows in the confusion from the first few commenters.

Google Web Fonts have always been available offline; when you start browsing the web fonts and then add one to your "collection", you should be able to find a link that says "download your collection". That's one way to do it.

Another way to do it is by accessing the font repository on Google Code, which is not new and has always been around; the fonts are stored in a Mercurial repo. This is mentioned in a link near the end of the article.

The point of this article, and what isn't really emphasized even though it should be, is that (apparently, I haven't used it) this SkyFonts tool _keeps the fonts updated on your local machine_. This might seem like a silly feature, but I can tell you from experience that the Google Web Fonts do actually update pretty frequently; not just new fonts getting added, but existing fonts will also receive updates. Having to pull down the deltas on the hg repo and then reinstall all of the changed fonts manually when you're using a large subset of the library is not a trivial task.

</comment><comment><USER4> The app keeps the fonts in the C:\Users\<user>\AppData\Local\Temp\skyfonts- google folder - that's a terrible idea, since the Temp folder might be nuked. It's meant to be for temp files, not cache.

Also the systray app does not respond correctly. And then the manual download of font? What gives... I'm totally missing what's this about (I'm not a designer, so I probably don't know much).

</comment><comment><USER5> Or likely better for HN: get a clone of the repo of google webfonts, maintained by that team: <https://code.google.com/p/googlefontdirectory/>

</comment><comment><USER6> I'm trying to find an easy way to get them all.... A quick search turned up this linux script which uses Mercurial.

[http://www.webupd8.org/2011/01/automatically-install-all- goo...](http://www.webupd8.org/2011/01/automatically-install-all-google- web.html)

<comment><USER7> I put this up a while back (and need to update it): <http://joemaller.com/1856/download-google-fonts/>

Google changed repository layout so there are a lot more subfolders to sort through but `hg clone <https://code.google.com/p/googlefontdirectory/`> still works fine.

(edit: just saw @shadowmint's update, agreed about <https://github.com/w0ng/googlefontdirectory> being better in every way.)

</comment></comment><comment><USER8> Somewhat off topic, but I recently found this article about Google Web Fonts (and the comments) very interesting:

<http://www.typographyforlawyers.com/?page_id=3207>

</comment><comment><USER9> Is there any use in this for someone that doesn't use said fonts for development? Will having them installed locally prevent websites from loading them as needed, thus speeding up browsing?

<comment><USER10> You are more then capable of serving the fonts via https with javascript. See: <https://developers.google.com/fonts/docs/webfont_loader>

Edit: Hmm, re-reading your question that may not have been what you looking for. I would still say yes, it would serve pages faster, as it would serve pages faster due to the lack of https connection to fetch the font.

Oh dear, I hope I'm helping. :/

</comment></comment><comment><USER11> What if I don't want SkyFonts? What if I just want to download the font file? Didn't seem very clear in the article.

<comment><USER12> Add them to your collection, then hit the download arrow in the top right corner.

<comment><USER11> Thanks!

</comment></comment></comment><comment><USER13> How about just bundling the fonts in a zip with a download link?

<comment><USER14> <https://github.com/w0ng/googlefontdirectory/zipball/master></comment></comment>