UK Wants EU to Block Russia from SWIFT Banking Network - <AUTHOR> http://www.bloomberg.com/news/2014-08-29/u-k-wants-eu-to-block-russia-from-swift-banking-network.html <comment><USER2> SWIFT used to cover the world because it was 'neutral' to all countries. By banning Iran from using SWIFT several years back, SWIFT went from covering 'all' to only covering 'some' and thereby lost it's 'indispensable' reputation. Now it looks like another country will be banned from using it. That's called 'shooting yourself in the foot'.

The BRICs are now in the process of setting up their own SWIFT-like system, and as time goes on more people will use that system and SWIFT will become less and less relevant, instead of remaining the 'only game in town'.

<comment><USER3> I feel that blocking Russia from SWIFT would be quite ballsy on the part of the EU, but ultimately Russia and its partners won't need SWIFT. With the development of the BRICS Bank [1], this eventuality is inevitable. Does it happen now, with some pain, or later is the question. But Putin is really testing the resolve of the West. Personally, if China and Russia develop serious business dealings we'll all be in for trouble, as it'll pull China into any conflict, and quite frankly all China wants to do is business while autocratically ruling its (large) corner of the globe. The gambit now is weighing the possibility of strong economic ties between China and Russia, and the longevity of Putin's leadership.

Overall, this conflict in Ukraine is really bad news. MH15 was only the tip of the iceberg. I personally feel it should be handled firmly and swiftly, before it escalates any further, because, quite frankly, what amount of territory is "enough" for Russia/the separatists.

[1] [https://en.m.wikipedia.org/wiki/BRICS_Bank](https://en.m.wikipedia.org/wiki/BRICS_Bank)</comment></comment>


Population Equivalence Maps with Geopandas - <AUTHOR> http://linderb.org/geopandas/pandas/2019/04/13/population-equivalence-maps.html <comment><USER2> I'm surprised geopandas is all under development. A couple years ago it seemed dead in the water.</comment>


Other top secret Apple projects found at a bar - <AUTHOR> http://slideshow.techworld.com/3221670/other-top-secret-apple-projects-found-at-a-bar/?cmpid=TD1F3&no1x1 <comment><USER2> Slow news day?</comment>


Digital Nomads: The Big Picture – A Comprehensive Look into the Lifestyle - <AUTHOR> https://medium.com/@fluidsonic/the-big-picture-of-digital-nomadism-9afb2208a76 <comment><USER2> I had a good chuckle at this posts tone -- it sorta make you think if digital nomads as some species.

I have never set foot in an office to work in my life (I'm 27), and I make 100% of my income from location-independent work.

I can agree with the point about loneliness - it truly can feel lonely sometimes in unexpected ways. I'm currently in Amsterdam where I feel quite OK, but Balkan countries are one of those places that you have to try hard to socialize.

There is also a point about costs. I tend to be on the budget traveler end, but I splurge once Inna while on a good place. However in the long end, it can save you a lot of money because I'm not paying rent, taxes (I only pay ~5% because I don't live in my own country for the majority of the tax year), car payments, internet, etc.

<comment><AUTHOR> Thanks for the heads up! I guess my life as a software developer made me a bit too scientific here.

Have you found a way that helps you with loneliness?

<comment><USER2> Hiking and cycling communities tend to be quite friendly. I cycled Europe quite a lot, and have met many similar minded people.

Long distance hikes are a good way to meet cool people too. Europe is full of them, and the longer hikes mean you get to meet and make friends with people for a few days.

For bigger cities, free walking tours are both fun and a socializing place.

</comment></comment></comment><comment><USER3> First off, thanks for the comprehensive articles. You did a much better job than most of the digital nomad pieces I've read.

Also thanks for the cautions on taxes, and advice to get professional advice. Americans especially can get surprised when they find out the IRS follows them everywhere.

But I do have to question the 4.1 million American DNs, and the 16 million aspiring American DNs. Those numbers don't come close to matching my own experience as a digital nomad, or itinerant freelancer as I call myself.

The numbers for US digital nomads don't make sense. Either the definition includes so many people as to make the term meaningless, or the survey methodology has severe flaws.

4.1 million digital nomads from the USA. Consider that the entire US military, all branches, including reserves, comprises just over 2 million Americans. Most Americans know someone in the active military or reserves, but few Americans even know what "digital nomad" means. Do you think the 4.1 million number seems credible? That would mean about 5% of all Americans who traveled in 2018, or 10% of non-business travelers, call themselves digital nomads. If you look at the nature and duration of American travel (a week or two in Europe, or a spring break trip to Mexico) it doesn't add up that a huge percentage of those travelers qualify as digital nomads.

The US State Dept. estimates that 9 million Americans live abroad, not including military posted overseas. The vast majority of those are retirees, followed by employees in overseas offices. Do they count as digital nomads? If grandma and grandpa retire to Ecuador, post photos to Instagram, and make a few dollars from a rental house back in the US, does that make them digital nomads? What about a 20-year-old taking a gap year frugally doling out savings from a summer job in Chiang Mai while trying to write a game for the app store. Digital nomad?

I have traveled and lived abroad for almost ten years, working as a freelancer, staying in many countries. I rarely meet digital nomads even though I sometimes stay in cities (Chiang Mai, Ubud, Ho Chi Minh City, Amsterdam, Prague, Austin) supposedly teeming with them. I don't meet many of them in co-working places or coffee shops either. And I'm not a shy introvert — I go out of my way to meet other travelers, and go out of my way to talk to other Americans when I hear them speak. Anecdotally they make up a tiny percentage of travelers I meet, far outnumbered by regular tourists. 4.1 million American nomads, and presumably an even larger number of digital nomads from other countries, should have more of a visible presence. I know more German, Russian, Finnish, even Estonian digital nomads than I know Americans doing it.

I don't mean to gatekeep, but the 4.1 million number just seem ridiculously high.</comment>


Your comfy chair might be making you soft - <AUTHOR> http://scopeblog.stanford.edu/archives/2010/06/hard-chair-hard-heart.html

<comment><USER2> The article is not about chairs people, read it first before commenting.

</comment><comment><USER3> * clicks through to article __*

* clicks through to full text of the research __*

* slams into paywall __*

Gah! And how is one supposed to verify any of the information in such an article, without paying the gatekeeper? It strikes me as the epitome of ethical failure in science to block access to knowledge, or to only sell it, or to somehow ration it to specific people.

<comment><USER4> Pay-per-view articles in scientific publications present a significant drag on progress and pose an ethical dilemma of the first order. Science (information and data ought to be free) and a market economy (I have something you want and will sell it to your for the highest price I can negotiate) are antithetical. See Robert Laughlin's _The Crime of Reason and the Closing of the Scientific Mind_.

</comment></comment><comment><USER5> I'd also like to throw in music in there. A friend of mine did some research [sorry don't remember the article link] on negotiation and music influence before the negotiation. If I remember correctly, classical music made the negotiator soft. So, next time you want to be a tough negotiator, don't listen to classical music before and give a comfy chair to your opponent at a soft musicky place :-)

<comment><USER6> Of course you should keep in mind, that the objective of (most) negotiations should be to find a solution that's equitable to all parties. And the other guy should still be happy with his position after he slept a night on it.

<comment><USER7> The object of the negotiation itself is to seek equilibrium. That goal is best _achieved_ , though, by each _party_ to the negotiation acting out of pure self-interest, not ceding any unnecessary ground to the other. (The Nash equilibrium of the Iterated Prisoner's dilemma has both sides defect on every turn, in other words.) To "win" the negotiation, both sides should be primed "hard."

<comment><USER6> > That goal is best achieved, though, by each party to the negotiation acting > out of pure self-interest, not ceding any unnecessary ground to the other.

That's true for certain cases. In an iterated game, possibly with more than two players, enlightened self-interest could make people seek a reputation for fairness and generosity.

Also -- have you looked at the ultimatum game? It works like this: Alice proposes how to split a dollar between herself and Bob. Bob can accept the division, and both get paid out by the game, or Bob can reject, and nobody gets anything.

Real Bobs tend to reject `unfair' proposals, even when the game is not iterated. That's not rational (at least within the game). So if Alice where driving as hard as she could go with a rational Bob, she won't get anything when playing with Humans.

<comment><USER7> > Real Bobs tend to reject `unfair' proposals, even when the game is not > iterated. That's not rational (at least within the game).

It is rational when you add to the game the human consideration of post-game retaliation. The threat of revenge for injustice (whether by the loser, or an arbiter) creates the potential for "fairness and generosity." Real Bobs are made to stop playing.

<comment><USER6> Yes. But Humans are so hard-wired on it, that this behaviour persists even in anonymous one-shot sessions.

<comment><USER6> Played via computer without any face to face contact.

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> I've had like a $5 chair that kills my back (jabs into it) whenever I slouch against it for the past year. Every couple of weeks I think to myself: "I really should go buy a more comfortable chair" and then I reprimand myself because I feel like if I'm slouching I'm not working as hard as I could be. People have told me I'm masochistic, but I'm happy to see someone else who thinks I'm not being totally irrational.

<comment><USER9> I'm of the mindset that the objects you use most are the ones that should be given the most attention, simply because extra consideration and money spent is most likely to affect your quality of life.

For me, it would be:

* Bed * Computer * Desk chair * Shoes * Phone

... and so forth. So I won't think twice about spending extra on these objects since I spend the majority of my day interacting with them.

For seating, my rule is it has to be comfortable enough that I can fall asleep on/in it, and that I shouldn't notice the object I'm sitting on.

Besides, humanity has designed millions of chairs throughout history. Millions of iterations. There's no excuse that they still produce chairs that are genuinely uncomfortable, and you owe it to yourself to find one that really does work for you.

<comment><USER3> To drag this thread further off topic...

My wife and I hunted around for a while, and bought a frickin' _awesome_ mattress. It does wonders to your attitude to not wake up sore and stiff. And I like that my toes finally don't hold up the end of the sheets (california king FTW, though finding sheets for it is a PITA).

And, personally, I think people pay _far_ too little attention to their feet. Good shoes are worth the money (unless you're a barefooter, in which case no shoes is worth the money (or Vibrams)).</comment></comment></comment>


Speedy Scala Builds with Bazel at Databricks - <AUTHOR> https://databricks.com/blog/2019/02/27/speedy-scala-builds-with-bazel-at-databricks.html <comment><USER2> Super interesting, @lihaoyi do you think mill will stick around as an alternative tool or could you see it becoming a front end to bazel?

Were there any SBT plugins they found themselves really missing?</comment>


Coinbase – Ethereum Sends Delayed - <AUTHOR> https://status.coinbase.com/incidents/wvhcfrftmdlx <comment><USER2> Thank you... I have Ethereum at Coinbase. Disturbing. Hopefully accounts were not hacked. This is the downside of digital currency: all transactions are final whether you instigated them or not. Would be nice if you could yank your money back if can prove you got ripped off.</comment>


$5000 Reward: Best Acquisition Marketing Director on Earth - <AUTHOR> http://betashop.com/post/8103638456/5000-reward-im-looking-to-hire-the-best-darn

<comment><AUTHOR> Seriously, do you know such a person?

I'm looking for an analytical genius who has experience with cohort analysis and with managing large online ad campaigns, facebook ads especially.</comment>


Search 100M Identities in Less Than One Second with the Trueface SDK - <AUTHOR> https://medium.com/trueface-ai/search-100-million-identities-in-less-than-one-second-with-the-trueface-sdk-a-tutorial-98b1b95448a6 <comment><USER2> Very informative post! Looks like you guys are doing great work at Trueface!</comment>


CVE-2019-5736: runc container breakout - <AUTHOR> https://seclists.org/oss-sec/2019/q1/119 <comment><USER2> Amazon employee here: we have released a security bulletin covering how to update to the latest patched Docker on Amazon Linux, Amazon ECS, Amazon EKS, AWS Fargate, AWS IoT Greengrass, AWS Batch, AWS Elastic Beanstalk, AWS Cloud9, AWS SageMaker, AWS RoboMaker, and AWS Deep Learning AMI.

Please check out the bulletin and update if you are using one of these services.

[https://aws.amazon.com/security/security- bulletins/AWS-2019-...](https://aws.amazon.com/security/security- bulletins/AWS-2019-002/)

<comment><USER3> As far as I understand, EKS doesn't support PodSecurityPolicy yet so any user that can launch a pod can trivially root the host via host mounts already. This surprisingly isn't clearly documented.

<comment><USER2> ECS doesn't have a top level resource called "PodSecurityPolicy" but we do provide task level configuration options for all the major settings that you would normally put in your pod security policy, including including adding and dropping capabilities, privileged or unprivileged mode, docker security options for controlling SELinux or AppArmor, ulimits, sysctl settings, among others. You can find all these configuration options and more documented here: [https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definition_security)

It is definitely possible to prevent a task running in ECS from getting root access to the host. If there is something missing that you feel we need to add to ECS to better enable this, definitely reach out and let me know!

<comment><USER3> I'm referring to EKS here not ECS. EKS doesn't yet enable the PodSecurityPolicy admission controller, so any user that can launch a pod via EKS can root the EKS cluster regardless of RBAC rules. The main ask here is to just find a way to enable PodSecurityPolicy admission controller so that secure multi-user EKS clusters are possible like ECS.

</comment></comment></comment></comment><comment><USER4> Hey all - Seth from Google here. Please let us know if you have any questions regarding GKE or questions about the upgrade process. I'm also happy to escalate any feedback to our internal product and engineering teams.

Here's link to our security posting with more information and upgrade procedures: [https://cloud.google.com/kubernetes-engine/docs/security- bul...](https://cloud.google.com/kubernetes-engine/docs/security- bulletins#february-11-2019-runc)

<comment><USER4> One thing I want to emphasize: you are only affected if you're using Ubuntu base images for your node pools. If you're using COS, you are unaffected.

<comment><USER5> It would be great if folks could run COS and leverage tools like Falco. I realize there is a trade-off between having a totally locked down OS and being able to flexibly use such tools.

However, Google and Sysdig announced a partnership around Falco and GCSCC integration. It would make sense that such a tool would be able to be run on COS.

Perhaps I'm guilty wanting to have my cake and eat it, too. But this seems like an area where GKE and COS are somewhat limited.

<comment><USER6> Hi, Falco developer here. We do have support for running falco with an ebpf program taking the place of the kernel module. You can learn more about ebpf support at [https://github.com/draios/sysdig/wiki/eBPF](https://github.com/draios/sysdig/wiki/eBPF), and you should be able to run falco with ebpf by setting an environment variable SYSDIG_EBPF_PROBE="".

<comment><USER5> This is awesome news! I see it's still beta, which is probably why the Falco docs still say GKE users must run Ubuntu images. Adding this to my tracking list. Thanks.

</comment></comment></comment></comment></comment><comment><USER7> > However, it _is_ blocked through correct use of user namespaces (where the > host root is not mapped into the container's user namespace).

In other words, this won't affect anyone who understands the implications of running a process as root. Unfortunately, the sad truth is that most people I've come across who have "lots of experience" with implementing Docker containers, do not even understand the basics of how they work, let alone the implications of root access. I've interviewed candidates who claim to know Docker but can't even tell me how Docker differs from traditional virtualisation or how it achieves its isolation. The best explanation that most of them come up with is, "Docker containers are more lightweight".

This sort of vulnerability should have been a non-issue but it has gained attention due to the sheer amount of incorrectly configured containers in the wild. This was an accident waiting to happen, and I doubt we've heard the last of this sort of thing.

<comment><USER8> I think it's subtler than that. It is _mostly_ safe to run a contained process as "root" because in theory the ways that root access can be exercised is highly sandbox by the use of various namespaces, as well as things like capability restrictions (you generally don't have CAP_SYS_ADMIN or a few others), limited syscall attack surface (you generally have a syscall allowlist via seccomp-bpf), etc. Yes, it's wrong to not understand that the runc process runs as root. But I think it's only very slightly less wrong to claim that a process inside the container has root access in the way that, say, ssh root@host-system has root access. It mostly does not, and this vulnerability is notable precisely because it's one of the rare ways to exercise that root privilege outside the container.

We looked at this at $work and got into a serious rabbit hole about how exactly Linux capabilities work. I think if I started asking interviewees to explain permitted vs. effective capability sets and how file and process capabilities differ, I'd never hire anyone. (And I think to figure out yourself how to "correctly configure" a container, you need to have at least some understanding of that.)

<comment><USER7> > I think it's subtler than that. It is mostly safe to run a process as "root" > because in theory the ways that root access can be exercised is highly > sandbox by the use of various namespaces, as well as things like capability > restrictions (you generally don't have CAP_SYS_ADMIN or a few others), > limited syscall attack surface (you generally have a syscall allowlist via > seccomp-bpf), etc. Yes, it's wrong to not understand that the runc process > runs as root. But I think it's only very slightly less wrong to claim that > runc has root access in the way that, say, ssh root@host-system has root > access. It mostly does not, and this vulnerability is notable in that it's > one of the few ways to exercise that root privilege.

> We looked at this at $work and got into a serious rabbit hole about how > exactly Linux capabilities work. I think if I started asking interviewees to > explain permitted vs. effective capability sets and how file and process > capabilities differ, I'd never hire anyone. (And I think to figure out > yourself how to "correctly configure" a container, you need to have at least > some understanding of that.)

I think you've hit the nail on the head. I only ask those interview questions because I believe it's important to find out just how much a candidate understands. I have to admit I've let some of these things go, otherwise I'd never hire anyone either. I think in the end what it comes down to is that Docker is an ambitious project that is somewhat flawed from a security perspective. There have been numerous namespace vulnerabilities to date and I expect there will be plenty more found in the future.

<comment><USER9> I believe there was work on non-root container host processes going on at some point? Did that ever get to a usable state?

<comment><USER8> Rootless containers using user namespaces was merged into runc in March 2017: [https://github.com/opencontainers/runc/pull/774](https://github.com/opencontainers/runc/pull/774)

A week ago Docker gained support for running dockerd as non-root: [https://github.com/moby/moby/pull/38050](https://github.com/moby/moby/pull/38050)

And there is this project for running Kubernetes as non-root: [https://github.com/rootless- containers/usernetes](https://github.com/rootless-containers/usernetes)

</comment></comment></comment></comment></comment><comment><USER10> Looks like Docker 18.09.2 was released a few minutes ago to address this: [https://github.com/docker/docker- ce/releases](https://github.com/docker/docker-ce/releases)

<comment><USER11> PoC here:

[https://github.com/feexd/pocs/blob/master/CVE-2019-5736/expl...](https://github.com/feexd/pocs/blob/master/CVE-2019-5736/exploit.c)

</comment></comment><comment><USER12> Red Hat's page on the vulnerability: [https://access.redhat.com/security/vulnerabilities/runcescap...](https://access.redhat.com/security/vulnerabilities/runcescape)

RH CVE page, with the vulnerability's metrics and the list of RH packages affected (plus links to the errata pages that have details on fixed builds): [https://access.redhat.com/security/cve/cve-2019-5736](https://access.redhat.com/security/cve/cve-2019-5736)

</comment><comment><USER13> There are nearly 4,000 exposed Docker daemons: [https://www.shodan.io/report/ol761bRb](https://www.shodan.io/report/ol761bRb)

<comment><USER14> So inexperienced developers are just as likely to reach for Docker as experienced ones?

I wonder how many of those 4,000 docker daemons are running/managing containers of dubious origin.

<comment><USER13> Around 10% of them are running cryptominers so it looks like there are already people out there compromising these public Docker instances.

</comment></comment></comment><comment><USER15> For better isolation check out KataContainers: [https://github.com/kata- containers/runtime](https://github.com/kata-containers/runtime)

It's a drop-in replacement for runc. With KataContainers it runs docker containers in a lightweight VM so you get all the security benefits of a VM. The downside is slightly slower container start up times and might not work in nested virtualized environments.

<comment> justanother- Alternative idea: throw away docker and katacontainers and move to freebsd, where jails were introduced on 14 Mar 2000 (no, seriously, superior technology exists for 19 years - stable, time proven, working).

Some more info: [https://www.freebsd.org/doc/handbook/jails.html](https://www.freebsd.org/doc/handbook/jails.html)

And for quick start: [https://github.com/iocage/iocage](https://github.com/iocage/iocage)

<comment><USER16> Jails are virtually identical technology to Linux containers from a security point of view. They've had holes before and they likely will again, and a breakout like this (seems like the root cause here is a writable file descriptor to the host binary) can absolutely compromise the host system.

The upthread recommendation was using hardware VM technology, which is a fundamentally different isolation model from what software can provide and (at least in theory) makes that kind of exploit impossible. And while there are tradeoffs with everything, for you to throw that argument out due to personal platform loyalty is really, really bad advice.

<comment><USER17> My understanding is that jails were designed as a security boundary from the get go, unlike containers. Wouldn't that result in code that's less likely to be exploitable?

<comment><USER16> FWIW, "containers" aren't a thing. Namespaces, cgroups et. al. certainly were designed with security in mind, as was docker/runc.

Look, this isn't about whether jails are secure containers or not. I'm sure they're great. It's that responding to "if you want more isolation, try hardware virtualization" with "FreeBSD is just better because 19 years!" is not really enaging with the argument as framed.

</comment></comment></comment></comment></comment><comment><USER12> Debian's security tracker, showing the affected versions, and (when available) the fixed versions: [https://security- tracker.debian.org/tracker/CVE-2019-5736](https://security- tracker.debian.org/tracker/CVE-2019-5736)

And Ubuntu's: [https://people.canonical.com/~ubuntu- security/cve/2019/CVE-2...](https://people.canonical.com/~ubuntu- security/cve/2019/CVE-2019-5736.html)

Personally, I like these vs. RHEL, since all the info is on page.

<comment><USER18> In addition to the information all being on one page you can also:

        git clone https://salsa.debian.org/security-tracker-team/security-tracker.git
        git clone https://git.launchpad.net/ubuntu-cve-tracker

There is a lot of interesting things you can do with the data.

</comment></comment><comment><USER19> The vulnerability description seems to be lacking an explanation why the /proc/$PID/exe symlink is so special and why using the #!/proc/self/exe hashbang will work while using #!/usr/sbin/runc probably won't. Am I right that the proc filesystem in proc_exe_link() fills the file_operations struct in a way that causes open() not to go through a dereferencing procedure using the filesystem but just open the file used to run the executable?

<comment><USER19> So I will answer myself. Experiments suggest it is like that: [https://www.reddit.com/r/linux/comments/apmptq/cve20195736_r...](https://www.reddit.com/r/linux/comments/apmptq/cve20195736_runc_vulnerability_enabling_container/egcc313/)

</comment></comment><comment><USER20> The best fix is to upgrade to 18.09.2. For those that can't do that immediately, backported versions of runc for Docker releases going back to 1.12.6 are available from Rancher at [https://github.com/rancher/runc- cve](https://github.com/rancher/runc-cve). But please only do that as a temporary workaround until you can properly upgrade to 18.09.2.

Please patch if you don't 100% trust all users on your host.

<comment><USER21> For systems where you already enforce security policies, updating policy is faster than upgrading software, due to fewer build processes, quality control issues, and potential side-effects.

If you are using SELinux, verify your containers are running as _container_t_. If not, verify you are using user namespaces that don't map host root into the container user's namespace. These should mitigate the issue.

(as far as trust goes, just don't trust any local users. there's too many ways to privesc on Linux, and SELinux is the only thing that stops most of them)

<comment><USER8> I believe that if you're in an environment where users don't have (and can't gain) root inside the container, you're also fine, and if that's your theoretical policy, getting to the point where you 100% enforce that might also be easier than patching.

</comment></comment></comment><comment><USER22> Mesosphere employee here. We have released the product advisory on this CVE. Please check out the advisory and update your software.

[https://support.mesosphere.com/s/article/Known-Issue- Contain...](https://support.mesosphere.com/s/article/Known-Issue-Container- Runtime-Vulnerability-MSPH-2019-0003)

</comment><comment><USER23> Next year this exploit will still be in thousands if not millions of containers all over

There is a distinct lack of knowledge on how to manage a system in the container ecosystem

<comment><USER24> What do you mean _in_ containers?

</comment></comment><comment><USER25> Not that this shouldn't be patched and all, but this seems like it is being treated with more urgency that is required.

If I am understanding the CVE correctly, you need to be able to launch _privileged_ containers with an attacker-controlled image where the container user is root _and_ not namespaced (i.e., the same root as the outside root user). How is this not "on the wrong side of an airtight hatch[1]"?

Am I missing something here? If you can start privileged containers, why not just execute evil.exe directly?

[1]: [https://blogs.msdn.microsoft.com/oldnewthing/20060508-22/?p=...](https://blogs.msdn.microsoft.com/oldnewthing/20060508-22/?p=31283)

<comment><USER26> I run a privileged container, of which I am not the author. I also know many other people run this container in privileged mode. No one is paying that person for it, and if they want to or they get compromised at some point we all might get rooted when we update the image.

I think my OS is on a read only filesystem though, and maybe I've got it namespaced correctly as well, but still it's pretty dangerous.

</comment></comment><comment><USER27> Is this something that non-privileged containers mitigates? Curious what the big barriers are to this. I know they exist, but aren't used widely...I assume because some functionality doesn't work.

<comment><USER28> Yes. The lxc commit[1] states that this issue only affects privileged containers. No CVE has been issued for lxc because they consider privileged containers to be insecure.

In my experience unprivileged containers work for most tasks, but there is breakage in some areas. Usually the issues are simple to resolve, like disabling OOM adjustments in systemd or changing the idmap range in winbind to be within the namespace allotment.

[1] [https://github.com/lxc/lxc/commit/6400238d08cdf1ca20d49bafb8...](https://github.com/lxc/lxc/commit/6400238d08cdf1ca20d49bafb85f4e224348bf9d)

<comment><USER29> I've also written a smallish blogpost about this CVE. I'm a LX{C,D} maintainer and I've worked with Aleksa the runC maintainer together on a fix for this CVE: [https://brauner.github.io/2019/02/12/privileged- containers.h...](https://brauner.github.io/2019/02/12/privileged- containers.html)

<comment><USER30> Thank you @brauner for writing this blogpost!

IIUC, using Docker's userns-remap would protect against this CVE by making the containers run unprivileged (container's id 0 != host's id 0) and should generally be the industry's best practice.

</comment></comment></comment></comment><comment><USER31> Is this issue specific to containers running as root?

<comment><USER32> Yes. You need to be able to run a container as root (or rather, as a user which has write access to the host runc binary -- which is usually root). User namespaces protect you for this reason.

</comment></comment><comment><USER33> Yet to see an Ubuntu Security Notice released, I'm presuming an update to the docker.io package will be released?

<comment><USER34> docker.io and runc are in universe so not officially supported by the Ubuntu Security team so won't get a Ubuntu Security Notice

</comment></comment><comment><USER35> Yikes that's a big patch! Just on a meta-level, security vulnerabilities fixed with big patches are usually the least fun.

Also, I would bet that freshly written C code has about 1 RCE bug every 100 LoC. This patch has 236 LoCs so probably about 2.36 RCE's.

<comment><USER36> By your guess each new Linux release would have about 5000 new RCEs. So that's 25000-30000 new RCEs over the last year alone.

<comment><USER37> In 2018, 68% of Linux CVEs were caused by C's memory corruption features.

Source, Google talk at Linux Kernel Summit 2018.

<comment><USER38> And how many of those were related to the difficulty of managing IPC across user/kernel boundaries, DMA, mbuf-based networking code, eBPF JIT'ing, etc? So-called memory safe languages don't help with any of that.

I'll take your memory safe languages and up the ante with microkernels, which actually help immensely in all those cases. But Linux isn't going to go that route, either.

<comment><USER37> Plenty of them related to lack of bounds checking handling strings and arrays.

Google has been pushing for the Kernel Self Preservation project for quite a while now, which Android and ChromeOS make best use of, also a reason why the NDK is so constrained on Android.

Security in the kernel also had quite a few talks at Linux Conf 2019, just recently in New Zealand.</comment></comment></comment></comment></comment>


Yahoo! Auto-Resignation Tool - <AUTHOR> http://www.yahoorezinr.com/

<comment><USER2> Sorry but please can you stop this mindless Yahoo bashing? You can use Techcrunch for that if you are so inclined. Thank you.

<comment><USER3> It's disappointing that stories like this get 50+ points. I fear the non- commenting population here is much dumber than the commenters.

<comment><USER4> I imagine some of the posters have attempted similar jokes, or are even the authors...

</comment></comment></comment><comment><USER5> I'm disappointed that it doesn't encompass the "tin-smithing" letter.

</comment><comment><USER6> That doesn't pass the notability test in my book.

<comment><USER7> It is, however, hilarious.

<comment><USER8> I worry that votes based on hilarity alone are the downfall of social sites everywhere.

<comment><USER9> Defiantly.

Digg has coped "OK" because you can just go into the technology/science sections and find what your interested in, but the front page stories and the comments which go a long with them however, make the average Myspace page look sophisticated. I guess that's as good as it gets when you open your universal service to the world, along with the stupid people which occupy it, with all their cheap-gags.

I just hope Hacker News will stay as close to it is now, for years to come! :)

<comment><USER7> I'm trying to figure out whether your "Defiantly." was an accidental typo or an intentionally ironic nod to digg/youtube style comments where nobody actually knows how to spell "definitely."

edit: that is, <http://digg.com/odd_stuff/D_E_F_I_N_I_T_E_L_Y>

</comment></comment></comment></comment></comment><comment><USER10> ... and is that Jerry's real email?

<comment><USER11> hahahhahahaa

<comment><USER12> Damn, these comments are really faded.

Can they be set up so the text is visible when highlighted?

<comment><USER13> That depends on your OS/browser. You could set a user style up (with Stylish for FF for example)</comment></comment></comment></comment>


A Beginner's Guide to Scaling to 11M Users on Amazon's AWS (2016) - <AUTHOR> http://highscalability.com/blog/2016/1/11/a-beginners-guide-to-scaling-to-11-million-users-on-amazons.html <comment><USER2> Here's some anecdata from someone serving 1.5 million monthly users with a realtime (chat etc.) web app. Disregarding the file serving functionality, this is the entire infrastructure:

\- The frontend is running on a 30 bucks/mo VPS with cycles and RAM to spare. \- There's 2 DB servers for 5 bucks / mo for redundancy.

This thing would scale pretty linearly in theory, so for 10 million users just add like 4 servers.

So about 200 bucks / mo to run your average startup's web app at 10 mil users.

Alternatively write bad software and throw 50k/mo at AWS I guess.

<comment><USER3> What's the typical QPS like?

<comment><USER2> Since it's essentially an one-page app I can do a lot via persistent websocket connections (of which there are usually ~25k concurrently).

</comment></comment></comment><comment><USER4> > Amazon S3 is an object base store. > Highly durable, 11 9's of reliability.

My eye twitched when I read this, considering that I remember more than one aws outage that affected s3 availability/reliability.

AWS own pages mention 10 nines for /durability/, however, I doubt this durability number, this level is almost unachievable by any end customer product. Could it be some clever durability accounting?

I'm quite disappointed on AWS service status reporting when the service may be disrupted so so badly, to the point of not being serve a single request, however status pages not mentioning problems at all, or just reporting "increased error rates".

<comment><USER5> The quoted 11 nines durability number is saying that you won't lose your data. Availability has nothing to do with it. It seems the OP used the wrong word and should have said "durability" instead of "reliability".

> this level is almost unachievable by any end customer product.

Not sure what your point is here, unless you're highlighting the benefits of using a cloud provider. AWS is not using an "end customer product" to implement S3. They're providing a managed service, and their implementation and management is what allows it to achieve those levels of durability.

</comment></comment><comment><USER6> One important thing it doesn't discuss is the insane AWS costs compared to the other clouds and even regular self-managed solutions running on cheap VPSes and dedicated servers.

<comment><USER7> When you get network level ddos on digital ocean, they can't save you.

This rules out small clouds for us.

So, we use GCP Azure AWS exclusively because of their ability to defy network level ddos.

<comment><USER4> How do you manage to mitigate a DDOS on a public cloud?

When you say GCP, Azure and AWS have the capacity to defy the ddos, what capacity are you reffering to?

Are you talking about actually scaling and serving the bogus requests? Or capacity to have enoough bandwith and firewall power to fend it off?

<comment><USER8> I can only talk for AWS, not GCP or Azure, but there are services that can help mitigate DDOS attacks:

Shield ([https://aws.amazon.com/shield/](https://aws.amazon.com/shield/)) is AWS's DDOS protection service. It's free and is basic protection against L3/4 attacks.

Shield Advanced (same URL as above) is a big step up in price, but gives you access to 'improved protection' and a global response team.

Cloudfront ([https://aws.amazon.com/cloudfront](https://aws.amazon.com/cloudfront)) is a CDN with global edge locations.

WAF ([https://aws.amazon.com/waf](https://aws.amazon.com/waf)) is AWS's web application firewall service. It's less about DDOS and more about specific application attacks, but is part of the whole solution.

For more detail, you can have a look at AWS's DDOS whitepaper: [https://d0.awsstatic.com/whitepapers/Security/DDoS_White_Pap...](https://d0.awsstatic.com/whitepapers/Security/DDoS_White_Paper.pdf)

</comment></comment></comment></comment><comment><USER9> And just remember, public clouds are expensive. If you need to handle millions of requests per second then there is a very high probability that you can afford a dedicated infrastructure team and save a ton of money while working just as well.

Just look at the insane(!!!) bandwidth costs...

<comment><USER10> I'm a bit naive when it comes to infrastructure, since I've never managed it myself. I've always been in companies with dedicated infrastructure :).

Why does Netflix outsource to AWS?

The gripes I've had with dedicated infrastructure team is their knowledge/experience can be hit or miss. My current company is about saving money, so sometimes there's junior guys on there. Or there's guys who need to handle 20 different products, they're good at 10..but suck at the other 10. I personally don't believe they have best practices (but I'm not an infrastructure expert), because downtime happens.

<comment><USER11> AFAIK, Netflix does not _stream_ out of AWS, but instead "just" runs their application services on AWS (which is still a significant workload).

I seem to recall they offered a rack full of their own caching equipment to medium and larger ISPs (to save both sides streaming costs while providing better service to Netflix customers). Basically a private app-specific CDN.

<comment><USER12> It's true. If you open Web Inspector while streaming a Netflix movie, the URL it's getting the stream from probably contains your ISP name and region!

</comment></comment></comment></comment><comment><USER13> They say "you need to add monitoring, metrics and logging" at the 500k user mark. That should be happening with user 0 with few exceptions that I can think of.

<comment><USER14> I feel like this is the reason stuff doesn't get shipped. If you couldn't release v.01 without adding in a robust analytics suite, nothing would ever get released.

Nothing wrong with releasing a .01v and not putting priority on anything but basic functionality until you start to grow. Half of this means you need to build out functionality that doesn't require heavy monitoring, metrics, and logging to get the job done. All that infrastructure stuff is more fun to code when you've blown through the free tier of S3 storage anyway.

<comment><USER13> Nothing heavy needed. You should know that your thing is working. A basic health check. You should know what errors are happening with basic logging. As soon as you have a paying user, you should have some form of reporting or alerting on this. You should have some basic metric(s) in place very early on like requests per second or sign ups. This should take you in the order of hours, not days, to set up. Nothing heavy. But without basic visibility into your system, you are asking for problems.

<comment><USER15> What tools would you use to set this up?

<comment><USER16> Depends on your stack and requirements (do you want to know about errors ASAP, or is a 2-5 minute delay ok?), but I personally love NewRelic because of how easy it is to set up (and the number of features that it has).

If you want tools that you can manage yourself, then a combination of StatsD + Grafana for metrics, and Sentry for errors. For logs, Graylog if you want to set up something complicated but powerful, and syslog-ng if you just want to dump the logs somewhere (so you can simply grep them).

<comment><USER17> Most of these tools cost too much to scale past 1M users.

<comment><USER18> You could write your own service... some thin agent that runs on your boxes and dumps the files every hour to some storage optimized boxes (your data lake)... where another process picks up those files periodically (or by getting a notification that something new is present) and loads them into a Postgres instance (you actually probably want column oriented).

Running every hour, you won't get up to the second or minute data points. For more critical sections of code, maybe have your code directly log to an agent on the machine that periodically flushes those calls to some logging service.

<comment><USER17> 1\. Collectd and Graphite serve this well or can be modified

2\. Nine of these commercial services gives you per second granularity that I have seen

<comment><USER17> s/Nine/None/

#if there are 9 i'd like to know of them :-)

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER19> This is a written summary (like all High Scalability articles) of this video: [https://www.youtube.com/watch?v=vg5onp8TU6Q](https://www.youtube.com/watch?v=vg5onp8TU6Q)

</comment><comment><USER20> Some of this advice seems like wildly premature optimization. Splitting into multiple hosts and/or using a database service at 10 users?

At 10 users, you should be focusing on growth, not fussing with your tech stack.

</comment><comment><USER21> I wrote the first version of this deck back in 2013: [https://www.slideshare.net/AmazonWebServices/scaling-on- aws-...](https://www.slideshare.net/AmazonWebServices/scaling-on-aws-for-the- first-10-million-users) and it since then went on to become one of the most delivered talks by AWS across the world. I thought I'd pop into this thread and answer some questions/help provide guidance as to why it exists. Note that first deck was mostly tossed together while on an Amtrak from NYC to Philly in an early morning, so excuse the typos and graphics glitches.

Generally speaking we've/I've seen a number of customers struggle with pre- optimization of their infrastructure and/or just not knowing what they should think about and when, when it comes to scale and architecture patterns. Think of this deck as an 80/20 rule for the most general and basic of people. Also, 2013 was a very very different world in the cloud and in infrastructure than today.

This deck was all pre-Docker/k8s, pre-Lambda/FaaS, pre-alot-of-cool-db-tech. However as a general starting point a lot of it still holds just fine. You _should_ probably start with a relational DB if you have a more traditional background using them. You _probably_ don't need autoscaling Day 1 (but it's nice via things like ElasticBeanstalk).

Someone commented that metrics/logging/monitoring is a Day 0 thing, and it absolutely is, but it is the kind of thing most companies skimp out on until very late in the game. Not pushing for excellence in this area will block you from success down the line. The same is hiring someone with dedicated Ops/SRE responsibilities, and/or making sure someone on the team has the right experience in that space.

DB growth/scaling is now a lot more transparent than it used to be, but I still see people doing less than ideal things with them. This is an area the industry needs to be sharing more best practices on (while still respecting some of the best practices from decades of SQL).

On costs; today some of the biggest sites in the world run on a public cloud. They do that for many reasons, but cost effectiveness at scale is pretty well proven when measured the right way. Most math ignores people and opportunity cost and that's what get's you more than metal and power. There is also now today the wealth of managed services that equate to replacement of people with a returned value greater than the people cost (essentially the ROI on managed services outpaces the value of you doing it yourself greatly). The SaaS ecosystem is also vastly larger than it was in 2013. I hope to never run a logging cluster, monitoring systems, backup tools, etc, myself again.

Anyway, glad to see this deck still kicks around and that people are discussing it. Happy to try and answer more questions on this. - munns@

</comment><comment><USER22> A glaring omission in AWS offerings is a NewSQL database that let's you start with SQL, have redundancy and scale horizontally having consistent reads the whole time. Scaling to 10M+ users and then switching datastores makes for good story writeups because it's hard. It's also avoidable.

<comment><USER5> Doesn't Aurora fit that bill?

<comment><USER22> Aurora has good replication but it's still just master/slaves rather than sharded. DynamoDB is sharded but nosql.

<comment><USER5> Thanks, wow, I guess I just never dug deep enough past the Aurora marketing, which I see now actually talks about its "distributed storage system".

</comment></comment></comment></comment><comment><USER23> I don't think it's necessarily a good idea to use SQL for everything. If you are constantly inserting records and don't need relational structures, DynamoDB might be a better choice. DynamoDB scales quickly and cheaply, so it's potentially the preferable option if it works in your architecture.

<comment><USER24> It's relatively easier to iterate your data storage with relational tables though. Dynamite is great if you are sure a component just needs a k/v store, and you are sure you will never want to do a join.

</comment></comment><comment><USER25> I think the guide does not work when you have less users and more data, for instance if you are a b2b saas provider. I think the way you architect an infrastructure should be driven by the value of the user.

</comment><comment><USER26> (2016)

<comment><USER27> Does anybody have an equally short summary of any differences today?

By the way, there are links below the text, here's the one to the HN discussion of 2016: [https://news.ycombinator.com/item?id=10885727](https://news.ycombinator.com/item?id=10885727)

<comment><USER7> Today we use

For database, we use RDS/Dynamodb

Redis for cache.

Dynamodb is better in cases where we want to localize the latency of our regional lambdas.

RDS for everything else like dashboard entity storage etc..

Cloudwatch prints logs, kinesis takes the log to s3 where it's transformed in batches with lambda then data is moved to Redshift. Redshift for stats/reports.

Converted whole ad network to Serverless. Used Rust Lambda runtime for CPU intensive tasks.

Using Go for the rest of the Lambdas.

I love Go and Rust and optimizing just one Lambda at a time brought back the programme joy in my life

Used Apex+Terrform to manage whole infra for the ad network.

We managed to deploy Lambda in all AWS regions ensuring minium latency for the ad viewers.

The thing which took over 50 (tech side only) person team, now takes 10 people to run the whole ad network.

Network is doing 20M profit per year/9 billion clicks per day.

It's my greatest achievement so far because we made it from scratch without any investor money.

But one the other side, we'll have to shrink our team size next year as growth opportunity is limited and we want to optimize efficiency further.

<comment><USER28> Pretty awesome!

I'm currently planing to write a book about AWS. It should teach people how to build their MVPs without restricting themselves in the future.

Are you available for an interview in January?

<comment><USER29> I've been looking for a book on this topic for a while now!

<comment><USER28> Some people told me they were searching for this.

I think I'll put up a small splashpage for email gathering in the next days to keep people up to date :)

</comment></comment></comment></comment></comment></comment><comment><USER30> Is this still valid ?

<comment><USER31> Yeah absolutely. Things change fast but not _that_ fast :-)

That said it's not a perfect guide. You need to adjust based on area of expertise and comfort, and also any business requirements. For example in most businesses protecting the database is important, so running only one instance with no backups/snapshotting would be bad, reckless even. If it's just a hobby project tho, sure. I've done it several times just to deploy a simple POC that can be actually used.

Is there anything specific you had a question about?

<comment><USER32> Offtopic: I've never worked in anything that has > 100 (200 on good days) users. When you have >10k or 100k users, what really changes? Do you have to rewrite everything? Do you spin up more machines? Profile & optimise any/all bottlenecks to eek out more perf? Am really curious as, am not sure how something of that scale looks like. I always imagine those always are rewritten in C/C++/Java or Rust(current years) or have very special architecture etc. :)

<comment><USER31> haha, nope I've scaled ruby on rails apps (ruby is one of my favorite languages so I say this with love: it is fat and resource hungry) to huge levels. Basically:

> _Do you spin up more machines?_

Is correct. We make sure that the app itself is stateless (very important for horizontal scalability), and then I set up auto-scaling groups in AWS (or more recently actually, pods in kubernetes that auto-scale). For database I use sharding, tho lately I've gotten away from sharding because it complicates the database queries and you would be amazed at how far you can vertically scale an RDS Aurora instance.

You do have to profile expensive database queries tho. Basically if your app is stateless and your database queries are performant, you can scale really far by just adding instances (more machines).

That said, there is a point where re-writing is attractive: cost. Adding these instances is expensive. There was a great blog post that I'll try to find where a service in RoR had an app server for every 400 to 500 concurrent users, and rewrote in Go and was able to use one app server for every 10,000 concurrent users.

<comment><USER32> wow! Thanks for taking the time to explain. Your idea of stateless apps gave me a new perspective, something to try on next project. The article you mention should also be an interesting read. :)</comment></comment></comment></comment></comment>


An exercise in profiling a Go program - <AUTHOR> http://thornydev.blogspot.com/2015/07/an-exercise-in-profiling-go-program.html

<comment><USER2> In my experience, almost all overhead in Go programs will result from being bitten by memmove, alloc and GC. Profiling any program that uses interface{} and byte buffers will quickly expose you to how much the runtime loves to reallocate and copy objects. In one instance I got 10x throughput improvement by switching from a generic interface (take an interface{} and detect the type) to using typed methods and hand-rolling a shim that called the appropriate methods.

<comment><USER3> Do people actually use `interface{}` so liberally? I've never encountered it in the wild (except by newbies or as an example of what _not_ to do) and I've always been under the impression that it's been heavily discouraged for a very long time.

<comment><USER4> interface{} is heavily used in many of the Go std libraries. Particularly for what I call "convenience methods" where you want to be able to pass in various types and it will do a type switch for you and then determine what to do.

Part of what I was discovering in this blog post is that if you look under the hood at these convenience methods, and you know in advance what type are passing (i.e., you are not using interface{}), you can often find the "direct" call to use that uses a concrete type and likely get better performance out of it.

</comment></comment></comment><comment><USER5> Nice read. I love the pprof tool: being able to pinpoint which line of code (or line of assembly code even) takes the most time is awesome.

<comment><USER6> While very neat in my experience this has two issues (using Python's line_profiler so it may not have the same problems):

1\. it adds significant overhead to execution speed which means savings under line_profiler and savings without it may only be distantly related, not sure how much overhead pprof adds

2\. it requires knowing which functions should be line_profiled, because when you have thousands or millions of LOCs, you've got no idea what to line- profile

I've never found "usual" whole-program profilers to be great at the latter, it may just be that I'm bad at reading them but they never really click, and when you've got a few "leaf functions" called from basically everywhere, they end up having a low SNR. Recently however I've started using sampling profilers and flamegraph representations[0] (or sunbursts, but there's no standard tool for that one) and found it to be a significantly superior way of identifying bottlenecks with very high SNR.

[0] [http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html](http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html)

<comment><USER7> "While very neat in my experience this has two issues (using Python's line_profiler so it may not have the same problems)... Recently however I've started using sampling profilers"

pprof is a sampling profiler. You can get it from Google for C, too. I'm not sure if Go has a "port" or just wrote something with the same ideas, but, well, I guess you could say I don't know precisely because it hardly matters.

<comment><USER6> > pprof is a sampling profiler.

Good to know.

</comment></comment></comment></comment><comment><USER8> You should look into the GODEBUG env var ([https://golang.org/pkg/runtime/](https://golang.org/pkg/runtime/)) for ways to get an idea of what all your goroutines are doing

</comment><comment> zupa-hu the comments on the page seem to go in blackhole

<comment><USER4> I'm the blog post author. I haven't disabled comments (I welcome them) and I don't see any pending ones that I need to approve. I'm not super happy with Blogger, so I'm going to blame it on that. There is one other comment on this post, so the problem probably has to do with the "Comment As" choice and whether you are logged in to that service.

<comment> zupa-hu I tried posting before authenticating with my google account; it sent me to google to log in, I did, I returned, comment nowhere, but I _was_ logged in this time. So I went ahead again, commented again, didn't appear. Weird.

Btw I read somewhere that it's best use N-1 cpu-s for actual work and spare the last one for the scheduler so that the goroutines will not be reassigned to different threads thus cpus all the time because that means lots of memory movement. I suspect that was your 9th goroutine.</comment></comment></comment>


Chinese auto giant pours $29B into dystopian, blockchain powered mega city - <AUTHOR> https://decrypt.co/7539/chinese-auto-giant-pours-29-billion-into-dystopian-blockchain-powered-mega-city <comment></comment>


Dijkstra's algorithm and the Fibonacci heap - <AUTHOR> http://maryrosecook.com/blog/post/the-fibonacci-heap-ruins-my-life <comment><USER2> Please do note that, depending on the dataset size, it may be faster to just use a plain old binary heap. In this case, the big O cost for Dijkstra's algorithm is O((E + V) log V), as opposed to the better O(E + V log V) for a Fibonacci heap. Note, also, that Fibonacci heap operations are amortized time, so if you're not operating on a large enough dataset, the costs will not amortize enough and you may end up with a slower real time.

A binary heap also has better locality of reference, because you have more chances of a cache hit, especially when you're near the root. The Fibonacci heap, in contrast, keeps a number of pointers to memory locations that are all allocated on-the-go and possibly very far apart. If the binary heap is implemented as a B-heap, as described by PHK [1], it can be made even faster.

My point is: please benchmark. Always benchmark. "Stupid" algorithms and data structures may work much better for your case than you would have thought. Programmers are notoriously bad at finding the real bottlenecks... so please benchmark and profile. Fibonacci heaps, AVL trees, KMP, Coppersmith-Winograd, these are all little works of art and theoretically great solutions. With big data, they _will_ perform better than more naive solutions. But that big data doesn't happen as often as you might think. Computers are weird, have too many quirks, and their operation will surprise you, no matter how skilled you are. So profile your programs.

[1]: [http://queue.acm.org/detail.cfm?id=1814327](http://queue.acm.org/detail.cfm?id=1814327)

<comment><USER3> Thanks, this is a pet peeve of mine. Theoretical bounds are all nice and such, and it's cool if you can prove a better running time using some algorithm, but if it doesn't actually perform better in practice most of the time then it's not really worth much. I blame computer scientists who are in a rush to publish the best theoretical solution but never bother to consider whether it has practical impacts. Also textbooks who blithely claim you can do better with such and such a data structure using a certain algorithm.

See this StackOverflow question for some nice insights: [https://stackoverflow.com/questions/504823/has-anyone- actual...](https://stackoverflow.com/questions/504823/has-anyone-actually- implemented-a-fibonacci-heap-efficiently)

<comment><USER4> It is worth pointing out here that Quicksort has complexity of Ɵ(n²) in the worst case. Yet it has worked out as the best one in practice in many applications.

<comment><USER5> That's true, but it's also worth pointing out that the worst case can be trivially prevented by shuffling the array before you Quicksort it. Which is maybe a confirmation of your larger point.

<comment><USER6> Strictly speaking, it is not true that shuffling first avoids the worst case. Given a deterministic shuffling function, some permutation of input needs to shuffle to sorted order, which will again trigger the worst case.

Of course, shuffling is still potentially helpful because ascending/descending order is often common in many applications.

<comment><USER7> Randomization of the pivots produces O(n log n) average running time with a very low probability of any other running time for any value of n where the worst case running time matters.

<comment><USER6> Even without randomization of pivots expected run time is O(n log n). If the order of the input data is random, I don't believe randomizing pivots changes the distribution of running times.

What changes is that one very common permutation of data (ascending data) does not have O(n * 2) performance.

<comment><USER7> There's little justification for expecting data to behave randomly unless our algorithm introduces it.

<comment><USER8> Correct. In general, real data will seldom, if ever, look like random data.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> Cool article.

Dijkstra's algorithm factors into two components: (1) time spent selecting best paths and (2) time spent updating the best currently known paths.

In a naive implementation, a best path to every unvisited vertex is kept. Selection of a best path to a vertex v_0 is made by a linear scan of the paths (O(V) each time). Locking in a path causes us to possibly update paths to every vertex v_1 that is touched by an edge from v_0 (constant time per out edge of v_0).

Overall, time complexity is O(|V||V|) selecting paths, and O(|E|) updating paths. Since |E| is bounded by |V|(|V| - 1), the time is O(|V| |V|) (insensitive to density).

Using a min-heap to store best known paths, we spend O(log(|V|)) time selecting each path, but an update takes O(log(|V|)). This means the total time complexity is O(|V|log(|V|) + |E|log(|V|)).

In the case of dense graphs, this is worse: O(|V| |V| log(|V|)). We're trying to keep the heap organized to allow fast extract, but there are too many updates and the heap is changing too much iteration to iteration. OTOH, if the grpah is sparse, |E| is in O(|V|), so we reduce the time complexity to O(|V|log(|V|)).

A fib heap keeps the same extract time, but updates are O(1) amortized. Thus the time complexity is O(|V| log(|V|) + |E|). If the graph is dense, this is O(|V| |V|) (as good as naive). If the graph is sparse, this is O(|V|log(|V|)) (as good as bin heap).

This is useful if we do not know whether the graph is sparse or dense. However, I am not sure what the constants are on Fib heaps. If you know the density of the graph, I certainly expect using the appropriate of the first two ways is superior. Another thought: you could always speculatively execute both algorithms and see which finishes first :P

Anyway, I hope that's not too boring of me.

Edit: Grr. Asterisks.

</comment><comment><USER7> I've been taking Roughgarden's Algorithms: Design and Analysis on Coursera and implementing the assignments in Racket. So in a sense, I identified the author's plight with my own recent and current experiences.

One of the early lessons I learned is that designing functional versions of algorithms is a whole higher level of hard over and above implementing conventional versions. For me, getting to a functional or less imperative version is an iteration of an imperative preliminary design. There's a stage at which it is easier not to get bogged down with data structures as values and instead let them be variables because it allows me to simplify the transition to the next state as (next-state). I don't have to add additional parameters to every function and coordinate them between functions.

I think that it easy to lose site of the fact that functional programming is supposed to make our lives easier by making it easier to reason about our code. Jumping from imperative pseudo-code to an idiomatic functional implementation is not easier for me to reason about than from the pseudo-code to It's direct implementation. And it's easier to reason from a direct implementation to a more functional one after I've got a feel for the direct implementation.

Probably it's an intellectual shortcoming on my part that functional implementations don't come readily to me when looking at imperative pseudo- code. I just have to live with the fact that even money says I am below average and will probably have to be naughty and use mutation sometimes.

<comment><USER9> I highly recommend Tim Roughgarden. Best class I've taken on Coursera by far.

<comment><USER7> I took Discrete Optimization. and it's pretty good, too. But I was way out of my depth The cool thing is though that it gave me anchors for a lot of the algorithms in Roughgarden- I look at the algorithm and see an application for it from that class

But Roughgarden is top drawer, too.

</comment></comment></comment><comment><USER10> This isn't a problem; you can get around it by indirecting such "pointers" through an (immutable) array. When you want to reference something, make sure it's in the array and reference its location. When you want to "mutate" something, update the array with that thing's old entry pointing to its new value instead.

(Side note: the union-find algorithm is another interesting algorithm with the same "problem".)

<comment><USER11> "This isn't a problem"? You're so sure?

How many nodes are there? Let's presume there are a lot, otherwise the problem is trivial and speed doesn't matter anyway. So now you have an N-long immutable array that you are copying every time you want to change a node pointer? So you have changed the operation of pointer-changing from O(1) to O(N)? What does that do to the run time of the algorithm?

Also, your garbage velocity just went WAY up. What does this do for the runtime of your program generally?

<comment><USER11> (The language implementation can of course engage in strategies to avoid a full array copy, that you as the user have little insight into, but these are often questionable as they slow down run time in other cases, and anyway, they can only mitigate this problem, which is not going to go away.)

<comment><USER10> Uh, most good implementations of languages with immutable arrays perform O(1) or O(log n) array updates. The two I use most, Mercury and Erlang, both do (O(1) and O(log n) respectively).

If you, the user, has "little insight" into the runtime of data structures you're using, you have bigger issues. Every standard library I've ever seen guarantees the asymptotic behavior of hash tables, sets, linked lists, etc. Immutable arrays are no different.

</comment></comment></comment></comment><comment><USER12> There is no reason to implement fibonacci heap for Dijkstra. Either you have relatively small data (up to 10 mil. nodes and edges) and normal heap is a way faster or you have bigger data and you should start thinking about things like A* where fibonacci heap is again useless.

You should note, that log n factor is so small (up to 40 on the biggest data you will ever see), that it can be easily dominated by other constant factors in implementation (like cache locality, ...).

<comment><USER13> Why would you not use A* from the beginning? It's a trivial extension to Dijkstras and is often orders of magnitude faster when an informative heuristic is available.

Also, both algorithms require identical data structures. After all, Dijkstras is just A* with a zero heuristic.

I do agree about the constant factor, though: it's likely that a binary heap would be faster on most data sets.

</comment></comment><comment><USER14> Keep in mind that functional programming is inherently less efficient than a random access machine[1]:

... And it is not easy to create their equally efficient general-purpose immutable counterparts. For purely functional languages, the worst-case slowdown is logarithmic in the number of memory cells used, because mutable memory can be represented by a purely functional data structure with logarithmic access time (such as a balanced tree).

In this case I think you could use (a pair of?) balanced trees to keep a mapping between the graph and the FIbonnaci heap, as well as for any "mutable" state you need to keep per node.

[1] [http://en.wikipedia.org/wiki/Functional_programming#Efficien...](http://en.wikipedia.org/wiki/Functional_programming#Efficiency_issues)

<comment><USER15> Fortunately, most functional languages (including Haskell) provide mutable random access arrays, so the slow-down is theoretical.

<comment><USER16> But then the code is not "pure" anymore and it even isn't different than the non-functional one.

<comment><USER15> This just proves that "pure" in this sense is a useless concept.

<comment><USER17> How so? All I'm seeing is that some speedups aren't possible. It still seems like a useful concept, particularly since "faster" was never one of its promises (maybe "easier to parallellize", though.).

<comment><USER18> Interestingly, in some cases it is actually faster. Modern hardware is peculiar in that respect.

</comment></comment></comment></comment></comment></comment><comment><USER19> And zippers are just beyond cool. You can formally take a derivative of data structure to find one.

List is L(a) = 1 + a L(a) (≡ Nil or Cons(a, L(a))

L(a) = 1/(1-a)

L'(a) = 1/(1-a)^2 = L(a)L(a) (≡ Pair of L(a) and L(a))

This is our zipper. One list holding for the tail and one list to remember how we get to where we are.

This one is trivial, but for e.g. trees it will be utterly fascinating.

<comment><USER20> Why does 1/(1-a)^2 = L(a)L(a) ?

<comment><USER21> I'm not familiar with this calculus of data structures, but I'd surmise it's because L(a) = 1 + a L(a) implies that (1 - a) L(a) = 1.

</comment></comment></comment><comment><USER22> > _Immediately, I discovered that tree structures are more complicated in > languages like Clojure that have immutable state. This is because changing a > node requires rebuilding large parts of the tree._

This must have implications for implementing a Merkle tree as an immutable data structure. Anyone know how a bitcoin client in clojure might deal with this?

<comment><USER23> You'd use mutable state, either in the form of java collections, or atoms and family.

Even Haskell, where the State monad itself is actually immutable, provides an escape hatch into mutable state for when it's really, truly required. Just if you do use it when it's not, great shame shall be visited on you and your kin. By which I mean someone in #haskell will very politely point out how you could have done it purely.

<comment><USER22> Right, using a mutable structure is the easy way. But is there a way that would maintain the advantages of immutability? (eg memory-efficient and easy undo)

<comment><USER24> Are those truly the advantages of immutability?

Easy undo can be achieved just as easily by reversing operations, in many cases. No need to keep two copies of a potentially large structure when you could just keep the diff and reverse apply it. Oddly, I think typically you would do both. Keep a few large snapshots with small diffs between stages. That is digressing, though.

As for memory-efficiency... not sure how keeping many potentially large copies is more efficient than just modifying a single copy.

Now, I can agree that in many cases it is nice that it prevents you from worrying about race conditions across threads. Though, I'm also not convinced that immutable things are any better than traditional locking strategies. Is that race truly won?

<comment><USER25> One doesn't need to keep large copies, since immutability allows common substructures to be shared. (That said, it still seems rather unlikely that an immutable implementation will use less memory.)

<comment><USER26> That's not a property of immutability. That's a property of certain specialized data structures, see "Purely Functional Data Structures" for a start, implemented in clojure.

<comment><USER27> Well, it's a property of basically everything except arrays. So it's more than just "certain specialized data structures".

<comment><USER24> To be fair, many folks learn immutable data structures with the so called "persistent" data structures.

Which are cool, to be sure. They are far from new, however. So, I'll be a little surprised if it turns out they are a panacea.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER28> In the opposite direction of pragmatically avoiding "fancy" data-structures, you can find a description of a persistent min-heap via Brodal queues in Chris Okasaki's wonderful book _Purely Functional Data Structures_. Otherwise, it doesn't make too much sense to use the Fibonacci heap algorithm without mutable update.

<comment><USER29> Speaking of, I had Mr. Brodal in my algo class. That man thinks in algorithms. He can make it seem so intuitive and easy to understand, when he's explain it at the blackboard. That lasts until you're trying to implement it and get bogged down by details. I miss the algorithm classes.

<comment><USER28> That's so cool! Any fun stories or obscure algorithms that came up?

</comment></comment></comment><comment><USER30> You may be interested in this paper www.cs.ox.ac.uk/ralf.hinze/publications/ICFP01.pdf. It solves Dijkstra's algorithm as an example for the implementation of a priority search queue data structure, which is very similar to a priority search tree. These structures act as a heap on one index and a search tree on a second index. So you get both efficient update and find-min.

<comment><USER31> I think your link is the perfect answer to the problem highlighted in the post (which is, incidentally, a problem I was facing myself, so thank you!). There's also an Haskell implementation of priority search queues: [http://hackage.haskell.org/package/PSQueue-1.1/docs/Data- PSQ...](http://hackage.haskell.org/package/PSQueue-1.1/docs/Data-PSQueue.html)

</comment></comment><comment><USER32> Heaps are the the most elegant data structure, IMO.

So many ways to prioritize. So many ways to support the same basic operations, with different tradeoffs.

My semi-outdated site: [http://leekillough.com/heaps/](http://leekillough.com/heaps/)

<comment><USER33> I love that page! I link to it from my own semi-outdated page: [http://theory.stanford.edu/~amitp/GameProgramming/Implementa...](http://theory.stanford.edu/~amitp/GameProgramming/ImplementationNotes.html#data- structure-comparison)

<comment><USER5> I in turn love your page -- so useful how you show all the data structures working together to enable a larger task. Bookmarked.

</comment></comment></comment><comment><USER34> How about adding new elements with lower keys instead of decreasing the existing elements? Then, once we count that more than half of the n elements are duplicates, we could spend O(n log n) operations cleaning up the heap by recreating it.

</comment><comment><USER35> Fibonacci heaps are very interesting beasts, but implementation is slightly complicated. For C++ developers there's one in Boost that is not so easy to use quickly.

I've recently put an alternative one in C++11 here: [https://github.com/beniz/fiboheap](https://github.com/beniz/fiboheap)

</comment><comment><USER36> Ah, I went on this journey a decade or so ago when building a social network - "how are we connected" type feature, and tried to go down the same route with a Fibonacci heap - and eventually threw in the towel and implemented A* , then D* a while later when scale started to be an issue.

</comment><comment><USER37> Why can't the Fibonacci heap node store a pointer to the graph node and when it is updated, use that pointer to store the new pointer to the heap node in the graph node?

</comment><comment><USER38> Someone else mentioned this below, but it deserves repeating: Okasaki's Purely Functional Data Structures is definitely worth reading.

</comment><comment><USER39> Why was the title changed? It was originally the same as the post's; "The Fibonacci heap ruins my life".

<comment><USER40> From the guidelines: "Otherwise please use the original title, unless it is misleading or linkbait." I think the original title qualifies as at least one of those.</comment></comment>


Netflix and Ch-Ch-Chilly - <AUTHOR> https://backchannel.com/the-internet-really-has-changed-everything-here-s-the-proof-928eaead18a8#.vrhmwhsi2 <comment><USER2> Really interesting content, it is fascinating and somehow heartwarming to see a picture of an environment like this one. Unfortunately this is in stark contrast to the narrator, who at times seems like a total jerk. Why be so snarky to the photographers? They're just trying to do their job! I have to admit, I had to stop reading and start skimming when I got to this line:

> I consider asking the boys if they appreciate music beyond the norm, maybe > bebop or grime or chiptune, but luckily realize the ridiculousness of that > line of questioning. So instead I ask about their favorite rappers. They > amass a respectable list: Kendrick, Wiz, Jeezy, Kanye, Juicy J.

There are many more lines just like it scattered throughout the article, and I hope I do not have to explain why they leave a negative impression of the author. I am sorry Mr. Sorgatz but your culturally-savvy, hard-edged commentary makes me feel like you are a phony.

<comment><USER3> The line you quote seems completely innocuous to _me_. If he actually asked teenagers if they like bebop or grime or chiptune, I imagine the answer would be something like, "Uh, noooo, I don't listen to anything weird" or "I don't know what those are." So he asks a safe question about rappers. The writer is explaining his thought process in a lighthearted way.

We both read the same thing, but I like the author and you dislike him. Which is really interesting, because this is a totally uncontroversial article. It's funny that subtle choice of words have hugely different effect on different people.

<comment><USER4> I read the comments before reading the piece, as one tends to do, and I started out on the side of the narrator being pretentious, if that's the best word to describe enthdegree's feelings on the conveyed tone. Interestingly, however, I am now most the way through the piece and am strongly identifying with the narrator, have warm fuzzies, and completely trusting of his or her good intentions.

Maybe I should start reading posted content more often, rather than reading only the comments?

<comment><USER5> Not that it matters a lot, but isn't this pretentious:

> Every spring, around 20 new kids don graduation caps, celebrate their > nascent adulthood with a class party in a rye field, and return to the farm > the next day to plant corn or milk holsteins.

It's like milking holsteins and planting corn is inferior to complaining about it. As if the author is the only one smart enough to complain.

<comment><USER6> I take that as just an honest description of what happened.

</comment></comment></comment></comment></comment><comment><USER7> Interesting read.

I grew up in a rural town just outside the suburbs of a US rust-belt city. I had a similar experience, but not nearly so severe. Luckily for me, the suburbs were a 40 minute drive, and we had access to 2 metro areas worth of radio stations (thanks to being across lake Ontario from Toronto). So I not only heard the talking heads thanks to CFNY, I saw Depeche Mode, New Order, etc in concert with my high school friends.

The point I was going to make is that I just returned to the town for the first time in 20 years (my parents moved away in the mid 90s) due to a family vacation to Niagara Falls. I was AMAZED at how little the town had changed. If I squinted, I could be back in high school in the mid 80s. The only real differences were some of the shops had changed hands and the cars were newer. This is in stark contrast to the places where I've lived in my adult life (Raleigh-Durham, the Bay Area, Virginia) where things change at a rapid clip.

</comment><comment><USER8> Photog2 (aka Andrew Spear) really is "annoyingly good at his job". I especially liked the disheveled gaze of "Rex Sorgatz on West Lake outside Napoleon" in contrast to the author's cheery, preppy profile picture.

<comment><USER9> The photographs really made this piece for me. The church is fantastic.

<comment><USER10> Agreed. That's a tricky shot and he nailed it.

The piece was a little too rambling for me, but the mix with the good pictures made it all work.

<comment><USER11> What parts of the picture make it tricky? Is it balancing the roofing with the whites of the building/snow?

Genuinely curious, I don't know enough about photography to tell what shots would be hard to capture.

<comment><USER10> The tough part about that shot is, imo, figuring out what you want to do. You've got a white church all alone with a black roof on a white background.

There's not a lot of information in the scene. It's way easy to burn out big hunks of the image and try to grab some contrast in the boards of the building or in the roof.

What I'm guessing he did -- and this is only a guess -- is take a multiple- exposure shot and then do some magic post-production. But instead of trying to pull more information out? He left it mostly a wash, with only the roof having detail. (I'm thinking he pumped that up a bit)

This makes your eye struggle to find information and meaning in the walls of the church, or the surrounding land. You see the roof, it is interesting, so you assume there are other items of interest there. But there isn't any -- which nicely tells the story of a remote community that's timeless, the gist of the writing.

He formed the shot in an original way to have the viewer's mind play into the theme of the text, while still making a nice image. Very cool.

ED: Actually the more I look at it, the more I like it. It's almost like he used a gauze of soft focus on one of his shots, with the church in focus and everything else just a little blurry.

<comment><USER3> Keegan, AI-based image analysis that's on the front page of Hacker News right now, also liked the photo:

[https://keegan.regaind.io/p/XlF3k_kKSMqjacY8minUhQ](https://keegan.regaind.io/p/XlF3k_kKSMqjacY8minUhQ)

</comment></comment></comment></comment></comment></comment><comment><USER12> Why don't I read more articles like this? I always enjoy it, instead I read some inconsequential piece about nothing. This was great, personal, journalism.

<comment><USER13> It reminds me of a Jack Reacher novel, minus the fistfights of course.

In answer to your question: HN has stories like this every day, maybe you haven't scrolled through the 2nd/3rd/4th pages enough?

</comment></comment><comment><USER14> I really enjoyed reading this.

I grew up in Spokane, and graduated from high school there in the late 80s. At the time, the place seemed just as boring and backwater as Napoleon...though it retrospect it was far from it. I was eager to get out and go to "real" places, so I joined the military. I saw new states, countries, _continents_.

I still went back to Spokane, though. Up until this year, I had family there. My mental map, however, was frozen in the early 90s; 1993-94, specifically...the year I lived in town after getting out of the Army. Subsequent visits to Spokane over the next 20 years became increasingly jarring, as that city moved on but the image in my head did not.

In that respect, I envy the author because sometimes a lack of change can be comforting. The fast-food place where I worked my first job for two years? Demolished. The same for movie theaters where I took dates, pizza parlors and bowling alleys where I hung out with friends. That's progress, of course, and if Spokane hadn't changed at all that would probably be even more disquieting.

After reading the article, I think the residents of Napoleon may have the advantage over the rest of us "big city" dwellers. While modern technology has made it into their world, they still have developed interpersonal skills that a close-knit community fosters. In my neighborhood, in contrast, I barely know my neighbors. We all have Netflix. I've worked on sexy, shiny things in my career but as the cutting edge has moved on, they've faded away; folks in places like Napoleon are more likely to focus on hardy things that _last_.

Small, remote towns have obvious disadvantages but I think sometimes their benefits are overlooked.

<comment><USER15> _Small, remote towns have obvious disadvantages but I think sometimes their benefits are overlooked._

I'm probably blindered by my asocial nature, but I strongly agree. "Internet culture" is often considered synonymous with urbanization, but improvements in communication and delivery make rural living (at least in the developed world) much more appealing.

For me at least, technology removes the need for proximity. If I were incubating a technology company and working with a small team, a small town in the Dakotas would be an ideal location. Yet so far as I know all of the incubators are encouraging companies to base themselves in expensive (and distracting) urban areas.

Assuming you have an idea, and a small team, and need a place to implement it, what are the actual disadvantages to a rural location? Sure, maybe bustling cities are great for generating new ideas, but what startup has ever been long on time but at a lack for ideas?

Veering off in analogy, my guess is that there is probably a correspondence between those who like "open office layouts" and those who want to live in dense cities. Many productive engineers prefer offices with doors, and fewer distractions. Am I wrong to associate "rural life" to a "an office with a door"?

</comment></comment><comment><USER16> The second picture perfectly sets the mood for this piece.

The 1970s Plymouth Sapporo in the foreground (a rebranded Mitsubishi Galant). Haven't seen one of those in decades, and this one is rustless. Behind it a 1990s Chevy Caprice, a 20000s Chevy Impala, and then the grain elevators. It is the land that time has forgotten.

</comment><comment><USER17> Fascinating article. I would have expected that modern technology would tear old fashioned communities like this apart, but it can actually enable them to survive. People don't have to leave to see the world, it comes to them.

<comment><USER18> Reminds me of the Tao Te Ching:

      Not venturing out-of-doors, one may know the world

Having technology actually support this is fascinating.

</comment></comment><comment><USER19> I'm really glad the editor was insistent that he have a professional "photo". Those photos were beautiful and affective.

</comment><comment><USER15> The photographs look hauntingly familiar. I lived for a few years in a mirror image town just across the South Dakota line an hour due south. Some stray thoughts:

The area was mostly settled by "German-Russians". These were Germans who moved to Russia under Catherine the Great, and then re-immigrated to America in the late 1800's: [https://en.wikipedia.org/wiki/Volga_Germans](https://en.wikipedia.org/wiki/Volga_Germans). German was the primary spoken language in the area until World War Two (1940's) when it became too politically unpopular to continue. Which is to say, they made it through World War One before making this change.

Pheasant hunting is probably the primary activity that brings visitors in to this part of the country. Pheasants were introduced from Asia about the time the German-Russians came, although they nativized faster. There is lots of public land available for hunting, much of it enrolled in the Conservation Reserve Program, which pays farmers to keep land out of production and often opens it up to public access: [https://en.wikipedia.org/wiki/Conservation_Reserve_Program#E...](https://en.wikipedia.org/wiki/Conservation_Reserve_Program#Enrollment_Procedures).

The town's approach too "development" was different than other parts of the US. Primarily, they were concerned with tearing down houses before they became dilapidated, or worse, "bought up by some California trucker passing through who'll move in with his Go-Go girlfriend" (close to exact quote). You could (and still can) buy a very nice house in town for less than $50,000, or a livable fixer-upper for much less. There's probably better potential for being a remote technology worker now than when I was there trying to do it by dialup.

I was back a couple falls ago, visiting from California for pheasant hunting. The town I was in looks exactly like the photos in the story, enough so that I wondered at first whether Napoleon was a pseudonym. Visually, it was almost identical to how it had been when I was last there a decade before. I had been wondering how much impact the internet has had on the culture, and this article gives nice insight. Thanks Rex!

</comment><comment><USER20> "no drags to cruise"

Strange. That was the only daytime form of entertainment for a large portion of the teenage population of towns like this: driving up and down Main Street and in front of the school, either showing off your car (your status) for the girls or showing off that you have a girl in the passenger seat.

</comment><comment><USER21> Maybe next time someone on HN claims that FB is dying among teens it would be possible to point them at this story.</comment>