Inside the race to rescue a health site, and Obama - <AUTHOR> http://www.nytimes.com/2013/12/01/us/politics/inside-the-race-to-rescue-a-health-site-and-obama.html?pagewanted=all <comment><USER2> Does anyone know what the author meant by this?

" _Software that assigned identities to enrollees and ensured that they saw only their own personal data, known internally as the EIdM, was being quickly overwhelmed. Customers could not log in to create accounts._ "

" _Experts disagree on what went wrong. But several said that errors in the software code written to stitch the Oracle product into the online system and improperly configured hardware trapped users in endless technological loops. It would take eight days to resolve just that one bottleneck._ "

<comment><USER3> The first part sounds like they created a user management system: normally a single table with fields like username, password-hash, real name, address, etc, and sometimes a couple of other tables for groups/roles and cross-ref tables for relationships. In this case the user table would also probably have ids/references to the various government backend systems that the site used to verify your identity, income, etc as you signed up.

The rest of it sounds like they botched making this system scalable, to handle both the number of possible users (every adult in the US) and the rate at which they'd be signing up. "Endless technological loops" might refer to either a circular reference in an overly-complex schema, or to normal dependency checking in the schema which takes too long to evaluate within the database when the database is under enormous load.

<comment><USER4> no offense, man, it is just sound like you've never developed or deployed Enterprise Identity and Access Management. I envy you :) The keyword here is "Enterprise" \- insanely complex [lets just don't touch whether such complexity is really necessary, it is just the fact of a design to meet any possible RFP in the space :) ] and [in many aspects as a result of the complexity] slow non-scalable products.

>The rest of it sounds like they botched making this system scalable

these systems just don't scale. In a BigCo with local replication to regional sites it takes noticeable time for such systems to perform. Trying to build such a website - well, it is a government procurement, though i don't think this is a worse one around (Halliburton's and the likes' cost+ in Iraq/Afganistan definitely bigger :).

>normally a single table with fields like

It is "normal" from another world :) The simple way you described would be done by some startup, and they would horizontally partition it when time come to scale. It is not the way the "Enterprise" things are done.

<comment><USER5> This is just fascinating. This must be why I've seen many a store clerk / govt employee / postal office worker / bank worker / etc. sit there want wait for 10 minutes while the "system is processing"?

> "Software that assigned identities to enrollees and ensured that they saw > only their own personal data ..."

I guess the simplest EIDM is a where clause in your SQL query :P Just kidding - I do wonder what this thing does more than just a regular authentication / authorization system though. Maybe sandboxes data so that it's impossible to even query for data that doesn't belong to you?

<comment><USER6> In part because it handles a separate authentication questionnaire with Experian. That's a PII sensitive integration that needs dynamic content.

</comment></comment></comment></comment></comment><comment><USER7> The folks at MarkLogic must be unhappy with being singled out in the following paragraph:

 _Some of the companies building the system opposed an early decision by the Medicare agency to use database software from a company called MarkLogic, which handles data differently from systems by companies like IBM and Oracle. Some suggest that its unfamiliar nature slowed their work. By mid-November, more than six weeks after the rollout, the MarkLogic database -- essentially the website 's virtual filing cabinet and index -- continued to perform below expectations, according to one person who works in the command center._

The company's response is recorded as:

 _In interviews, MarkLogic's executives faulted inadequate computing power and instability at the site's data center, as well as the failure to properly integrate their product, problems repeatedly cited by other website vendors._

Any insight on what is meant by how MarkLogic "handles data differently"?

<comment><USER8> Not really a slam at all. Customers choose MarkLogic to replace Oracle and DB2 specifically because it "handles data differently".

"Schemaless" aka "Schema on read" is the biggest difference between Only "SQL" and "Not Only SQL" systems. Scale-out on commodity hw vs. scale-up on shared storage is another. Customers often choose MarkLogic instead of other NoSQL systems because of it's ACID capabilities, ability to run multi-statement transactions between multiple databases (including Oracle and MarkLogic in the same transaction) and a multitude of enterprise capabilities from government grade security to point-in-time restore.

Read more: [http://developer.marklogic.com/pubs/architecture/inside- mark...](http://developer.marklogic.com/pubs/architecture/inside-marklogic- server-r7.pdf)

<comment><USER9> Totally, not a slam at all. As long as you don't think a company name and "perform below expectations" sitting next to each other in a NYT long-form piece is a BAD thing...

<comment><USER8> I'll repeat what our CEO said in the NYTimes article "He said MarkLogic is performing up to standard, but "the network and the storage systems are not properly sized and not properly run."

Put _any_ database on top of the resources that were provided and I think performance "below expectations" would have been likely.

</comment></comment></comment></comment><comment><USER10> As a Canadian I wonder, with all the American technology giants, why was this project outsourced to Canada? If the idea was that outsourcing would lower cost, why were the contracts no-bid?

<comment><USER11> CGI Federal is a big-ish Fed contractor in the U.S. and, though a subsidiary of a Canadian firm, for various legal reasons is owned and operated as an American firm. It performs quite a bit of tech work for the U.S. government, work won in generally open competition. Because the U.S. Fed space is such a big market, it's not unusual to have locally owned subsidiaries in the mix e.g. BAE, Quinetiq, etc. all have similar groups.

My understanding is that CGI was already handling various health care related work and was selected to do the work without a proper competitive bidding process. But having seen these processes, they definitely wouldn't have resolved the kinds of widescale technical issues.

<comment><USER12> Addressing you and growupkids, CGI Federal is responsible for the Medicare.gov site, which I can attest to being a bit clunky but otherwise working well for claims and Part D selection and application (the prescription drug benefit program).

They were on a list of already vetted vendors, plus if "a proper competitive bidding process" had been done it could easily be hung up in the courts by losers lawsuits.

As you note, and as I've been noting, given the structure and processes of the project, especially it being run by political and bureaucratic types from the White House on down to CMS (the integrator prior to getting replaced in late October), there's no way they and the other contractors could have possibly succeeded, especially with change orders continuing though the week before launch. Heck, the integration tests CMS ran in the last week or two before the launch told them it couldn't possibly work, but it was launched anyway....

<comment><USER12> Per this article [http://www.nytimes.com/2013/11/23/us/politics/tension-and- wo...](http://www.nytimes.com/2013/11/23/us/politics/tension-and-woes-before- health-website-crash.html?hpw&rref=) there were three other bidders: IBM, QSSI and Computer Sciences Corporation.

All I can say is thank god CSC wasn't chosen! And to the extent CGI Federal retained some of the qualities of the civilian part of AMS the company bought, again, not an obvious bad move (I worked for a short time in a national security part of AMS, which went to the American firm CACI because foreigners couldn't own it).

</comment></comment></comment></comment><comment><USER13> Amazingly, the website appears to be down again! [https://www.healthcare.gov/](https://www.healthcare.gov/)

Looks like they had planned that downtime, but I wonder how messed up this whole project must be that they can't even manage to have a static maintenance page up!

<comment><USER14> It up for me. I was just able to register an account. After looking at all the steps to do that

1\. This is already too complicated. ( why a username when you know I am going to need to use my SSN ) 2\. This is not a complicated web app. I am now fully convinced that this could have been done with better results by anyone other then the government.

<comment><USER15> > I am now fully convinced that this could have been done with better results > by anyone other then the government.

But it wasn't built by the government. Do you think Obama asked for a username field?

It was built by a private contractor ie. your usual capitalist, profit- oriented enterprise IT company.

<comment><USER16> You've never built a system for the government I take it? You don't build these things in a vacuum, they approve and co-design the system, drive the technology decisions, require you to use "approved" technologies, ban the use of icky technologies because they aren't on the interoperability list (which locks you into a few vendors, the usual suspects, Microsoft, oracle, etc.), slap on all kinds of additional requirements, force you to use hardware from a cut rate vendor so they can make their 8A quota, and so on.

It's nothing like a private company building a system. It's more like being a peasant to a lord. Sure you can do what you want, as long it's exactly what they want to you, when, where and how they want you to do it.

</comment></comment></comment></comment><comment><USER17> Data In. Data Out. I have seen very little of exactly which databases are being integrated. I saw a reference a few days ago that there were many arguments about using SSN in the system. Does that mean that some of systems being pinged for data don't use it? Very hard with SSN, I can't imagine trying to wing a match with similar name, similar address, similar birthday.

</comment><comment><USER18> Anyone know what tech stack they're using? Linux or some other *nix or Microsoft?

<comment><USER19> [http://medcitynews.com/wp- content/uploads/healthcaredotgovin...](http://medcitynews.com/wp- content/uploads/healthcaredotgovinfographiceduardogarcia-588x968.jpg)

Mirror: [http://web.archive.org/web/20131201071252/http://medcitynews...](http://web.archive.org/web/20131201071252/http://medcitynews.com/wp- content/uploads/healthcaredotgovinfographiceduardogarcia-588x968.jpg)

</comment></comment><comment><USER20> I'm not from the US. I wonder, isn't this overly dramatic? So it takes a while to fix the web site. People have presumably been waiting for health insurance for decades, what is one more month? Also, really, people get worked up about government processes that are difficult to use?

<comment><USER21> If your previous policy was cancelled due to ACA and you can't get a new one because the enrollment processes for the exchanges are broken, you get to enjoy a gap in coverage. That is never a good thing.

<comment><USER22> That's nonsense. All of the previous ways to sign up for health insurance are still around, and the policies that were cancelled basically didn't give any coverage anyway (that is, they didn't cover medication or hospital visits and allowed insurance companies to drop the owners of those policies at any time).

<comment><USER23> What's nonsense is to claim "the policies that were cancelled basically didn't give any coverage anyway." The ACA added several minimum requirements to private insurance plans, such as maternity and prescription drug coverage, which people could reasonably choose to forgo in exchange for the lower premiums.

FYI, not having "prescription drug coverage" does not mean insurance does not cover any medication. It is a _pharmacy_ benefit, in other words, over-the- counter medication. In my personal experience, I pay significantly _less_ without the coverage than with it. I also don't have to worry about formularies, prior authorization, or generic swap-outs.

To give you a specific example, in 2013 I paid $107/mo for a private individual policy with a $2,700 deductible, $5,250 max out-of-pocket (inclusive of the deductible) and unlimited lifetime benefits. Next year under ACA I'll be paying $283/mo for a policy with a $4,500 deductible and $6,350 max out-of-pocket (inclusive of the deductible) and unlimited lifetime benefits.

The most important clause in the ACA is limiting the rating factor due to age to a maximum of 3x. That clause alone makes it the single largest tax increase ever passed on young middle class families with children. By my estimates, it will cost families like mine about $50,000 in increased insurance premiums over the next 10 years, because the _least_ we can now pay is 1/3rd the rate of unhealthy 64 year-olds.

<comment><USER24> > FYI, not having "prescription drug coverage" does not mean insurance does > not cover any medication. It is a pharmacy benefit, in other words, over- > the-counter medication.

Am I parsing that incorrectly?

"Prescription drug coverage" has always covered prescription drugs, in my experience. I've never had a policy that would (to my knowledge) cover OTC meds.

<comment><USER23> Too late to edit my original post, but as you suspect, I'm trying to differentiate between prescription drugs like birth control, anti-depressants, statins, etc. which you buy at a pharmacy with a prescription, and medication administered in an inpatient/outpatient setting, which is also _prescribed_ of course, but covered by the medical benefit not the pharmacy benefit.</comment></comment></comment></comment></comment></comment>


Containers Are Not the Future - <AUTHOR> https://www.linkedin.com/pulse/containers-future-ian-eyberg/ <comment><USER3> If we take "Container" to specifically mean OS-level isolation of applications, then there is a strong likelihood that Containers will be replaced by lightweight VMs.

To me, the rise of containers has been about packaging software alongside all its dependencies in a standalone package that is (to varying degrees) isolated from the underlying host. In that sense, lightweight VMs are an evolution of the container model we already use.

The author fails to explain how these lightweight VMs would be orchestrated. Maybe Kubernetes sucks, but if something more secure comes along, is that a failure of the container approach, or is it an evolution?

All in all, this feels like someone writing a post saying "The aircraft created by the Wright Brothers was terrible, therefore powered flight was a failure". That, an attempt to sell something by bashing popular techs and millennial coders.

<comment><USER4> _The author fails to explain how these lightweight VMs would be orchestrated._

There are two directions that most organizations can go from containers, and they have opposite answers to this question.

The first direction is back to a monolithic application running in a VM. Orchestration no longer exists as a problem. What else do we get? Development is easier. Debugging is easier. We have literally decades of experience figuring out how to make it work.

Very, very few of the people reading this post are solving problems harder than ones I was solving decades ago using that kind of stack. Other than the fact that your resume won't look as good, there is zero reason not to do this.

The other is, as the post's author recommends, even further into isolating the app. Down to something even simpler than a container. In which case you will need a solution for orchestration, but it won't let you leverage a single running VM to a whole cluster. Limiting the attack area will be good for security.

As a bonus things that container folks say work, but don't, will work. For example autoscaling fails in adtech with containers fails hard because the time that it takes to boot up new instances exceeds the time that you are getting flooded by your ad. But if your unikernel boots fast enough, now autoscaling _does_ work.

Now there is a third use case. And that is organizations like Google whose codebase is so big that they can't run it monolithically on a single machine. Containers solve that in a distributed environment by letting them run only some of their code on each machine. There is a hidden cost that RPCs over a network are inherently an order of magnitude more expensive (in time, latency, and resources) than a function call inside of a running application. So you order an order of magnitude more hardware, and containers are the sweet spot for you.

But that problem is of a kind that very few organizations have. And making it actually work takes a lot of thought and organization that very few in the containerization world seem to understand. Certainly if you read a few blog posts, install Docker and Kubernetes, then start deploying through Helm, I am willing to bet large sums of money that you are almost certainly making rookie mistakes.

<comment><USER5> > The first direction is back to a monolithic application running in a VM. > Orchestration no longer exists as a problem

Of course it does, VMware makes a boatload of money with VM orchestration. Or you setup OpenStack which is free but a nightmare in itself.

You do _not_ want a hand-rolled KVM virtualization stack. I've been there, I've seen the horrors.

<comment><USER4> In 2003 I was a developer on a website that served a million pages per hour using 2 databases, 4 webservers, and associated networking. Most of that hardware was there for automated failover.

At that scale, orchestration is simply not a challenge. We wrote a deployment script to roll out code changes and life was good.

2020 hardware is not worse than 2004 hardware. And while that business eventually failed, it was not due to technology. (And it made a lot of money before it failed.) As I said, most here are not doing something harder now than we did then. The same class of solution works.

As the old saying goes, KISS. Know when it works, and when it doesn't. As another saying says, YAGNIY. (You Ain't Gonna Need It Yet.) Deal with the limits of being monolithic after you encounter them. Don't complicate your life to tackle problems that don't exist.

<comment><USER5> > In 2003 I was a developer on a website that served a million pages per hour > using 2 databases, 4 webservers, and associated networking. Most of that > hardware was there for automated failover.

fwiw, Stackoverflow runs on a similarly sized/deployed stack.

But that is because Stackoverflow is fine with their employees running a cheap, un-complex system that _works_ (I can't remember the last SO outage!), and their employees are also happy with this.

Others, especially startups? Gotta chase these hipster credentials and use the free AWS/Azure/GCP credits somehow... and the employees don't want boring old school stuff, they want the newest hipster crap that's barely alpha tested, but in production. Utterly ridiculous. Most of them would be fine with three sub-$10k AMD servers and a bit of work by a seasoned Linux sys-admin. But eww. Linux management. Not cool/hip.

<comment><USER6> Did that stack include daily traffic analytics hardware?

<comment><USER4> See [https://nickcraver.com/blog/2016/02/17/stack-overflow-the- ar...](https://nickcraver.com/blog/2016/02/17/stack-overflow-the- architecture-2016-edition/) for an overview of their stack.

They do NOT need a stack of daily traffic analytics hardware. As I said above, remote RPCs take an order of magnitude more resources than function calls. If you are used to using various distributed tools to do analytics, you are used to requiring an order of magnitude more hardware than a simpler solution. And if there are inefficiencies in your solution, it is easy for a second order of magnitude to sneak in. Therefore your experience is going to seriously mislead you about the likely limits of a simpler solution.

Making this concrete, if your job requires a Hadoop cluster of less than 50 machines, odds are good that it could be run just fine on a single machine with a different technology choice. But if you are fully utilizing hundreds of machines in your cluster, then you actually need to be distributed. (Though you could achieve considerable savings for some of your jobs could be run in a more efficient architecture. Amazon in particular makes excellent use of this kind of hybrid architecture in places.)

Google is making excellent decisions for the scale that they operate on. You just are unlikely to operate at a scale that makes their decisions make much sense for you.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> Pardon the ad hominem, but this is mostly marketing FUD from someone with a vested interest in you not using Kubernetes. Kubernetes may not be perfect, but I guarantee you the licenses are a lot cheaper than whatever "NanoVMs" are.

The tone is also very arrogant and condescending. Not a great way to win mind share in this industry.

<comment><USER8> I had to stop reading after the insinuation that Hacker News comment sections are filled with trash. To the contrary, I find that half of the time the comments sections are more thoughtful and enlightening than the article posted. Very regularly I read the comments first to see if the post is worth diving into.

<comment><USER9> Yeah, that's where I knew something was wrong. There are some very genuine and useful conversations that happen in the comments at HN. They aren't universally awesome, but when I see something linked, I frequently jump straight to the comments to see the thoughts of the HN users.

</comment></comment></comment><comment><USER10> I think the three usages of containers often get conflated.

1) Packaging 2) Isolation 3) Secure isolation

Containers have dong a good job of packaging and isolating applications. This is has been a boon to development. Particularly with testing and deploying applications in a more or less consistent state and managing those applications. Note that these benefits have nothing to do with security.

Containers have done a poor job of being secure against malicious actors but that doesn't mean the isolation is useless. From a security point of view you just have to treat it the same as if it was running outside the container(s).

One day we'll hopefully get something better from a security POV but that doesn't mean packaging and isolation are worthless in themselves, even if they don't add any security. The problem comes when people assume containers add more security than they do.

<comment><USER11> Exactly, this is what has me so confused. The value I get from containers is the series of steps that create the container, not the underlying implementation. I don't really care what substrate the software sits in, be it a unimicronanofirecrackerkernelVM or LXC or a chroot jail, besides security implications of isolation escapes. I just like having a deterministic Dockerfile-like file. If I can keep that, sure, I'll buy into your Containers But Not But Really Cloud Native For Actual schtick and use it, especially if it actually is more secure (I'm sure it very likely is). It's the abstraction and model that makes containers easier to work with, not the implementation.

I find it very irritating and deceptive that the author wildly equivocates between containers and Kubernetes almost every sentence. And it's incredibly condescending that they suggest people buying the container Kool-Aid are simply too young to have done development and operations on regular servers/VMs, "duh!" Or that they're just trying to pad their resumes. No, I've had plenty of years of experience doing that, and I find containers far simpler to develop with, deploy, maintain, and update. And I've never used Kubernetes.

As most others have said, this comes across as extremely patronizing and demeaning marketing FUD from a company so certain that they're obviously right and almost everyone else is mind-numbingly wrong that even if that's actually the case, no one is ever going to believe them; and very justifiably. This is not how you do marketing, or write prose or communicate in general.

<comment><USER12> > wildly equivocates between containers and Kubernetes

The conflation of containers and K8s is pretty damning. I personally believe k8s is an early realization of orchestration and will be superseded by better ones. Building yet another orchestration system is a cottage industry at the moment and I've noticed that the existence of k8s hasn't impeded it.

And "spin up a linux box" is deeply naive. It is the habit of spinning up linux instances that leave companies and other institutions with huge collections of neglected, mis-configured, vulnerable linux VMs spread all over the planet with 0.1% average utilization.

Developers popularized containers, independent of the cloud or the troubles of operations. Containers solve tough development and packaging problems and have value whether the ultimate deployment abstraction is 'docker' or some microVM system. The most likely use case for whatever microVM product one might imagine is _running containers_.

<comment><USER13> * I don't see the conflation of k8s/containers. Many/most? production container users are using kubernetes. Not sure how that is debatable. Mesosphere changed their name to D2IQ and Docker sold off their enterprise arm.

* As for microvms running containers -- in this case Nanos doesn't run Linux so it can't run containers.

</comment></comment></comment></comment><comment><USER14> I've been saying for years that containers are nothing more than an ugly hack to get around the fact that OSes are broken. More specifically modern OSes still carry 1980s assumptions about multi-tenancy, system administration, application management, state management, etc.

In the 1980s a "server" was a big expensive special snowflake. Each server had a "system administrator." The system administrator would "install" software "on" the machine.

I quoted all those words to highlight no longer valid concepts.

Boxes are ephemeral now. Systems have more than one administrator, sometimes belonging to a different organization or department. Applications are not installed "on" the system in the sense that they root into it and modify its state. If they do this makes them a terrible hassle to manage. Applications should be ephemeral things that can float freely between machines and store their state in some kind of distributed or remote data store or database, or at least can be easily migrated between hosts. Everything is mobile. Everything is commodity. Everything is liquid.

OSes are just not designed this way, so we have an ugly hack called a container that basically amounts to "let's tar up images of operating systems and just treat them like giant static binaries." Onto this is bolted a whole array of other ugly hacks to make that easy to deal with, but the concept is fundamentally ugly as hell.

<comment><USER15> Those are valid points,

Have you seen or heard of any alternatives? I mean, software getting more and more complicated, maybe even more complicated than necessary, however running apps natively has its own pitfalls (we've all been there). It's like choosing necessary evil and if there is a better way of doing things we should evaluate it and talk louder about it

<comment><USER14> The only attempts I've seen are advanced package managers for Linux, like the stuff used by NixOS:

[https://en.wikipedia.org/wiki/NixOS](https://en.wikipedia.org/wiki/NixOS)

Unfortunately these tend to be overly complex and have too steep of a learning curve, and they don't address _all_ the issues.

The closest is probably apps on mobile, but again that doesn't address all the issues and comes with a whole set of restrictions that are very problematic outside small single-user devices.

<comment><USER16> I'd say NixOS probably is not the right entry point if you are looking for a softer learning experience. NixOS is built on top of Nixpkgs package manager, which itself could be thought of as a side-effect of Nix - the DSL and a runtime for building software from source in a reproducible manner. If you start by learning how to employ Nix-the-build-tool in your daily software development practices, you will be amazed how little you need to learn in order to be productive with it, because there's essentially just one new concept - a derivation [1], the rest are just community-driven infrastructure and utility methods on top of it.

[1] [https://nixos.org/nixos/nix-pills/our-first- derivation.html#...](https://nixos.org/nixos/nix-pills/our-first- derivation.html#idm140737320503024)

</comment></comment></comment></comment><comment><USER17> K8 specifically might not be the future, but containers are the past (chroot) present (docker) and future. Sure, containers are a hacky shell script... but a hacky shell script that _always runs the same way_. I put in my time chasing down RPM dependencies, and debugging weird behavior because of a different version of some shared library I previously didn't know existed. I think the arguments presented here are focusing on implementation details, while the higher level picture is that containers-the-concept save a TON of developer (and devops) time and pain.

<comment><USER18> I haven't delved that deep into containerland but to me, it seems the main downside of using k8s is the complexity of set up and maintenance.

I'm not sure what's stopping people from developing tools to make the whole containerization process invisible to developers who don't need that much control. I'm sure stuff like that already exists/is being worked on, and I don't know why it shouldn't work.

<comment><USER19> The thing is that k8s != containers. I've been using containers for years, but I don't use k8s in any of my current projects (although that might change at some point)

That's why I had to stop reading this article. They repeatedly conflate k8s with containers. From the users perspective, one is an extraordinarily complex orchestration framework, and the other is an incredibly simple packaging system.

</comment></comment></comment><comment><USER20> The company I work at started with the idea that we would build an orchestration system for Unikernels. Very much in the same space as the NanoVMs folks. I totally applaud their work. We took a different path once our small team worked with unikernels for a bit (mostly OSv). During the early days we noticed that the unikernel landscape had technical shortcomings that would take an incredible amount of engineering effort to overcome and we found convincing users to trade Linux for a (mostly but not totally) compatible unikernel based system was an insurmountable hurdle. It was a fun experiment but, after timeboxing our work and taking stock of the landscape, we fell back to one of our original sentiments: A stripped down Linux is actually a pretty good substitute for a Unikernel for most applications (emphisis on "most).

We ended up pivoting to running a lightweight linux, based on Alpine and orchestrating everything using Kubernetes and Virtual Kubelet [1]. Shameless Plug! Pods are isolated on their own virtual machines that are booted for the pod, the underlying OS is rock solid and gives users all the great tools, bells and whistles you'd expect from a linux based system. Fewer surprises, easier development. We actually open sourced the system today.

[1] [https://github.com/elotl/kip](https://github.com/elotl/kip)

<comment><USER21> Great intel from on the ground. The story arc makes total sense. I also have starry eyes for orchestrated unikernels, but otoh linux is so configurable scaling down the kernel also seems very reasonable. How do the boot times compare? To me, the unikernel's chief appeal is that it could potentially start up in about the same amount of time it takes to load a native binary. Thanks for sharing.

<comment><USER20> On bare-metal boot times are drastically different. OSv was sub-second while our Alpine images were 3-5 seconds depending on what services we needed. However, we were focused on running our system on cloud VMs, not bare-metal. In the cloud you can't get a VM to start booting in less than 20-30s so that order of magnitude difference turned into, at most a 10% difference in boot times.

In 2017 we measured the restart time in of our unikernel images in AWS to be 22 seconds, all that time was waiting for Zen (2017... no KVM yet) to get around to getting to the place where we could run our image. So for our use case, the boot time didn't actually matter, it was far overshadowed by everything else happening under the hood.

I should say: Unikernels do have their advantages and should be used in areas that can exploit those advantages: Fast boot, low attack surface, way better performance for some workloads. We had trouble finding the specialized customers in the cloud that needed any of a unikernel's positives so badly that they would take on a unikernel's shortcomings.

</comment></comment></comment><comment><USER22> The tone of this article is very condescending, it contains a lot of absolute statements and most of its reasoning is done by linking somewhere else.

This reads like a marketing piece ("CEO at NanoVMs") by someone who is betting on the frustration of Developers and Operators with the pace at which "DevOps tech" is moving.

Maybe MicroVMs are the future (whatever that means) but this is not the way to push it.

</comment><comment><USER23> I wrote a satirical blog post on Docker/k8s in 2015 [1]. At the time, I actually felt docker was the future, but it became clear over the next few years that it was too much. The complexity of docker /infra/devops is overwhelming, and something much much simpler is clearly needed.

I don't know if Dark (my startup trying to solve this) is going to take over, but it's pretty clear that the current complexity is unsustainable.

This goes double if recession/layoffs/etc is happening due to COVID. People are going to need to spend a lot less on engineering, and the amount being spent on engineers that maintain infra is an obvious starting point. I would expect people to switch to much simpler infrastructures, and reduce the complexities of their stacks by necessity.

[1] [https://circleci.com/blog/its-the-future/](https://circleci.com/blog/its- the-future/)

<comment><USER24> Be careful. The whole point of Docker/k8s was to _simplify_ operational complexity - and to that end it's done a great job. Yes k8s has complexity, but it's a _known and knowable_ complexity vs. the unknown complexities of the processes of yore.

Does that mean there can't be improvements? Of course not. If you're working on simplifying things even further then great. But I'm concerned with your ability to execute if you believe Docker/k8s have made things more complicated because that signals to me you may not realize where the complexities lie.

Meanwhile I'd love to hear about your ideas to make things less complex.

<comment><USER25> > Yes k8s has complexity, but it's a known and knowable complexity vs. the > unknown complexities of the processes of yore.

TL;DR -> I see what you are saying, but there are methods to make your infrastructure consistent and keep the manual drift away that do not require k8s-level complexity. Both for monoliths and microservices.

Because we talk of the processes of yore, I will use a simple application architecture in the form of a web site that clients access that CNAMEs to a load balancer.

The load balancer distributes traffic to web servers, which each make back-end connections to a database.

This could also be load balancer to hosts that just run single-host instances of docker CE/EE as well - microservices can be done through this setup as well. If they are designed correctly, they shouldn't "need" to all be in one cluster, after all, these are durable microservices, not sharded bits of a monolith, right? :)

You have 4 basic tools, which with not a lot of thought can be used as a foundational platform for many, many workloads.

Aside from your codebase, you have the following (and/or their competitors)

Hashicorp Terraform

JetBrains TeamCity

Octopus Deploy

With a build server, a deployment platform, and a configuration management platform, you can get 98% of 'web scale' (all the scale you are really gonna get + plenty of runway for all but the FAANGs among us) for WAY less daily cognitive load. This applies to the DevOps team, the developers, the support cannon fodder (note if this is too true, and you are in leadership - <USER2> help this by empowering your cannon fodder), the DBAs, all of 'em.

1\. Programmers check in their code to source

2\. TeamCity runs unit/integration/regression tests. Pass? Publish to nuget feed for Octopus

3\. Terraform sets up automated DB, web server, load balancer infrastructure for the servers, configured for whatever your company's definition of 'production server' is

4\. Octopus deploy takes the newest build artifact from TC nuget package and deploys it to those servers.

In the above process, if you need to scale up:

1\. modify the count of web servers in terraform

2\. Terraform apply

3\. wait for TF to complete its provisioning of new servers

4\. wait for Octopus to recognize the new servers as deployment targets for your code (in this scenario, you write TF such that it will install the Octopus agent and set deployment target)

5\. Deploy your code to the new servers

6\. Have Terraform reconfigure the load balancer to expect these web servers to work but start in a disabled state

7\. the load balancer will detect they are now passing health checks, enable them, and poof, they are serving traffic.

You have infrastructure as code, you have CICD, you have scalability for miles, and not a docker registry, helm chart, istio policy, or rancher needed. Best of all, your helpdesk, intern developers, even non-tech managers can cognitively follow the process.

Need to patch? Cattle, not pets says:

1\. Terraform: reconfigure load balancer to drain connections on the boxes to patch

2\. Terraform: reconfigure lb to remove from LB when done

3\. Terraform: destroy the boxes

4\. Terraform: spin me up same number of new boxes. Using packer or a fresh AMI, the new box will be patched, then have TF set configs, install etc. as done with the original boxes

5\. Octopus: deploy code to all known web servers

6\. TF: add boxes to load balancer

PS: sorry this grew into a wannabe blog post with handwave set to maximum.

</comment></comment></comment><comment><USER26> There's a good reason I start reading articles and papers from the end to the beginning.

"Conclusion Containers are clearly not the future. You will eventually need to migrate completely off of containers and kubernetes and we are here to help."

While this does not negate the core idea, it summarizes the motivation and bias behind the arguments.

</comment><comment><USER27> Operating systems abstract resources.*

Unixes got the abstraction boundary not-quite-right, by excluding networking and filesystems from the concept of process.

Containers are a correction to this.

i.e. not quite as catchy, but:

"Containers: the present is what the past should have been."

* Windows abstract screens, Files abstract storage Sockets abstract connections, Threads abstract CPUs, etc.

<comment><USER28> Maybe so, but somehow containers aren't really there yet. As a trivial example, when I 'lsof' a container process (as root), it often cannot "see" all of the information about the process' open files, apparently because they're mapped into a different chroot, or because of other cgroups/namespace issues.

This is one of a number of issues that make containers hard to deal with.

(We also see an issue where containers interact badly with the OOM killer, leading the entire system to hang because /proc/*/cmdline cannot be read at all.)

<comment><USER29> The failure of userland tools (such as lsof) to correctly show work with containers is unfortunate, but this is partly an artifact of the Linux kernel's somewhat ramshackle approach to containerization.

The kernel doesn't have a concept of containers, but rather of kernel namespaces. For example, it's possible for a process to create a new namespace that is the same as that of a default process, except with a new file system namespace — i.e., a chroot.

Hopefully we'll see these tools updated to better understand kernel namespaces. In the meantime, if you need to interact with containers, use the nsenter [1] tool. For example, it allows you to enter the same PID namespace as the container, but still use the host file system.

For example, run to lsof against a container's PID 1 (corresponding to the host PID 1234):

$ nsenter -p 1234 lsof -p 1

[1] [http://man7.org/linux/man- pages/man1/nsenter.1.html](http://man7.org/linux/man- pages/man1/nsenter.1.html)

</comment></comment></comment><comment><USER30> I think this is largely correct, but I think we have to add that POSIX is not the future either. The model of process isolation and shared resources that is built in to the POSIX model is just too hard to get right.

I think the post-POSIX world will have to look somewhat like the IOS model of isolation, something like the NIX level of packaging, and something like the Docker model of running. A shared filesystem where certain information is held in certain locations in the filesystem is just a nightmare to manage, but it is currently not possible to replace it.

<comment><USER31> > The model of process isolation and shared resources that is built in to the > POSIX model is just too hard to get right.

isn't that the basis of like, every modern operating system? Haven't we been working on getting it right since the '70s?

</comment></comment><comment> 100-xyz This phrase from the article "Resume Driven Development" is 100% true where I work.

<comment><USER32> It's true at a lot of places.

I know one developer who is obviously planning to go to Amazon because _every single application_ he works on immediately starts with looking at which AWS services he can glue together. I'm to the point where I want to create and AWS bingo card and when hes on a conference call, pass it around to see if anyone wins by the time he describes some solution set.

 _" Well I'm starting with [AWS service here] but it doesn't support [major, normally functional breaking requirement], so I'll use [this crazy set of JS NPM packages] to do the work in between"_

Wish people would develop solutions, not resumes.

</comment></comment><comment><USER33> Containers have uses, but yes if you expect them to be secure, you're in for a big surprise.

And last time I checked Kubernetes basically washes its hands of the problem, and maybe makes it worse by adding layers that obscure the security properties.

I worked on a Kube-like project from 2010-2015 and I was definitely horrified by the state of containers in the Linux kernel. Security is basically what made me give up on the project.

<comment><USER34> You can run containers in userspace, use app armor, etc and you get better security. Aside from maybe one or two major security exploits, what issues have you want into?

</comment></comment><comment><USER35> I'm not so sure that unikernels are the way forward. And in general, angry rants about technologies from the CEO of a company entirely devoted to some competing technology is generally best taken with a large grain of salt.

Undoubtedly there are risks to having the flexibility that a full OS provides, but the benefits to troubleshooting, incident-response, resource-sharing, and just plain operational flexibility are huge. Until there's a good story for managing unikernel-based microVMs at scale, they aren't going to make a dent in container-world.

I suspect the path we'll actually see pursued is more along the lines of Bottlerocket ([https://github.com/bottlerocket- os/bottlerocket](https://github.com/bottlerocket-os/bottlerocket)), which is basically the Linux kernel plus a containerd API that works with Kubernetes but doesn't expose nearly as much surface area.

</comment><comment><USER36> Nothing in software is ever "the future." How many things change as fast as software technology? For many of us not concerned with "building our brands" through "resume driven development", but instead focused on enabling rapid and reliable production deployments containers and orchestration have improved our results. Obviously this will not remain the dominant paradigm forever. The next thing may be an evolution of it, or it may be a take in a different direction that leaves kubernetes a dusty and unused side-branch in history. Wouldn't be the first time. Regardless of how likely that is, or what the drivers of the eventual result are, I am not sure what the author wishes people to have done instead of employing containers? Should we have just kept on installing dependencies and services onto VMs while awaiting the one true new thing that would last forever?

</comment><comment><USER37> Containers are like front-end frameworks: they are great tools to have, and solve certain problems better than the previous options, but there is a tendency among advocates to assume they are the only tool in the toolbox.

</comment><comment><USER38> > If their own developer evangelists are going to conferences and calling the > software a "clusterfuck" why on earth would you bring this clusterfuck into > your organization?

Most software sold to large organizations are tire fires/clusterfucks, Kris is just saying up front that complex software is hard.

Everyone knows reasonably complex software tends to be a clusterfuck -- if you're making purchasing decisions based on a vendor telling you their software does all these things and it's a perfectly engineered masterpiece then ...

<comment> kris-nova thanks jcastro - if anything here is a clusterfuck it was this bro's article - If you actually watch my talks you will see that I love Kubernetes and just want to get people ready and excited about new technologies. The word clusterfuck is just funny to say...

</comment></comment><comment><USER39> Security-as-a-second-thought also doomed Java Applets and Flash. If Cloudville doesn't focus heavily on security, it's doomed. History has given you at least 3 lessons.

</comment><comment><USER40> I feel this was a bit scattershot, more about Kubernetes than containers per se.

Kubernetes isn't directly responsible for the sins of container runtimes and vice versa. Lightweight VMs solve some of these and I don't think it's controversial at this point to say so.

I may have reservations about the design of Kubernetes but I don't need self- serving conspiracy amongst the designers to hold those reservations.

Disclosure: I work for VMware, we do a fair amount of Kubernetes stuff, including lightweight VMs.

<comment><USER41> Are we not able to have polite discourse anymore? Sheesh..

Container orchestration is not the first silver bullet that has grown a large following that makes money off of it, whether reasonable or not. I do have a question though. Say you have a bunch of microvms processing your workload or serving dynamic pages. How are you going to manage your fleet of MicroVMs? Would you perhaps write a microvm orchestrator of some sort?

Got a huge kick out of the first comment on the linked page:

> You know what you call a "micro-VM that's purpose-built to run one > application" if you're a Unix person?

> A process.

> And now we've come full circle. Everything old is new again!

<comment><USER13> For everyone asking about a "microvm orchestrator" \- the implication here is that the public cloud (where the vast majority of people reading this will be deploying too) _is_ the orchestrator.

Public cloud like AWS and GCP does all the networking, volume management and everything else for you - for free. That's arguably one of the reasons for using it in the first place. There is no need to put something on top. That's kinda the whole point here.

<comment><USER42> People abandoned Packer+Terraform style workflows in favor of containers. Why? Why change back?

<comment><USER13> I think there are few thoughts here.

a) I don't necessarily agree with that statement - witness HashiCorp's recent raise:

[https://www.sdxcentral.com/articles/news/hashicorp- scores-](https://www.sdxcentral.com/articles/news/hashicorp-scores-) 175m-funding-round-5b-valuation/2020/03/

b) The vast majority of container users run their workloads on public clouds like AWS and GCP which run on top of virtual machines to begin with.

I'm not stating you need to use terraform (I definitely don't) - what I'm saying is that you can use the underlying orchestration provided by the cloud of choice - be it the networking or the volumes. There's no need to replicate that on top as container workloads do. When people who speak of kubernetes/container complexity - this is what they are complaining about.

If you look at a lot of the configuration mgmt software out there - chef/puppet/ansible/salt/terraform/etc. it all involves configuring a linux system whereas with unikernels it's literally just one application and instead of having to plumb your network together or ensure that your workloads or 'stateless' vs 'stateful' you just use the underlying cloud facilities. That's the magic. It actively removes complexity from the provisioning.

</comment></comment></comment></comment></comment><comment><USER43> I'm confused why the author keeps associating containers with total insecurity. Docker's approach to whitelisting access to the filesystem has been great in my experience. I've managed classic multi-tenant Linux servers before, and I've pretty much always found lots of examples of services having read or even write access to each other's files unnecessarily. Docker's strong whitelisting-based approach to filesystem access seems so much more ideal.

In the video of someone breaking into a Kubernetes cluster, they start off with restricted kubectl access. Okay, so restricted kubectl access isn't a strong security boundary. I don't think it's a big deal to a lot of people's use-cases. I didn't know it was a feature; the lack of a strong feature like that is only a dealbreaker to certain kinds of setups. It doesn't seem necessary for sandboxing different services apart from each other; it's for giving different sysadmins/developers access to deploying to different parts of the cluster. I'm going to make a wild guess that the issue isn't core to and unfixable in Kubernetes, because the person in the video shows a way to plug the issue. Summarizing this as calling Kubernetes security a clusterfuck is way over the top.

I could believe that the author's NanoVM project is even better at security out of the box, but I find the way they present the argument as overblown, and they don't introduce NanoVM for comparison until the end, so I think the arguments are doubly misleading because most people are going to be comparing containers to the classic plain multi-tenant Linux server model, and Docker and Kubernetes seem like at least an improvement to that in security.

</comment><comment><USER44> "CEO at NanoVMs"

OK.

<comment><USER45> CEO of Redmeat corp says that Soja isn't the future.

</comment></comment><comment><USER46> Eh, hard to care when Docker works well for what I need it to. I'll stick with containers until something _significantly_ better comes along.

</comment><comment><USER47> Ehh if microvms are the future, we are going to need software to schedule and orchestrate them across large clusters. Software with this capability happens to already exist and it's called Kubernetes.

I feel as though the author is only thinking about Kubernetes as an application platform. However, you can build platforms on top of Kubernetes (knative, Istio, etc) which solve many of the issues the author has brought up.

</comment><comment><USER48> Some truths in there...but unfortunately drowned out by the FUD marketing spiel

</comment><comment><USER49> I agree with the article entirely. Just the other day/week I was down voted for lamenting K8s and pointing out how dumb it is.

When I saw and read about unikernels a few years ago I knew they'd be the future. I've explained several times in the past: why did we create containers instead of working hard to make VMs better, lighter, faster? Now the time has come.

K8s will be dead in 3-4 years.

</comment><comment><USER50> I used to be a die-hard advocate of containers, back when they were Solaris Zones. They could destroy the performance of Xen VMs (and I wrote posts explaining the technical reasons why.) But a lot of things changed with HW hypervisors, with the leaders nowadays being Amazon Nitro and Firecracker.

It's something the article doesn't address. Containers used to be a slam dunk, so I'm not sure I'd say that we were wrong _at the time_. But HW hypervisors evolved, so now I can see HW hypervisors like Nitro and Firecracker being compelling for some* workloads (e.g., to give the end user a dedicated kernel where they can do anything, including run any BPF program, run PGO kernels, etc.)

* I'm not sure yet whether it's "some" or "most".

</comment><comment><USER51> It's almost as if there's multiple solutions for multiple problems.

Containers are just a solution to a problem. I've been around long enough to know that containers aren't novel either.

Novelty is the future. It always is.

</comment><comment><USER52> The biggest reason I have to concern myself with deployments nowadays is to quickly spin up local projects. So far, nothing has offered me the speed of docker-compose to achieve that.

Sure, I can take a look at these microvms, but if they're not as convenient to run multiple projects simultaneously—I'm a freelancer and have my own hobby projects—then I see no reason to switch.

After all, I used to use vagrant, but spinning out containers is so much faster.

</comment><comment><USER53> Unrelated to the content of this post, but look at how terrible Linked in is:

[https://share.getcloudapp.com/JruWePeK](https://share.getcloudapp.com/JruWePeK)

Scrolling pushes state to your history, and it's so poorly done that hitting back doesn't even restore previous scroll positions (I'm assuming that was the intent).

</comment><comment><USER54> And yet, you could probably with a bit of work swap out the container runtime for a VM manager in Kubernetes and most users wouldn't even notice.

</comment><comment><USER55> This article is enthusiastically declaring that the CEO of NanoVMs doesn't understand the audience he is addressing.

Containers were never meant to be "secure" and Kubernetes has nothing to do with security in general. Application level containerization solves 3 problems: 1- repeatable builds 2- software distribution 3- infrastructure mutability

Docker solves brilliantly all these problems: we can build software in controlled environments that we call images and the product is an image itself, that we can share by means of software catalogs called registries and that we can deploy the same way we can play CDs in a computer, without modifying the host system. Security was never meant to be a concern, it was never part of the plan, at least no more than running a process in an operating system. And containment systems are just a smart way to launch a process, that's it. They can't replace security functionalities beyond what is already provided by host OS.

Is this a fault? Would you blame a hammer for not being able to unscrew a cork?

NanoVMs CEO spends some time also making fun of "stripped down systems", accusing them of recklessly "ripping out components of a system that simply don't belong in the virtualized world of 2020", namely users, SSH logins and desktops. Again, there is a big misunderstanding here. Linux processes expect some facilities to be present at runtime, namely system calls to retrieve Process IDs and user IDs. But these facilities were merely retrofitted in containment systems to avoid processes crash upon syscall invocations. As all retrofitted functionalities, their implementation is barely beyond the dummy level. Therefore, we have PID 1 everywhere, no SSH logins and dummy users.

Now, Kubernetes. K8s is just a scheduler, an orchestrator. It has nothing to do with security, it is mainly concerned with resource optimal usage and container placement. And it is a fundamental component in microservice world, otherwise we would be paralized by operations at each deployment. "You spin up a linux box and call it a day - done". Seriously? Is this really the whole understanding of the world of operations? I've been in ops for 12 years and I'd never want to go back to the old bad days of 2008, where most of us managed an handful of servers and services and, yet, datacenter ops were at one with deployment ops. That was the very world of manual and error prone ops. Orchestration has been a bliss but requires a unit of scheduling. A container, in this case. But it may be a VM as well.

So, the saddest thing here is that NanoVMs could be a perfect replacement for containers while retaining the benefits of orchestration (as in Kubevirt, for example), but the CEO prefers to throw away the dirty water along with the baby for marketing hype.

</comment><comment><USER45> The CEO from a VM oriented Start-up says that Container are not the future..

There is no way to go back to VM deployment and management. I used to be openstack wizard, proxmox ninja and kvm/vagrant contributor and I cannot really think about giving up my whole container orchestration (ci, deploy, update, introspection , monitoring, you name it) to use back VMs.. I'm 40, doing sysadmin since 2002 and computer scientist since 2006.

</comment><comment> Eric-Guo Why should I click the LinkedIn URL to read a tech article?

</comment><comment><USER56> There is no way to secure k8s. Really now. REALLY NOW.

<comment><USER57> Care to expand ?

<comment><USER34> [https://github.com/kubernetes/enhancements/commit/c8b1f15359...](https://github.com/kubernetes/enhancements/commit/c8b1f153592c9499d1b44c2a0a88a3dde6726652)</comment></comment></comment>


Ask HN: Who are some of the must-follow Twitter users? - <AUTHOR>

No, not Justin Bieber. Let's post links to profiles of developers/hustlers on Twitter who have amazing things to say, but don't have as much exposure.
Requirements:
- must be active user
- may/may not be very popular
- must be in software/startup scene
GO. <comment><USER2> Here are a few of the software/startup people I follow. (I don't know any of them personally but I've met/emailed a few)

@patio11

@asmartbear

@raganwald

@hnshah

@MikeIsaac

@shanley

@hunterwalk

@steveklabnik

@paulg

@pmarca

@jedgar

@ryanbigg

@zedshaw

@peterc

@DanielleMorrill

@naval

@VCdelta (this is a bot that tweets changes in VC portfolio pages)

<comment><USER3> @pmarca is pretty great at Twitter, if a bit high-volume.

Also recommend:

@coda (Coda Hale)

@matthew_d_green (Matthew Green; if you follow him, you don't need to follow any other crypto people)

@pinboard/@baconmeteor (Maciej Ceglowski)

@KirinDave (Dave Fayram)

<comment><USER2> I meant to put you in there too ... sorry :/

<comment><USER3> I am super noisy on Twitter. I've never recommended that anyone follow me. :)

</comment></comment></comment></comment><comment><USER4> Aral Balkan - [https://twitter.com/aral](https://twitter.com/aral)

A very humble person, who hopes to empower people to own their own data. Currently working on [http://indiephone.eu/](http://indiephone.eu/)

</comment><comment><USER5> I've found out that following users in bulk will bring a lot of noise in my Twitter feed. I suggest you make a separate list for new accounts that seem interesting, and if someone is really worth following, go for it.

</comment><comment><USER6> Patrick Walton (@pcwalton) for Rust and general Mozilla stuff.

</comment><comment><USER7> Danielle Morill and Marc Andreesen are pretty popular, but still worth mentioning here, some of the most valuable content on my feed comes from them.

</comment><comment><USER8> John Carmack</comment>


Ship.io: CI for mobile apps - <AUTHOR> https://ship.io/

<comment><USER2> cool. and its free including on-device testing</comment>


Skype launches free one click conference calls that don't require accounts - <AUTHOR> https://alternativeto.net/news/2020/4/skype-launches-free-one-click-conferece-calls-that-don-t-require-accounts-to-combat-zoom/ <comment><USER2> Does it work with any browser? Last time I checked you could only chat using the web app, for anything else, you needed either Chrome or the Native app</comment>


Coda 2.5 Released - <AUTHOR> https://www.panic.com/blog/introducing-coda-2-5/

<comment><USER2> I really, really enjoyed Coda for quite a few years in the 1.x days. Panic makes really good stuff.

But had a hard time figuring out how to work Coda into a more "modern" workflow with Git, and moved on to Sublime because it has a more active plugin community and is cross-platform.

Maybe I should take another look at it. Coda definitely excels at the "I have a local copy of a website, and need to sync it with what's on the server over SFTP" workflow.

That was back in the 1.x days, though. I haven't looked at it in ages. Should I? I do mostly Rails these days.

<comment><USER3> Coda 2, released a few years ago, brought a lot of git-related changes. Things like branching, merging, and committing are integrated in the UI. Personally, I like to have a terminal window open in Coda for git, but the integrated solution works reasonably well.

<comment><USER2> Thank you for the reply! Now I'm going to give it another spin...

</comment></comment></comment><comment><USER4> I definitely outgrew Coda a few years ago although I occasionally fire it up to use its visual grep. However, in the time since I switched to other editors (Chocolat, Sublime, and now Atom), my workflow has changed significantly.

I'm ready to do a clean install for Yosemite, and maybe it's also time to give Coda another look. I had been planning to switch back to Sublime after giving Atom a shot (there's _a lot_ to like, but it's slow as molasses and the autocomplete is total garbage). So much of my workflow lives outside of my editor these days, so maybe it makes sense.

<comment><USER5> What did you think of Chocolat, compared to the others? I've been meaning to give it a try.

<comment><USER6> I have had chocolat installed for a long time and I sometimes use it when I don't want to look at a file in vim, which is pretty often when I just want to read the code. I hang out in the #chocolat channel on Freenode. My take on it is the developer sincerely cares about making a great editor, and it has always been pretty simple to use. The economics of being yet another code editor are not great and Alex is the only owner and developer yet he is still dedicated to the project. I am not sure you will ever see the kind of ecosystem that TextMate and Sublime Text enjoy, but if you can see past its faults, chocolat is a pretty nice little editor.

<comment><USER4> I started using it shortly after I outgrew Coda. I went to Textmate at first, but it was around the time that Textmate seemed totally dead. Its interface reeked of 2004 but Chocolat was quite modern. I liked it a lot but I was eventually lured away by Sublime's ecosystem. Then I got sick of dealing with JSON for my settings and checked out Atom. But I find that I'm not really taking all those rich ecosystems that Sublime and Atom offer. It's probably time for me to take another look at Chocolat as well.

</comment></comment></comment></comment><comment><USER7> Perhaps I'm wrong, but I see Coda fitting in between the poweful IDE's produced by JetBrains (ItelliJ, WebStorm, PHPStorm, etc.) and text editors. I think one issue for Coda is that development workflows (especially web development) changed significantly since version 2 was released in 2012. Coda is still a very nice application if it fits in your workflow.

</comment><comment><USER8> I've used Coda for a few years, during which time I started using Sublime + wbond's FTP plugin. Coda is still crucial to my remote PHP freelance workflow. For remote work, it's a much more polished experience when dealing simultaneously with multiple active/live projects.

Sublime FTP's experience can improve slightly when using it in mirror mode, but in practice I've found issues with very slight time differences between my local computer and the remote computer, which can cause unnecessary or possibly dangerous syncing.

Can't really think of any Coda negatives other than lack of git integration. I generally just keep an SSH open to commit anything.

Edit: nvm, it does have git integration. I'll have to try it out.

<comment><USER6> > FTP

Please do not use FTP, FTP sends everything over the wire in clear text, across the entire internet. You should not be using FTP for anything ever. Use SFTP, scp, HTTPS from the server, but not FTP.

<comment><USER8> I don't use FTP. The plugin for sublime I was referring to is called Sublime SFTP. It supports SFTP, which I use for all my projects.

Check it out here: [http://wbond.net/sublime_packages/sftp](http://wbond.net/sublime_packages/sftp)

</comment></comment></comment><comment><USER9> For my day to day work, I use PHPStorm and PyCharm. But, I've been working on a bunch of exercism.io problems in various languages, and figured I'd give Coda a shot for that. I ended up absolutely hating it. Couldn't get the split windows to work the way I liked, and I felt like I was fighting with the editor every step of the way. It's a shame, because it would have been a nice way to manage all the little exercise folders.

</comment><comment><USER10> Sneaky pricing trick there. Coda 2 wasn't $99. It was $75 or possibly $50, due to a never ending discount.

<comment><USER11> This was due to the lack of upgrade pricing in the App Store; they essentially gave everyone the upgrade price for a while. $99 was always the intended "retail price"; now that they've abandoned the App Store, I expect Coda 3 will have separate pricing for upgrading vs. buying new.

</comment></comment><comment><USER12> I used this years ago and just tried it out this version. I really wanted to like it. Sadly, it still lives in a world where we all use MySQL, SVN, and FTP. Git may be supported, but it's an afterthought. For $99, Panic should include a time machine back to 2004.

</comment><comment><USER13> Love Coda 2 and am very excited about Coda 2.5. The one major annoyance for me with 2 was not being able to search remote locations. I don't see this in the release notes. If the Panic team is listening, can we get this in version 3?

<comment><USER11> I would like this feature too (as well as indexing symbols from remote files), butI wouldn't expect it, due to the potential for overwhelming a server by downloading hundreds or thousands of files. Doing the search server-side would be prohibitively complicated, due to the variety of shells and environments.

The workaround would be to use a setup that mounts SFTP drives locally (ExpanDrive, MacFusion, etc).

<comment><USER2> You're absolutely right about the challenges, but mounting SFTP drives locally wouldn't solve the problem - in that scenario, every file would still have to be downloaded to your machine in order to be searched.

It doesn't seem _totally_ unsolvable, though. I mean, if Coda can connect via SFTP then it can almost certainly connect to the server and run commands like awk, grep, find, Silver Searcher, etc. There's no technical reason why Coda couldn't run those commands _on the remote host_ and return their results to you.

That's still quite a frigging challenge, obviously - there's a huge range of Unix utils that may or may not be available on each individual remote host. Coda would have to do some trial-and-error and/or probing on each remote host to see what's available.

<comment><USER11> Yeah, local SFTP would still be horribly inefficient; presumably you'd be aware of the leaky abstraction and use it lightly, whereas things would get messy if Coda tried to abstract it for you.

> There's no technical reason why Coda couldn't run those commands on the > remote host and return their results to you.

The other obstacle I see is Windows-based S/FTP servers, which presumably lacks grep, et al.

(One other little gripe: I wish I could open remote files directly from Coda's terminal, which would sidestep half the inconvience of remote searching. If it's not in 2.5, I'll have to remember to submit a feature request.)

<comment><USER2>
      The other obstacle I see is Windows-based S/FTP servers, which presumably lacks grep, et al.

Fortunately, not always!

I believe a lot of Windows users who run SFTP do so via Cygwin's sshd, which for most intents and purposes is a full Posix implementation. At the very least, you can grep your heart out! (I've run it on a few Windows servers myself)

I don't have the slightest clue what Cygwin's relative marketshare is in the Windows SFTP space but I'm guessing it's pretty high because all the other solutions I found were from weirdo vendors I'd never heard of, very expensive, or both. I also had a hunch that a lot of the commercial solutions were wrapping/using Cygwin but I never looked into it.

Regardless, yeah, some servers would not be remotely searchable! This doesn't seem unsolveable from a UI perspective. Coda could disable the search box for those hosts and provide an explanatory note. Or maybe via a little red/yellow/green searchability indicator next to the remote search box.

</comment></comment></comment></comment></comment><comment><USER14> Looking forward to giving this a spin, I moved over to using brackets a lot but remember liking Coda a lot. I think I will have another go at using it the changes sound like they have some pretty big improvements.

</comment><comment><USER15> I'm new to programming. Is Coda worth the $99 price compared to Sublime?

<comment><USER16> I have both, Sublime seems to be leaner, but Coda has some neat features. If you are new to programming I would personally use Sublime since it is free, and then decide from there after a month or so.

<comment><USER17> Ahem, Sublime Text costs 70 USD at the moment.</comment></comment></comment>


"I have seen so many young entrepreneurs and..." - <AUTHOR> http://37signals.com/svn/posts/1999-i-have-seen-so-many-young-entrepreneurs-and <comment><USER2> Ironically, this post's signal (business leaders should focus on fundamental questions of profitability) gets lost in the inevitable noise that accompanies a rant (put-downs, despair, negativity).

To me, this post demonstrates that ranting to make a point is normally counterproductive. As further evidence, I point to the comments (albeit only 4 of them at the time of this response), which deal not with the subject of the post (whatever you make that to be), but with the way it is presented.

My philosophy professor in school once said (and this was years ago, so I recognize that this is paraphrased and maybe even misremembered) that he wouldn't ignore spelling and grammar mistakes in essays, because substance is inseparable from style. I tend to agree. If you want people to listen to you and engage in meaningful discussion with you (and if you're writing a blog, I assume this is what you want), you have to present your substance in a style which is conducive to this end. Ranting is rarely this style.

<comment><USER3> Agreed, however they rarely pull punches at the SvN blog. This is not the first time they make comments about the "others" (even if they're successful "others").

</comment></comment><comment><USER3> What's wrong with being young? I feel old at 27, but because I have 5 less years now to think of, develop, and launch new projects compared to a 22 year old.

There's nothing wrong with being young and a CEO. In fact, you'll probably have to step up to the plate because no old fart will work for free and helm you startup (not that you'd want them to anyway).

</comment><comment><USER4> > ... and 22 year old CEOs use buzz terms like scalable, robust and enterprise > but there is no meat to anything ....

A peeve: More than once recently I've seen articles indicating that words like these are meaningless. I beg to differ. For the record:

"Scalable" describes things that work well when dealing with increasingly large problems.

"Robust" means it can take anything you throw at it, without breaking.

"Enterprise" means for a business, particularly a sizable one.

These are all meaningful, useful words, and the fact that some people _mis_ use them, does not change that.

(OTOH, I must admit that, when I hear the phrase "enterprise computing", my brain automatically translates it to "I'm trying to rip you off", but maybe my brain just needs rewiring.)</comment>


First space, then auto–now Elon Musk quietly tinkers with education - <AUTHOR> https://arstechnica.com/science/2018/06/first-space-then-auto-now-elon-musk-quietly-tinkers-with-education/ <comment><AUTHOR> "science, math, engineering and ethics... [no] sports or music at all, and languages fall by the wayside because Musk believes we will all soon have immediate, real-time computer-aided translation"</comment>


Newest U.S. Counterterrorism Strategy: Trolling - <AUTHOR> http://www.wired.com/dangerroom/2012/07/counterterrorism-trolls/all/ <comment><USER2> Israel has been using/endorsing similar tactics with software-assisted, crowd- sourced trolling tool(s) since ages. The assistance is in the form of curation of targets to troll against, not generate troll speech.

<http://en.wikipedia.org/wiki/Megaphone_desktop_tool>

<http://www.theregister.co.uk/2006/09/06/giyus_megaphone/>

<http://www.jpost.com/Israel/Article.aspx?id=42902>

<comment><USER3> Israel?

Or pro-Israel-ists?

</comment></comment><comment><USER4> This is similar to one of the tactics credited with marginalizing the KKK

[http://en.wikipedia.org/wiki/Superman_%28radio%29#.E2.80.9CC...](http://en.wikipedia.org/wiki/Superman_%28radio%29#.E2.80.9CClan_of_the_Fiery_Cross.E2.80.9D)

The impact it had was discussed in the book freakonomics.

</comment><comment><USER5> Having been involved in various online activist forums including high-profile Anonymous and Occupy groups as well as low-profile environmentalist and political theater groups, I'm pretty sure this kind of thing has been going on for a while and not just for supposed terrorists groups. And sadly it can be very effective. When an online forum has dedicated professional accounts that post 24 hours a day 7 days a week (meaning there are multiple people operating them) and they are very familiar with all the internal group dynamics and personalities, and they use multiple demoralizing techniques from subtle misinformation to playing off different personalities to crass attacks, the effects can really destroy the cohesion of a group and drive people away. It's a struggle figuring out how to combat this.

<comment><USER6> I don't know which forums you visit, but in my experience trolls are always a failure of modding. Activists are vulnerable to strange notions of censorship (as if everyone should be able to scribble everywhere on the internet), and this bias makes them ascribe supernatural powers to trolls (like they can defeat any moderation).

</comment></comment><comment><USER7> This is a fundamental misunderstanding of how human psychology works. Mocking and marginalizing a group makes them more likely to engage in extremist acts, not less. [http://lesswrong.com/lw/lr/evaporative_cooling_of_group_beli...](http://lesswrong.com/lw/lr/evaporative_cooling_of_group_beliefs/)

<comment><USER8> The point is, if you make them look stupid to the broader population, people are less likely to come in. It's about recruitment, not mitigation of existing terrorists, at least how I understood it.

<comment><USER9> Right. So the question to ask, then, is: will the potential shorter-term increase in attacks from trolled terrorists be more or less than the longer- term decrease in attacks due to fewer new terrorists being around?

If it were Americans (or anyone else who worries about being a terrorist target already) doing the trolling, I would seriously worry about that. But if it's local people discouraging each other, I figure the risk is pretty small.

</comment></comment></comment><comment><USER10> Is it possible that the trolling we experience on English language forums is really the rest of the world using this strategy to tie us up in petty arguments so we don't have time to do any real work?

<comment><USER11> Take responsibility for your own society's trolls, and for our gullibility to them.

It would be easy to point the finger at outsiders hating us for our freedom. In order to defeat the troll, we must recognize him in ourselves, however.

</comment></comment><comment><USER12> It's funny that this has been going on 'underground' for a few years (at least 2009) by a bunch of different groups. These two knuckleheads (hi HackMiami!) give a talk in 2010 about terrorist forums and how to troll them: <http://www.tagtele.com/videos/voir/62093>

</comment><comment><USER13> Just a personal observation: On usenet, it looks to me like the political and religious right have been using trolling for years. IMHO, its a bizarre thing that political trolling looks almost exclusively right wing. Heh, what ever the politics, at least they put the leg work in. Dunno if it has any material effect though.

<comment><USER14> the left has Colbert -- who a quantifiable number of people from the right feel is an actual conservative

<comment><USER15> He can be conservative and fun at the same time. It is almost like he is trolling the left, the reverse of his intended mission.

<comment><USER16> If you think he is trolling the left, your the one being trolled.

</comment></comment></comment></comment><comment><USER17> Making already pent-up wanna-be jihadis angry and look foolish is like shooting fish in a barrel.

</comment><comment><USER18> They learned it from the chinese. In china,this trolling scheme has been employed for years. There is even a name given (five cents) - meaning they are hired for cheap to praise everything the government does and demonize the activists.

</comment><comment><USER19> Anonymous: the offense that America needs, but not the offense that America wants.

</comment><comment><USER20> This is a cross between propaganda and the infiltration of extreme domestic organizations by the FBI in the 70's.

Yet more snipping at leaves while the roots continue to grow.

<comment><USER21> The roots are the underlying discontent that motivates radicalization? Western diplomacy is not going to eliminate mid-east discontent any time soon, I agree.

This effort seeks to replace the radicalization option with something else. What's wrong with that? There are lots of discontents in the world, and not many of them kill other people.

</comment></comment><comment><USER22> They are implementing the SNL skit where Mayor Giuliani fights graffiti by adding "sucks" right under it.

<http://www.hulu.com/watch/277719>

</comment><comment><USER23> Nope, this will never catch on. The minute they realize that the exact same principles they use to subvert al-Qaeida propaganda can be used against their own propaganda, they will have to shut down the program. You can't teach critical thinking, humour and satire to your army and not have them use the same tools against you.

<comment><USER24> But, in a country designed like the United States, wouldn't it be better for our denizens to have critical thinking, humor and satire? I mean, we're supposed to be a country that's full of people that say "no" to stupidity and that can laugh at -- and then work to fix our faults, through the channels that were established at out creation.

It's the democratic republic part of this country that makes it so great --- when used by critical thinkers.

</comment></comment><comment><USER25> <http://en.wikipedia.org/wiki/Groupthink>

Use groupthink to ease the angries out into tge open. Way to raise the bar on discourse, military-industrial complex.

<comment><USER26> I'm sorry, I can't tell what your point is. Please explain what about groupthink and/or this situation you mean.

</comment></comment><comment><USER27> Lame. They could address the reasons that people hate them but instead they invade foreign lands then use propaganda to convince people that "it's OK, our bombs contain freedom."

</comment><comment><USER28> Isn't this exactly what China does to catch dissidents?</comment>


Show HN: Findyour.blog – Get your blog rated and reviewed by authors/readers - <AUTHOR> https://www.findyour.blog/ <comment><USER2> Great work on the site. It's way easier to use than mine.

After reflecting a bit from my old approach of using [0]json files in a directory + everything on git for easy forkability.

I have been experimenting with activitypub or other protocol by which these little sites can share entries with each other.

It's would be less work for the bloggers as they only need to add their blog once but for user and curators as well. No segregation.

It removes centralisation and one bad actor from harming the others by pushing for profitability at the cost of intended purpose. (This doesn't mean they can't monetize their curation effort. Just not in excess or becoming a closed wall)

technology.blog, political.blog, cookingrecipe.blog etc.

0] [https://github.com/searchableguy/awesomeblog- club/tree/maste...](https://github.com/searchableguy/awesomeblog- club/tree/master/blogs)

<comment><AUTHOR> That sounds interesting, i've added you on Discord.

</comment></comment><comment><USER3> This is really cool. Just clicking around to various blogs and scanning for articles that perk my interest. It would be pretty cool for blog authors to pick like a top 3 articles or something to help me quickly gauge on top of their description.

</comment><comment><USER4> Note that you can't sign up or login on a mobile device (tested on mobile android chrome and firefox).

<comment><AUTHOR> Hey, that's really strange I just tried IOS chrome and safari. Do you mind telling me what error you are getting?

The navbar icon is hidden automatically and needs to be expanded by clicking the top right blue icon.

Maybe i need to hire a UI designer..

<comment><USER4> Here are two screenshots from my phone. I don't see an error. I don't see any way to sign up from the navbar. Maybe the problem is user error, but I tried for a bit.

[https://i.imgur.com/Fv5YcDq.png](https://i.imgur.com/Fv5YcDq.png)

[https://i.imgur.com/dpIIQnh.png](https://i.imgur.com/dpIIQnh.png)

<comment><AUTHOR> Ah yes, might not be too intuitive but if you click the blue icon with the three dashes on the top left and scroll down a bit you should be able to see the signup/login option.

I'll note that this is a ui bug and will try and get it fixed asap. I didnt intend for the footer to be a sticky

</comment></comment></comment></comment><comment><USER5> A language property would be nice to encourage e.g. German bloggers to add their blogs, too.

</comment><comment><USER6> Please consider un-gating the submission. I tried the password test123 which was too short and test1234 which wasn't accepted either.

Please add my blog in my profile to the corpus is possible. : )

<comment><AUTHOR> Sorry, the submission needs to be reviewed right now as there's a chance your blog might appear in the front page and someone might post something inappropriate.

I'm currently working on a submission filter that should automate the gating and it should be merged by the end of the week.

I also approved a bunch of blogs just then, so hopefully yours is public.

<comment><USER7> I had no issue with the password at all. I tend to set long passphrases with unicode characters and it worked like a charm.

</comment></comment></comment><comment><USER8> Ah yes, David Lebovitz. I had forgotten about him. Thanks for putting this together, it was worth it to me just reminding me of his blog.

</comment><comment><USER9> Anyone knows of anything similar for portfolio websites? Aimed at front end developers

<comment><USER10> awwwards.com has been getting pretty popular</comment></comment>


Show HN - SuggestMeLearn, My first web application - <AUTHOR>

	- After starting and quitting so many times, I finally decided to start small project and created SuggetMeLearn - This is my first ever attempt to a complete web application - The idea is to - let people suggest the way they learnt a particular language or technology - let people seeking to learn decide based on the suggestion to pick resources and start learning rather than browsing through infinite search results. - people can vote if they think they also believe that a particular suggestion is a great resource(and suggestions are sorted by votes) - All suggestions are welcome. - URL : http://suggestmelearn.appspot.com/ <comment><USER2> Clickable link: <http://suggestmelearn.appspot.com/>

Good luck!

</comment><comment><USER3> I love it! We are doing a similar thing here in Germany. Would you be interested in working together on the project?

<comment><AUTHOR> Hi Nurik! Sure, let me know how can I help!

</comment></comment><comment><USER4> Super cool, but miss a lot of data, maybe you should start by filling something out yourself.. :)

<comment><AUTHOR> I will definitely add some more content. thank you

</comment></comment><comment><USER5> Hey - one small bug I found, when I suggest a resource, i can upvote my own suggestion.

You do allow only one upvote, though. Seems like the error is that the initial suggestion doesn't count as an upvote.

This is a nice start. Are you testing any assumptions with this first version? Also, may also want to add something that auto finds urls and auto changes them to links.

<comment><AUTHOR> Hey rcavezza, I fixed the bug that you found, now you can no longer upvote your own suggestion. Thank you for pointing that out I will look for auto url finder thing now

Thanks again!</comment></comment>


Is Free Software Daily dead? - <AUTHOR> http://www.tech-no-media.com/2009/06/is-free-software-daily-dead.html

<comment><AUTHOR> The site is down for quite some time, anyone knows why?</comment>


Some of My Best Friends Are Rich (1998) - <AUTHOR> http://www.matthewklam.com/nonfiction/bestfriends.html <comment><USER2> This article made me really nostalgic for a interview on Q that really stood out to me.

[http://www.cbc.ca/radio/q/schedule-for-thursday- august-6-201...](http://www.cbc.ca/radio/q/schedule-for-thursday- august-6-2015-1.3181153/journalist-vs-juiceboxxx-can-the-world-be-divided- into-geniuses-and-critics-1.3181161)

This may seem really random but I honestly feel the NYT Magazine profile is really too aspirational for most people, the author is a Guggenheim Fellow who has the material support of his psychiatrist fiance; and frankly, it sounded all of his friends went to fancy prep schools and colleges that enabled them to embark on professional careers.

Sure hard choices had to made between moderately rich and creative and extremely rich and respectable. It leaves one wondering if the author wrote this piece trying to _rationalize_ his life-choices to himself in comparison not with average people, but with the stratosphere social circle of insurance executives, dentists with private practices and lawyers he still chooses to upkeep with.

This interview between Juiceboxxx and his author friend at Slate really humanized the comfortable yuppie with a starter-house in his 30's, towing the line though in the media industrial complex and still conscientious about art, and Juiceboxx who most people would view as a loser who should've quite after late 20's in his music career.

I find this piece much more relatable as both features two arguably "losers" in society and how they move forward with their life and reality into their 30's.

I think it answered the motivation and confirmed the life choices both people have made much better than the posted NYT magazine article.</comment>