Why Kubernetes is winning the container war - <AUTHOR> http://www.infoworld.com/article/3118345/cloud-computing/why-kubernetes-is-winning-the-container-war.html <comment><USER2> I've used both mesosphere and Kube now (in production) and I feel I can safely comment on this.

Kube is winning for the same reason React/Redux (and now Mobx) is winning and why Rails was winning at the time. Community.

The community for Kube is awesome and the work people are doing in the field is being noticed all over the place.

I've seen people (myself included) that moved production clusters from mesos to Kube just because the activity of the development and how secure they feel with the community and the project going forward.

React and Rails (at the time) had the same sort of pull towards the community and why a lot of people on-boarded.

Golang is most likely a factor here too. I feel most people find Golang friendlier than Scala/Java. That's why Kube has many more contributors, the hurdle for contributing is easier to jump

<comment><USER3> >I've seen people (myself included) that moved production clusters from mesos to Kube just because the activity of the development and how secure they feel with the community and the project going forward.

This is a bit disappointing and disconcerting to hear! I understand that technology doesn't happen in a vacuum, but when you have two well supported/known technologies, picking the most 'popular' one rather than something based on technical merit is a huge issue for me.

<comment><USER4> The popularity is great, but Kubernetes also happens to have an awesome technology too. Where it intersects with the community is that the technical direction actually listens to real-world use-cases that members of the community brings up. Combined with the amount of activity, Kubernetes is evolving fast.

For example, the PetSet feature of Kubernetes 1.3 involved a really long discussion with people trying to work out how and why to use it.

I remember back in the Kubernetes 1.1 days when I was trying to get AWS ELBs to talk with Kubernetes and possibly finding drivers to automatically set up proxy points. In my search, I found the Github issues where people were discussing that very thing. The person who wanted AWS integration was there talking about the specific needs and quirks of the AWS ecosystem. There were a series of patches made to beef it up, and the discussion on Github documented how the feature was conceptualized (so as to be part of a more general solution on port management).

In contrast, I don't see that with Docker. I can't say anything about Mesos, as I've never tried using it.

<comment><USER2> I agree, that's exactly the proof of a strong community.

Community goes a long way but has to co-exist with strong technology and being open to change.

</comment></comment></comment></comment><comment><USER5> For a devOps fan like me, k8s has been a godsend, and what I like in particular is their 3 month release schedule. There are still some hiccups like no good documentation (or a tutorial really) on setting up shared writeable storage and how to handle databases, or more importantly replication.

The k8s team is very responsive and I'm sure these will be ironed out in the near future so we can all adore each other's cattle :)

<comment><USER6> How did you handle shared writable storage and databases?

<comment><USER7> We (actor.im) also moved from google cloud to our servers + k8s. Shared persistent storage is a huge pain. We eventually stopped to try to do this, will try again when PetSets will be in Beta and will be able to update it's images.

We tried:

* gluterfs - cluster can be setup in seconds, really. Just launch daemon sets and manually (but you can automate this) create a cluster, but we hit to that fact that CoreOS can't mount glusterfs shares at all. We tried to mount NFS and then hit next problem.

* NFS from k8s are not working at all, mostly this is because kubelet (k8s agent) need to be run directly on a machine and not via rkt/docker. Instead of updating all our nodes we mounted NFS share directly to our nodes.

* PostgreSQL we haven't tried yet, but if occasional pod kill will take place and then resyncing database can became huge issue. We ended up in running pods that is dedicated to specific node and doing manual master-slave configuration. We are not tried other solutions yet, but they also questionable in k8s cluster.

* RabbitMQ - biggest nightmare of all of them. It needs to have good DNS names for each node and here we have huge problems on k8s side: we don't have static host names at all. Documentation said that it can, but it doesn't. You can open kube-dns code it doesn't have any code at all. For pods we have only domain name that ip-like: "10-0-0-10". We ended up with not clustering rabbitmq at all. This is not very important dataset for us and can be easily lost.

* Consul - while working around problems with RabbitMQ in k8s and fighting DNS we found that Consul DNS api works much better than built-in kube-dns. So we installed it and our cluster just goes down when we kill some Consul pods as they changed it's host names and ip. And there are no straightforward way to fix IP or hostnames (they are not working at all, only ip-like that can easily changed on pod deletion).

So best way is to have some fast(!) external storage and mount it via network to your pods, this is much much slower than direct access to Node's SSD but it give you flexibility.

<comment><USER8> As long as you associate a separate service with each RabbitMQ pod, you can make it work without petsets. (Setting the hostname inside the pod is trivial, just make sure it matches.) Then you can create a "headless" service for clients to connect to, which matches against all the pods.

If you set it up in HA mode, then in theory you don't need persistent volumes, although RabbitMQ is of course flaky for other reasons unrelated to Kubernetes -- I wouldn't run it if I didn't have existing apps that relies on it.

<comment><USER9> > RabbitMQ is of course flaky for other reasons unrelated to Kubernetes -- I > wouldn't run it if I didn't have existing apps that relies on it.

I'm surprised because I know teams which are very satisfied with running RabbitMQ at scale. Could you elaborate?

<comment><USER8> RabbitMQ doesn't have a good clustering story. The clustering was added after the fact, and it shows. I've written about it on HN several times before, e.g. [1]. Also see Aphyr's Jepsen test of RabbitMQ [2], which demonstrates the problem a bit more rigorously.

With HA mode enabled, it will behave decently _during_ a network partition (which can be caused by non-network-related things: high CPU, for example), but there is no way to safely recover without losing messages. (Note: The frame size issue I mention in that comment has been fixed in one of the latest versions.)

We have also encountered multiple bugs where RabbitMQ will get into a bad state that requires manual recovery. For example, it will suddenly lose all the queue bindings. Or queues will go missing. In several cases the RabbitMQ authors have given me a code snippet to run with the Erlang RELP to fix some internal state table; however, even if you know Erlang, you have to know the deep internals of RabbitMQ in order to think up such a code snippet. There have been a couple of completely unrecoverable incidents where I've simply ended up taking down RabbitMQ, deleted its Mnesia database, and started up a new cluster again. Fortunately, we use RabbitMQ in a way that allows us to do that.

The bugs have been getting fewer over the years, but they're not altogether gone. It's a shame, since RabbitMQ _should_ have a model showcase for Erlang's tremendous support for distribution and fault-tolerance. You're lucky if you've not had any issues with it; personally, I would move away from RabbitMQ in a heartbeat, if we had the resources to rewrite a whole bunch of apps. We've started using NATS for some things where persistence isn't needed, and might look at Kafka for some other applications.

[1] [https://news.ycombinator.com/item?id=9448258](https://news.ycombinator.com/item?id=9448258)

[2] [https://aphyr.com/posts/315-jepsen- rabbitmq](https://aphyr.com/posts/315-jepsen-rabbitmq)

<comment><USER9> Thanks a lot for elaborating. This is exactly the kind of insights I wanted to know.

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> Disclaimer: I work for Mesosphere (Champions of Apache Mesos and DC/OS)

We have total respect for K8s, but I don't think you can claim winning just based on the community and stars.

OpenStack has a huge larger community of developers and advocates, but it is still haven't reached it's potential despite many years and incredible effort, seminars and summits

Also most of these next gen infr projects now (DC/OS which is powered by Mesos, K8s, docker swarm ) are converging feature wise, but they also have their strengths and weaknesses, some of these are temporary, some of these are structurally by design

Mesos (and DC/OS) were not just designed for scale, but also for extensibility and different workloads, which is why you can run Cassandra, Kafka, Spark, ..etc In production. None of these workloads run as traditional containers, with DC/OS they have their own operational logic such as honoring placement, simple installation, and upgrade, which is a core design function of the two- level scheduler

People usually complain that standing up Mesos is hard, which is why we we built DC/OS and open sourced it to give you the power of mesos and its awesome capabilities in managing containers and data workloads without going through all the effort to stitch everything yourself. Check it out at DCOS.io, I am sure you guys will be blown away.

<comment><USER11> The problem I always here with Mesos is that it doesn't scale down well. I.E., that it is not developer friendly or, to put it another way, you can't run it on your laptop.

<comment><USER12> Actually that isn't true. Minimesos runs quite well on your laptop if you wish to test that way:

[https://minimesos.org/](https://minimesos.org/)

</comment></comment></comment><comment><USER13> I know it's young still, but I think Nomad is going to get a share of this market with little effort.

I played with Mesos & k8s and I picked Nomad instead. Now I'm not managing a huge fleet of servers I want to abstract away as much as I wanted a simple, consistent framework for allocating resources to containerized tasks across a small group of nodes. And I don't think that use case is anything to sneeze at and for a new user there just isn't anything out there as easy as nomad IMO.

[https://www.nomadproject.io/](https://www.nomadproject.io/)

<comment><USER14> For a simple solution, also see Rancher. I've used them about 2 years ago and even then it was a very useful and stable tool. [http://rancher.com/](http://rancher.com/)

<comment><USER15> I also use Rancher and quite happy with it.

The cool thing is that you can set up and environment using cattle (their own orchestration system), k8s, mesos or swarm. Up to you and your preferences.

</comment></comment></comment><comment><USER16> It pretends to compare Kubernetes, Apache Mesos and Docker Swarm. This article says Kubernetes has a lot of stars on github (doesn't compare it to Docker or Mesos, only says Kubernetes has a lot), same for Slack/Stack Overflow and number of CVs mentioning the tech ... I will pass Infoworld opinion from now on.

<comment><USER17> I completely agree that this is an article only intended to praise Kubernetes. Stating Google invented Linux containers alone is wrong as well.

</comment></comment><comment><USER18> I can bring up an app on Linux or Windows from bare metal in minutes by hand. But the way it's supposed to be done now is something like this, right:

      1) Use one of Chef/Puppet/Salt/Ansible to orchestrate
      2) One if those in item 1 will use Vagrant which
      3) Uses Docker or Kubernetes to
      4) Set up a container which will
      5) finally run the app

Really?

<comment><USER19>
      1) Developer pushes code to repo, its tested, if pass>
      1) Rkt/Docker image made by CI/CD system and pushed to docker registry
      2) Automatically deploy new image to staging for tests and master for production (can be a manual step)
      3) Sit back and relax because my time invested up front saves me hassle in the future

I have 30+ nodejs apps that can be updated on a whim in under 5 seconds each.

Setting up bare metal instances, even when using something like ansible is slow and cumbersome compared to what I'm now doing with k8s

Doing an apply/patch to kubernetes nodes takes seconds (assuming services aren't affected).

edit: Sorry, for the unfamiliar by services I mean load balancers/ingress, it's a k8s term. It takes 45s-1minute to update glbcs so I modify them as rarely as possible.

<comment><USER20> How do you solve your #2?

We push our docker images to quay.io but pull+deploy is still manual. How does the target platform detect a new container on the registry in order to kick off a pull+deploy?

<comment><USER8> Take a look at Drone ([http://readme.drone.io](http://readme.drone.io), not to be confused with the hosted Drone app). It allows you to execute arbitrary commands after publishing the Docker image. You can tell it to run kubectl to perform a deploy, for example.

Drone can also run CI tests and build stuff in temporary containers, which avoids the need to compilers etc taking up space in the final image, and negates the need for squashing (which busts your cache).

Much faster than Quay. Quay starts a new VM in AWS every time it builds, which is super slow. Drone just starts containers.

(You have to self-host Drone, though. Since it requires running Docker in privileged mode it's not a good fit for running under Kubernetes, unfortunately.)

(Kubernetes on GKE always runs in privileged mode, but that doesn't mean it's a good idea!)

<comment><USER21> Quay.io has been beta testing a new build system (based on Kubernetes under the hood) that should make builds a bit faster. If you're interested in testing it out, your organization can be whitelisted. Tweet at @quayio and we can get you set up.

</comment></comment></comment></comment></comment><comment><USER22> _It 's all about knowing how to build an open source community_

This. Engineering excellence is secondary. You can get away with complete craptitude in your tech if you can build community. (I won't name examples.) Of course, it's better if you also have technical excellence. On the other hand, you can have technical excellence, but it will come to naught if you have community destroying anti-patterns.

</comment><comment><USER23> In my experience, I haven't been coming to k8s because I particularly like the developer experience (despite their efforts to focus heavily on it), but because it cleanly supported some things that I need.

For instance, with k8s, out of the box every running container in a clustered system is discoverable and has its own IP. If you're writing distributed applications, and you're using containers principally as a tool to make your life easier (and not as part of an internal paas or for handling data pipelines or some other use case), having that sort of discovery available out of the box is great.

</comment><comment><USER24> One thing I've found extremely difficult to handle is the Zookeeper cluster model of containers. Where when a thing dies, a thing has to come back and be able to referred to as "zookeeper-1" forever. The way to do this currently is to use a service in front of a replication controller with one pod. This feels wrong all over. Supposedly they have a thing called Pet Sets [1] coming to solve this, but it's been in the works for an eternity. Also we've started to outgrow the load balancing simplicity that the k8 load balancer gives you, and I have not seen a nice migration path to something like HAProxy in Kubernetes. All that said, we like kubernetes a lot.

[1] To distinguish from cattle. If you have a red blue and green goldfish, and your red goldfish dies, you can replace with another red fish and not really notice, but if it's purple, the others won't play with it.

<comment><USER25> Re: load balancing. Out of curiosity, why not just use haproxy? Or nginx, or your load balancer of choice.

Run haproxy in host networking mode, as a daemonset, and use it as your ingress point?

<comment><USER24> So it would basically be daemon set sitting on top of all the other services, with their own load balancing turned off? That should work. I like that

<comment><USER25> Email's in profile. I'm happy to discuss specifics offline, as I have some experience with this. Can show some example configs of how we've employed it.

</comment></comment></comment></comment><comment><USER2> One thing that I think Kube (and dc/os) are missing is what Chef is working on right now. Application definition should live within the app and consumed by the scheduler.

Chef's product is called Habitat [https://www.habitat.sh/](https://www.habitat.sh/) and it has some VERY interesting concepts that if Kube will implement it will be much more interesting (to a lot of people).

Right now, the deployment and configuration of an application is supposed to be separated but I feel they need to be just a bit more coupled. An engineer that develops the application will define a few things like "domain" and how you connect to the application and this will be consumed by the scheduler.

Right now, dc/os and mesos are really fine grained around the DevOps people and I feel that the first that will crack the "batteries included" approach will win the fight by a knock-out.

Imagine something like Heroku on your own infrastructure, if an engineer can launch and deploy a micro-service with the same ease they deploy and access a Heroku application. That will be awesome.

<comment><USER8> While coupling is convenient, there is a problem when you are not the only consumer of your code. If you have an open source project that is to be deployable by anyone, you can't put Kubernetes files there (there's not really a way to write a generic Kubernetes manifest -- in addition to volumes and secrets, things like labels are environment-specific). This is also true for other things like .drone.yml for Drone.

So you have to maintain a private "release fork" that you periodically rebase into. This becomes a workflow bottleneck, and the VCS history will not look good, either. Even worse if you have multiple branches you deploy from.

<comment><USER2> This is true. It's especially true if you have services depending on one another. I think there's a middle-ground there.

Right now, 99% of people I know use Github-Workflow, master is always deployable (often CD), other branches are able to deploy using "hubot deploy branch-name to xxx"

I think if you take the latter and you make it easier to deploy an application on top of your cluster, you can gain a lot.

The workflow today is more cumbersome. If you could create your application, attach some definitions to it and launch it to the scheduler, it would be better.

</comment></comment></comment><comment><USER26> Just as an outside observer developing on a platform, I see my fellow devOps team members working in Kubernetes and it's been a shit storm for the most part where containers disappear randomly, stuff breaks and they are on call on the weekends not having fun. I have my own clients on other projects using AWS where I just upload and click buttons like a dumb monkey and I end up looking more competent even though I'm completely not. I've consequently not been motivated to dive into these DIY deployments just yet.

</comment><comment><USER27> I haven't dived into Kubernetes yet, but I set up Rancher for our new application and it has been nothing short of amazing so far. I can't express how happy we've been with it.

I previously tried the Mesos/Marathon route (with Mesosphere and then again with Mantl) and that was nothing but a huge waste of time due to all the maintenance that was necessary for all the required servers. With Rancher, it's just spin up a container for each host with a single command and you're done.

<comment><USER28> I've been looking at Rancher for a few weeks, haven't put an application yet. How do you perform rolling updates with Rancher+Kubernetes? With Kubernetes itself, currently I just do `kubectl rolling-update <rc> \--image <new-image>`

</comment></comment><comment><USER29> From the subtitle:

 _It 's all about knowing how to build an open source community -- plus experience running applications in Linux containers, which Google invented_

<comment><USER30> It kinda did, right? Maybe, "invented" is to bold of a term, maybe there were others before. But, I've heard folk tales that Google has been running stuff in containers since the early to mid 2000s.

<comment><USER31> Google contributed cgroups to Linux, so saying that it invented Linux containers is a fair statement. Other operating systems had containers before Linux.

<comment><USER12> Not exactly. Google contributed the namespace support to Linux and IBM contributed much of the initial control group bits to Linux. Together, they form the modern "container" that we have all came to know and love.

</comment></comment></comment></comment><comment><USER32> The setup to get k8s running isn't great, but once it's running and you understand it's config files it makes things so much easier. We're getting ready to deploy k8s at work soon and begin moving more there as we can.

From what I understand, and is completely not in the article, Mesos is designed for scale while most start-ups (and even established companies) can't afford or justify. K8s is simpler but still robust. Better than just fleet or compose and clearly still better than swarm (based on posts read here on hn).

<comment><USER33> Disclaimer: I work for Red Hat.

> The setup to get k8s running isn't great,

+1 But they are greatly improving.

For a local test environment have a look at OpenShifts 'oc cluster up' [https://github.com/openshift/origin/blob/master/docs/cluster...](https://github.com/openshift/origin/blob/master/docs/cluster_up_down.md)

<comment><USER34> Someone elsewhere in this thread complained about the NFS experience with k8s. I know OpenShift contributed some or all of that code so does it improve the NFS experience as a layer on top of k8s?

<comment><USER35> Disclaimer: I work at Red Hat on OpenShift.

All of the code Red Hat contributed around Kubernetes storage plugins went into Kubernetes upstream. If you have any questions or problems, feel free to raise an issue or reach out to us in the community. Red Hat has a large number of contributors on the Kubernetes project and we are big fans of the community and the technology!

Red Hat OpenShift provides a an enterprise-ready Kubernetes distribution that also builds on top of Docker, RHEL/Fedora/CentOS and includes integrated networking (OVS), routing (HAProxy), logging (ELK stack), metrics, image registry, automated image builds & deployments, integrated CI/CD (Jenkins), and self-service UX (Web, CLI, IDE). You can check out the free Origin community distro here - [https://github.com/openshift/origin](https://github.com/openshift/origin) or sign up for our free hosted developer preview here: [https://www.openshift.com/devpreview/](https://www.openshift.com/devpreview/). We also offer commercially supported solutions - [https://www.openshift.com/container- platform/](https://www.openshift.com/container-platform/).

</comment></comment></comment></comment><comment><USER36> Seems like "Why Kubernetes is winning the orchestration war" is a more appropriate title for this?

</comment><comment><USER37> I use Mesos and K8s heavily as well as contribute back to the projects, and while I do agree this is leaning towards being fan-fare there is a bit of truth to this.

Community is a big deal, people tend to underestimate this; Putting aside newer companies, when a larger enterprise ventures out to open source they do take community as a major factor as you have to consider the tool you build will have to last 5 years, maybe 10, maybe more.

To delve further into the Docker side of things, I personally wish that the company would focus on its core business instead of stretching itself with extra things. I get the need to improve the UX, which they do very well considering how far we have come from LXC.

I feel Mesosphere starting to go down the feature creep route as well, but I wish them all the best as I loved Mesos since the beginning all those years ago.

</comment><comment><USER17> Why is no one mentioning Docker's response with version 1.12, the new built-in orchestration called swarm mode (different than just swarm): [http://blog.nigelpoulton.com/docker-launches-kubernetes- kill...](http://blog.nigelpoulton.com/docker-launches-kubernetes-killer/)

Granted, the title is fanboyish, but it really seems to be a significant response to Kubernetes.

</comment><comment><USER38> IMHO nobody is winning anything that matters right now because the current transition is a transition to an additional level of abstraction which is _definitely_ not properly met by any of the tools available.

What we now need is _tools that allow architectural fences around components, and reliability guarantees around subsystems_ ... versus not only technical but also business-level risks (including state-level actors)... often across borders, including for example exchange rate risk. This is based on business- level risk models, not some engineer feels X or Y type reasoning which is (often very well, but broad-picture uselessly) based on technical know-how.

I prototyped such a system pretty successfully, you can read the design @ [http://stani.sh/walter/cims/](http://stani.sh/walter/cims/) .. it's incomplete (critically hard to explain investment utility for non-tech business types) but at least infrastructure agnostic and practically informed.

NB. To be non-humble, by way of demonstration I am a guy who has been called to conceive/design/set up/manage from-scratch datacenters for extremely visible businesses with highly motivated attackers with USD$Ms at stake for penetration. Systems I personally designed run millions of USD/month and have many times that in investment. And it's obvious that both Google ("compete with amazon on cloud business... high-level desperate for non-advertising income!") and Docker ("VC! growth! growth! growth!") have their own agendas here... we should trust no-one. It's early days. Bring on the ideas, build the future. Hack, hack, hack. We the little people own the future. It's ideas.

</comment><comment><USER39> Looking through all of these comments we should be glad, as a community, that there are a number players in this burgeoning orchestration space. Nobody has won and nobody should win. They each have their strengths and weaknesses and there is no one size that fits all.

</comment><comment><USER40> Sad to see Mesos losing steam. My understanding was that Mesos subsumes the functionality of Kubernetes thanks to its Aurora scheduler. But it has much more customized schedulers, for different purposes, that might make it more efficient to run complicated pieces of software.

For instance, it certainly is possible to run a Cassandra cluster by having each instance run in its own Docker container. My understanding is that it would be much more efficient to run this cluster with a dedicated Cassandra cluster instead.

Is this right? Or are the performance benefits of running a dedicated Cassandra scheduler on Mesos negligible compared to running them in containers?

</comment><comment><USER41> It would be nice if open source projects (especially popular databases) came with k8s definition files so that you wouldn't have to write the yaml yourself.

<comment><USER42> I did that for my open source project SocketCluster. See [https://github.com/SocketCluster/socketcluster/tree/master/k...](https://github.com/SocketCluster/socketcluster/tree/master/kubernetes)

Also, if you look at the main Kubernetes repo on GitHub, you can find the yaml files for various popular open source projects in there as well. Some of them are out of date though - Things have been changing so fast.

I think a problem that still comes up is that you still need yaml files to account for the interactions between different services too. Each service doesn't live in a vacuum.

We're currently working on building an complete auto-scalable stack on top of Kubernetes and a deployment service to go with it. See [https://baasil.io](https://baasil.io)

Everything we do is open source except the actual website.

<comment><USER43> I use SC for my startup and it works great! Thanks. I'll have a look at the Baasil service, it sounds interesting.

</comment></comment></comment><comment><USER44> I wish kubernetes has more examples for example their vsphere volume driver has almost 0 documentation/tutorials on how to set that up.

[http://kubernetes.io/docs/user-guide/volumes/#vsphere- vmdk-e...](http://kubernetes.io/docs/user-guide/volumes/#vsphere-vmdk-example- configuration)

I believe is inadequate

<comment><USER45> Agree - better doc is a focus for us. We've added providers so fast the doc has not caught up.

</comment></comment><comment><USER46> The author is just gaga over google. He presents no comparison, no benchmarks and no mention of alternatives except docker swarm.

</comment><comment><USER47> What is the best resource to learn Kubernetes like a boss? I like ebooks, but will take anything, as long as it's up-to-date and easy to follow without being a long-term sysadmin.

<comment><USER25> You might not like this answer, but... play with it. In our environment, we're compiling and deploying Kubernetes by hand (I mean, not entirely by hand, but you get the idea). It was daunting at first but is getting easier and easier all the time, and we have a better handle on how things work as a result. What's going on under the hood, what the mass of command line parameters actually mean, and so on.

The documentation is lacking. Badly. And doesn't seem to keep pace with the features that are added in every point release, or in the alpha/beta versions. A lot of what we've learned has been by experimentation, poking at things, following the k8s user discussions, watching Github, reading code.

<comment><USER48> Can you elaborate on the information you wish was documented but isn't?

Also, have you seen the box at the bottom of the docs that says "I wish this page ..."? It goes right into their issue tracker, which increases the likelihood of something getting fixed.

<comment><USER25> To be fair, I have not seen or used that link, but I will take note of it for the future - thank you, that's very helpful.

I should keep notes on specifics, so I apologize that I can't highlight a particular thing that I've been frustrated by in a given moment. I'll certainly say that the documentation is improving.

As a general item, I think my biggest struggle has been hitting a wall with the documentation - there are some things that are left almost as exercises to the reader, especially getting into more advanced topics (how does one do rolling upgrades or canary deployments in more esoteric situations, how might one do load balancing properly in a master-slave style configuration versus something more round-robin oriented, etc.)

And I don't like to levy a complaint without acknowledging that anything opensource is ripe for improvement by contribution, but my professional situation prevents me from doing so. Trust that I would love nothing more than to help and not just make an empty observation or moan about it.

<comment><USER48> So I typed out this whole answer and only realized at the end that I should have asked: have you tried their Slack channel: [https://kubernetes.slack.com/messages/kubernetes- users/](https://kubernetes.slack.com/messages/kubernetes-users/) and/or their mailing list [https://groups.google.com/forum/#!forum/kubernetes- users](https://groups.google.com/forum/#!forum/kubernetes-users) for getting answers to your specific concerns?

 _how does one do rolling upgrades or canary deployments in more esoteric situations_

So there are a couple of moving parts to that: foremost, Kubernetes is only a tool, and so I would doubt there is The One True Way(tm) of doing canary deployments -- and I'm taking liberties with your comment in that I presume you don't mean the literal rolling-upgrade which is ``kubectl rolling-update`` and is _very smart_. Having said "it's just a tool," the first thing that sprung to mind when reading canary deployments was the health checker built into the ReplicationControllers. There are several ways one can check on the "health" of a Pod, and k8s will remove any Pod from service that has declared itself ineligible for service.

If the QoS metrics are too broad for any one given Pod to have insight into, k8s has a very robust API through which external actors can manipulate the state of the cluster, rolling back the upgrade if it turns out the new deploy is proving problematic.

I hope he doesn't throw tomatoes at me for suggesting this, but Kelsey Hightower <[https://github.com/kelseyhightower>](https://github.com/kelseyhightower>) is an amazing, absolutely amazing, community resource. If you don't find inspiration from the talks he has given, then reach out on Twitter and ask for a good place to read up on your concerns. I bet if he doesn't know the answer, he will know the next contact for you to try.

 _load balancing properly in a master-slave style configuration versus something more round-robin oriented_

Now there is something you and k8s may genuinely have a disagreement about - in its mental model (to the best of my knowledge) every Pod that matches the selector for a Service is a candidate to receive traffic. So in that way, there is no "slave" because everyone is equal. However, if I am hearing the master-slave question correctly, failover is automagic because of the aforementioned health check.

<comment><USER25> I have, yes, and there's no doubt they're both crucial in getting answers to some of those more advanced topics.

 _edit_ I want to also add that the Kubernetes community meeting is fantastic. I don't attend every week but I do play catchup on the videos released after.

Regarding rolling updates, I have found that looking at the kubernetes API - and discerning what kubectl is doing under the hood - is helpful in some cases. We've taken to API inspection to assist with some of the more complex update patterns we're trying to address.

Completely agreed on the value of Kelsey to the community. I hesitate about contacting any one person directly on a topic, but his guides and github repo are just outstanding.

On the load balancing/master slave thing - I would have agreed completely with you. Master/slave configurations seem antithetical to some of the core Kubernetes concepts... but then the waters are becoming muddied because it seems like PetSets are a response to that missing pattern. I think you're right on the mental model. Every pod is (from what I understand), or at least should be, seen as an equal candidate, the only thing being "can it receive and service requests, if yes, kubernetes doesn't care".

Failover by letting Kubernetes do the dirty work is, of course, an option. If the master or a slave dies - it'll come back up. Except... it's also at odds with anything that handles its own clustering model (thinking Redis Sentinel or Redis Cluster, or InfluxDB when it had clustering support). Sometimes "coming back up" in a system that has an orthogonal view to Kubernetes is challenging.

It also doesn't accommodate for the situation where... what if Kubernetes _doesn 't_ bring it back up for some reason? If the pod was killed for resource contention, or something along those lines. And now I have two slaves, no master, nowhere to feed writes?

I dont't complain too loudly about these things, because most often the solution is (correctly) to think differently about it, but there are real- world considerations that don't always fit perfectly with the existing Kubernetes model - but we may still have to use them from time to time.

<comment><USER45> PetSet co-author here:

Think of PetSet as a design pattern for running a cluster on Kubernetes. It should offer a convenient pattern that someone can use to Do-The-Right-Thing.

The hard part of clustering software is configuration change management. If an admin adds a new member to the cluster by hand, they're usually correct (the human "understands" that the change is safe because no partition is currently going on). But an admin can also induce split brain. PetSets on Kube try to make configuration changes predictable by leveraging their own strong consistency (backed by the etcd quorum underneath the Kube masters) to tell each member the correct configuration at a point in time.

The PetSet goal is to allow relative experts in a particular technology to make a set of safe decisions for their particular clustered software that behaves _exactly_ as they would predict, so that they can make a set of recommendations about how to deploy on the platform.

For instance, CoreOS is working to provide a set of recommendations for etcd on pet sets that would be correct for anyone who wants to run a cluster safely, and encode that into tutorials / docs / actual pet definitions. They're also driving requirements. Other folks have been doing that for elastic search, zookeeper, galera, cassandra, etc. PetSets won't be "done" until it's possible to take the real world considerations into account safely when writing one.

<comment><USER25> That's really valuable insight, thank you. The biggest struggle so far has been with zookeeper and Kafka. Things that are stateful have posed a lot of difficulty - from a mental perspective more than fighting against Kubernetes specifically, just trying to think and adhere more to microservices principles.

I'm following PetSets very closely and I think they're going to help a great deal.

Is it accurate that PetSets were introduced to compensate for the type of thing I've singled out? That certain models just don't quite fit with the "everything is equal and interchangeable" notion? Or does it just feel that way? I don't want to end up missing the point of PetSets or using them for something they're not truly intended for, or leaning on them only to hit a stumbling block.

<comment><USER45> Yes. PetSets are intended to be step one of "some things need to be stable and replaceable at the same time". So that unit can then be composed into apps (i.e. Zookeeper + Kafka might be a PetSet for ZK and multiple scale groups for Kafka). basically trying to boil the problem space down into "what do cluster aware software need to be safe", like protection from split brain and predictable ordering.

There is likely to be more needed (on top or around) because we need to also solve problems for highly available "leaders" and active/passive setups. Think of what something like pacemaker does to ensure you can run two Linux systems in an HA fashion, and then map the necessary parts to Kube (fencing, ip failover, cloning, failure detection). Still a lot of hard problems to solve - PetSets are the first building block.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER49> These articles never mention the elephant in the room, AWS.

How many containers are running on Elastic Beanstalk and ECS?

I'd wager magnitudes more are running containers by reading the docs and clicking around than those that mastering the cutting edge challenges getting running on Mesos, Kube or Swarm.

Another blind spot is Heroku. Every day new businesses spin up containers with a 'git push Heroku master' and don't even know about it.

All these providers, platforms and tools have their place.

I simply don't think the "winning the war" talk is accurate or constructive.

Disclaimer: I worked on the container systems at Heroku and now manage 100s of production ECS clusters at Convox.

<comment><USER50> What "cutting edge challenges" are there to get running on Mesos/Kube/Swarm? Are you just referring to getting the cluster booted? There are just so many options for that these days.

Just curious, I was skimming through ECS docs last night and it seems like some of the basic building blocks look very similar to their equivalent constructs in Kube/Nomad.

<comment><USER49> Automating VMs with Elastic Beanstalk and deploying to a platform like Heroku are effectively solved problems.

Container orchestration brings in new challenges like operating a consensus database, managing data volumes and load balancing.

And this is on top of VM management.

Yes, ECS, Kube, Nomad are very similar. All relatively hard to use right now compared to their simpler ancestors.

Here is more in depth info about the challenges of ECS:

[https://convox.com/blog/ecs-challenges/](https://convox.com/blog/ecs- challenges/)

</comment></comment></comment><comment><USER51> On Safari OS X, the article is blocked by a full-page ad that I can't dismiss.

<comment><USER52> Don't see any ads on Chrome with Adblock Plus.

It shows 32 blocked requests though (ouch).

<comment><USER53> That seems light for most pages...

<comment><USER52> About average, I'd say.

Quick check:

* cnn.com: 30-40 * bbc.com: 10-15 * washingtonpost.com: 10-25 * medium.com: 1-2 * teccrunch.com: 25-40

</comment></comment></comment></comment><comment><USER54> Innovation is what makes this industry so exciting to be a part of. I joined the tech world, via Holberton School[1] just a mere nine months ago, and already so much has changed. It makes it challenging to keep up, but that's the fun part.

The future of technology is held in the hands of open source projects.

[1] [https://www.holbertonschool.com](https://www.holbertonschool.com)</comment>


Why I Switched to Octopress - <AUTHOR> http://grigy.com/blog/2012/01/29/why-i-switched-to-octopress/ <comment><USER2> What is your process for publishing new blog posts or editing an old one?

<comment><AUTHOR> Maybe I don't get the question right, but for now I follow the standard flow of Octopress:

<http://octopress.org/docs/blogging/>

What specifically do you want to know?

<comment><USER2> This answers my question, thanks. I wanted to know if you had set up your own customised flow for publishing.</comment></comment></comment>


Eliminating the Human - <AUTHOR> https://www.technologyreview.com/s/608580/eliminating-the-human/ <comment><USER2> Yeah, I really miss going to Best Buy and having "meaningful" interactions with the salesperson who knows nothing about what they're selling.</comment>


Ask HN: Finding a CPA for my startup - <AUTHOR> I'm running a solo startup at the seed stage, and the C Corp structure has brought so many new tax concepts that I'm still trying to get my head around: payroll taxes, franchise taxes, etc.<p>I tried finding a local CPA through yelp, but was disappointed with the results. Do you guys have any recommendations for a great CPA who has experience dealing with startups? Or any advice on how I might find one? <comment><USER2> Have you checked your local CPA firms? They typically will take on all kinds of clients, and if they don't feel they can meet your needs they will generally recommend you other firms who might be able to help you.

</comment><comment><USER3> early growth financial services is one of the larger tax / accounting firms focused on startups. ive used them with a few startups and they are responsive and offer a lot of services. there are tons of freelance / boutique startup CPA firms that may be cheaper but that all depends on your location</comment>


Best programming language for high performance (January 2017)? - <AUTHOR> http://lemire.me/blog/2017/01/16/best-programming-language-for-high-performance-january-2017/ <comment><USER2> For high performance, assembly language is frequently the best choice, at least for the central algorithmic core. Performance frequently demands access to the raw machine and a careful balance between the hardware and the algorithm not available in a modular, flexible, easy-to-understand compiled programming language. See, for example, the techniques used for the Intel CPUs in Lee, Victor W., et al. "Debunking the 100X GPU vs. CPU myth: an evaluation of throughput computing on CPU and GPU." ACM SIGARCH Computer Architecture News 38.3 (2010): 451-460.</comment>


Show HN: Daily Stock Fluctuation (web App Using IEX Free Trading API) - <AUTHOR> https://stockfluctuation.com/ <comment><AUTHOR> Hello HN. I stumbled upon a free trading data API ([https://iextrading.com/developer](https://iextrading.com/developer)) a few weeks ago and decided to write a small web app that utilized their data to display some helpful and easy to digest statistics. Some of my calculations might be off so any feedback and suggestion would be great! Thanks!</comment>


Facebook recommended that this psychiatrist's patients friend each other - <AUTHOR> https://splinternews.com/facebook-recommended-that-this-psychiatrists-patients-f-1793861472 <comment><USER2> Uninstalled the Facebook app a while ago — constant geolocation is too high a price. Also, file access could allow Facebook to read photo geotags when location was turned off. Constant clipboard access from the FB app was excessively invasive as well. Still use Facebook, but from a browser, making this sort of tracking only possible when I'm actually using Facebook.

</comment><comment><USER3> it's like all these self learning algorithms are doing things their creators never intended or understand.

It's kinda like that thing a while ago with Google correctly matching witness protection before-names with their new names.

Spooky shit happens when you let the computers do the thinking

</comment><comment><USER4> (2016), please.</comment>


You can donate Ethereum for Beer - <AUTHOR> http://ethforbeer.com <comment><AUTHOR> Amazing, right?

<comment><USER2> It's a one-pager with an ETH address. When people send you money you buy yourself beer. Might as well be a paypal donate button. I don't see anything amazing.</comment></comment>


Apple slows down iPhones to prevent sudden shut-downs - <AUTHOR> https://daringfireball.net/2017/12/iphone_battery_throttling <comment><USER2> Let's cut the bullshit. LiOn ageing is not unique to Apple. The problem is that their batteries are so horribly low powered (1800 mAh for iPhone 6s), especially for the price of the overall phone, that normal known ageing patterns would render the battery below spec of max power the iPhone draws at peak performance. It's BAD design.

Note 8 ships with 3300mah and iPhone X with 2700mah. Both cost a $1000. Which one do you think will last longer if 2500mah were the max power threshold?</comment>


Linux TCP/IP Tuning for Scalability - <AUTHOR> http://www.engineyard.com/blog/2012/linux-scalability/ <comment><USER2> Perhaps I'm crazy but didn't this just make the front page a couple weeks ago? Interesting though it is.. has anything changed?

Edit: I must be crazy because the date on that says 14..

<comment><USER3> Was this it? <https://news.ycombinator.com/item?id=4582253>

</comment></comment><comment><USER4> I do not think everyone fully appreciates how much overall application performance could be network-bound.

This is definitely the sort of tuning everyone should be doing on their production instances.</comment>


How Hacker News and FreshDesk led me to quit my job and launch my own startup - <AUTHOR> http://www.maaxmarket.com/marketing-automation/story-of-maaxmarket/ <comment><AUTHOR> a prototype of the product can be found at [http://demo.maaxmarket.com/app](http://demo.maaxmarket.com/app)</comment>


How to learn how to write great tests in Python? - <AUTHOR> I need help. I want to improve. I feel that I'm not writing the best unit tests. Can you guys recommend any books or articles where I can learn how especially in Python? I'm more concerned about strategies about knowing what tests to write and when is the best scenario to write them. I really want to be good at development and this area is what I think I'm lacking.
I'd really appreciate your help. Thank you. <comment><USER2> My best advice for you is to write modular functions that can be tested separately rather than integrated stuff that can't be disentangled.

A unit test should test the functionality of a single chunk of code. What kind of stuff are you writing?

I write webservers most of the time. So what I do is, at the top level of the request, immediately pull data off of the request so that I can access to the data in native python types that are easy to mock for testing. Each layer of my webserver does only one thing which could be gluing the auth+services together, but is still easy to mock.

</comment><comment><USER3> You might have a look at the O'Reilly book from August 2, 2017: "Test-Driven Development with Python: Obey the Testing Goat: Using Django, Selenium, and JavaScript" 2nd Edition</comment>


David Attenborough: collapse of civilisation is on the horizon - <AUTHOR> https://www.theguardian.com/environment/2018/dec/03/david-attenborough-collapse-civilisation-on-horizon-un-climate-summit <comment><USER2> Whatever's happening with the climate, we should be working to mitigate the risks - decrease the cost of food while increasing resilience of production; same goes for our infrastructure, what else? I get the feeling we're being snowed with all this hyperbole around CO2 and rhe warming planet. Bottom line is we need to become stewards of the Earth and Sea; fretting about carbon emissions is a distraction.

</comment><comment><USER3> Don't worry, rogue robots will kill us first ;-)

</comment><comment><USER4> Yea, we've heard this story numerous times. Has yet to happen, and dare I say, it never will.

<comment><AUTHOR> Granted there is a tendency to hyperbole that foresees a kind of "Mad Max" level of devastation but it would be unwise to wholly discount the collapse of certain parts of the eco-system - especially outside the sphere of wealthy countries. There's historical precedent. Parts of North Africa and Middle East that are perceived as desolate were once, in fact, green with reasonable levels of water access. Humans changed that.</comment></comment>


Doubts About Twitter the Business - <AUTHOR> http://www.twitterrati.com/2009/03/19/doubts-about-twitter-the-business/

<comment><USER2> You know what is most amusing about this? I have more doubts about Facebook's long term business plans than Twitters. Twitter seems easily monetizable, and the fact that others are already doing it outside (twitpay, etc) is some validation. Facebook seems to be monetizable only through ads. Annoying, annoying ads.

Both companies have potential valuation issues, however.

<comment><USER3> But isn't it trivial for Facebook to offer Twitter's functionality (or do they already? Not an FB user ;)) and instantly have about x30 (AFAIK) larger userbase?

<comment><USER4> They don't offer exact functionality (they're not aimed at the same market), but they've got something close.

Facebook already has ads, though, and they're evolving a business plan constantly. Twitter has nothing. So they have less experience with raising money and they've got more to lose.

<comment><USER5> I disagree. Twitter is already testing its adverts, and masterfully so imho.

I've clicked on those ads that they have now already. And since they were interesting things, I'm likely to keep an eye on that area for the foreseeable future, to see if anything else neat crops up.

Seems like an excellent way to roll-out ads... first, make the users look at the area where you'll place the ads, by putting interesting stuff there, and then put paid ads there.

So long as most ads remain relatively interesting/targeted, I'll keep looking. And probably clicking, too.

Consider also that Twitter users are very used to clicking on external links, because we do it all the time for links tweeted by others. I am much less likely to click on an external link in FB than on Twitter.

<comment><USER4> Huh! I didn't know Twitter was rolling out ads at all.

Facebook has some excellent targeting. I've gone to a few Facebook links and if I wasn't broke I'd have bought some products, too. (Right now they insist on showing me the Twilight DVD, though, which irritates me.)

<comment><USER5> Click here to see the ads: <http://skitch.com/swombat/beiik/twitter-now-with- ads>

<comment><USER6> Those are not ads. Those are just little promos for twitter features. By this logic, every link in the topnav and footer of twitter is an "ad".

[http://www.itworld.com/business/64445/twitter-house-ads- appe...](http://www.itworld.com/business/64445/twitter-house-ads-appear)

<comment><USER5> And you really think they're not going to use that piece of prime real-estate for text advertising? Wake up and smell the coffee :-)

<comment><USER6> I don't doubt that they are, but these are not ads.

HN has a lot of prime homepage real estate to the right, but if they put a link there to the submission form, it's hardly an ad :)

<comment><USER5> Yes, and so my point is, this is a masterful approach to pave the way for placing ads there.

The standard way is to place ads and juggle them about until you've figured out where they attract the more clicks. Twitter's way appears to first manufacture a high-interest area (by putting cool twitter stuff there), then, once people are used to seeing nifty stuff there, start selling that nifty space to others.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> The fact that we have invented the term "monetization" to delineate the point at which fun web projects try and become true businesses only underscores the fact that much of Web 2.0 is about getting rich (relatively) quickly, rather than building sustainable businesses.

</comment><comment><USER3> Favorite quote: "Sanford Bernstein's bearish approach is based on its belief the Web 2.0 model of building something and then figuring out a way to make money is no longer valid."

IIRC, that was supposed to be our lesson from the dot.com bubble :) The early hype about "Web 2.0" was that this time around we're doing it for cheap and with a revenue model...

<comment><USER7> I disagree. The lesson from the dot.com bubble was that anything put on the Internet could be monetized and would become profitable. That was (and is) silly.

The "Web 2.0" mindset, at least the mindset I've seen most often on HN, is build it fast and get it out there to see if it has value. I also think there is some thought to how the solution can be monetized, but plans change in response to changing environments.

<comment><USER3> Yes, but how do you judge if it has value? It seems that, as in the dot.com era, it's still not judged by income, but by number of users (or just level of hype - Twitter only has ~6 million users).

Current technology does make this valid in some sense - you can launch very cheaply and, if you attract enough interest, get bought by a larger company. As I understand this is YC's general approach. But it seems that Twitter tends to refuse buyout offers, so I they are more of a classic dot.com.</comment></comment></comment>


How E-Commerce Is Transforming Rural China - <AUTHOR> https://www.newyorker.com/magazine/2018/07/23/how-e-commerce-is-transforming-rural-china <comment><USER2> A tad too sensational and exaggerating, it's off-putting. I actually appreciate very much that the author went to China to get first-hand experience for the article. Too bad it's written as if the author was afraid that tech/business content would not have necessary sensation to sell the article.

JD has been doing great but I think it (and Taobao of Alibaba) has the great risk of losing the young generation (born after year 2000) who are less interested in these big box online sellers but more likely to buy from small invidual sellers, based on friends' recommendations or the sellers' promotions on social networks. For example, some of the new thriving places like Pinduoduo and Douyin (similar to Musical.ly, in fact Douyin bought Musical.ly recently) are very popular nowadays. The old giants might be gone within ten years because the ways to do ecommerce are changing so fast in China.

<comment><USER3> What do you think is exaggerated or sensational? It sounds like you wanted to read a different article about something else.

It seems to me that this article is mostly about technological and social changes as seen by people from small villages. Since the author was born in China and grew up in a small village, this is the ideal person to talk to others and tell their stories.

<comment><USER2> The part about JD.com's business and ground level operations are very informative and interesting to read, props to the author.

But on the other hand, details of personal stories of various characters in the article made me grumbling "Really?! Is this from the New Yorker?".

Just one example, in the story about her parents sending her to stay with relatives in an extremely poor village before they left China for the US, the author wrote, "I wouldn't see meat for three months;" and "that hunger could feel like a demon clawing at your stomach. ".

Come on, that was in 1992 when China had already got into economic reform for fourteen years(starting in 1978). Not matter how remote the village is, three months without meat for the daughter of your powerful relative? To make things even less believable, don't forget her mother was a doctor at an Army hospital in a big city, that's absolutely in the upper echelon in China financially in the 1980s. It's probably comparable to a $300K/year doctor in the US today relative to the rest of population, not to mention how few Chinese had the means and connections to immigrate (not merely visit) to US in that period of time, so how can you not be shocked to read that her parents would let their daughter go hungry and have no meat for three months?

These and some other passages made me enjoy a lot less of the article. I don't blame her because first the country and the people have gone through dramatic changes since she came to the states, and secondly even the New Yorker has to make things attractive to the mass in order to sell.

<comment><USER3> Of course memories can be fuzzy and reporters do sometimes exaggerate or get things wrong. Also, I certainly don't know one way or another (never been to China). But you seem overly confident making indirect inferences by analogy about what happened in a small village somewhere in China 25 years ago. The world isn't that neat and tidy.

</comment></comment></comment></comment><comment><USER4> this should be happening here in America more often to. There is no reason why we can't utilize our cheap vast farm land, a lot of it empty, for people who want to build things like robots, start bio-labs, do that sort of research, make interesting discoveries, build and manufacture products, work remotely, etc. There's already a huge supply chain network, and with 10kw of solar panels being under $5k (not including the inverter and wiring, I'm being very optimistic here:), and a diesel generator for other things, we should be encouraging it. The suppose next generation of satellite broadband is the icing on the cake. Right now, you're going to spend $150 for 2-3mb at 800ms, but hopefully that changes.

With the lowered cost of living, this lowers the barrier to entry to start a business. Imagine starting a business as an entrepreneur with your land and house already paid off, you'd be able to take far greater risks. I just spoke with a machine shop who complained that their prices on lathe parts are going up 25% due to the Trump China tariffs, and I asked them why don't they just make them here? The lathe parts are literally made on the lathe themselves. I pointed out, that their current product offering is basically a part-by-part clone of a popular lathe from the 1920s and is extremely outdated. Newer machines with less parts and higher precision could be more effective.

</comment><comment><USER5> Bit of a meta-comment, I wonder why there are so few comments on a submission with so many points, compared to others in the same point range. (100 pts and 5 comments at the time of me typing this comment.)

<comment><USER6> I find that the submissions I notice with the highest upvote-to-comment ratios are among the most thoughtful.

</comment></comment><comment><USER7> Immense oligopoly on its way</comment>


Track the time of your life on Earth just in your mac menu bar - <AUTHOR> http://time-of-your-life.com/ <comment><USER2> Nice, electron?

<comment><AUTHOR> yes, it is!</comment></comment>


How to be a faster code reviewer - <AUTHOR> http://blog.codacy.com/top-10-faster-code-reviews/ <comment><USER2> As a less experienced coder reviewing coders with half-decades or more experience on me, I always had a difficult time knowing whether I was looking at something that I just didn't understand the purpose of, or if I should question their wisdom.

For awhile, this resulted in me effectively checking to be sure it looked 'grammatically correct'\- as in, followed style guides, was linted right, etc. The bullshit stuff.

Then a few got through where I had reviewed them, wrote down "What's this for?" as a thing to review a month or so later, and they turned out to be _huge_ problems- sometimes performance, sometimes functionality that would break under the right circumstances.

Have others gone through this, and if so, how would you recommend 'teaching' a newer coder how to do quality code reviews of the elder coders around him?

<comment><USER3> You can ask the senior engineers, but nothing beats rolling up your sleeves, diving in and peeling the onion back. If you have a good manager, you need to tell them you need more time to absorb and understand the code base you're working on. They should be more than willing to accommodate, particularly for a more a coder with less experience.

My biggest pet peeve when I send code out for review is someone responding with the superficial bullshit stuff, so it's great that you identified it as a problem. You do tend to get reviews like that from junior folks, but I also see it with really insecure engineers. People start using code reviews as a weapon to take pot shots at each other.

I also think it becomes a problem when you're doing these smaller code reviews as the blog post is recommending; if you're doing bite sized chunks of code, it's really easy to lose context and overlook things which might be performance bottle necks.

<comment><USER4> _My biggest pet peeve when I send code out for review is someone responding with the superficial bullshit stuff_

Send something out with a fairly complex algo and you get back people complaining about a brace being 2 spaces instead of 4 or something else that will be automatically cleaned up by your editor before commit.

<comment><USER5> You should have an automatic code formatter / checker if possible. Plus code linter. Automate everything if possible and make build fail in case of discrepancies. Then there'll be no discussions like that.

</comment></comment></comment></comment><comment><USER6> The first point isn't something you can do: that's something you others have to do for you. Unless you count convincing people to change their workflow, which I've found to be near equivalent to moving mountains.

In my experience, it seems to boil down to not being effective with their VCS (or in some cases, the VCS not being an effective tool). But it's hard to convince people to go through the trouble of learning something: the short- term cost is just too much, and they seem to be blind to the long term gains. It's like convincing someone to use vim (who isn't an emacs or equivalently- powerful-editor user).

</comment><comment><USER7> A related question. I'm wondering what other people do to ensure quality of outsourced code. I'm intrigued at Codacy. A lint in the cloud could solve some of my problems, but probably not all.

I'm in the position of occasionally outsourcing non-critical development but not having time myself (I used to be technical; now focused on marketing) or able to distract the dev team to look at code in depth.

Do you outsource development? How often do you code review that code? Never, once in a while, on delivery, frequently? Do you do anything else to ensure outsourced code quality?

<comment><AUTHOR> Hi Sunir. Great questions. Hope to get more people answering this.

Please also take a look at a parallel discussion in proggit: [http://www.reddit.com/r/programming/comments/1zqbx1/10_ways_...](http://www.reddit.com/r/programming/comments/1zqbx1/10_ways_to_be_a_faster_code_reviewer/cfvymys)

From my experience in Codacy, some of our users use it to grade and quickly send out comments from the tool to the outsourced team. They like being able to review the code as it's being done (since it's per commit) with the help of our linting code patterns.

We also give out grades which is an overall vision of the project without having to go into much technical detail. "Why is this a C?" is a great question to ask :) I'd love to hear more about your experience and your pains and check if we can be of assistance. Here's my email: jaime at codacy dot com.

</comment></comment><comment><USER8> They missed the most important one, practice. It might sound like I am being snarky or stating the obvious, but like any new task at first you will be slow at code review and it will take up "too much time". Its a skill, in some situations, its not really too much code to review once you have gotten good at reading code. Everyone can do with being better at reading code too- traditionally people have been better at writing their own new code realtive to their ability to reading and grok their teammates code- and that doesn't scale well.

</comment><comment><USER9> Be sure to add tasks to the sprint for the code review to happen. If it's not an actual sprint item, I find it likely won't get done effectively. If you have a couple hours, it's a great time to train the developers on some of the libraries, code standards, practices, too.

<comment><USER10> > Be sure to add tasks to the sprint for the code review to happen.

Just make sure it is a sprint and not a sprintf.

</comment></comment><comment><USER11> I don't understand the argument of keeping your commits small. In my team we do code review after a task is done, not for every commit so the size of the commits won't affect the review.

<comment><USER5> A commit should be like UNIX tools: should do one thing and do it well. If the commit is adding hell lots of new features, fixing bugs, and doing heavy refactoring all at once, it's way too much.

When developing say a new widget (or heavily extending an existing one), you can develop basic functionality, commit, add bells and whistles, commit, and some more stuff, commit, send for code review, do the fixes, commit, ask for re-review maybe, and then squash the commits into one commit if needed. With DVCSes like Git it's extremely easy, and reviewing smaller commits is easier than having to go through a behemoth (of course it always depends on your particular situation).

I sometimes tend to open half-baked pull requests to ask for quick early feedback if I'm doing things the right way, and what to be careful about, and continue developing in background while the more senior person takes a look at the code.

<comment><USER12> Does that really work well in practice, though? I find that I might get the base functionality down to a workable state, but then as you add the bells and whistles you're likely to find and fix your own bugs, realize you need to refactor, etc.

You can make all those changes in distinct, clean commits, but if the reviewer works commit by commit, they have to follow the evolution of your code rather than simply review the final state. You might argue that provides a basis for a better review, but it's also much more time consuming for the reviewer.

<comment><USER5> When you find an issue with previous commit while adding new things, you can make a temporary commit, fix the issue, commit once again and reorder commits, then meld the fix into the previous one. I'm pretty comfortable with doing things like that but some guys not that much. You have to like this workflow and work from the start like this.

You can of course split a huge commit into smaller ones using sth like `git add -p` but it's way more work than just doing lots of small commits regularly and then squashing them into medium-sized commits before the review. I like to do temporary commits regularly as "checkpoints", meaning, "this code works", and anytime I make a stupid mistake and break something, I can analyze a short diff to find the bug I just introduced rather than trying to figure out the bug just by looking at the code. It's usually a lot faster for me.

</comment></comment></comment></comment><comment><USER13> The link to codacy at the bottom doesn't work.

<comment><AUTHOR> Thanks for heads up. Updated!</comment></comment>


What Would Happen if Amazon Ruled Publishing? - <AUTHOR> http://www.theatlantic.com/technology/archive/2011/10/what-would-happen-if-amazon-ruled-publishing/246854/#.Tp2Tm7PKn4Y.hackernews <comment><USER2> _If Amazon consolidates its power in the publishing industry, what would become of a book criticizing Amazon?_

It would be published as a simple PDF or mobile-friendly ebook format on the open web. Or, you know, just blogged.

Or sold as an app on iOS and Android.

Or sold by Pragmatic Publishers, or O'Reilly, or any of the other similar small publishers that will soon be all over the landscape, particularly in the all-electronic publishing era.

Or sold by Barnes and Noble. Does anybody remember Barnes and Noble?

Who knows, maybe Facebook or Google will start an e-publishing arm. Wouldn't be too difficult.

One shouldn't minimize Amazon's mighty marketing power, or their mighty infrastructure, or their customer goodwill, or their good hardware, or the extent to which they have a good eye in a market where everyone else is blind. And these things are going to continue to give them a mighty share of the publishing industry, because writers and vendors go where the money is. But they don't have a monopoly, and I don't see how they have an incipient monopoly, because publishing text is so easy that it literally can't be stopped by _major governments_.

<comment><USER3> Or sold on Amazon.com:

[http://www.amazon.com/Inside-Giant-Machine-Amazon-com- Story/...](http://www.amazon.com/Inside-Giant-Machine-Amazon-com- Story/dp/145153082X/)

</comment></comment><comment><USER4> Technically it would be their channel to control, just as Wal*Mart controls what inventory it carries (in music, for example).

Free speech would not apply as it's not government --but PR would suffer if they did restrict content, I think.

Still, I think if they decided to get into the business of gatekeeper, the gov (and the public) would want to investigate their actions --whether it would be legally actionable I have no idea. I think it would be foolish of them to go in that direction --it would be an opportunity for a smaller player to establish themselves as the non-paternalistic alternative.

</comment><comment><USER5> If Amazon dominates publishing, to the point of driving a noticeable number of publishers out of business, we'll have fewer choices in books. There will be more titles, and we'll find less worth reading.

Amazon sells an uncountable number of products because other companies make those products. Amazon _cannot_ make all those products themselves.

Similarly, Amazon _cannot_ shepherd within an order of magnitude the same number of more or less worthwhile books into the market as current publishers are doing today.

"Self publish yadda yadda whatever." Yes, Amazon will offload much of the production of books to the authors. And yes, Amazon can just say "yes" and print any book on demand. Paradoxically, I have a lot of books _actually available to me_ because publishers have acted as gatekeepers/curators. I can't possibly find as many mostly-self-published printed on demand or kindle Amazon books worth my attention as I can currently by choosing from the pre- filtered publisher-produced books of now.

</old man>

</comment><comment><USER6> When digital publishing and DRM technologies will become mainstream and standardized enough, I guess that the industry will have to agree on an open standard for e-books - or risk that governments will impose one by law.

The latter is what I think will - and in my opinion should - happen if Amazon and/or Apple ended up in a monopolistic/duopolistic position with their ebook/pad gadgets while keeping a strict control over what can be sold there. Which is probably what Amazon is trying to do, but in the long run I don't think that would be sustainable (either by market forces or government intervention).

So for publishers the long term real issue is most likely to be how to add value as (mostly) digital publishers, but not how to survive against a monopoly.

</comment><comment><USER7> _What are the services that publishers provide that we really value? ... Is it thoughtful editing and careful editing? Maybe, but many people accustomed to reading lightly-edited posts on the web may think that this sort of production is overvalued._

This gives me an idea: use the readers as editors. For every mistake found, offer $0.X rebate off their next book by the author.

<comment><USER8> This is an interesting idea, but many people don't realize that editing a book is more than just "finding mistakes." That's a proof reader's job. A good developmental editor helps shape a book from the ground up, and can make a _significant_ impact on the final product.

(Disclaimer: I edit books for a living.)

</comment></comment><comment><USER9> The article fails to answer its own question. It just does a bit of hand- waving and points at the apple app store.

</comment><comment><USER10> They wouldn't care

If it sells they make money - laugh at that suckers!

If it doesn't sell then nobody read it.

As some heavy metal band said about their albums being burned in the bible belt - "Those sales still count for the charts"!</comment>


US FAA Grounds Boeing 787 Over Battery Concerns - <AUTHOR> http://www.cnbc.com/id/100385850 <comment><USER2> I have traveled few times on ANA's 787 between Narita/Tokyo and Seattle. While sitting in emergency exit row at window seat, I could feel that the floor in front of the seat was unusually hot. After seeing location of the fire in pictures of JAL's 787, I get the impression that the emergency exit row seat was just above the cargo bay where fire originated. I am glad that ANA 787 I was on didn't catch fire.

I am glad to finally see FAA making a decision to ground 787. I hope 787 don't fly until these issues are resolved.

<comment><USER3> <http://en.wikipedia.org/wiki/Swissair_Flight_111>

<comment><USER4> In-flight fires cause many many crashes.

<https://en.wikipedia.org/wiki/Air_Canada_Flight_797>

<https://en.wikipedia.org/wiki/British_Airtours_Flight_28M>

<https://en.wikipedia.org/wiki/ValuJet_Flight_592>

[https://en.wikipedia.org/wiki/South_African_Airways_Flight_2...](https://en.wikipedia.org/wiki/South_African_Airways_Flight_295)

<https://en.wikipedia.org/wiki/Saudia_Flight_163>

</comment></comment></comment><comment><USER5> OK, from the horse's mouth:

[http://www.faa.gov/news/press_releases/news_story.cfm?newsId...](http://www.faa.gov/news/press_releases/news_story.cfm?newsId=14233)

And here's a good document dealing with Lithium-Ion battery hazards:

[http://www.nfpa.org/assets/files/pdf/research/rflithiumionba...](http://www.nfpa.org/assets/files/pdf/research/rflithiumionbatterieshazard.pdf)

I have a lot of experience with Lithium Polymer batteries as I fly eletric- powered radio-controlled helicopters and planes that use them. There are always reports of LiPo (as they are commonly known) packs puffing-up and sometimes catching on fire and even exploding (more like the popping of a pop- corn bag except that you have flames shooting out of it). I have personally exploded a small 3S (three cells in series) pack just to see the mechanism and the magnitude of the effect. I have to say that it took a LOT of work to explode the pack. I over-charged it to a ridiculous degree.

Now, of course, there are also reports of packs catching on fire for no apparent reason at all. There are many explanations out there. The best I came across was from a PhD Chemist who explained it something like this (paraphrasing, of course): When these batteries are assembled there's moisture in the surrounding environment. And, while moisture is kept in check, some of it stays in the pack. They can't run a super-dry environment because it could be very dangerous. One of the electrodes is lithium-oxide infused in carbon. Lithium reacts with water, which gives you hydrogen. Carbon reacts with hydrogen to give you methane. As methane is generated the cell puff-up. If thermal runaway is triggered through other mechanisms and the methane ignites: kaboom!

I keep about thirty LiPo packs in strong locked and vented steel container (heavy toolbox) in the garage away from flammable materials and flame sources. When we go to the flying field the same container is used to transport to the field. While at the field the container is always locked and, if possible, removed from the car. I have never had an incident I did not cause.

As far as the Boeing issues. Hard to say from the outside. It could boil down to quality issues at their battery supplier. It is probably impossible to perform a full inspection on these kinds of batteries on delivery. How do you determine if there's too much moisture inside a sealed package?

<comment><USER6> I think your PhD chemist needs to rethink his theory.

Lithium oxide is not the same as lithium metal. Lithium metal reactions with water to create hydrogen and heat, but lithium oxide is a salt. It generates no hydrogen when it contacts water, but it might generate some heat.

Carbon will react with hydrogen to give you methane, but not with the conditions inside the battery. You need a ton of pressure, heat and some catalyst to do that.

After reading the wikipedia article, apparently the cathode can actually produce oxygen which may explain the bursting of the cells.

<comment><USER5> Well, let's blame me for not recalling the details of the conversation. I'm sure what he said was correct, which probably isn't exactly what I posted.

</comment></comment></comment><comment><USER7> They are shooting for a 330 minute ETOPS for the 787. That means a plane can be up to 6.5 hours from the closest airport on long haul flights. ETOPS has more to do with engine reliability than fire safety, but I can't help wonder if this will affect things.

Incidentally, I flew on two United 787s in December and I thought they were fantastic planes. The electrochromic windows were a real treat.

<comment><USER8> 5.5 hours? Or is there an extra hour there above the ETOPS rating?

<comment><USER7> Woops you're right. Math failure on my part.

</comment></comment></comment><comment><USER9> Will this affect non-US airlines that flight to the U.S.? A 787 from a Polish airline (LOT) is about to land in Chicago. Will it be allowed to fly back to Poland with passengers on board?

<http://fr24.com/LOT3>

<comment><USER10> Right now, it will affect all flights, including those not flying to/from the US.

Only countries that are not members of the ICAO can still fly 787s after receiving the notification. (Congo?)

ICAO defines the rules and countries can change/add some parts of it using amendments (PANS). But those about safety/certifications cannot be changed.

When a plane is certified in a country, its local safety board is responsible for its safety certifications, decisions made by this safety board are reflected to all countries.

So, in fact, NTSB has jurisdictions over boeing certifications worldwide. If a boeing airplane crashes in brazil, NTSB will also participate in the investigation. If an Airbus crashes in the US, french authorities will also be envolved.

EDIT: (as noted by binarycrusader)

This seems to be a immediate service bulletin. So, the planes can still fly, they just need to be serviced before the next flight.

<comment><USER11> So can any national safety body of any member country of the ICAO ground an airplane model worldwide? Or can the U.S. do so in this case because Boeing is a U.S. company? What's the process a national agency goes through to ground an aircraft worldwide?

<comment><USER10> Only the country that emitted the certifications can cancel them. (models, not individual aircrafts).

Examples of certifications DO-254 and DO-178, those are required for building avionics and general software.

Link: <http://en.wikipedia.org/wiki/DO-178B>

Cool stuff. When I quit flying I thought about building software for aviation. Didn't work out.

</comment></comment></comment></comment><comment><USER12> Just a random piece of pedantry; the article states that Poland's LOT is the only european airline operating the 787 right now.

I was at Heathrow maybe 90 days ago and British Airways had a brand new 787 (Painted in fancy "Dreamliner" dress and BA logo) parked at a gate as if it was in use. Might be that they were still demoing it/showing it off deliberately but I had the impression it was an active aircraft.

<comment><USER13> The crew haven't been trained on it yet so I don't think it was in service.

If anything it was probably a meet and greet where the executives (and staff) can walk around the cabin to get a feel for it. They run their entire operation out of T5 now so getting a an unused gate for a 787 wouldn't be impossible. Other airlines are still jockeying for gate space at T1-T4.

<comment><USER12> And it _was_ Terminal 5, so that makes sense. It looks like they're not getting their 787s (and an order of 'superjumbo' A380s) until this year.

A training/demo aircraft makes sense. Not to mention free advertising people like me, sitting in a window seat, seeing that beautiful new aircraft with "DREAMLINER" painted across her body.

Granted, BA is the only Airline besides Virgin Atlantic/America where I have felt like I was treated like a human being... so convincing me to fly on them isn't _that_ difficult.

</comment></comment></comment><comment><USER14> There's a nice interactive guide to the 787's reported problems on the Guardian's site today:

[http://www.guardian.co.uk/business/interactive/2013/jan/11/b...](http://www.guardian.co.uk/business/interactive/2013/jan/11/boeing-787-dreamliner- faults-interactive)

</comment><comment><USER15> Does anyone know why the 787 seems to have gone so wrong? Wasn't it tremendously delayed and then it seems like I'm hearing about problem after problem?

Is this a sign our civilization is in technological decay and we're losing our ability to build advanced machinery?

<comment><USER16> 1.) General consensus is "too many outside contractors, not enough QA." Outsourcing taken too far, essentially.

2.) No.

<comment><USER17> 1+) The 787 was _three_ BHAGs[1] in terms of...

a) Fuel economy: drove the primarily composite structure (causing much angst during development), drove using the lighter weight lithium batteries which is the current issue, etc. A 20% fuel improvement is incredibly ambitious goal (I say _is_ because last I heard they were still falling short, although still remarkable at ~15%(?)).

b) Cost (sharing): drove the huge number of outside contractors, all or most of whom invested a substantial amount of their own money in the pieces parts that they designed and built for the 787.

c) Schedule. Total miss. The schedule was predicated on no major problems... in hindsight the (a) and (b) BHAGs guaranteed the schedule miss.

[1] <http://en.wikipedia.org/wiki/Big_Hairy_Audacious_Goal>

<comment><USER18> ANA said in a statement that their 787s are 21% more fuel efficient per seat than a 767

</comment></comment></comment></comment><comment><USER19> How long until United stops running the 787 ad at the beginning of every flight?

<comment><USER20> Just flew United on the 14th and didn't see the ad.</comment></comment>


Skype Expired Credit Class Action Settlement - <AUTHOR> https://creditexpirationclassaction.com/

<comment><USER2> This is one thing I don't like about our legal system as it stands. Skype cannot come out and say, yeah, we probably shouldn't have done that, because it could be used against them some day in the future. So instead they have to pretend that, or at least publicly declare, that they think the plaintiffs' claims are meritless, which any idiot can see they aren't. It's legally mandated suppression of honesty.

<comment><USER3> I remember getting told by a lawyer that you shouldn't even say 'sorry' after a car accident, because that is an admission of guilt which would damage any defense against lawsuits that might be brought against you.

</comment></comment><comment><USER4> I had some Skype credit expire on me. I don't intend to join in on this madness though. Skype sent me about 7 emails about this. Something like two months before expiration, one month before expiration, then two weeks, one week, then 5 days, then every day for the next 5. To save your credit all you have to do is make a call, it can last for less than a second if you want. I think they give reasonable notifications of expiring credit.

<comment><USER5> I got the same emails and DID make a call and sent an SMS text like the directions said and my credits expired anyway. I remember being irritated. Were they actively deceiving the public?

</comment></comment><comment><USER6> Also got this in my inbox. Haven't seen any news articles from legit sources referencing it though...

Edit: found [http://www.bjtlegal.com/classactions/barker/Barker_Complaint...](http://www.bjtlegal.com/classactions/barker/Barker_Complaint.pdf)

<comment><AUTHOR> Also: [http://www.gizmodo.com.au/2010/01/skype-class-action- settlem...](http://www.gizmodo.com.au/2010/01/skype-class-action-settlement- offers-us4-for-expired-credits/)

</comment></comment><comment><USER7> I just got an email about this. I assumed that it was a scam that Gmail had missed. Huh.

<comment><AUTHOR> Me too, actually. I haven't put my information in yet... still waiting to see if I can corroborate the story. The whois on the site seems to check out though. It's registered to a "Rust Consulting", whose website talks about class action settlements (<http://www.rustconsulting.com/>).</comment></comment>