<story><title>Kubernetes Failure Stories</title><url>https://github.com/hjacobs/kubernetes-failure-stories</url></story><parent_chain><item><author>AaronFriel</author><text>Kubernetes does use dead simple off-the-shelf software for ingress and load balancing. That software though, unfortunately, has a lot of knobs, and what &amp;quot;Ingress&amp;quot; and &amp;quot;Service&amp;quot; resources do is make sure those knobs are turned to the right settings.&lt;p&gt;The nginx ingress controller for example, under the hood, just generates and applies a big ol&amp;#x27; nginx config! You can extract it and view it yourself, or extend it with a Lua script if you want to be fancy and do some FFI inline with processing the request, etc.</text></item><item><author>hinkley</author><text>Possibly edgy opinion:&lt;p&gt;Load balancing should be a solved problem already.  Swarm and Kubernetes should be using dead simple off-the-shelf software for ingress and load balancing.  Any competitor should be able to use the same solutions.  To put it another way, this shouldn&amp;#x27;t be a differentiator.&lt;p&gt;The problem is that the functionality in tools like nginx are still tied to static network architectures that evolve slowly, and don&amp;#x27;t take advantage of things like diurnal variability in workloads.</text></item><item><author>y4mi</author><text>But swarm only has overlay network iirc?&lt;p&gt;Swarm is great to get simple orchestration going, but it doesn&amp;#x27;t really do the same thing.&lt;p&gt;With k8s you can configure multi-point routing, storage backends and load balancing among other things.&lt;p&gt;With swarm, you get ... Overlay network (which is shit if you actually need to &lt;i&gt;really&lt;/i&gt; scale) and &amp;#x27;scaling&amp;#x27;, which starts n containers of the same service.&lt;p&gt;Swarm feels more like an mvp which actually works even for small teams. K8s is more like a behemoth which only dedicated personal can fully tame with a silly amount of features most people don&amp;#x27;t need.&lt;p&gt;We&amp;#x27;ve used both at my current job for toy projects (self hosted). Never in production however.&lt;p&gt;And I&amp;#x27;m a personal user of gcp - which works wonderfully .. albeit more expensive than simple root servers</text></item><item><author>theflyinghorse</author><text>Essentially why we made a choice to go with Swarm - less features, although all the features we needed are there (restarting containers, scaling them up or down and across several machines).</text></item><item><author>ben_jones</author><text>I feel like their needs to be a sanity check. Realistically what does kubernetes do to &amp;quot;not make software fail&amp;quot;? Health checks? Autorestarting containers that crash? Enforce various timeouts? Relatively quick distributed configuration deployments? These can be hard problems for sure, but do you need all of kubernetes to get these benefits in a production application?</text></item><item><author>dullgiulio</author><text>Nice, exactly what I needed today, after firefighting a latency spike because of a compute node after a separate ingress node was restarted.&lt;p&gt;I would make the taxionomy a bit more precise: failures at running software on the platform and failures at running the platform itself (which of course affect the software running over it: high latency, network packets dropped, DNS not reolved, ...)&lt;p&gt;In general I find that when the platform works as expected, it is not that easy to make software run on it fail. That is, it is harder than without what Kubernetes provides (granted: you can have it without Kubernetes, but many of us didn&amp;#x27;t bother to have capable admins setting things up the right way).&lt;p&gt;What I find extremely fragile is the platform itself. It works, but you have to start and stop things in the right order, navigate a sea of self-signed TLS certificates that can expire, iptable rules and services and logs.&lt;p&gt;All have failure modes you need to learn: it takes a dedicated team. And once you have that, you&amp;#x27;ll need another team and cluster to perform an upgrade.&lt;p&gt;But hey, when it works, it is really cool to deploy software on it.</text></item></parent_chain><comment><author>jhanschoo</author><text>&amp;gt; The nginx ingress controller for example, under the hood, just generates and applies a big ol&amp;#x27; nginx config!&lt;p&gt;I learned the hard way that GKE, in using GCP&amp;#x27;s load balancers don&amp;#x27;t support the same syntax for Ingress patterns as when you use an nginx Ingress. Definitely read the documentation thoroughly!</text></comment>
<story><title>Kubernetes Failure Stories</title><url>https://github.com/hjacobs/kubernetes-failure-stories</url></story><parent_chain><item><author>AaronFriel</author><text>Kubernetes does use dead simple off-the-shelf software for ingress and load balancing. That software though, unfortunately, has a lot of knobs, and what &amp;quot;Ingress&amp;quot; and &amp;quot;Service&amp;quot; resources do is make sure those knobs are turned to the right settings.&lt;p&gt;The nginx ingress controller for example, under the hood, just generates and applies a big ol&amp;#x27; nginx config! You can extract it and view it yourself, or extend it with a Lua script if you want to be fancy and do some FFI inline with processing the request, etc.</text></item><item><author>hinkley</author><text>Possibly edgy opinion:&lt;p&gt;Load balancing should be a solved problem already.  Swarm and Kubernetes should be using dead simple off-the-shelf software for ingress and load balancing.  Any competitor should be able to use the same solutions.  To put it another way, this shouldn&amp;#x27;t be a differentiator.&lt;p&gt;The problem is that the functionality in tools like nginx are still tied to static network architectures that evolve slowly, and don&amp;#x27;t take advantage of things like diurnal variability in workloads.</text></item><item><author>y4mi</author><text>But swarm only has overlay network iirc?&lt;p&gt;Swarm is great to get simple orchestration going, but it doesn&amp;#x27;t really do the same thing.&lt;p&gt;With k8s you can configure multi-point routing, storage backends and load balancing among other things.&lt;p&gt;With swarm, you get ... Overlay network (which is shit if you actually need to &lt;i&gt;really&lt;/i&gt; scale) and &amp;#x27;scaling&amp;#x27;, which starts n containers of the same service.&lt;p&gt;Swarm feels more like an mvp which actually works even for small teams. K8s is more like a behemoth which only dedicated personal can fully tame with a silly amount of features most people don&amp;#x27;t need.&lt;p&gt;We&amp;#x27;ve used both at my current job for toy projects (self hosted). Never in production however.&lt;p&gt;And I&amp;#x27;m a personal user of gcp - which works wonderfully .. albeit more expensive than simple root servers</text></item><item><author>theflyinghorse</author><text>Essentially why we made a choice to go with Swarm - less features, although all the features we needed are there (restarting containers, scaling them up or down and across several machines).</text></item><item><author>ben_jones</author><text>I feel like their needs to be a sanity check. Realistically what does kubernetes do to &amp;quot;not make software fail&amp;quot;? Health checks? Autorestarting containers that crash? Enforce various timeouts? Relatively quick distributed configuration deployments? These can be hard problems for sure, but do you need all of kubernetes to get these benefits in a production application?</text></item><item><author>dullgiulio</author><text>Nice, exactly what I needed today, after firefighting a latency spike because of a compute node after a separate ingress node was restarted.&lt;p&gt;I would make the taxionomy a bit more precise: failures at running software on the platform and failures at running the platform itself (which of course affect the software running over it: high latency, network packets dropped, DNS not reolved, ...)&lt;p&gt;In general I find that when the platform works as expected, it is not that easy to make software run on it fail. That is, it is harder than without what Kubernetes provides (granted: you can have it without Kubernetes, but many of us didn&amp;#x27;t bother to have capable admins setting things up the right way).&lt;p&gt;What I find extremely fragile is the platform itself. It works, but you have to start and stop things in the right order, navigate a sea of self-signed TLS certificates that can expire, iptable rules and services and logs.&lt;p&gt;All have failure modes you need to learn: it takes a dedicated team. And once you have that, you&amp;#x27;ll need another team and cluster to perform an upgrade.&lt;p&gt;But hey, when it works, it is really cool to deploy software on it.</text></item></parent_chain><comment><author>hinkley</author><text>&amp;gt; That software though, unfortunately, has a lot of knobs&lt;p&gt;Lots of people have different definitions of &amp;#x27;easy&amp;#x27; which is why I didn&amp;#x27;t say &amp;#x27;easy&amp;#x27;.  But how did you get this far off the rails with &amp;#x27;dead simple&amp;#x27;?</text></comment>