<p dir="auto">We are running kubernetes 1.2.4 on GKE. We observed that when we increased load on our system by 20% our outbound connections from our pods started to fail 80% of the time. After much diagnosing we found that our cluster was running with just one <code class="notranslate">kube-dns</code> pod. This pod was being hammered and resulted in failures to resolve DNS requests.</p>
<p dir="auto">Our cluster started smaller (3x VM 8 core if i remember correctly) and is currently almost doubled in size/cores (10x VM 4 core).</p>
<p dir="auto">Creating a new cluster in GKE of the same size gave us three <code class="notranslate">kube-dns</code> pods. My expectations would be that the amound of <code class="notranslate">kube-dns</code> pods would scale by load (using a HPA?) or that the pods would scale by cluster size (using deamonsets?). Not that the amound of <code class="notranslate">kube-dns</code> pods would be based on the initial cluster size when creating it in GKE.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;2&quot;, GitVersion:&quot;v1.2.4&quot;, GitCommit:&quot;3eed1e3be6848b877ff80a93da3785d9034d0a4f&quot;, GitTreeState:&quot;clean&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;2&quot;, GitVersion:&quot;v1.2.4&quot;, GitCommit:&quot;3eed1e3be6848b877ff80a93da3785d9034d0a4f&quot;, GitTreeState:&quot;clean&quot;}"><pre class="notranslate"><code class="notranslate">Client Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.4", GitCommit:"3eed1e3be6848b877ff80a93da3785d9034d0a4f", GitTreeState:"clean"}
Server Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.4", GitCommit:"3eed1e3be6848b877ff80a93da3785d9034d0a4f", GitTreeState:"clean"}
</code></pre></div>
<p dir="auto">NOTE: I do not know if this issue is superseded by the DNS changes discussed in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="140012079" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/22823" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/22823/hovercard" href="https://github.com/kubernetes/kubernetes/issues/22823">#22823</a></p>
<p dir="auto">DNS SPOF, Autoscaling and GKE concerns - SPOF and upgrade path without downtime.</p>
<p dir="auto">Investigate switching to deployments or HPA.<br>
Fix addon manager upgrade path to allow rolling upgrades of DNS pods - SLO is 100% DNS service availability during rolling node upgrades and master upgrades.</p>