<p dir="auto">From examples:</p>
<p dir="auto">{<br>
"id": "redis-master-2",<br>
"desiredState": {<br>
"manifest": {<br>
"id": "redis-master-2"</p>
<p dir="auto">This stems from the fact that Kubelet takes ContainerManifest structs and needs them to have an ID, but none of the frontends massage the info from the Pod struct into the Manifest struct.</p>
<p dir="auto">We should think hard about whether using the same structs for the user API and the Kubelet API makes sense.  Or maybe we can restructure them to nest better, so the kubelet API looks more like:</p>
<p dir="auto">{<br>
id<br>
manifest<br>
}</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.): No</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): kube-dns stale/wrong</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): BUG REPORT</p>
<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>): 1.3.5</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: AWS</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): Ubuntu Xenial</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): Linux 4.4.0-21-generic <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35324390" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/37" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/37/hovercard" href="https://github.com/kubernetes/kubernetes/pull/37">#37</a>-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x86_64 GNU/Linux</li>
<li><strong>Install tools</strong>: Custom install (fluentd + etcd via ELB + kube-master via ELB)</li>
<li><strong>Others</strong>: The custom install isn't terribly "custom" - it's based on an older version of Kelsey's Kubernetes the hard way. The major difference is that both etcd and the kube-master go through an ELB that terminates idle connections after 60s.</li>
</ul>
<p dir="auto"><strong>What happened</strong>: kube-dns didn't refresh the DNS entry of a pod that got recreated.</p>
<p dir="auto"><strong>What you expected to happen</strong>: kube-dns to be up to date</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):<br>
I don't have solid reproduction steps since this is very intermittent, but here is how this particular instance happened:</p>
<ol dir="auto">
<li>Create a PetSet (say <code class="notranslate">petset</code>) and Headless service (say <code class="notranslate">disco</code>)</li>
<li>Start a different pod with nslookup</li>
<li><code class="notranslate">nslookup petset-0.disco</code></li>
<li>Delete the PetSet pod. It should be recreated with the same name.</li>
<li><code class="notranslate">nslookup petset-0.disco</code> See the same IP as before (rather than the new IP).</li>
</ol>
<p dir="auto"><strong>Anything else do we need to know</strong>:<br>
Here's the reality I'm seeing now:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# nslookup kafka-0.kafka-discovery.default.svc.cluster.local
Server:     10.254.0.10
Address:    10.254.0.10#53

Name:   kafka-0.kafka-discovery.default.svc.cluster.local
Address: 10.9.13.13"><pre class="notranslate"><code class="notranslate"># nslookup kafka-0.kafka-discovery.default.svc.cluster.local
Server:     10.254.0.10
Address:    10.254.0.10#53

Name:   kafka-0.kafka-discovery.default.svc.cluster.local
Address: 10.9.13.13
</code></pre></div>
<p dir="auto">But really:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl get pod kafka-0 -o yaml | grep podIP
  podIP: 10.9.13.5"><pre class="notranslate"><code class="notranslate">$ kubectl get pod kafka-0 -o yaml | grep podIP
  podIP: 10.9.13.5
</code></pre></div>
<p dir="auto">Interestingly, both instances of kube-dns are wrong on this one. It looks like the endpoints are correct though:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl get --context prod endpoints kafka-discovery
NAME              ENDPOINTS                                      AGE
kafka-discovery   10.9.13.5:9092,10.9.37.5:9092,10.9.73.5:9092   68d"><pre class="notranslate"><code class="notranslate">$ kubectl get --context prod endpoints kafka-discovery
NAME              ENDPOINTS                                      AGE
kafka-discovery   10.9.13.5:9092,10.9.37.5:9092,10.9.73.5:9092   68d
</code></pre></div>
<p dir="auto">Restarting both kube-dns pods fixed the issue. Is it perhaps a refresh triggering an <code class="notranslate">OnUpdate</code> rather than an <code class="notranslate">OnDelete</code>+<code class="notranslate">OnAdd</code> &amp; the handler doesn't expect the pod ip to change?</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/justinsb/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/justinsb">@justinsb</a></p>