<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<ul dir="auto">
<li>"too old resource version"</li>
</ul>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>
<p dir="auto">BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;4&quot;, GitVersion:&quot;v1.4.0&quot;, GitCommit:&quot;a16c0a7f71a6f93c7e0f222d961f4675cd97a46b&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-09-26T18:16:57Z&quot;, GoVersion:&quot;go1.6.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;darwin/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;4&quot;, GitVersion:&quot;v1.4.1&quot;, GitCommit:&quot;33cf7b9acbb2cb7c9c72a10d6636321fb180b159&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-10-10T18:13:36Z&quot;, GoVersion:&quot;go1.6.3&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:16:57Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.1", GitCommit:"33cf7b9acbb2cb7c9c72a10d6636321fb180b159", GitTreeState:"clean", BuildDate:"2016-10-10T18:13:36Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="gcloud container clusters list"><pre class="notranslate"><code class="notranslate">gcloud container clusters list
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="NAME  ZONE        MASTER_VERSION  MASTER_IP       MACHINE_TYPE   NODE_VERSION  NUM_NODES  STATUS
k0    us-west1-b  1.4.1           104.199.114.88  n1-standard-1  1.4.1         3          RUNNING"><pre class="notranslate"><code class="notranslate">NAME  ZONE        MASTER_VERSION  MASTER_IP       MACHINE_TYPE   NODE_VERSION  NUM_NODES  STATUS
k0    us-west1-b  1.4.1           104.199.114.88  n1-standard-1  1.4.1         3          RUNNING
</code></pre></div>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">When the resource version is too old the API server returns HTTP 200 OK.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ curl -i http://127.0.0.1:8001/api/v1/watch/namespaces/default/endpoints/nginx?resourceVersion=240385"><pre class="notranslate"><code class="notranslate">$ curl -i http://127.0.0.1:8001/api/v1/watch/namespaces/default/endpoints/nginx?resourceVersion=240385
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="HTTP/1.1 200 OK
Content-Type: application/json
Date: Tue, 18 Oct 2016 21:47:16 GMT
Content-Length: 176

{&quot;type&quot;:&quot;ERROR&quot;,&quot;object&quot;:{&quot;kind&quot;:&quot;Status&quot;,&quot;apiVersion&quot;:&quot;v1&quot;,&quot;metadata&quot;:{},&quot;status&quot;:&quot;Failure&quot;,&quot;message&quot;:&quot;too old resource version: 240385 (250923)&quot;,&quot;reason&quot;:&quot;Gone&quot;,&quot;code&quot;:410}}"><pre class="notranslate"><code class="notranslate">HTTP/1.1 200 OK
Content-Type: application/json
Date: Tue, 18 Oct 2016 21:47:16 GMT
Content-Length: 176

{"type":"ERROR","object":{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"too old resource version: 240385 (250923)","reason":"Gone","code":410}}
</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">When the resource version is too old the API server returns HTTP 410 to match the error code in the response body.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<ul dir="auto">
<li>Create a deployment and expose it.</li>
<li>Scale up the deployment to 3 nodes</li>
<li>Wait ~30 mins and get the endpoints backing the service</li>
<li>Perform a watch on the endpoint using the resource version from the previous get</li>
</ul>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">No.</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto"><code class="notranslate">no</code></p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<p dir="auto"><code class="notranslate">upgrade dns</code><br>
<code class="notranslate">upgrade external ip</code></p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>
<p dir="auto"><code class="notranslate">BUG REPORT</code></p>
<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto"><code class="notranslate">1.3.7</code></p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: <code class="notranslate">Google Cloud Container Engine</code></li>
<li><strong>OS</strong> (e.g. from /etc/os-release):  <code class="notranslate">Debian GNU/Linux 7 (wheezy)</code></li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): <code class="notranslate">Linux gke-trace-pool-1-dc497b52-0ptp 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux</code></li>
<li><strong>Install tools</strong>: -</li>
<li><strong>Others</strong>: -</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<ol dir="auto">
<li>new node pool added to the cluster via GC Container Service Dashboard (new pool 1.3.7, default pool is: 1.3.6, master is: 1.3.7)</li>
<li>nodes in old (default) pool set to un-schedulable and drained</li>
<li>default pool removed via GC Container Service Dashboard</li>
<li>Services with External IP stopped working from the public IP, all requests were hanging (while pods could reach each other via internal IP and service worked with internal IP as well)</li>
<li>I could solve the issue only with removing and re-creating all services with External IP</li>
</ol>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">"Zero down time" upgrading with no manual step involved</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">I didn't try it again, probably with upgrade and pool remove.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">At my previous upgrade I also experienced issues with the DNS service, that time I simply pressed upgrade on the GC Container Engine UI, but after the upgrade, some of the pods had to be manually re-created otherwise they weren't able to communicate with each other.</p>