<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):<br>
1.3.4</p>
<p dir="auto"><strong>What happened</strong>:<br>
Performing a <code class="notranslate">kubectl get jobs -l &lt;any label selector&gt;</code> returns no jobs despite their existence.</p>
<p dir="auto"><strong>What you expected to happen</strong>:<br>
List of jobs matching the label selector</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):<br>
Create a job with a label. Call <code class="notranslate">kubectl get jobs -l &lt;selector&gt;</code></p>
<p dir="auto"><strong>Anything else do we need to know</strong>:<br>
I am not sure if this is similar to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="138448991" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/22517" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/22517/hovercard" href="https://github.com/kubernetes/kubernetes/issues/22517">#22517</a> or not</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.): no</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.): conntrack</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one): BUG REPORT</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>): tested on the following:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.6&quot;, GitCommit:&quot;ae4550cc9c89a593bcda6678df201db1b208133b&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-08-26T18:13:23Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.5&quot;, GitCommit:&quot;b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;1970-01-01T00:00:00Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6", GitCommit:"ae4550cc9c89a593bcda6678df201db1b208133b", GitTreeState:"clean", BuildDate:"2016-08-26T18:13:23Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.5", GitCommit:"b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5", GitTreeState:"clean", BuildDate:"1970-01-01T00:00:00Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.6&quot;, GitCommit:&quot;ae4550cc9c89a593bcda6678df201db1b208133b&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-08-26T18:13:23Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;3&quot;, GitVersion:&quot;v1.3.0&quot;, GitCommit:&quot;283137936a498aed572ee22af6774b6fb6e9fd94&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2016-07-01T19:19:19Z&quot;, GoVersion:&quot;go1.6.2&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}"><pre class="notranslate"><code class="notranslate">Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6", GitCommit:"ae4550cc9c89a593bcda6678df201db1b208133b", GitTreeState:"clean", BuildDate:"2016-08-26T18:13:23Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"283137936a498aed572ee22af6774b6fb6e9fd94", GitTreeState:"clean", BuildDate:"2016-07-01T19:19:19Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
</code></pre></div>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: tested on minikube and an AWS cluster</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): tested on boot2docker and debian jessie</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): tested on <code class="notranslate">Linux 4.4.15-boot2docker #1 SMP Fri Jul 15 18:11:45 UTC 2016 x86_64 GNU/Linux</code> and <code class="notranslate">Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u3 (2016-07-02) x86_64 GNU/Linux</code></li>
<li><strong>Install tools</strong>: minikube and custom</li>
<li><strong>Others</strong>:</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">When attempting to connect to a service which transiently has no running pods, the connect operation hangs, and continues to hang even once brand new connections are able to connect.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">The connect operation should hang until a pod is running, then succeed</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">create the foo http service with no pods (for now)</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ cat &gt; svc.yaml &lt;&lt;EOF
apiVersion: v1
kind: Service
metadata:
  name: foo
spec:
  ports:
    - port: 80
  selector:
    name: foo
EOF
$ kubectl create -f svc.yaml"><pre class="notranslate"><code class="notranslate">$ cat &gt; svc.yaml &lt;&lt;EOF
apiVersion: v1
kind: Service
metadata:
  name: foo
spec:
  ports:
    - port: 80
  selector:
    name: foo
EOF
$ kubectl create -f svc.yaml
</code></pre></div>
<p dir="auto">start a pod (just to be a shell inside the cluster), and install curl</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ kubectl run -it bar --image=ubuntu"><pre class="notranslate"><code class="notranslate">$ kubectl run -it bar --image=ubuntu
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="bar# apt-get update &amp;&amp; apt-get install curl"><pre class="notranslate"><code class="notranslate">bar# apt-get update &amp;&amp; apt-get install curl
</code></pre></div>
<p dir="auto">attempt to curl the foo service, it should hang trying to connect (this is expected)</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="bar# curl -v foo.default.svc.cluster.local/"><pre class="notranslate"><code class="notranslate">bar# curl -v foo.default.svc.cluster.local/
</code></pre></div>
<p dir="auto">(while the curl is still hanging, before the connect operation times out)<br>
create the foo deployment as a basic nginx pod</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ cat &gt; dep.yaml &lt;&lt;EOF
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: foo
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: foo
    spec:
      containers:
      - name: foo
        image: nginx
        ports:
        - containerPort: 80
EOF
$ kubectl create -f dep.yaml"><pre class="notranslate"><code class="notranslate">$ cat &gt; dep.yaml &lt;&lt;EOF
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: foo
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: foo
    spec:
      containers:
      - name: foo
        image: nginx
        ports:
        - containerPort: 80
EOF
$ kubectl create -f dep.yaml
</code></pre></div>
<p dir="auto">EXPECTED:<br>
Since the service is now available, the connect should stop hanging and instead succeed.</p>
<p dir="auto">ACTUAL:<br>
The connect continues to hang. If you abort and retry with a brand new connection, it succeeds immediately.</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">The above is a minimal case, but I originally hit this issue when testing rolling upgrades of a deployment with 1 replica. Since there's a brief window where there are no running pods for the service, there is a race where a connection can enter this state.</p>
<p dir="auto">I have even observed this happening in cases where only one replica is running at some stage, eg. when a deployment with default strategy has replicas=2. However, this is much harder to trigger, suggesting a much smaller race window.</p>
<p dir="auto">I think I already know what's actually happening - this is the same bug as <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="174796521" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/31983" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/31983/hovercard" href="https://github.com/kubernetes/kubernetes/issues/31983">#31983</a> but applied to TCP, not UDP. See also <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="46911842" data-permission-text="Title is private" data-url="https://github.com/moby/moby/issues/8795" data-hovercard-type="issue" data-hovercard-url="/moby/moby/issues/8795/hovercard" href="https://github.com/moby/moby/issues/8795">moby/moby#8795</a>.</p>
<p dir="auto">When the opening SYN packet for the TCP connection arrives, it gets passed to an empty service routing chain, and subsequently is not rewritten. In my tests, this caused it to go out the default route instead (ie. out of the minikube vm and into the host OS), but that may depend on your exact networking setup. In any case the effect is the same - the SYN packet has nowhere to go, and either gets dropped or goes somewhere nonsensical (that's probably a seperate bug, come to think). The issue comes when we add a pod to that service routing chain, and now any new connections will follow that rule and successfully connect.<br>
However, since our first SYN packet went somewhere else, conntrack ensures that it continues to be sent the same place for the duration of that particular TCP connection (even though the "connection" has been nothing but outgoing SYNs so far). So all subsequent SYN retries also end up dropped or otherwise ignored.</p>
<p dir="auto">I confirmed this was the issue by running <code class="notranslate">conntrack -F</code> on the minikube host - this allowed the hung connection to successfully connect to the newly running pod.</p>
<p dir="auto">I believe the proper fix for this would be something like "when a service goes from having no pods to having at least one, clear all conntrack entries to that service ip in state SYN_SENT".</p>