<p dir="auto">From the logs, it looks like the pod was determined to be <code class="notranslate">terminated</code> before it was started running. There are actually 2 issues here:</p>
<ol dir="auto">
<li>The pod was killed on the before it was started</li>
<li>After killing the pod, kubelet continued to download the image and attempted to bring up the container.</li>
</ol>
<p dir="auto">Relevant log snippets from <a href="https://pantheon.corp.google.com/m/cloudstorage/b/kubernetes-jenkins/o/pr-logs/3c72506f10986713974d403387a51ccf914e8dd3/kubernetes-pull-build-test-e2e-gce/22725/artifacts/104.197.180.217%3A22-kubelet.log" rel="nofollow">kubelet.log</a>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="I0107 20:55:17.628805    3397 manager.go:366] Status for pod &quot;nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed(f3de47bb-b580-11e5-93b1-42010af00002)&quot; updated successfully: {status:{Phase:Pending Conditions:[] Message: Reason: HostIP: PodIP: StartTime:0xc208bb1c60 ContainerStatuses:[]} version:1 podName:nginx-deployment-7txis-ge6lu podNamespace:e2e-tests-deployment-jyfed}
I0107 20:55:17.629165    3397 config.go:252] Setting pods for source api
I0107 20:55:17.738594    3397 config.go:252] Setting pods for source api
I0107 20:55:17.776348    3397 manager.go:377] Pod &quot;nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed(f3de47bb-b580-11e5-93b1-42010af00002)&quot; fully terminated and removed from etcd
I0107 20:55:17.791774    3397 server.go:563] Event(api.ObjectReference{Kind:&quot;Pod&quot;, Namespace:&quot;e2e-tests-deployment-mb56s&quot;, Name:&quot;nginx-controller-yee2t&quot;, UID:&quot;ea4a9603-b580-11e5-93b1-42010af00002&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;1019&quot;, FieldPath:&quot;spec.containers{nginx}&quot;}): type: 'Normal' reason: 'Started' Started container with docker id 327d0a201fe8
I0107 20:55:17.792989    3397 kubelet.go:2338] SyncLoop (UPDATE, &quot;api&quot;): &quot;nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed(f3de47bb-b580-11e5-93b1-42010af00002)&quot;
I0107 20:55:17.793594    3397 config.go:252] Setting pods for source api
I0107 20:55:17.827530    3397 kubelet.go:2341] SyncLoop (REMOVE, &quot;api&quot;): &quot;nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed(f3de47bb-b580-11e5-93b1-42010af00002)&quot;
...
I0107 20:55:18.083187    3397 manager.go:802] Added container: &quot;/55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2&quot; (aliases: [k8s_POD.6059dfa2_nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed_f3de47bb-b580-11e5-93b1-42010af00002_ba926a42 55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2], namespace: &quot;docker&quot;)
...
I0107 20:55:18.226164    3397 manager.go:857] Destroyed container: &quot;/55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2&quot; (aliases: [k8s_POD.6059dfa2_nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed_f3de47bb-b580-11e5-93b1-42010af00002_ba926a42 55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2], namespace: &quot;docker&quot;)
...
I0107 20:55:18.794119    3397 volumes.go:214] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-wjp6m of pod f3de47bb-b580-11e5-93b1-42010af00002
I0107 20:55:18.794143    3397 volumes.go:250] Used volume plugin &quot;kubernetes.io/secret&quot; for f3de47bb-b580-11e5-93b1-42010af00002/kubernetes.io~secret
I0107 20:55:18.794170    3397 kubelet.go:1909] volume &quot;f3c0705a-b580-11e5-93b1-42010af00002/default-token-etatl&quot;, still has a container running &quot;f3c0705a-b580-11e5-93b1-42010af00002&quot;, skipping teardown
W0107 20:55:18.794190    3397 kubelet.go:1914] Orphaned volume &quot;f3de47bb-b580-11e5-93b1-42010af00002/default-token-wjp6m&quot; found, tearing down volume
I0107 20:55:18.794222    3397 secret.go:202] Tearing down volume default-token-wjp6m for pod f3de47bb-b580-11e5-93b1-42010af00002 at /var/lib/kubelet/pods/f3de47bb-b580-11e5-93b1-42010af00002/volumes/kubernetes.io~secret/default-token-wjp6m
I0107 20:55:18.816071    3397 kubelet.go:1844] Orphaned pod &quot;f3de47bb-b580-11e5-93b1-42010af00002&quot; found, removing
...
E0107 20:55:19.907248    3397 manager.go:1889] Error running pod &quot;nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed(f3de47bb-b580-11e5-93b1-42010af00002)&quot; container &quot;nginx&quot;: impossible: cannot find the mounted volumes for pod &quot;nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed(f3de47bb-b580-11e5-93b1-42010af00002)&quot;
I0107 20:55:19.907312    3397 kubelet.go:3112] Generating status for &quot;nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed(f3de47bb-b580-11e5-93b1-42010af00002)&quot;
I0107 20:55:19.907687    3397 server.go:563] Event(api.ObjectReference{Kind:&quot;Pod&quot;, Namespace:&quot;e2e-tests-deployment-jyfed&quot;, Name:&quot;nginx-deployment-7txis-ge6lu&quot;, UID:&quot;f3de47bb-b580-11e5-93b1-42010af00002&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;2035&quot;, FieldPath:&quot;spec.containers{nginx}&quot;}): type: 'Normal' reason: 'Pulled' Successfully pulled image &quot;nginx&quot;
I0107 20:55:19.934736    3397 manager.go:339] Container inspect result: {ID:55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2 Created:2016-01-07 20:55:16.836072085 +0000 UTC Path:/pause Args:[] Config:0xc209357040 State:{Running:false Paused:false Restarting:false OOMKilled:false Pid:0 ExitCode:0 Error: StartedAt:2016-01-07 20:55:17.82618655 +0000 UTC FinishedAt:2016-01-07 20:55:18.176860955 +0000 UTC} Image:8950680a606cf7a0b7916dbf4a435b35d28d75c705999847eddb5ed38eb53204 Node:&lt;nil&gt; NetworkSettings:0xc209604b00 SysInitPath: ResolvConfPath:/var/lib/docker/containers/55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2/resolv.conf HostnamePath:/var/lib/docker/containers/55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2/hostname HostsPath:/var/lib/docker/containers/55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2/hosts LogPath:/var/lib/docker/containers/55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2/55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2-json.log Name:/k8s_POD.6059dfa2_nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed_f3de47bb-b580-11e5-93b1-42010af00002_ba926a42 Driver:aufs Mounts:[] Volumes:map[] VolumesRW:map[] HostConfig:0xc2082a8c80 ExecIDs:[] RestartCount:0 AppArmorProfile:}
E0107 20:55:19.934991    3397 pod_workers.go:125] Error syncing pod f3de47bb-b580-11e5-93b1-42010af00002, skipping: not all containers have started: 0 != 1
I0107 20:55:19.935295    3397 server.go:563] Event(api.ObjectReference{Kind:&quot;Pod&quot;, Namespace:&quot;e2e-tests-deployment-jyfed&quot;, Name:&quot;nginx-deployment-7txis-ge6lu&quot;, UID:&quot;f3de47bb-b580-11e5-93b1-42010af00002&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;2035&quot;, FieldPath:&quot;&quot;}): type: 'Warning' reason: 'FailedSync' Error syncing pod, skipping: not all containers have started: 0 != 1
I0107 20:55:20.023722    3397 manager.go:346] Pod &quot;nginx-deployment-7txis-ge6lu&quot; (f3de47bb-b580-11e5-93b1-42010af00002) does not exist on the server"><pre class="notranslate"><code class="notranslate">I0107 20:55:17.628805    3397 manager.go:366] Status for pod "nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed(f3de47bb-b580-11e5-93b1-42010af00002)" updated successfully: {status:{Phase:Pending Conditions:[] Message: Reason: HostIP: PodIP: StartTime:0xc208bb1c60 ContainerStatuses:[]} version:1 podName:nginx-deployment-7txis-ge6lu podNamespace:e2e-tests-deployment-jyfed}
I0107 20:55:17.629165    3397 config.go:252] Setting pods for source api
I0107 20:55:17.738594    3397 config.go:252] Setting pods for source api
I0107 20:55:17.776348    3397 manager.go:377] Pod "nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed(f3de47bb-b580-11e5-93b1-42010af00002)" fully terminated and removed from etcd
I0107 20:55:17.791774    3397 server.go:563] Event(api.ObjectReference{Kind:"Pod", Namespace:"e2e-tests-deployment-mb56s", Name:"nginx-controller-yee2t", UID:"ea4a9603-b580-11e5-93b1-42010af00002", APIVersion:"v1", ResourceVersion:"1019", FieldPath:"spec.containers{nginx}"}): type: 'Normal' reason: 'Started' Started container with docker id 327d0a201fe8
I0107 20:55:17.792989    3397 kubelet.go:2338] SyncLoop (UPDATE, "api"): "nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed(f3de47bb-b580-11e5-93b1-42010af00002)"
I0107 20:55:17.793594    3397 config.go:252] Setting pods for source api
I0107 20:55:17.827530    3397 kubelet.go:2341] SyncLoop (REMOVE, "api"): "nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed(f3de47bb-b580-11e5-93b1-42010af00002)"
...
I0107 20:55:18.083187    3397 manager.go:802] Added container: "/55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2" (aliases: [k8s_POD.6059dfa2_nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed_f3de47bb-b580-11e5-93b1-42010af00002_ba926a42 55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2], namespace: "docker")
...
I0107 20:55:18.226164    3397 manager.go:857] Destroyed container: "/55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2" (aliases: [k8s_POD.6059dfa2_nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed_f3de47bb-b580-11e5-93b1-42010af00002_ba926a42 55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2], namespace: "docker")
...
I0107 20:55:18.794119    3397 volumes.go:214] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-wjp6m of pod f3de47bb-b580-11e5-93b1-42010af00002
I0107 20:55:18.794143    3397 volumes.go:250] Used volume plugin "kubernetes.io/secret" for f3de47bb-b580-11e5-93b1-42010af00002/kubernetes.io~secret
I0107 20:55:18.794170    3397 kubelet.go:1909] volume "f3c0705a-b580-11e5-93b1-42010af00002/default-token-etatl", still has a container running "f3c0705a-b580-11e5-93b1-42010af00002", skipping teardown
W0107 20:55:18.794190    3397 kubelet.go:1914] Orphaned volume "f3de47bb-b580-11e5-93b1-42010af00002/default-token-wjp6m" found, tearing down volume
I0107 20:55:18.794222    3397 secret.go:202] Tearing down volume default-token-wjp6m for pod f3de47bb-b580-11e5-93b1-42010af00002 at /var/lib/kubelet/pods/f3de47bb-b580-11e5-93b1-42010af00002/volumes/kubernetes.io~secret/default-token-wjp6m
I0107 20:55:18.816071    3397 kubelet.go:1844] Orphaned pod "f3de47bb-b580-11e5-93b1-42010af00002" found, removing
...
E0107 20:55:19.907248    3397 manager.go:1889] Error running pod "nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed(f3de47bb-b580-11e5-93b1-42010af00002)" container "nginx": impossible: cannot find the mounted volumes for pod "nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed(f3de47bb-b580-11e5-93b1-42010af00002)"
I0107 20:55:19.907312    3397 kubelet.go:3112] Generating status for "nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed(f3de47bb-b580-11e5-93b1-42010af00002)"
I0107 20:55:19.907687    3397 server.go:563] Event(api.ObjectReference{Kind:"Pod", Namespace:"e2e-tests-deployment-jyfed", Name:"nginx-deployment-7txis-ge6lu", UID:"f3de47bb-b580-11e5-93b1-42010af00002", APIVersion:"v1", ResourceVersion:"2035", FieldPath:"spec.containers{nginx}"}): type: 'Normal' reason: 'Pulled' Successfully pulled image "nginx"
I0107 20:55:19.934736    3397 manager.go:339] Container inspect result: {ID:55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2 Created:2016-01-07 20:55:16.836072085 +0000 UTC Path:/pause Args:[] Config:0xc209357040 State:{Running:false Paused:false Restarting:false OOMKilled:false Pid:0 ExitCode:0 Error: StartedAt:2016-01-07 20:55:17.82618655 +0000 UTC FinishedAt:2016-01-07 20:55:18.176860955 +0000 UTC} Image:8950680a606cf7a0b7916dbf4a435b35d28d75c705999847eddb5ed38eb53204 Node:&lt;nil&gt; NetworkSettings:0xc209604b00 SysInitPath: ResolvConfPath:/var/lib/docker/containers/55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2/resolv.conf HostnamePath:/var/lib/docker/containers/55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2/hostname HostsPath:/var/lib/docker/containers/55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2/hosts LogPath:/var/lib/docker/containers/55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2/55b4df411b855303ec4383bb23621a64e5ae0acfb8475e18c91ead73069b3af2-json.log Name:/k8s_POD.6059dfa2_nginx-deployment-7txis-ge6lu_e2e-tests-deployment-jyfed_f3de47bb-b580-11e5-93b1-42010af00002_ba926a42 Driver:aufs Mounts:[] Volumes:map[] VolumesRW:map[] HostConfig:0xc2082a8c80 ExecIDs:[] RestartCount:0 AppArmorProfile:}
E0107 20:55:19.934991    3397 pod_workers.go:125] Error syncing pod f3de47bb-b580-11e5-93b1-42010af00002, skipping: not all containers have started: 0 != 1
I0107 20:55:19.935295    3397 server.go:563] Event(api.ObjectReference{Kind:"Pod", Namespace:"e2e-tests-deployment-jyfed", Name:"nginx-deployment-7txis-ge6lu", UID:"f3de47bb-b580-11e5-93b1-42010af00002", APIVersion:"v1", ResourceVersion:"2035", FieldPath:""}): type: 'Warning' reason: 'FailedSync' Error syncing pod, skipping: not all containers have started: 0 != 1
I0107 20:55:20.023722    3397 manager.go:346] Pod "nginx-deployment-7txis-ge6lu" (f3de47bb-b580-11e5-93b1-42010af00002) does not exist on the server
</code></pre></div>
<p dir="auto">If the timestamps are accurately synced, it looks like a DELETE request went through the API server before the kubelet even started to sync the pod. From <a href="https://pantheon.corp.google.com/m/cloudstorage/b/kubernetes-jenkins/o/pr-logs/3c72506f10986713974d403387a51ccf914e8dd3/kubernetes-pull-build-test-e2e-gce/22725/artifacts/23.236.58.56%3A22-kube-apiserver.log" rel="nofollow">kube-apiserver.log</a>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="I0107 20:55:15.764214       6 handlers.go:130] DELETE /api/v1/namespaces/e2e-tests-deployment-jyfed/pods/nginx-deployment-7txis-ge6lu: (39.143436ms) 200 [[kube-controller-manager/v1.2.0 (linux/amd64) kubernetes/762c4d1/replication-controller] 127.0.0.1:53073]"><pre class="notranslate"><code class="notranslate">I0107 20:55:15.764214       6 handlers.go:130] DELETE /api/v1/namespaces/e2e-tests-deployment-jyfed/pods/nginx-deployment-7txis-ge6lu: (39.143436ms) 200 [[kube-controller-manager/v1.2.0 (linux/amd64) kubernetes/762c4d1/replication-controller] 127.0.0.1:53073]
</code></pre></div>
<p dir="auto">Jenkins report: <a href="http://kubekins.dls.corp.google.com:8081/job/kubernetes-pull-build-test-e2e-gce/22725/" rel="nofollow">http://kubekins.dls.corp.google.com:8081/job/kubernetes-pull-build-test-e2e-gce/22725/</a><br>
Logs gist (incase jenkins results are removed): <a href="https://gist.github.com/timstclair/7a8b573c7d82c092a775">https://gist.github.com/timstclair/7a8b573c7d82c092a775</a></p>
<p dir="auto">/cc @kubernetes/goog-node @kubernetes/goog-control-plane</p>
<p dir="auto">Example failure:<br>
<a href="http://kubekins.dls.corp.google.com:8081/job/kubernetes-pull-build-test-e2e-gce/21458/" rel="nofollow">http://kubekins.dls.corp.google.com:8081/job/kubernetes-pull-build-test-e2e-gce/21458/</a></p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/nikhiljindal/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/nikhiljindal">@nikhiljindal</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/janetkuo/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/janetkuo">@janetkuo</a></p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/nikhiljindal/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/nikhiljindal">@nikhiljindal</a> - can you please take a look?</p>