<p dir="auto"><a href="http://pr-test.k8s.io/22807/kubernetes-pull-build-test-e2e-gce/32271/build-log.txt" rel="nofollow">http://pr-test.k8s.io/22807/kubernetes-pull-build-test-e2e-gce/32271/build-log.txt</a></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="STEP: Destroying namespace &quot;e2e-tests-services-1ymgz&quot; for this suite.
STEP: Destroying namespace &quot;e2e-tests-services-kdrdp&quot; for this suite.
Mar 10 11:15:17.444: INFO: Pod e2e-tests-services-kdrdp mutability-test-r7zn3 on node e2e-gce-builder-3-3-minion-gt43 remains, has deletion timestamp &lt;nil&gt;
Mar 10 11:15:17.444: FAIL: Couldn't delete ns &quot;e2e-tests-services-kdrdp&quot;: pods remained within namespace e2e-tests-services-kdrdp after deletion: [mutability-test-r7zn3]


â€¢ Failure in Spec Teardown (AfterEach) [417.838 seconds]
Services
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:902
  should be able to change the type and ports of a service [AfterEach]
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:716

  Mar 10 11:15:17.444: Couldn't delete ns &quot;e2e-tests-services-kdrdp&quot;: pods remained within namespace e2e-tests-services-kdrdp after deletion: [mutability-test-r7zn3]

  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework.go:177"><pre class="notranslate"><code class="notranslate">STEP: Destroying namespace "e2e-tests-services-1ymgz" for this suite.
STEP: Destroying namespace "e2e-tests-services-kdrdp" for this suite.
Mar 10 11:15:17.444: INFO: Pod e2e-tests-services-kdrdp mutability-test-r7zn3 on node e2e-gce-builder-3-3-minion-gt43 remains, has deletion timestamp &lt;nil&gt;
Mar 10 11:15:17.444: FAIL: Couldn't delete ns "e2e-tests-services-kdrdp": pods remained within namespace e2e-tests-services-kdrdp after deletion: [mutability-test-r7zn3]


â€¢ Failure in Spec Teardown (AfterEach) [417.838 seconds]
Services
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:902
  should be able to change the type and ports of a service [AfterEach]
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/service.go:716

  Mar 10 11:15:17.444: Couldn't delete ns "e2e-tests-services-kdrdp": pods remained within namespace e2e-tests-services-kdrdp after deletion: [mutability-test-r7zn3]

  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework.go:177
</code></pre></div>
<p dir="auto">It seems there is some bug around namespace controller. What happened is:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="08:05:39 STEP: Collecting events from namespace &quot;e2e-tests-density-b5lqr&quot;.
08:05:39 STEP: Destroying namespace &quot;e2e-tests-density-b5lqr&quot; for this suite.
09:05:40 Feb 25 09:05:40.322: FAIL: Couldn't delete ns &quot;e2e-tests-density-b5lqr&quot;: namespace e2e-tests-density-b5lqr was not deleted within limit: timed out waiting for the condition, pods remaining: []"><pre class="notranslate"><code class="notranslate">08:05:39 STEP: Collecting events from namespace "e2e-tests-density-b5lqr".
08:05:39 STEP: Destroying namespace "e2e-tests-density-b5lqr" for this suite.
09:05:40 Feb 25 09:05:40.322: FAIL: Couldn't delete ns "e2e-tests-density-b5lqr": namespace e2e-tests-density-b5lqr was not deleted within limit: timed out waiting for the condition, pods remaining: []
</code></pre></div>
<p dir="auto">So basically, namespace wasn't deleted for the whole hour.<br>
Then the next test actually timed out</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="09:05:40 Feb 25 09:05:40.997: INFO: &gt;&gt;&gt; testContext.KubeConfig: /jenkins-master-data/jobs/kubernetes-kubemark-gce/workspace/kubernetes/test/kubemark/kubeconfig.loc
09:05:40 
09:05:41 Feb 25 09:05:41.182: INFO: Waiting for terminating namespaces to be deleted...
09:46:36 
Build timed out (after 120 minutes). Marking the build as failed."><pre class="notranslate"><code class="notranslate">09:05:40 Feb 25 09:05:40.997: INFO: &gt;&gt;&gt; testContext.KubeConfig: /jenkins-master-data/jobs/kubernetes-kubemark-gce/workspace/kubernetes/test/kubemark/kubeconfig.loc
09:05:40 
09:05:41 Feb 25 09:05:41.182: INFO: Waiting for terminating namespaces to be deleted...
09:46:36 
Build timed out (after 120 minutes). Marking the build as failed.
</code></pre></div>
<p dir="auto">So in total, namespace wasn't deleted for more than 90 minutes (note that usually it takes ~2-3 minutes for density test).</p>
<p dir="auto"><a href="https://storage.googleapis.com/kubernetes-jenkins/logs/kubernetes-kubemark-gce/5970/build-log.txt" rel="nofollow">https://storage.googleapis.com/kubernetes-jenkins/logs/kubernetes-kubemark-gce/5970/build-log.txt</a></p>
<p dir="auto">The problem is that we can't really debug it, since we don't have logs from k8s components if build times out. - <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ixdy/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ixdy">@ixdy</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/spxtr/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/spxtr">@spxtr</a> @kubernetes/goog-testing</p>