<p dir="auto">The first failure happened at ~9:20 PST 29/04/2015. The failure looks like that:</p>
<p dir="auto">/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/density.go:158<br>
Expected error:<br>
&lt;*errors.errorString | 0xc20b84daa0&gt;: {<br>
s: "Controller my-hostname-density3000-75cb549b-ee8e-11e4-ae78-42010af01555: Only found 3052 replicas out of 3000",<br>
}<br>
Controller my-hostname-density3000-75cb549b-ee8e-11e4-ae78-42010af01555: Only found 3052 replicas out of 3000<br>
not to have occurred</p>
<p dir="auto">It seems that because of some reason we are starting much more pods within a replication controller than expected. Another related failure:</p>
<p dir="auto">Expected error:<br>
&lt;*errors.errorString | 0xc2083b3050&gt;: {<br>
s: "Number of reported pods changed: 3222 vs 3000",<br>
}<br>
Number of reported pods changed: 3222 vs 3000<br>
not to have occurred</p>
<p dir="auto">cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/quinton-hoole/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/quinton-hoole">@quinton-hoole</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/lavalamp/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/lavalamp">@lavalamp</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/bprashanth/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/bprashanth">@bprashanth</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/fgrzadkowski/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/fgrzadkowski">@fgrzadkowski</a></p>
<p dir="auto">I haven't looked into the details yet, but it seems that a PR went in at around 16:30 PDT today (Wed 04/29) that is causing significantly more e2e failures than usual in our continuous integration.</p>
<p dir="auto">I'll dig into it in the morning unless one of the oncalls gets there before me.</p>