<p dir="auto"><code class="notranslate">error: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.</code> appears in a lot of flakes. Speculation is that the metadata server is slow to respond.</p>
<p dir="auto">So far, these are the issues where this message has popped up:<br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="164220053" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/28569" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/28569/hovercard" href="https://github.com/kubernetes/kubernetes/issues/28569">#28569</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="164204998" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/28565" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/28565/hovercard" href="https://github.com/kubernetes/kubernetes/issues/28565">#28565</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="163991819" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/28523" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/28523/hovercard" href="https://github.com/kubernetes/kubernetes/issues/28523">#28523</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="163933215" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/28507" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/28507/hovercard" href="https://github.com/kubernetes/kubernetes/issues/28507">#28507</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="163811600" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/28493" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/28493/hovercard" href="https://github.com/kubernetes/kubernetes/issues/28493">#28493</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="163579268" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/28439" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/28439/hovercard" href="https://github.com/kubernetes/kubernetes/issues/28439">#28439</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="163578278" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/28437" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/28437/hovercard" href="https://github.com/kubernetes/kubernetes/issues/28437">#28437</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="163545161" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/28429" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/28429/hovercard" href="https://github.com/kubernetes/kubernetes/issues/28429">#28429</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="163541783" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/28426" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/28426/hovercard" href="https://github.com/kubernetes/kubernetes/issues/28426">#28426</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="163533035" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/28420" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/28420/hovercard" href="https://github.com/kubernetes/kubernetes/issues/28420">#28420</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="163207971" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/28293" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/28293/hovercard" href="https://github.com/kubernetes/kubernetes/issues/28293">#28293</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="162029512" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/27976" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/27976/hovercard" href="https://github.com/kubernetes/kubernetes/issues/27976">#27976</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="161598746" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/27839" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/27839/hovercard" href="https://github.com/kubernetes/kubernetes/issues/27839">#27839</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="161178789" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/27715" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/27715/hovercard" href="https://github.com/kubernetes/kubernetes/issues/27715">#27715</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="159529335" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/27156" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/27156/hovercard" href="https://github.com/kubernetes/kubernetes/issues/27156">#27156</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="158173781" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/26715" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/26715/hovercard" href="https://github.com/kubernetes/kubernetes/issues/26715">#26715</a><br>
<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="157357173" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/26490" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/26490/hovercard" href="https://github.com/kubernetes/kubernetes/issues/26490">#26490</a></p>
<p dir="auto">Sample failed run: <a href="https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10664" rel="nofollow">https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10664</a></p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/cjcullen/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/cjcullen">@cjcullen</a> since this is dealing with auth, I'm initially assigning to you. Feel free to reassign to someone more appropriate.</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto">no</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<p dir="auto">fatal error: concurrent map writes<br>
i found this <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="166696390" data-permission-text="Title is private" data-url="https://github.com/kubernetes/minikube/issues/368" data-hovercard-type="issue" data-hovercard-url="/kubernetes/minikube/issues/368/hovercard" href="https://github.com/kubernetes/minikube/issues/368">kubernetes/minikube#368</a> not the same problem.</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):<br>
BUG</p>

<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto">Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.7", GitCommit:"a2cba278cba1f6881bb0a7704d9cac6fca6ed435", GitTreeState:"clean", BuildDate:"2016-09-12T23:15:30Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}<br>
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.7", GitCommit:"a2cba278cba1f6881bb0a7704d9cac6fca6ed435", GitTreeState:"clean", BuildDate:"2016-09-12T23:08:43Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: virtual machine ,4 cpu cores,8G RAM</li>
<li><strong>OS</strong> (e.g. from /etc/os-release):CentOS Linux release 7.2.1511</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>):Linux k8s-master-1 3.10.0-327.10.1.el7.x86_64 <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="35192559" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/1" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/1/hovercard" href="https://github.com/kubernetes/kubernetes/issues/1">#1</a> SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux</li>
<li><strong>Install tools</strong>:none</li>
<li><strong>Others</strong>:two kubernetes master(both with api,controller,scheduler) node use election feature.</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">panic frequently,sometimes 3 times a day.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">not panic</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<p dir="auto">not known</p>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">here is the dump stack:</p>
<p dir="auto">Nov 23 06:03:13 centos7-image kube-controller-manager: fatal error: concurrent map writes<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 19396 [running]:<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: runtime.throw(0x2c86560, 0x15)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc820cbbab0 sp=0xc820cbba98<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: runtime.mapassign1(0x1e59780, 0xc8201892f0, 0xc820cbbbf8, 0xc820cbbc48)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/runtime/hashmap.go:445 +0xb1 fp=0xc820cbbb58 sp=0xc820cbbab0<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.(*DeltaFIFO).queueActionLocked(0xc8200e0a50, 0x2a67378, 0x4, 0x29fe980, 0xc821752b60, 0x0, 0x0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:314 +0x5d1 fp=0xc820cbbcc0 sp=0xc820cbbb58<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.(*DeltaFIFO).Resync(0xc8200e0a50, 0x0, 0x0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:498 +0x4f7 fp=0xc820cbbe38 sp=0xc820cbbcc0<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch.func1(0xc8201912b0, 0xc82009ad80, 0xc8201503c0, 0xc82124eb40, 0xc8201912d0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:289 +0x252 fp=0xc820cbbf88 sp=0xc820cbbe38<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: runtime.goexit()<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820cbbf90 sp=0xc820cbbf88<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:296 +0xde3<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 1 [chan receive]:<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.WaitFor(0xc8213b7220, 0xc821389780, 0xc821196180, 0x0, 0x0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:209 +0x64<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.pollInternal(0xc8213b7220, 0xc821389780, 0x0, 0x0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:170 +0x8a<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.Poll(0x77359400, 0x2540be400, 0xc821389780, 0x0, 0x0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:164 +0x4f<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/leaderelection.(*LeaderElector).renew.func1()<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:224 +0x7d<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.JitterUntil.func1(0xc821389918)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:86 +0x4b<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.JitterUntil(0xc821389918, 0x0, 0x0, 0x435701, 0xc820500900)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:87 +0xb4<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.Until(0xc821389918, 0x0, 0xc820500900)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:49 +0x43<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/leaderelection.(*LeaderElector).renew(0xc8201b0000)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:235 +0x64<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/leaderelection.(*LeaderElector).Run(0xc8201b0000)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:178 +0x9f<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/leaderelection.RunOrDie(0x2c9c620, 0x17, 0x0, 0x0, 0x2a9cb40, 0xb, 0x0, 0x0, 0x0, 0x0, ...)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:189 +0x98<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/cmd/kube-controller-manager/app.Run(0xc8201758c0, 0x0, 0x0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:194 +0x781<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: main.main()<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/controller-manager.go:53 +0x9b<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 18 [chan receive]:<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x3f051a0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x67<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.1<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x297<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 11 [select, 385 minutes]:<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch.func1(0xc82008a0d8, 0xc82009ad80, 0xc820150d20, 0xc820672540, 0xc82008a0e0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:283 +0x3c8<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:296 +0xde3<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 25 [select]:<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.JitterUntil(0x317bb30, 0x12a05f200, 0x0, 0x434301, 0xc82009ad80)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:101 +0x191<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.Until(0x317bb30, 0x12a05f200, 0xc82009ad80)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:49 +0x43<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/util.InitLogs<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/logs.go:50 +0xba<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 26 [IO wait, 275 minutes]:<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: net.runtime_pollWait(0x7fc2c66cc4f0, 0x72, 0x0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/runtime/netpoll.go:160 +0x60<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: net.(*pollDesc).Wait(0xc8202fc450, 0x72, 0x0, 0x0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: net.(*pollDesc).WaitRead(0xc8202fc450, 0x0, 0x0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: net.(*netFD).accept(0xc8202fc3f0, 0x0, 0x7fc2c64ed078, 0xc820e74700)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/fd_unix.go:426 +0x27c<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: net.(*TCPListener).AcceptTCP(0xc82008a058, 0x453cb0, 0x0, 0x0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: net/http.tcpKeepAliveListener.Accept(0xc82008a058, 0x0, 0x0, 0x0, 0x0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/http/server.go:2427 +0x41<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: net/http.(*Server).Serve(0xc820194280, 0x7fc2c64ed040, 0xc82008a058, 0x0, 0x0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/http/server.go:2117 +0x129<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: net/http.(*Server).ListenAndServe(0xc820194280, 0x0, 0x0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/http/server.go:2098 +0x136<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/cmd/kube-controller-manager/app.Run.func1(0xc8201758c0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:153 +0x30b<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/cmd/kube-controller-manager/app.Run<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:154 +0x333<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 27 [chan receive, 385 minutes]:<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/watch.(*Broadcaster).loop(0xc8202b5240)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/watch/mux.go:201 +0x58<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/watch.NewBroadcaster<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/watch/mux.go:75 +0x129<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 28 [chan receive, 385 minutes]:<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x7fc2c6697df0, 0xc8201734a0, 0xc8202f3b40)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/record/event.go:227 +0x97<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/record.(*eventBroadcasterImpl).StartEventWatcher<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/record/event.go:239 +0x70<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 29 [chan receive, 385 minutes]:<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x7fc2c6697df0, 0xc8201735f0, 0xc8201735c0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/record/event.go:227 +0x97<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/record.(*eventBroadcasterImpl).StartEventWatcher<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/record/event.go:239 +0x70<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 40 [semacquire]:<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: sync.runtime_Semacquire(0xc8200e0a58)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/runtime/sema.go:47 +0x26<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: sync.(*RWMutex).Lock(0xc8200e0a50)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/sync/rwmutex.go:88 +0xa1<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: sync.(*Cond).Wait(0xc8200e0a68)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/sync/cond.go:64 +0xc1<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.(*DeltaFIFO).Pop(0xc8200e0a50, 0xc820189350, 0x0, 0x0, 0x0, 0x0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:399 +0xc4<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/controller/framework.(*Controller).processLoop(0xc8202dbc00)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:127 +0x4a<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/controller/framework.(*Controller).(k8s.io/kubernetes/pkg/controller/framework.processLoop)-fm()<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:103 +0x20<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.JitterUntil.func1(0xc8202cbf88)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:86 +0x4b<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.JitterUntil(0xc8202cbf88, 0x3b9aca00, 0x0, 0x1e5f501, 0xc82009ad80)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:87 +0xb4<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.Until(0xc8202cbf88, 0x3b9aca00, 0xc82009ad80)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:49 +0x43<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/controller/framework.(*Controller).Run(0xc8202dbc00, 0xc82009ad80)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:103 +0x1c2<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/controller/endpoint.(*EndpointController).Run<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/endpoint/endpoints_controller.go:157 +0x79<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 65 [select (no cases), 385 minutes]:<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/cmd/kube-controller-manager/app.StartControllers(0xc8201758c0, 0xc8202b5200, 0xc82006a9c0, 0xc8205008a0, 0x0, 0x0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:484 +0x62e2<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/cmd/kube-controller-manager/app.Run.func2(0xc8205008a0)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:157 +0x4b<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/leaderelection.(*LeaderElector).Run<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:177 +0x91<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 6692 [select, 269 minutes]:<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch.func1(0xc820026ef0, 0xc82009ad80, 0xc82019f680, 0xc820996f00, 0xc820026ef8)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:283 +0x3c8<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:296 +0xde3<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 7159 [select, 259 minutes]:<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch.func1(0xc820812400, 0xc82009ad80, 0xc8202ee640, 0xc821046660, 0xc820812410)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:283 +0x3c8<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:296 +0xde3<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 3403 [select]:<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch.func1(0xc820812430, 0xc82009ad80, 0xc82019ec80, 0xc820e6ab40, 0xc820812460)<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:283 +0x3c8<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:296 +0xde3<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 68 [select]:<br>
Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.JitterUntil(0xc820188150, 0x12a05f200, 0x0, 0x7fc2c64eb601, 0xc82009ad80)</p>