<p dir="auto">I created an e2e cluster with <code class="notranslate">REGISTER_MASTER_KUBELET=true</code>, and the master node is extremely unstable with docker/kubelet being restarted frequently.<br>
I ended up <em>fixing</em> this by removing <code class="notranslate">/etc/supervisor/conf.d/docker.conf</code> to prevent docker from being restarted again. The cluster was stable and functioning after the fix.<br>
Here are my questions:</p>
<ul dir="auto">
<li>Is REGISTER_MASTER_KUBELET=true supposed to work in this case, or was there any other configurations that I needed to change?</li>
<li>Do we run any e2e suite with master node registered?</li>
</ul>
<p dir="auto">The docker health check performed by supervisord seems defective on the master node in this case, but I am not sure why.</p>
<p dir="auto">/cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/roberthbailey/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/roberthbailey">@roberthbailey</a></p>
<p dir="auto">Running kube-up on a clean build of <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/kubernetes/kubernetes/commit/6afbaf6bf6ac8d1d71307819f21d56d7946c522c/hovercard" href="https://github.com/kubernetes/kubernetes/commit/6afbaf6bf6ac8d1d71307819f21d56d7946c522c"><tt>6afbaf6</tt></a>, I'm seeing the docker on the master crashing, and kube-apiserver crash with it.</p>
<p dir="auto">This is the standard 4-node GCE cluster, I haven't touched any config.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubernetes·(6afbaf6...) $ make clean release-skip-tests &amp;&amp; kup &amp;&amp; kdown
[...]
Waiting for cluster initialization.

  This will continually check to see if the API for kubernetes is reachable.
  This might loop forever if there was some uncaught error during start
  up.

................................Kubernetes cluster created.
Wrote config for devbox-964_kubernetes to $HOME/.kube/config

Kubernetes cluster is running.  The master is running at:

  https://IP

The user name and password to use is located in $HOME/.kube/config.

... calling validate-cluster
Waiting for 5 ready nodes. 1 ready nodes, 5 registered. Retrying.
Waiting for 5 ready nodes. 1 ready nodes, 5 registered. Retrying.
Waiting for 5 ready nodes. 1 ready nodes, 5 registered. Retrying.
error: couldn't read version from server: Get https://IP/api: dial tcp IP:443: connection refused"><pre class="notranslate"><code class="notranslate">kubernetes·(6afbaf6...) $ make clean release-skip-tests &amp;&amp; kup &amp;&amp; kdown
[...]
Waiting for cluster initialization.

  This will continually check to see if the API for kubernetes is reachable.
  This might loop forever if there was some uncaught error during start
  up.

................................Kubernetes cluster created.
Wrote config for devbox-964_kubernetes to $HOME/.kube/config

Kubernetes cluster is running.  The master is running at:

  https://IP

The user name and password to use is located in $HOME/.kube/config.

... calling validate-cluster
Waiting for 5 ready nodes. 1 ready nodes, 5 registered. Retrying.
Waiting for 5 ready nodes. 1 ready nodes, 5 registered. Retrying.
Waiting for 5 ready nodes. 1 ready nodes, 5 registered. Retrying.
error: couldn't read version from server: Get https://IP/api: dial tcp IP:443: connection refused
</code></pre></div>
<p dir="auto">List of all pods:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubernetes·(6afbaf6...) $ k get pods --all-namespaces
NAMESPACE     NAME                                           READY     STATUS                                                                  RESTARTS   AGE
kube-system   etcd-server-kubernetes-master                  1/1       Running                                                                 3          16m
kube-system   fluentd-cloud-logging-kubernetes-master        1/1       Running                                                                 3          16m
kube-system   fluentd-cloud-logging-kubernetes-minion-5h5s   1/1       Running                                                                 0          15m
kube-system   fluentd-cloud-logging-kubernetes-minion-e3p1   1/1       Running                                                                 0          15m
kube-system   fluentd-cloud-logging-kubernetes-minion-k0jp   1/1       Running                                                                 0          15m
kube-system   fluentd-cloud-logging-kubernetes-minion-ykur   1/1       Running                                                                 0          15m
kube-system   kube-apiserver-kubernetes-master               1/1       Running                                                                 3          16m
kube-system   kube-controller-manager-kubernetes-master      1/1       Running                                                                 3          16m
kube-system   kube-dns-v8-v7svt                              4/4       Running                                                                 0          16m
kube-system   kube-scheduler-kubernetes-master               1/1       Running                                                                 3          16m
kube-system   kube-ui-v1-65m9j                               0/1       Image: gcr.io/google_containers/kube-ui:v1.1 is not ready on the node   0          16m
kube-system   monitoring-heapster-v7-3xn8i                   0/1       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-heapster-v7-6h6d2                   0/1       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-heapster-v7-79ozz                   0/1       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-heapster-v7-8jffe                   0/1       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-heapster-v7-95i3o                   0/1       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-heapster-v7-aur4k                   0/1       Running                                                                 9          15m
kube-system   monitoring-heapster-v7-e7wwf                   0/1       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-heapster-v7-g1px0                   0/1       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-heapster-v7-gbyfh                   0/1       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-heapster-v7-mppao                   0/1       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-influx-grafana-v1-421pp             0/2       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-influx-grafana-v1-qhvpi             2/2       Running                                                                 0          15m"><pre class="notranslate"><code class="notranslate">kubernetes·(6afbaf6...) $ k get pods --all-namespaces
NAMESPACE     NAME                                           READY     STATUS                                                                  RESTARTS   AGE
kube-system   etcd-server-kubernetes-master                  1/1       Running                                                                 3          16m
kube-system   fluentd-cloud-logging-kubernetes-master        1/1       Running                                                                 3          16m
kube-system   fluentd-cloud-logging-kubernetes-minion-5h5s   1/1       Running                                                                 0          15m
kube-system   fluentd-cloud-logging-kubernetes-minion-e3p1   1/1       Running                                                                 0          15m
kube-system   fluentd-cloud-logging-kubernetes-minion-k0jp   1/1       Running                                                                 0          15m
kube-system   fluentd-cloud-logging-kubernetes-minion-ykur   1/1       Running                                                                 0          15m
kube-system   kube-apiserver-kubernetes-master               1/1       Running                                                                 3          16m
kube-system   kube-controller-manager-kubernetes-master      1/1       Running                                                                 3          16m
kube-system   kube-dns-v8-v7svt                              4/4       Running                                                                 0          16m
kube-system   kube-scheduler-kubernetes-master               1/1       Running                                                                 3          16m
kube-system   kube-ui-v1-65m9j                               0/1       Image: gcr.io/google_containers/kube-ui:v1.1 is not ready on the node   0          16m
kube-system   monitoring-heapster-v7-3xn8i                   0/1       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-heapster-v7-6h6d2                   0/1       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-heapster-v7-79ozz                   0/1       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-heapster-v7-8jffe                   0/1       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-heapster-v7-95i3o                   0/1       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-heapster-v7-aur4k                   0/1       Running                                                                 9          15m
kube-system   monitoring-heapster-v7-e7wwf                   0/1       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-heapster-v7-g1px0                   0/1       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-heapster-v7-gbyfh                   0/1       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-heapster-v7-mppao                   0/1       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-influx-grafana-v1-421pp             0/2       InsufficientFreeCPU                                                     0          16m
kube-system   monitoring-influx-grafana-v1-qhvpi             2/2       Running                                                                 0          15m
</code></pre></div>
<p dir="auto">Not sure why so many heapster pods are scheduled.</p>
<p dir="auto">Watching the status of the apiserver and docker on the master:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="uluyol@kubernetes-master:~$ while true; do pgrep kube-apiserver &amp;&gt;/dev/null; kube_up=$?; [[ $(pgrep docker | wc -l) == 2 ]]; docker_up=$?; if [[ $kube_up == 0 ]]; then echo -n apiserver up; else echo -n apiserver down; fi; if [[ $docker_up == 0 ]]; then echo ' docker up'; else echo ' docker down'; fi; sleep 1; done
apiserver down docker down
apiserver down docker down
apiserver down docker down
apiserver down docker down
apiserver down docker down
apiserver down docker down
apiserver down docker down
apiserver down docker up
apiserver down docker down
apiserver up docker up
apiserver up docker up
[ +35 more ]
apiserver up docker up
apiserver down docker down
apiserver down docker down
apiserver down docker down
[...]"><pre class="notranslate"><code class="notranslate">uluyol@kubernetes-master:~$ while true; do pgrep kube-apiserver &amp;&gt;/dev/null; kube_up=$?; [[ $(pgrep docker | wc -l) == 2 ]]; docker_up=$?; if [[ $kube_up == 0 ]]; then echo -n apiserver up; else echo -n apiserver down; fi; if [[ $docker_up == 0 ]]; then echo ' docker up'; else echo ' docker down'; fi; sleep 1; done
apiserver down docker down
apiserver down docker down
apiserver down docker down
apiserver down docker down
apiserver down docker down
apiserver down docker down
apiserver down docker down
apiserver down docker up
apiserver down docker down
apiserver up docker up
apiserver up docker up
[ +35 more ]
apiserver up docker up
apiserver down docker down
apiserver down docker down
apiserver down docker down
[...]
</code></pre></div>
<p dir="auto">After digging around with <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/mikedanese/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/mikedanese">@mikedanese</a> on Friday, looks like removing the <code class="notranslate">kubelet_api_servers</code> definition in /etc/salt/minion.d/grains.conf fixes this (I followed this by calling <code class="notranslate">salt-call state.highstate</code>). I have 25 min of apiserver/docker uptime so far, and the scheduled pods looks more like what it's supposed to</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubernetes·(6afbaf6...) $ k get pods --all-namespaces
NAMESPACE     NAME                                           READY     STATUS    RESTARTS   AGE
kube-system   fluentd-cloud-logging-kubernetes-minion-5h5s   1/1       Running   0          51m
kube-system   fluentd-cloud-logging-kubernetes-minion-e3p1   1/1       Running   0          51m
kube-system   fluentd-cloud-logging-kubernetes-minion-k0jp   1/1       Running   0          51m
kube-system   fluentd-cloud-logging-kubernetes-minion-ykur   1/1       Running   0          51m
kube-system   kube-dns-v8-v7svt                              4/4       Running   0          52m
kube-system   kube-ui-v1-kwrq6                               1/1       Running   0          25m
kube-system   monitoring-heapster-v7-aur4k                   0/1       Running   10         51m
kube-system   monitoring-influx-grafana-v1-qhvpi             2/2       Running   0          51m"><pre class="notranslate"><code class="notranslate">kubernetes·(6afbaf6...) $ k get pods --all-namespaces
NAMESPACE     NAME                                           READY     STATUS    RESTARTS   AGE
kube-system   fluentd-cloud-logging-kubernetes-minion-5h5s   1/1       Running   0          51m
kube-system   fluentd-cloud-logging-kubernetes-minion-e3p1   1/1       Running   0          51m
kube-system   fluentd-cloud-logging-kubernetes-minion-k0jp   1/1       Running   0          51m
kube-system   fluentd-cloud-logging-kubernetes-minion-ykur   1/1       Running   0          51m
kube-system   kube-dns-v8-v7svt                              4/4       Running   0          52m
kube-system   kube-ui-v1-kwrq6                               1/1       Running   0          25m
kube-system   monitoring-heapster-v7-aur4k                   0/1       Running   10         51m
kube-system   monitoring-influx-grafana-v1-qhvpi             2/2       Running   0          51m
</code></pre></div>
<p dir="auto">It seems the <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/kubernetes/kubernetes/commit/8df33bc1a75ca7e81a07b49c033ae94e04dd66f4/hovercard" href="https://github.com/kubernetes/kubernetes/commit/8df33bc1a75ca7e81a07b49c033ae94e04dd66f4"><tt>8df33bc</tt></a> was the commit that changed this. EDIT: For clarification, I mean that this commit is when we started setting kubelet_api_servers on the master.</p>