<p dir="auto">Learning more Kubernetes, I wanted to follow the documentation through "Securing the Service". This example references another and opaquely, this second reference references code that is broken :-(</p>
<p dir="auto"><a href="http://kubernetes.io/v1.1/docs/user-guide/connecting-applications.html#securing-the-service" rel="nofollow">http://kubernetes.io/v1.1/docs/user-guide/connecting-applications.html#securing-the-service</a></p>
<p dir="auto"><a href="https://github.com/kubernetes/kubernetes/blob/release-1.1/docs/user-guide/connecting-applications.md#securing-the-service">https://github.com/kubernetes/kubernetes/blob/release-1.1/docs/user-guide/connecting-applications.md#securing-the-service</a></p>
<p dir="auto">The page references "Nginx https service":</p>
<p dir="auto"><a href="http://kubernetes.io/v1.1/examples/https-nginx/README.html" rel="nofollow">http://kubernetes.io/v1.1/examples/https-nginx/README.html</a></p>
<p dir="auto"><a href="https://github.com/kubernetes/kubernetes/blob/release-1.1/examples/https-nginx/README.md">https://github.com/kubernetes/kubernetes/blob/release-1.1/examples/https-nginx/README.md</a></p>
<p dir="auto">Which provides instructions on generating the secret. It was not immediately obvious from the page that there's an assumption that the reader will have downloaded Kubernetes/examples sources.</p>
<p dir="auto">make keys secret KEY=/tmp/nginx.key CERT=/tmp/nginx.crt SECRET=/tmp/secret.json</p>
<p dir="auto">After attempting and failing to download/build the sources, I believe that this script no longer works:</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="108337767" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/14558" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/14558/hovercard" href="https://github.com/kubernetes/kubernetes/issues/14558">#14558</a></p>
<p dir="auto">I'll break the issue into separate issues, since, by fixing the make, the "Nginx https service" will be fixed (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="135129277" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/21633" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/21633/hovercard" href="https://github.com/kubernetes/kubernetes/issues/21633">#21633</a>). However, I wasted quite some (free time) to realize this mistake and was blocked for some time.</p>
<p dir="auto">Ironically, I found generate-secrets-from-file on the following page and, through grokking this and the make_secret.go file, I was able to muddle through and create a secret.json file for Kubernetes.</p>
<p dir="auto"><a href="https://tectonic.com/enterprise/docs/latest/deployer/tls.html" rel="nofollow">https://tectonic.com/enterprise/docs/latest/deployer/tls.html</a></p>
<p dir="auto">Recommend:</p>
<p dir="auto">Add documentation that explicitly documents (separately) how to generate a Kubernetes secret from self-signed certificates.</p>
<p dir="auto">Please add (if there's not already) an easier way to submit bugs against kubernetes.io by helping the reader reference the github source directly. I googled for duplicates to find the correct github page.</p>
<p dir="auto">It's challenging in documentation but please be mindful of introducing nested assumptions to do this, do this, which requires doing this..... as, if any part of the chain fails...</p>
<p dir="auto"><strong>Is this a request for help?</strong> (If yes, you should use our troubleshooting guide and community support channels, see <a href="http://kubernetes.io/docs/troubleshooting/" rel="nofollow">http://kubernetes.io/docs/troubleshooting/</a>.):</p>
<p dir="auto">Yes</p>
<p dir="auto"><strong>What keywords did you search in Kubernetes issues before filing this one?</strong> (If you have found any duplicates, you should instead reply there.):</p>
<p dir="auto">I am aware of similar<br>
issue <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="166242705" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/29166" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/29166/hovercard" href="https://github.com/kubernetes/kubernetes/issues/29166">#29166</a> which was fixed in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="188655883" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/36616" data-hovercard-type="pull_request" data-hovercard-url="/kubernetes/kubernetes/pull/36616/hovercard" href="https://github.com/kubernetes/kubernetes/pull/36616">#36616</a> and v1.4.6. However, I can<br>
still reproduce as of v1.4.6.</p>
<hr>
<p dir="auto"><strong>Is this a BUG REPORT or FEATURE REQUEST?</strong> (choose one):</p>
<p dir="auto">Bug Report</p>
<p dir="auto"><strong>Kubernetes version</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto">v1.4.6</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: AWS</li>
<li><strong>OS</strong> (e.g. from /etc/os-release): CoreOS</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): 4.7.3-coreos-r2</li>
<li><strong>Install tools</strong>: kube-aws v0.9.1</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">Periodically, petsets will drop below the number of desired replicas<br>
and be unable to restore themselves.</p>
<p dir="auto">The petset shows the following error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Unable to mount volumes for pod &quot;petset-min-repro-0_default(xxx...)&quot;: timeout expired waiting for volumes to attach/mount for pod &quot;petset-min-repro-0&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[storage]
Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod &quot;petset-min-repro-0&quot;/&quot;default&quot;. list of unattached/unmounted volumes=[storage]"><pre class="notranslate"><code class="notranslate">Unable to mount volumes for pod "petset-min-repro-0_default(xxx...)": timeout expired waiting for volumes to attach/mount for pod "petset-min-repro-0"/"default". list of unattached/unmounted volumes=[storage]
Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "petset-min-repro-0"/"default". list of unattached/unmounted volumes=[storage]
</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">I expect EBS volumes to reattach to the correct pod automatically<br>
following node failure.</p>
<p dir="auto"><strong>How to reproduce it</strong> (as minimally and precisely as possible):</p>
<ol dir="auto">
<li>Apply the below YAML to bring up the test PetSet</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# Define storage class first so it can be used later
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: ebs-encrypted-storage

# Launch in AWS
provisioner: kubernetes.io/aws-ebs

# EBS-specific settings
# http://kubernetes.io/docs/user-guide/persistent-volumes/#aws
parameters:
  type: io1
  encrypted: &quot;true&quot;
  zone: us-west-1b
  iopsPerGB: &quot;10&quot;


---

# PetSet boilerplate
apiVersion: apps/v1alpha1
kind: PetSet
metadata:
  name: petset-min-repro
  labels:
    component: test
    role: reprpoduce

spec:
  serviceName: petset-min-repro
  replicas: 2

  template:
    metadata:
      labels:
        component: test
        role: reprpoduce
      annotations:
        pod.alpha.kubernetes.io/initialized: &quot;true&quot;

    spec:

      # One container with an image that does nothing
      containers:
      - name: es-data
        image: alpine:latest
        command:
        - tail
        - -f
        - /dev/null

        # Attach persistent storage
        volumeMounts:
        - name: storage
          mountPath: /data

  volumeClaimTemplates:
  - metadata:
      name: storage
      annotations:
        # The storage should use the below defined storage class
        # Use both alpha and beta annotations for compatibility
        # http://blog.kubernetes.io/2016/10/dynamic-provisioning-and-storage-in-kubernetes.html
        volume.alpha.kubernetes.io/storage-class: ebs-encrypted-storage
        volume.beta.kubernetes.io/storage-class: ebs-encrypted-storage

    spec:
      # Volume should only be mountable to one pod at a time
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          # Smallest allowed io1 volume size
          storage: 4Gi
"><pre class="notranslate"><code class="notranslate"># Define storage class first so it can be used later
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: ebs-encrypted-storage

# Launch in AWS
provisioner: kubernetes.io/aws-ebs

# EBS-specific settings
# http://kubernetes.io/docs/user-guide/persistent-volumes/#aws
parameters:
  type: io1
  encrypted: "true"
  zone: us-west-1b
  iopsPerGB: "10"


---

# PetSet boilerplate
apiVersion: apps/v1alpha1
kind: PetSet
metadata:
  name: petset-min-repro
  labels:
    component: test
    role: reprpoduce

spec:
  serviceName: petset-min-repro
  replicas: 2

  template:
    metadata:
      labels:
        component: test
        role: reprpoduce
      annotations:
        pod.alpha.kubernetes.io/initialized: "true"

    spec:

      # One container with an image that does nothing
      containers:
      - name: es-data
        image: alpine:latest
        command:
        - tail
        - -f
        - /dev/null

        # Attach persistent storage
        volumeMounts:
        - name: storage
          mountPath: /data

  volumeClaimTemplates:
  - metadata:
      name: storage
      annotations:
        # The storage should use the below defined storage class
        # Use both alpha and beta annotations for compatibility
        # http://blog.kubernetes.io/2016/10/dynamic-provisioning-and-storage-in-kubernetes.html
        volume.alpha.kubernetes.io/storage-class: ebs-encrypted-storage
        volume.beta.kubernetes.io/storage-class: ebs-encrypted-storage

    spec:
      # Volume should only be mountable to one pod at a time
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          # Smallest allowed io1 volume size
          storage: 4Gi

</code></pre></div>
<ol start="2" dir="auto">
<li>Identify the EBS volume bound to one of the pods</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="PVC=$(kubectl describe pvc storage-petset-min-repro-0 | grep Volume | awk '{ print $2 }')
VOLUME_ID=$(kubectl describe pv $PVC | grep VolumeID | awk -F '/' '{ print $NF }')"><pre class="notranslate"><code class="notranslate">PVC=$(kubectl describe pvc storage-petset-min-repro-0 | grep Volume | awk '{ print $2 }')
VOLUME_ID=$(kubectl describe pv $PVC | grep VolumeID | awk -F '/' '{ print $NF }')
</code></pre></div>
<ol start="3" dir="auto">
<li>Detach the EBS volume from the node. One may reasonably ask why this would<br>
happen at all, but this is the most reliable way I've found<br>
to reproduce. It has also occurred randomly.</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="aws ec2 detach-volume --volume-id=$VOLUME_ID"><pre class="notranslate"><code class="notranslate">aws ec2 detach-volume --volume-id=$VOLUME_ID
</code></pre></div>
<ol start="4" dir="auto">
<li>Observe that the pod does not successfully restart and<br>
remains stuck in state <code class="notranslate">ContainerCreating</code>.</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="kubectl get pod petset-min-repro-0"><pre class="notranslate"><code class="notranslate">kubectl get pod petset-min-repro-0
</code></pre></div>
<p dir="auto"><strong>Anything else do we need to know</strong>:</p>
<p dir="auto">I have been able to so far work around this issue by terminating the node<br>
the pod is attempting to attach onto.</p>