title: Getting GKE secrets back even after deleting the KMS keys used for encryption
body: <p>I followed <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/encrypting-secrets" rel="nofollow noreferrer">this</a> document to create a GKE cluster (1.13.6-gke.6) with <code>--database-encryption-key</code> flag giving a KMS key for enabling <strong>Application-layer Secrets Encryption</strong>.</p>
  <p>I created a secret using the following command:<br></p>
  <pre><code>kubectl create secret generic dev-db-secret --from-literal=username=someuser --from-literal=password=somepass
  </code></pre>
  <p>So if my assumption is correct, these secrets are stored encrypted using the KMS key provided by me while creating the cluster. However, even after I have destroyed all the versions of the used key, I am able to see the secret stored inside the GKE <code>etcd</code> using <code>kubectl get secret dev-db-secret -o yaml</code> as well as I am able to see them in a pod created using the below manifest:</p>
  <pre><code>apiVersion: v1
  kind: Pod
  metadata:
    name: secret-env-pod
  spec:
    containers:
    - name: mycontainer
      image: redis
      env:
        - name: SECRET_USERNAME
          valueFrom:
            secretKeyRef:
              name: dev-db-secret
              key: username
        - name: SECRET_PASSWORD
          valueFrom:
            secretKeyRef:
              name: test-secret
              key: password
    restartPolicy: Never
  </code></pre>
  <p>If I exec into the above pod and do <code>echo SECRET_USERNAME</code> and <code>echo SECRET_PASSWORD</code> I get the username and password printed on my console in plain text.</p>
  <p>Is this the way the encryption supposed to work? If yes, where is the encryption happening exactly?
  What am I doing wrong? Are the secrets really encrypted?</p>