title: Rabbitmq pod on kubernetes struck in pod initialization state
body: <p>I am running 3 nodes rabbitmq cluster on the Kubernetes. Kubernetes cluster is running on the AWS spot instances and somehow one of the Kubernetes nodes got terminated unexpectedly on with one of the Rabbitmq pods was running. Now the pod git scheduled ona another node and Since then my rabbitmq pod is stuck in the pod initialization state.</p>
  <p>Kubernetes event says &quot;FailedPostStartHook&quot;.</p>
  <p>Logs:</p>
  <pre><code>9m46s       Warning   FailedPostStartHook      pod/rabbitmq-0   Exec lifecycle hook ([/bin/sh -c until rabbitmqctl --erlang-cookie ${RABBITMQ_ERLANG_COOKIE} node_health_check; do sleep 1; done; rabbitmqctl --erlang-cookie ${RABBITMQ_ERLANG_COOKIE} set_policy ha-all &quot;&quot; '{&quot;ha-mode&quot;:&quot;all&quot;, &quot;ha-sync-mode&quot;: &quot;automatic&quot;}'
  ]) for Container &quot;rabbitmq&quot; in Pod &quot;rabbitmq-0_devops(c96c1a6e-bf9a-450d-828d-ed0e8a0ad949)&quot; failed - error: command '/bin/sh -c until rabbitmqctl --erlang-cookie ${RABBITMQ_ERLANG_COOKIE} node_health_check; do sleep 1; done; rabbitmqctl --erlang-cookie ${RABBITMQ_ERLANG_COOKIE} set_policy ha-all &quot;&quot; '{&quot;ha-mode&quot;:&quot;all&quot;, &quot;ha-sync-mode&quot;: &quot;automatic&quot;}'
  ' exited with 137: Error: unable to perform an operation on node 'rabbit@rabbitmq-0.rabbitmq-service.devops.svc.cluster.local'. Please see diagnostics information and suggestions below.
  Most common reasons for this are:
   * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
   * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
   * Target node is not running
  In addition to the diagnostics info below:
   * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
   * Consult server logs on node rabbit@rabbitmq-0.rabbitmq-service.devops.svc.cluster.local
   * If target node is configured to use long node names, don't forget to use --longnames with CLI tools
  DIAGNOSTICS
  ===========
  attempted to contact: ['rabbit@rabbitmq-0.rabbitmq-service.devops.svc.cluster.local']
  </code></pre>
  <p>Kubernetes statefulset manifest:</p>
  <pre><code>apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    name: rabbitmq
    namespace: devops
  spec:
    podManagementPolicy: OrderedReady
    replicas: 3
    revisionHistoryLimit: 3
    selector:
      matchLabels:
        app: rabbitmq
    serviceName: rabbitmq-service
    template:
      metadata:
        annotations:
        labels:
          app: rabbitmq
        name: rabbitmq
      spec:
        containers:
        - env:
          - name: HOSTNAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          - name: RABBITMQ_USE_LONGNAME
            value: &quot;true&quot;
          - name: RABBITMQ_BASIC_AUTH
            valueFrom:
              secretKeyRef:
                key: password
                name: rabbitmq
          - name: RABBITMQ_NODENAME
            value: rabbit@$(HOSTNAME).rabbitmq-service.$(NAMESPACE).svc.cluster.local
          - name: K8S_SERVICE_NAME
            value: rabbitmq-service
          - name: RABBITMQ_DEFAULT_USER
            value: admin
          - name: RABBITMQ_DEFAULT_PASS
            valueFrom:
              secretKeyRef:
                key: password
                name: rabbitmq
          - name: RABBITMQ_ERLANG_COOKIE
            value: some-cookie
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          image: rabbitmq:3.8.1-management-alpine
          imagePullPolicy: IfNotPresent
          lifecycle:
            postStart:
              exec:
                command:
                - /bin/sh
                - -c
                - |
                  until rabbitmqctl --erlang-cookie ${RABBITMQ_ERLANG_COOKIE} node_health_check; do sleep 1; done; rabbitmqctl --erlang-cookie ${RABBITMQ_ERLANG_COOKIE} set_policy ha-all &quot;&quot; '{&quot;ha-mode&quot;:&quot;all&quot;, &quot;ha-sync-mode&quot;: &quot;automatic&quot;}'
          livenessProbe:
            exec:
              command:
              - rabbitmqctl
              - status
            failureThreshold: 3
            initialDelaySeconds: 60
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 30
          name: rabbitmq
          ports:
          - containerPort: 4369
            protocol: TCP
          - containerPort: 5672
            protocol: TCP
          - containerPort: 5671
            protocol: TCP
          - containerPort: 25672
            protocol: TCP
          - containerPort: 15672
            protocol: TCP
          readinessProbe:
            exec:
              command:
              - rabbitmqctl
              - status
            failureThreshold: 3
            initialDelaySeconds: 20
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 30
          resources:
            limits:
              cpu: &quot;2&quot;
              memory: 3Gi
            requests:
              cpu: &quot;1&quot;
              memory: 2Gi
          volumeMounts:
          - mountPath: /var/lib/rabbitmq/
            name: rabbitmq-data
          - mountPath: /etc/rabbitmq
            name: config
        dnsPolicy: ClusterFirst
        initContainers:
        - command:
          - /bin/bash
          - -euc
          - |
            rm -f /var/lib/rabbitmq/.erlang.cookie
            cp /rabbitmqconfig/rabbitmq.conf /etc/rabbitmq/rabbitmq.conf
            cp /rabbitmqconfig/enabled_plugins /etc/rabbitmq/enabled_plugins
          image: rabbitmq:3.8.1-management-alpine
          imagePullPolicy: Always
          name: copy-rabbitmq-config
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /rabbitmqconfig
            name: rabbitmq-configmap
          - mountPath: /etc/rabbitmq
            name: config
          - mountPath: /var/lib/rabbitmq
            name: rabbitmq-data
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext: {}
        serviceAccount: rabbitmq
        serviceAccountName: rabbitmq
        terminationGracePeriodSeconds: 10
        volumes:
        - configMap:
            defaultMode: 420
            items:
            - key: rabbitmq.conf
              path: rabbitmq.conf
            - key: enabled_plugins
              path: enabled_plugins
            name: rabbitmq-configmap
          name: rabbitmq-configmap
        - emptyDir: {}
          name: config
    updateStrategy:
      type: RollingUpdate
    volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: rabbitmq-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 20Gi
        storageClassName: gp2
        volumeMode: Filesystem
  </code></pre>
  <p>Things I have tried:</p>
  <ol>
  <li>Logged into the struck pod and executed(This command just struck without any response)</li>
  </ol>
  <blockquote>
  <p>rabbitmqctl stop_app</p>
  </blockquote>
  <ol start="2">
  <li><p>Tried deleting the pod forcefully but no luck.</p>
  </li>
  <li><p>Logged into the struck pod and executed</p>
  </li>
  </ol>
  <blockquote>
  <p>rabbitmqctl reset</p>
  </blockquote>
  <ol start="4">
  <li>Logged into the struck pod and executed</li>
  </ol>
  <blockquote>
  <p>rabbitmqctl force_boot</p>
  </blockquote>
  <ol start="5">
  <li>Logged into the struck pod and executed</li>
  </ol>
  <blockquote>
  <p>rm /var/log/rabbitmq/*</p>
  </blockquote>
  <p>None of the above things helped.</p>
  <p>Please note that the other 2 rabbitmq nodes are running fine and serving the traffic and showing the failed node as up:</p>
  <pre><code>rabbitmq-2 rabbitmq 2021-07-04 12:19:07.233 [info] &lt;0.490.0&gt; node 'rabbit@rabbitmq-0.rabbitmq-service.devops.svc.cluster.local' up
  rabbitmq-1 rabbitmq 2021-07-04 12:19:07.208 [info] &lt;0.494.0&gt; node 'rabbit@rabbitmq-0.rabbitmq-service.devops.svc.cluster.local' up 
  </code></pre>