title: mongo db pod on bare metal kubernetese cluster is always pending
body: <p>On my bare metal kubernetese cluster, I installed mongo db using <a href="https://bitnami.com/stack/mongodb/helm" rel="nofollow noreferrer">helm from bitnami on kubernetese</a> as follows.</p>
  <pre><code>helm install mongodb bitnami/mongodb
  </code></pre>
  <p>Immediately I get the following output as a result.</p>
  <pre><code>vagrant@kmasterNew:~$ helm install mongodb bitnami/mongodb
  NAME: mongodb
  LAST DEPLOYED: Tue May  4 12:26:58 2021
  NAMESPACE: default
  STATUS: deployed
  REVISION: 1
  TEST SUITE: None
  NOTES:
  ** Please be patient while the chart is being deployed **
  
  MongoDB(R) can be accessed on the following DNS name(s) and ports from within your cluster:
  
      mongodb.default.svc.cluster.local
  
  To get the root password run:
  
      export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace default mongodb -o jsonpath=&quot;{.data.mongodb-root-password}&quot; | base64 --decode)
  
  To connect to your database, create a MongoDB(R) client container:
  
      kubectl run --namespace default mongodb-client --rm --tty -i --restart='Never' --env=&quot;MONGODB_ROOT_PASSWORD=$MONGODB_ROOT_PASSWORD&quot; --image docker.io/bitnami/mongodb:4.4.5-debian-10-r21 --command -- bash
  
  Then, run the following command:
      mongo admin --host &quot;mongodb&quot; --authenticationDatabase admin -u root -p $MONGODB_ROOT_PASSWORD
  
  To connect to your database from outside the cluster execute the following commands:
  
      kubectl port-forward --namespace default svc/mongodb 27017:27017 &amp;
      mongo --host 127.0.0.1 --authenticationDatabase admin -p $MONGODB_ROOT_PASSWORD
      
  </code></pre>
  <p>Then upon inspecting the pods, I see that the pod is always pending however long I wait.</p>
  <pre><code>vagrant@kmasterNew:~$ kubectl get pod -o wide
  NAME                      READY   STATUS    RESTARTS   AGE     IP       NODE     NOMINATED NODE   READINESS GATES
  mongodb-d9b6d589c-zpmb6   0/1     Pending   0          9m21s   &lt;none&gt;   &lt;none&gt;   &lt;none&gt;           &lt;none&gt;
  </code></pre>
  <p>What am I missing?</p>
  <p>As indicated in the helm install output I run the following command to get the secret.</p>
  <pre><code>export MONGODB_ROOT_PASSWORD=$(kubectl get secret --namespace default mongodb -o jsonpath=&quot;{.data.mongodb-root-password}&quot; | base64 --decode)
  </code></pre>
  <p>It is successfully executed. And when I do</p>
  <pre><code>echo $MONGODB_ROOT_PASSWORD
  </code></pre>
  <p>I get the root password as</p>
  <pre><code>rMjjciN8An
  </code></pre>
  <p>As the instructions in the helm install output suggests, I tried to connect to the database by running</p>
  <pre><code>kubectl run --namespace default mongodb-client --rm --tty -i --restart='Never' --env=&quot;MONGODB_ROOT_PASSWORD=rMjjciN8An&quot; --image docker.io/bitnami/mongodb:4.4.5-debian-10-r21 --command -- bash
  
  mongo admin --host &quot;mongodb&quot; --authenticationDatabase admin -u root -p rMjjciN8An
  </code></pre>
  <p>And I get the following output.</p>
  <pre><code>MongoDB shell version v4.4.5
  connecting to: mongodb://mongodb:27017/admin?authSource=admin&amp;compressors=disabled&amp;gssapiServiceName=mongodb
  Error: couldn't connect to server mongodb:27017, connection attempt failed: SocketException: Error connecting to mongodb:27017 (10.111.99.8:27017) :: caused by :: Connection refused :
  connect@src/mongo/shell/mongo.js:374:17
  @(connect):2:6
  exception: connect failed
  exiting with code 1
  </code></pre>
  <p>As you can see the connection attempt failed. This I guess is because the pod itself is in the pending state in the first place.</p>
  <p>So to get more info about the pod, I exit out of the mongodb-client pod(I created in the step above), and run the following command.</p>
  <pre><code>kubectl get pod -o yaml
  </code></pre>
  <p>And I get the lengthy output.</p>
  <pre><code>apiVersion: v1
  items:
  - apiVersion: v1
    kind: Pod
    metadata:
      creationTimestamp: &quot;2021-05-04T12:26:59Z&quot;
      generateName: mongodb-d9b6d589c-
      labels:
        app.kubernetes.io/component: mongodb
        app.kubernetes.io/instance: mongodb
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: mongodb
        helm.sh/chart: mongodb-10.15.0
        pod-template-hash: d9b6d589c
      name: mongodb-d9b6d589c-zpmb6
      namespace: default
      ownerReferences:
      - apiVersion: apps/v1
        blockOwnerDeletion: true
        controller: true
        kind: ReplicaSet
        name: mongodb-d9b6d589c
        uid: c99bfa3e-9a8d-425f-acdc-74d8acaba71b
      resourceVersion: &quot;52012&quot;
      uid: 97f77766-f400-424c-9651-9839a7506721
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/component: mongodb
                  app.kubernetes.io/instance: mongodb
                  app.kubernetes.io/name: mongodb
              namespaces:
              - default
              topologyKey: kubernetes.io/hostname
            weight: 1
      containers:
      - env:
        - name: BITNAMI_DEBUG
          value: &quot;false&quot;
        - name: MONGODB_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: mongodb-root-password
              name: mongodb
        - name: ALLOW_EMPTY_PASSWORD
          value: &quot;no&quot;
        - name: MONGODB_SYSTEM_LOG_VERBOSITY
          value: &quot;0&quot;
        - name: MONGODB_DISABLE_SYSTEM_LOG
          value: &quot;no&quot;
        - name: MONGODB_DISABLE_JAVASCRIPT
          value: &quot;no&quot;
        - name: MONGODB_ENABLE_JOURNAL
          value: &quot;yes&quot;
        - name: MONGODB_ENABLE_IPV6
          value: &quot;no&quot;
        - name: MONGODB_ENABLE_DIRECTORY_PER_DB
          value: &quot;no&quot;
        image: docker.io/bitnami/mongodb:4.4.5-debian-10-r21
        imagePullPolicy: IfNotPresent
        livenessProbe:
          exec:
            command:
            - mongo
            - --disableImplicitSessions
            - --eval
            - db.adminCommand('ping')
          failureThreshold: 6
          initialDelaySeconds: 30
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: mongodb
        ports:
        - containerPort: 27017
          name: mongodb
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - bash
            - -ec
            - |
              mongo --disableImplicitSessions $TLS_OPTIONS --eval 'db.hello().isWritablePrimary || db.hello().secondary' | grep -q 'true'
          failureThreshold: 6
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        resources: {}
        securityContext:
          runAsNonRoot: true
          runAsUser: 1001
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /bitnami/mongodb
          name: datadir
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: kube-api-access-g5kx8
          readOnly: true
      dnsPolicy: ClusterFirst
      enableServiceLinks: true
      preemptionPolicy: PreemptLowerPriority
      priority: 0
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1001
      serviceAccount: mongodb
      serviceAccountName: mongodb
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 300
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 300
      volumes:
      - name: datadir
        persistentVolumeClaim:
          claimName: mongodb
      - name: kube-api-access-g5kx8
        projected:
          defaultMode: 420
          sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              items:
              - key: ca.crt
                path: ca.crt
              name: kube-root-ca.crt
          - downwardAPI:
              items:
              - fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
                path: namespace
    status:
      conditions:
      - lastProbeTime: null
        lastTransitionTime: &quot;2021-05-04T12:26:59Z&quot;
        message: '0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.'
        reason: Unschedulable
        status: &quot;False&quot;
        type: PodScheduled
      phase: Pending
      qosClass: BestEffort
  kind: List
  metadata:
    resourceVersion: &quot;&quot;
    selfLink: &quot;&quot;
  </code></pre>
  <p>But I feel that an important clue is at the end of the output.</p>
  <pre><code>message: '0/3 nodes are available: 3 pod has unbound immediate PersistentVolumeClaims.'
  </code></pre>
  <p>Looks like the some thing wrong with PVC. Now as I look the manifest generated by running</p>
  <pre><code>helm get menifest mongodb
  </code></pre>
  <p>I get the manifest as follows.</p>
  <pre><code>---
  # Source: mongodb/templates/serviceaccount.yaml
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: mongodb
    namespace: default
    labels:
      app.kubernetes.io/name: mongodb
      helm.sh/chart: mongodb-10.15.0
      app.kubernetes.io/instance: mongodb
      app.kubernetes.io/managed-by: Helm
  secrets:
    - name: mongodb
  ---
  # Source: mongodb/templates/secrets.yaml
  apiVersion: v1
  kind: Secret
  metadata:
    name: mongodb
    namespace: default
    labels:
      app.kubernetes.io/name: mongodb
      helm.sh/chart: mongodb-10.15.0
      app.kubernetes.io/instance: mongodb
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/component: mongodb
  type: Opaque
  data:
    mongodb-root-password: &quot;ck1qamNpTjhBbg==&quot;
  ---
  # Source: mongodb/templates/standalone/pvc.yaml
  kind: PersistentVolumeClaim
  apiVersion: v1
  metadata:
    name: mongodb
    namespace: default
    labels:
      app.kubernetes.io/name: mongodb
      helm.sh/chart: mongodb-10.15.0
      app.kubernetes.io/instance: mongodb
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/component: mongodb
  spec:
    accessModes:
      - &quot;ReadWriteOnce&quot;
    resources:
      requests:
        storage: &quot;8Gi&quot;
  ---
  # Source: mongodb/templates/standalone/svc.yaml
  apiVersion: v1
  kind: Service
  metadata:
    name: mongodb
    namespace: default
    labels:
      app.kubernetes.io/name: mongodb
      helm.sh/chart: mongodb-10.15.0
      app.kubernetes.io/instance: mongodb
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/component: mongodb
  spec:
    type: ClusterIP
    ports:
      - name: mongodb
        port: 27017
        targetPort: mongodb
        nodePort: null
    selector:
      app.kubernetes.io/name: mongodb
      app.kubernetes.io/instance: mongodb
      app.kubernetes.io/component: mongodb
  ---
  # Source: mongodb/templates/standalone/dep-sts.yaml
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: mongodb
    namespace: default
    labels:
      app.kubernetes.io/name: mongodb
      helm.sh/chart: mongodb-10.15.0
      app.kubernetes.io/instance: mongodb
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/component: mongodb
  spec:
    strategy:
      type: RollingUpdate
    selector:
      matchLabels:
        app.kubernetes.io/name: mongodb
        app.kubernetes.io/instance: mongodb
        app.kubernetes.io/component: mongodb
    template:
      metadata:
        labels:
          app.kubernetes.io/name: mongodb
          helm.sh/chart: mongodb-10.15.0
          app.kubernetes.io/instance: mongodb
          app.kubernetes.io/managed-by: Helm
          app.kubernetes.io/component: mongodb
      spec:
  
        serviceAccountName: mongodb
        affinity:
          podAffinity:
  
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      app.kubernetes.io/name: mongodb
                      app.kubernetes.io/instance: mongodb
                      app.kubernetes.io/component: mongodb
                  namespaces:
                    - &quot;default&quot;
                  topologyKey: kubernetes.io/hostname
                weight: 1
          nodeAffinity:
  
        securityContext:
          fsGroup: 1001
          sysctls: []
        containers:
          - name: mongodb
            image: docker.io/bitnami/mongodb:4.4.5-debian-10-r21
            imagePullPolicy: &quot;IfNotPresent&quot;
            securityContext:
              runAsNonRoot: true
              runAsUser: 1001
            env:
              - name: BITNAMI_DEBUG
                value: &quot;false&quot;
              - name: MONGODB_ROOT_PASSWORD
                valueFrom:
                  secretKeyRef:
                    name: mongodb
                    key: mongodb-root-password
              - name: ALLOW_EMPTY_PASSWORD
                value: &quot;no&quot;
              - name: MONGODB_SYSTEM_LOG_VERBOSITY
                value: &quot;0&quot;
              - name: MONGODB_DISABLE_SYSTEM_LOG
                value: &quot;no&quot;
              - name: MONGODB_DISABLE_JAVASCRIPT
                value: &quot;no&quot;
              - name: MONGODB_ENABLE_JOURNAL
                value: &quot;yes&quot;
              - name: MONGODB_ENABLE_IPV6
                value: &quot;no&quot;
              - name: MONGODB_ENABLE_DIRECTORY_PER_DB
                value: &quot;no&quot;
            ports:
              - name: mongodb
                containerPort: 27017
            livenessProbe:
              exec:
                command:
                  - mongo
                  - --disableImplicitSessions
                  - --eval
                  - &quot;db.adminCommand('ping')&quot;
              initialDelaySeconds: 30
              periodSeconds: 10
              timeoutSeconds: 5
              successThreshold: 1
              failureThreshold: 6
            readinessProbe:
              exec:
                command:
                  - bash
                  - -ec
                  - |
                    mongo --disableImplicitSessions $TLS_OPTIONS --eval 'db.hello().isWritablePrimary || db.hello().secondary' | grep -q 'true'
              initialDelaySeconds: 5
              periodSeconds: 10
              timeoutSeconds: 5
              successThreshold: 1
              failureThreshold: 6
            resources:
              limits: {}
              requests: {}
            volumeMounts:
              - name: datadir
                mountPath: /bitnami/mongodb
                subPath:
        volumes:
          - name: datadir
            persistentVolumeClaim:
              claimName: mongodb
  </code></pre>
  <p>To summarize, the following are the 5 different kinds of objects the above manifest represent.</p>
  <pre><code>kind: ServiceAccount
  kind: Secret
  kind: PersistentVolumeClaim
  kind: Service
  kind: Deployment
  </code></pre>
  <p>As we can see there is PersistentVolumeClaim, but no PersistentVolume.</p>
  <p>I think, I followed the instructions given <a href="https://bitnami.com/stack/mongodb/helm" rel="nofollow noreferrer">here</a> for installing the mongo db chart on kubernetese.</p>
  <p><a href="https://i.stack.imgur.com/BhnAQ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/BhnAQ.png" alt="mongo db installation on kubernetese using helm" /></a></p>
  <p>There is nothing there mentioning about PersistentVolume. Am I missing something here? Do I have to somehow create a Persistant Volume myself?</p>
  <p>So the questions are</p>
  <ol>
  <li><p>Why is the pod in pending state indefinitely?</p>
  </li>
  <li><p>Why is there no Persistant Volume object created(I checked with the command kubectl get pv --all-namespaces)</p>
  </li>
  <li><p>Finally what baffles me is when try to get logs, I see nothing!!</p>
  <pre><code>vagrant@kmasterNew:~$ kubectl get pods
  NAME                      READY   STATUS    RESTARTS   AGE
  mongodb-d9b6d589c-zpmb6   0/1     Pending   0          60m
  vagrant@kmasterNew:~$ kubectl logs mongodb-d9b6d589c-zpmb6
  vagrant@kmasterNew:~$ 
  </code></pre>
  </li>
  </ol>