title: K8s pod priority & outOfPods
body: <p>we had the situation that the k8s-cluster was running out of pods after an update (kubernetes or more specific: ICP) resulting in "OutOfPods" error messages. The reason was a lower "podsPerCore"-setting which we corrected afterwards. Until then there were pods with a provided priorityClass (1000000) which cannot be scheduled. Others - without a priorityClass (0) - were scheduled. I assumed a different behaviour. I thought that the K8s scheduler would kill pods with no priority so that a pod with priority can be scheduled. Was I wrong?</p>
  
  <p>Thats just a question for understanding because I want to guarantee that the priority pods are running, no matter what.</p>
  
  <p>Thanks</p>
  
  <hr>
  
  <p>Pod with Prio:</p>
  
  <pre><code>apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      kubernetes.io/psp: ibm-anyuid-hostpath-psp
    creationTimestamp: "2019-12-16T13:39:21Z"
    generateName: dms-config-server-555dfc56-
    labels:
      app: config-server
      pod-template-hash: 555dfc56
      release: dms-config-server
    name: dms-config-server-555dfc56-2ssxb
    namespace: dms
    ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicaSet
      name: dms-config-server-555dfc56
      uid: c29c40e1-1da7-11ea-b646-005056a72568
    resourceVersion: "65065735"
    selfLink: /api/v1/namespaces/dms/pods/dms-config-server-555dfc56-2ssxb
    uid: 7758e138-2009-11ea-9ff4-005056a72568
  spec:
    containers:
    - env:
      - name: CONFIG_SERVER_GIT_USERNAME
        valueFrom:
          secretKeyRef:
            key: username
            name: dms-config-server-git
      - name: CONFIG_SERVER_GIT_PASSWORD
        valueFrom:
          secretKeyRef:
            key: password
            name: dms-config-server-git
      envFrom:
      - configMapRef:
          name: dms-config-server-app-env
      - configMapRef:
          name: dms-config-server-git
      image: docker.repository..../infra/config-server:2.0.8
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 3
        httpGet:
          path: /actuator/health
          port: 8080
          scheme: HTTP
        initialDelaySeconds: 90
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 1
      name: config-server
      ports:
      - containerPort: 8080
        name: http
        protocol: TCP
      readinessProbe:
        failureThreshold: 3
        httpGet:
          path: /actuator/health
          port: 8080
          scheme: HTTP
        initialDelaySeconds: 20
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 1
      resources:
        limits:
          cpu: 250m
          memory: 600Mi
        requests:
          cpu: 10m
          memory: 300Mi
      securityContext:
        capabilities:
          drop:
          - MKNOD
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: default-token-v7tpv
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    nodeName: kub-test-worker-02
    priority: 1000000
    priorityClassName: infrastructure
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext: {}
    serviceAccount: default
    serviceAccountName: default
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoSchedule
      key: node.kubernetes.io/memory-pressure
      operator: Exists
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - name: default-token-v7tpv
      secret:
        defaultMode: 420
        secretName: default-token-v7tpv
  </code></pre>
  
  <p>Pod without Prio (just an example within the same namespace):</p>
  
  <pre><code>apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      kubernetes.io/psp: ibm-anyuid-hostpath-psp
    creationTimestamp: "2019-09-10T09:09:28Z"
    generateName: produkt-service-57d448979d-
    labels:
      app: produkt-service
      pod-template-hash: 57d448979d
      release: dms-produkt-service
    name: produkt-service-57d448979d-4x5qs
    namespace: dms
    ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicaSet
      name: produkt-service-57d448979d
      uid: 4096ab97-5cee-11e9-97a2-005056a72568
    resourceVersion: "65065755"
    selfLink: /api/v1/namespaces/dms/pods/produkt-service-57d448979d-4x5qs
    uid: b112c5f7-d3aa-11e9-9b1b-005056a72568
  spec:
    containers:
    - image: docker-snapshot.repository..../dms/produkt-    service:0b6e0ecc88a28d2a91ffb1db61f8ca99c09a9d92
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 3
        httpGet:
          path: /actuator/health
          port: 8080
          scheme: HTTP
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 1
      name: produkt-service
      ports:
      - containerPort: 8080
        name: http
        protocol: TCP
      readinessProbe:
        failureThreshold: 3
        httpGet:
          path: /actuator/health
          port: 8080
          scheme: HTTP
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 1
      resources: {}
      securityContext:
        capabilities:
          drop:
          - MKNOD
        procMount: Default
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: default-token-v7tpv
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    nodeName: kub-test-worker-02
    priority: 0
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext: {}
    serviceAccount: default
    serviceAccountName: default
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - name: default-token-v7tpv
      secret:
        defaultMode: 420
        secretName: default-token-v7tpv
  </code></pre>