title: kubernetes does not track pods success count for job completion
body: <p>I am running a job as defined <a href="https://kubernetes.io/docs/tasks/job/coarse-parallel-processing-work-queue/" rel="nofollow noreferrer">here</a> i.e. "parallel processing using work queue"
  on GKE</p>
  
  <p>Each pod runs a single container and That container has an R script, takes around 5 mins to finish and then Pod completes successfully.</p>
  
  <p>When I run the job for small numbers of completions like</p>
  
  <pre><code>completions: 606
  parallelism: 450
  backoffLimit: 1
  </code></pre>
  
  <p>Everything runs fine, Cluster scales up and down properly and Job gets finished.</p>
  
  <p>But when I run the job with specifications like</p>
  
  <pre><code>completions: 37572
  parallelism: 1610
  backoffLimit: 1
  </code></pre>
  
  <blockquote>
    <p>Pod succeeded count gets increased for a while but after that, it stays
    in between 1000 to 1500 and never reach to completions</p>
  </blockquote>
  
  <p><a href="https://i.stack.imgur.com/PsOlR.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/PsOlR.png" alt="enter image description here"></a></p>
  
  <p>Though pods are completing successfully, I can see them on google cloud kubernetes dashboard and output files are also being generated successfully.
  Queue is also showing the progress very accurately
  <a href="https://i.stack.imgur.com/M7JfT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/M7JfT.png" alt="enter image description here"></a></p>
  
  <p>And It has happened to me every time when I have run the job with high number of parallelism.
  I have tried different autoscaling node pool setup in my cluster for machine type with 64-CPUs, 32-CPUs, 16-CPUs. </p>
  
  <p>Currently, the way I am handling is<br>
  => When queue has number of consumers == parallelism OR files_in_output == completions<br>
  => I delete the job and delete the autoscaling node pool.</p>
  
  <p>Please find the cluster details
  <a href="https://i.stack.imgur.com/zvCH0.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/zvCH0.png" alt="enter image description here"></a></p>
  
  <p>Cluster status is always green during the run.</p>
  
  <p><strong>QUESTION</strong>         </p>
  
  <ul>
  <li>Why job completion count never increase after a certain point (i.e in my case below my parallelism count)? Even though pods are finishing successfully.</li>
  <li>And the worst thing is the job completion count decreases too? That I can't even understand. What will be the reason that kubernetes is behaving like so?</li>
  <li>Do I need to add some additional fields to my spec template so that
  it tracks job completions properly?</li>
  </ul>
  
  <p>Update:</p>
  
  <ul>
  <li>I have enough CPU Quota</li>
  <li>Each container(Pod) is limited to use 1 CPU and 1GB RAM.</li>
  <li>I have also upgraded cluster and node-pools to 1.10.6-gke.2 version. No Luck.</li>
  </ul>
  
  <p>GKE ISSUE REPORTED => <a href="https://issuetracker.google.com/issues/114650730" rel="nofollow noreferrer">https://issuetracker.google.com/issues/114650730</a></p>
  
  <p><strong>job.yml</strong></p>
  
  <pre><code>apiVersion: batch/v1
  kind: Job
  metadata:
    # Unique key of the Job instance
    name: my-job
  spec:
    completions: 37572
    parallelism: 1610
    backoffLimit: 1
    template:
      metadata:
        name: my-job
        labels:
          jobgroup: my-jobs
      spec:
        volumes:
        - name: jobs-pv-storage
          persistentVolumeClaim:
            claimName: fileserver-claim
            readOnly: false
        containers:
        - name: rscript
          image: gcr.io/project/image:v1
          resources:
            limits:
              cpu: "1"
              memory: 1200Mi
            requests:
              cpu: "1"
              memory: 1000Mi
          env:
          - name: BROKER_URL
            value: amqp://user:pwd@rabbitmq-service:5672
          - name: QUEUE
            value: job-queue
          volumeMounts:
          - mountPath: /opt/work/output
            name: jobs-pv-storage
        # Do not restart containers after they exit
        restartPolicy: Never
  </code></pre>