title: Possible OOM in GCP container – how to debug?
body: <p>I have celery running in a docker container on GCP with Kubernetes. Its workers have recently started to get <code>kill -9</code>'d – this looks like it has something to do with OOMKiller. There are no OOM events in <code>kubectl get events</code>, which is something to be expected if these events only appear when a pod has trespassed <code>resources.limits.memory</code> value.</p>
  
  <p>So, my theory is that celery process getting killed is a work of linux' own OOMKiller. This doesn't make sense though: if so much memory is consumed that OOMKiller enters the stage, how is it possible that this pod was scheduled in the first place? (assuming that Kubernetes does not allow scheduling of new pods if the sum of <code>resources.limits.memory</code> exceeds the amount of memory available to the system).</p>
  
  <p>However, I am not aware of any other plausible reason for these SIGKILLs than OOMKiller.</p>
  
  <p>An example of celery error (there is one for every worker):</p>
  
  <pre><code>[2017-08-12 07:00:12,124: ERROR/MainProcess] Process 'ForkPoolWorker-7' pid:16 exited with 'signal 9 (SIGKILL)'
  [2017-08-12 07:00:12,208: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: signal 9 (SIGKILL).',)
  </code></pre>