title: How can I make celery more robust with regards to dropped tasks?
body: <p>Occasionally (read: too often) my celery setup will drop tasks. I'm running the latest <code>celery 4.x</code> on <code>Django 1.11</code> with a <code>redis</code> backend for the queue and results.</p>
  <p>I don't know exactly why tasks are being dropped, but what I suspect is that a worker is starting a job, then the worker is killed for some reason (autoscaling action, redeployment, out-of-memory...) and the job is killed in the middle.</p>
  <p>At this point probably it has exited the <code>redis</code> queue and it won't be picked up again.</p>
  <p>So my questions are:</p>
  <ol>
  <li>How can I monitor this kind of thing? I use <code>celerymon</code>, and the task is not reported as failed, and yet I don't see in my database the data that I expected by the task that I suspect failed.</li>
  <li>How can I make <code>celery</code> retry such tasks without implementing my own &quot;fake queue&quot; with flags in the database?</li>
  <li>How do I make <code>celery</code> more robust and dependable in general?</li>
  </ol>
  <p>Thanks for any pointers!</p>