title: Django filesystem/file-based cache failing to write data 5-10% of the time
body: <p>We are doing background data processing with Django Celery, taking a CSV file (up to 15MB), converting it into list of dict data (which also includes some Django model objects), and breaking it up into chunks to process in sub tasks:</p>
  
  <pre><code>@task
  def main_task(data):
    i = 0
    for chunk in chunk_up(data):
      chunk_id = "chunk_id_{}".format(i)
      cache.set(chunk_id, chunk, timeout=FIVE_HOURS)
      sub_task.delay(chunk_id)
      i += 1
  
  @task
  def sub_task(chunk_id):
    data_chunk = cache.get(chunk_id)
    ... # do processing
  </code></pre>
  
  <p>All tasks run in concurrent processes in the background managed by Celery. We originally used Redis backend but found it would <a href="https://stackoverflow.com/a/29834386/199754">routinely run out of memory during peak load scenarios</a> and high concurrency. So we switched to <a href="https://docs.djangoproject.com/en/3.0/topics/cache/#filesystem-caching" rel="nofollow noreferrer">Django's filebased cache backend</a>. Although that fixed the memory issue, we saw that 20-30% of the cache entries never got written. No error thrown, just silent failure. When we go back and look up the cache from CLI, we see that for e.g. chunk_id_7 and chunk_id_9 would exist, but chunk_id_8 would not. So intermittently, some cache entries are failing to get saved.</p>
  
  <p>We swapped in <a href="http://www.grantjenks.com/docs/diskcache/" rel="nofollow noreferrer">diskcache</a> backend and are observing the same thing, though cache failures seem to be reduced to 5-10% (very rough estimate).</p>
  
  <p>We noticed that in past <a href="https://code.djangoproject.com/ticket/20536" rel="nofollow noreferrer">there where concurrent process issues with Django filebased cache</a>, but it seems to have been fixed many years ago (we are on v1.11). One comment says that this cache backend is more of a POC, though again not sure if it's changed since then.</p>
  
  <p>Is filebased cache a production-quality caching solution? If yes, what could be causing our write failures? If not, what's a better solution for our use case?</p>