title: Redis memory usage continues to climb when using task.forget()
body: <p>I have a mysql database which stores thousands of stock OHLC data for 2 years. Data is read from MySQL in the form of pandas dataframes and then submitted to celery in large batch jobs which eventually lead to "OOM command not allowed when used memory > 'maxmemory'".</p>
  
  <p>I have added the following celery config options. These options have allowed my script to run longer however redis inevitably reaches 2gb memory and celery throws OOM errors.</p>
  
  <pre><code>result_expires = 30
  ignore_result = True
  worker_max_tasks_per_child = 1000
  </code></pre>
  
  <p>From the redis side I have tried playing with the maxmemory policy using both allkeys-lru and volatile-lru. Neither seem to make a difference.</p>
  
  <p>When celery hits the OOM error the redis cli shows max memory usage and no keys?</p>
  
  <pre><code># Memory
  used_memory:2144982784
  used_memory_human:2.00G
  used_memory_rss:1630146560
  used_memory_rss_human:1.52G
  used_memory_peak:2149023792
  used_memory_peak_human:2.00G
  used_memory_peak_perc:99.81%
  used_memory_overhead:2144785284
  used_memory_startup:987472
  used_memory_dataset:197500
  used_memory_dataset_perc:0.01%
  allocator_allocated:2144944880
  allocator_active:1630108672
  allocator_resident:1630108672
  total_system_memory:17179869184
  total_system_memory_human:16.00G
  used_memory_lua:37888
  used_memory_lua_human:37.00K
  used_memory_scripts:0
  used_memory_scripts_human:0B
  number_of_cached_scripts:0
  maxmemory:2147483648
  maxmemory_human:2.00G
  maxmemory_policy:allkeys-lru
  allocator_frag_ratio:0.76
  allocator_frag_bytes:18446744073194715408
  allocator_rss_ratio:1.00
  allocator_rss_bytes:0
  rss_overhead_ratio:1.00
  rss_overhead_bytes:37888
  mem_fragmentation_ratio:0.76
  mem_fragmentation_bytes:-514798320
  mem_not_counted_for_evict:0
  mem_replication_backlog:0
  mem_clients_slaves:0
  mem_clients_normal:2143797684
  mem_aof_buffer:0
  mem_allocator:libc
  active_defrag_running:0
  lazyfree_pending_objects:0
  </code></pre>
  
  <p>And there are zero keys?</p>
  
  <pre><code>127.0.0.1:6379[1]&gt; keys *
  (empty list or set)
  </code></pre>
  
  <p>When I run this same code in subsets of 200*5 requests (then terminate) everything runs successfully. Redis memory usage caps around 100mb and when the python process terminates all the memory usage drops as expected. This leads me to believe I could probably implement a handler to do 200*5 requests at a time however I suspect that the python process (my script) terminating is what is actually freeing memory in celery/redis...</p>
  
  <p>I would like to avoid subsetting this and process everything in MySQL in one shot. About 5000 pandas dataframes * 5 tasks total.</p>
  
  <p>I do not understand why the memory usage in redis continues to grow when I am forgetting all results immediately following retrieving them? </p>
  
  <p>Here is an example for how this is done in my code:</p>
  
  <pre><code>def getTaskResults(self, caller, task):
      #Wait for the results and then call back
      #Attach this Results object in the callback along with the data
      ready = False
      while not ready:
          if task.ready():
              ready = True
              data = pd.read_json(task.get())
              data.sort_values(by=['Date'], inplace=True)
              task.forget()
              return caller.resultsCB(data, self)
  </code></pre>
  
  <p>This is probably my ignorance with redis but if there are no keys how is it consuming all that memory, or how can I validate what is actually consuming that memory in redis?</p>
  
  <p>Since I store the taskID of every call to celery in an object I have confirmed that trying to do a task.get after adding in task.forget throws an error. </p>