title: How to add retry for celery backend connection?
body: <p>I am using celery 5.0.1 and using <code>CELERY_BACKEND_URL</code> as <code>redis://:password@redisinstance1:6379/0</code>. It works fine, but when there is a Redis instance loose connection, it breaks out tasks with an error.</p>
  <pre><code>Exception: Error while reading from socket: (104, 'Connection reset by peer')
  Traceback (most recent call last):
    File &quot;/usr/local/lib/python3.7/dist-packages/redis/connection.py&quot;, line 198, in _read_from_socket
      data = recv(self._sock, socket_read_size)
    File &quot;/usr/local/lib/python3.7/dist-packages/redis/_compat.py&quot;, line 72, in recv
      return sock.recv(*args, **kwargs)
  ConnectionResetError: [Errno 104] Connection reset by peer
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File &quot;/usr/local/lib/python3.7/dist-packages/celery/app/trace.py&quot;, line 477, in trace_task
      uuid, retval, task_request, publish_result,
    File &quot;/usr/local/lib/python3.7/dist-packages/celery/backends/base.py&quot;, line 154, in mark_as_done
      self.store_result(task_id, result, state, request=request)
    File &quot;/usr/local/lib/python3.7/dist-packages/celery/backends/base.py&quot;, line 439, in store_result
      request=request, **kwargs)
    File &quot;/usr/local/lib/python3.7/dist-packages/celery/backends/base.py&quot;, line 855, in _store_result
      current_meta = self._get_task_meta_for(task_id)
    File &quot;/usr/local/lib/python3.7/dist-packages/celery/backends/base.py&quot;, line 873, in _get_task_meta_for
      meta = self.get(self.get_key_for_task(task_id))
    File &quot;/usr/local/lib/python3.7/dist-packages/celery/backends/redis.py&quot;, line 346, in get
      return self.client.get(key)
    File &quot;/usr/local/lib/python3.7/dist-packages/redis/client.py&quot;, line 1606, in get
      return self.execute_command('GET', name)
    File &quot;/usr/local/lib/python3.7/dist-packages/redis/client.py&quot;, line 901, in execute_command
      return self.parse_response(conn, command_name, **options)
    File &quot;/usr/local/lib/python3.7/dist-packages/redis/client.py&quot;, line 915, in parse_response
      response = connection.read_response()
    File &quot;/usr/local/lib/python3.7/dist-packages/redis/connection.py&quot;, line 739, in read_response
      response = self._parser.read_response()
    File &quot;/usr/local/lib/python3.7/dist-packages/redis/connection.py&quot;, line 324, in read_response
      raw = self._buffer.readline()
    File &quot;/usr/local/lib/python3.7/dist-packages/redis/connection.py&quot;, line 256, in readline
      self._read_from_socket()
    File &quot;/usr/local/lib/python3.7/dist-packages/redis/connection.py&quot;, line 223, in _read_from_socket
      (ex.args,))
  redis.exceptions.ConnectionError: Error while reading from socket: (104, 'Connection reset by peer')
  
  
  Celery worker: None
  Celery task id: 244b56af-7c96-56cf-a01a-9256cfd98ade
  Celery retry attempt: 0
  Task args: []
  Task kwargs: {'address': 'ipadd', 'uid': 'uid', 'hexID': 'hexID', 'taskID': '244b56af-7c96-56cf-a01a-9256cfd98ade'}
  </code></pre>
  <p>When I run the second tasks, it works fine, there is some glitch in the connection for a short period of time.</p>
  <p>Can I set something by which, when celery tries to update the results to Redis, if it returns an error, it will retry after 2-5 seconds?</p>
  <p>I know how to set retry in the task, but this does not task failure. My tasks work fine and it returns the data, but celery is losing connection while updating to the backend.</p>