title: How to pass large chunk of data to celery
body: <p>I am using celery worker for getting results from my machine learning models.
  </p>
  
  <p>What I am doing is sending big numpy arrays(few megabytes) from client to celery task and back.</p>
  
  <p>Currently I am serializing in client numpy arrays as base64. When i store/get the data directly from/to Redis on client or celery worker the performance of the system is much faster than/as when i let the celery do all passing of arguments(base64 of numpy).</p>
  
  <p>I would like to use celery(with 'redis' broker) also for passing args/numpy arrays and not redis directly in client. Do you know where can be problem? How can i set the configuration of celery to do this more effectively(do the passing of data between client->broker->worker and back to client). </p>
  
  <pre class="lang-py prettyprint-override"><code>        serialized = np.asarray(images).reshape((number_of_records, size)).ravel().tostring()
          serialized = base64.b64encode(serialized)
          #self.redis.set(key, serialized)
  
          print('calling celery processor')
          result = self.celery.send_task('process', args=[number_of_records, serialized], kwargs={})
          returncode, result = result.get(timeout=1000, interval=0.1) 
  </code></pre>
  
  <p>vs(this is faster, directly usage of redis storage):</p>
  
  <pre class="lang-py prettyprint-override"><code>        serialized = np.asarray(images).reshape((number_of_records, size)).ravel().tostring()
          serialized = base64.b64encode(serialized)
          self.redis.set(key, serialized)
  
          print('calling celery processor')
          result = self.celery.send_task('process', args=[number_of_records, key], kwargs={})
          returncode, result = result.get(timeout=1000, interval=0.1) 
  
          resultc= self.redis.get(key)
  </code></pre>
  
  <p>Any tips for performance of celery for serialization, configuration settings, ...? I would like to have this system fast and simple. Should I really use redis directly as it is in second example?</p>