title: Sending inputs and image processing jobs to multiple machines using Celery and Python
body: <p>Recently I have been playing with celery and flower (for dashboard and task visualisation on a single machine) in Ubuntu using python 3.x. First I have installed rabbitmq-server, radis, celery and flower. Then I have created a script called <code>tasks.py</code> containing the following:</p>
  
  <pre><code>from celery import Celery
  
  # py-advanced-message-queuing-protocol
  app = Celery('tasks', backend='redis://localhost', broker='pyamqp://localhost//')
  
  @app.task
  def intensive_sum1(num):
      val = sum(x**4 for x in range(num))
      return val
  
  
  @app.task
  def intensive_sum2(num):
      val = sum(x**4 for x in range(num))
      return val
  
  @app.task
  def intensive_sum3(num):
      val = sum(x**4 for x in range(num))
      return val
  </code></pre>
  
  <p>Then I have created a script <code>run.py</code> containing</p>
  
  <pre><code>from tasks import intensive_sum1, intensive_sum2, intensive_sum3
  import time
  
  start = time.time()
  result1 = intensive_sum1.delay(100000000)
  result2 = intensive_sum2.delay(100000000)
  result3 = intensive_sum3.delay(100000000)
  print(result1.get(), result2.get(), result3.get())
  end = time.time()
  print('time: ', end - start)
  
  start = time.time()
  result1 = sum(x**4 for x in range(100000000))
  result2 = sum(x**4 for x in range(100000000))
  result3 = sum(x**4 for x in range(100000000))
  print(result1, result2, result3)
  end = time.time()
  print('time: ', end - start)
  </code></pre>
  
  <p>Before running the latter I started two distinct terminals and changed the directory to the location of the two scripts. I then ran <code>sudo celery -A tasks flower</code> in one terminal and <code>celery -A tasks worker --loglevel=info</code> in the other terminal. It turns out that (surprise surprise) celery can distribute each task to an individual core resulting in tremendous time saving. Of course this time saving is only to be expected for large functions as smaller ones will incur thread generation overhead which brings no benefit. </p>
  
  <p>This has made me think about another problem. Let's say that instead of a single machine I have 3 machines connected to the same WIFI router. I can work out the IP address for each of these Ubuntu machines using <code>ifconfig</code> command. Lets us say that one of these machines is a master machine that contains a <code>main.py</code> script which captures real-time images using Opencv-Python capture object. It is then taking each image, serialising it and sending it as a message to two worker machines. Both worker machines work independently and both de-serialise the same image. One worker machine does cat classification and returns a probability of cat, the other machine does dog classification and returns a probability of dog. One worker machine may take longer to reach a conclusion than the other. However, for that particular frame the master machine needs to wait for both classification results before overlaying some results on top on that particular frame. Instinctively, I am led to believe that master machine needs to check if both jobs are ready before moving forward (<code>e.g. result_worker_one.ready() == result_worker_two.ready() == True</code>). How can I achieve this behaviour ? How can I serialise one RGB image in master machine and de-serialise it in worker machines ? What <code>backend</code> and <code>broker</code> need each machine ? How can this be set up as a client server architecture ?</p>