Optimal framework to distribute a Python program across n cores
Question: I'm new to distributed systems and have been tasked with the objective of
distributing a piece of existing Python code. The goal is to treat the code as
a binary or a library and author two different kinds of wrappers:

  * **Wrapper 1:** Receives large datastreams from the environment, _invokes the Python code_ to perform some computation on it and then breaks it up and sends the chunks of data (and some other things) to the worker nodes. Runs in the master node.
  * **Wrapper 2:** Receives those chunks of data, _invokes the Python code_ to do some computation on them and when a particular condition is met, sends data back to the master node. 

The process is repeated until no further data comes to the master node. It can
be exemplified by the following figure:[![A high-level view of the desired
distributed
architecture.](http://i.stack.imgur.com/vH5Uq.png)](http://i.stack.imgur.com/vH5Uq.png)

So there exists both

(1) The need for communication between the workers and the master node as well
as

(2) The need for invocation of existing Python code.

It is also important that the entire framework views the notion of "node"
agnostically, since it needs to be run on either a personal computer where
nodes equate cores (physical or virtual) or on a cluster, where nodes can be
entire computers with a number of cores each. I'm therefore looking for a
technology that can help me achieve this plethora of goals. I'm already
studying up on Apache Spark, yet I'm not entirely sure whether Spark will
allow me to execute Python code in a streamlined fashion, and was looking for
ideas.


Answer: Check out [`celery`](http://www.celeryproject.org/) as an easy option:

> **Celery: Distributed Task Queue**
>
> Celery is an asynchronous task queue/job queue based on distributed message
> passing. It is focused on real-time operation, but supports scheduling as
> well. The execution units, called tasks, are executed concurrently on a
> single or more worker servers using multiprocessing, Eventlet, or gevent.
> Tasks can execute asynchronously (in the background) or synchronously (wait
> until ready).
>
> Celery is used in production systems to process millions of tasks a day.

