Converting a serial task to parallel to map inputs and outputs
Question: I have tens of thousands of simulations to run on a system with several cores.
Currently, it is done in serial, where I know my input parameters, and store
my results in a dict.

# Serial version

    
    
    import time
    import random
    
    class MyModel(object):
        input = None
        output = None
    
        def run(self):
            time.sleep(random.random())  # simulate a complex task
            self.output = self.input * 10
    
    
    # Run serial tasks and store results for each parameter
    
    parameters = range(10)
    results = {}
    
    for p in parameters:
        m = MyModel()
        m.input = p
        m.run()
        results[p] = m.output
    
    print('results: ' + str(results))
    

Which takes <10 seconds, and displays correct results:

    
    
    results: {0: 0, 1: 10, 2: 20, 3: 30, 4: 40, 5: 50, 6: 60, 7: 70, 8: 80, 9: 90}
    

# Parallel version

My attempts to parallelize this procedure are based on the example in the
`multiprocessing` module near the text ["An example showing how to use queues
to feed tasks to a collection of worker processes and collect the
results"](http://docs.python.org/2/library/multiprocessing.html#examples)
(sorry, no URL anchor available).

The following builds on the top half of the serial version:

    
    
    from multiprocessing import Process, Queue
    NUMBER_OF_PROCESSES = 4
    
    def worker(input, output):
        for args in iter(input.get, 'STOP'):
            m = MyModel()
            m.input = args[0]
            m.run()
            output.put(m.output)
    
    
    # Run parallel tasks and store results for each parameter
    
    parameters = range(10)
    results = {}
    
    # Create queues
    task_queue = Queue()
    done_queue = Queue()
    
    # Submit tasks
    tasks = [(t,) for t in parameters]
    for task in tasks:
        task_queue.put(task)
    
    # Start worker processes
    for i in range(NUMBER_OF_PROCESSES):
        Process(target=worker, args=(task_queue, done_queue)).start()
    
    # Get unordered results
    for i in range(len(tasks)):
        results[i] = done_queue.get()
    
    # Tell child processes to stop
    for i in range(NUMBER_OF_PROCESSES):
        task_queue.put('STOP')
    
    print('results: ' + str(results))
    

Takes only a few seconds now, but the mapping orders between inputs and
results are mixed up.

    
    
    results: {0: 10, 1: 0, 2: 60, 3: 40, 4: 20, 5: 80, 6: 30, 7: 90, 8: 70, 9: 50}
    

I realise that I'm populating the `results` based on an unordered
`done_queue.get()`, but I'm not sure how to get the correct mapping to
`task_queue`. Any ideas? Any other way to make this somehow cleaner?


Answer: A-ha! The worker needs to embed some kind of ID, such as the input
parameter(s) used to return to the output queue, which can be used to identify
the returned process. Here are the required modifications:

    
    
    def worker(input, output):
        for args in iter(input.get, 'STOP'):
            m = MyModel()
            m.input = args[0]
            m.run()
            # Return a tuple of an ID (the input parameter), and the model output
            return_obj = (m.input, m.output)
            output.put(return_obj)
    

and

    
    
    # Get unordered results
    for i in range(len(tasks)):
        # Unravel output tuple, which has the input parameter 'p' used as an ID
        p, result = done_queue.get()
        results[p] = result
    

