python 27 - Creating and running instances of another script in parallel
Question: I'm attempting to build a multiprocessing script that retrieves dicts of
attributes from a MySQL table and then runs instances of my main script in
**parallel** , using each dict retrieved from the MySQL table as an argument
to each instance of the main script. The main script has a method called
queen_bee() that's responsible for ensuring that all the other methods have
the correct information and are executed in the proper order.

I have tried to iterate through the list of dicts in order to create/run
parallel processes of the main script using the multiprocessing library. But
they end up running consecutively, not concurrently:

    
    
    from my_main_script import my_main_class as main
    import multiprocessing as mp
    
    def create_list_of_attribute_dicts():
        ...
        return list_of_dicts
    
    for each_dict in list_of_dicts:
        instance = main(each_dict)
        p = mp.Process(target=instance.queen_bee(),args=(each_dict,))
        p.start()
        ...
    

I have also tried using the multiprocessing library's Pool.map() method. But I
can't figure out how to instantiate the main script one time for each dict
using Pool.map():

    
    
    ...
    pool = mp.Pool()
    jobs = pool.map(main.queen_bee(),list_of_dicts)
    

The Pool.map method seems to be the cleanest, most pythonic way to get these
instances to run in parallel, but I'm hung up on the proper way to do that in
this case. I know the above 'jobs' variable will fail because 'main' has not
been instantiated. However, I can't figure out how to pass each dict as an
argument to separate instances of the main class and then run those instances
using the map method. I'm open to trying a different approach. Thanks in
advance for your help.


Answer: You could store your dictionaries in a list and then try something like that:

    
    
    from multiprocessing import Pool as ThreadPool
    
    # ...
    
    def parallel_function(list_of_dictionaries,threads=20):
        thread_pool = ThreadPool(threads)
        results = thread_pool.map(SC.queen_bee() ,list_of_dictionaries)
        thread_pool.close()
        thread_pool.join()
        return results
    

**Note:** I had problem using multiprocessing due to my pc, so for me worked
by replacing `multiprocessing` with the `multiprocessing.dummy` in the
`import` statement.

