basic multiprocessing with python
Question: I have found information on multiprocessing and multithreading in python but I
don't understand the basic concepts and all the examples that I found are more
difficult than what I'm trying to do.

I have X independent programs that I need to run. I want to launch the first Y
programs (where Y is the number of cores of my computer and X>>Y). As soon as
one of the independent programs is done, I want the next program to run in the
next available core. I thought that this would be straightforward, but I keep
getting stuck on it. Any help in solving this problem would be much
appreciated.

Edit: Thanks a lot for your answers. I also found another solution using the
joblib module that I wanted to share. Suppose that you have a script called
'program.py' that you want to run with different combination of the input
parameters (a0,b0,c0) and you want to use all your cores. This is a solution.

    
    
    import os
    from joblib import Parallel, delayed
    a0 = arange(0.1,1.1,0.1)
    b0 = arange(-1.5,-0.4,0.1)
    c0 = arange(1.,5.,0.1)
    params = []
    for i in range(len(a0)):
        for j in range(len(b0)):
            for k in range(len(c0)):
                params.append((a0[i],b0[j],c0[k]))
    
    def func(parameters):
        s = 'python program.py %g %g %g' % parameters[0],parameters[1],parameters[2])
        command = os.system(s)
        return command
    
    output = Parallel(n_jobs=-1,verbose=1000)(delayed(func)(i) for i in params)
    


Answer: You want to use
[`multiprocessing.Pool`](http://docs.python.org/library/multiprocessing.html#using-
a-pool-of-workers), which represents a "pool" of workers (default one per
core, though you can specify another number) that do your jobs. You then
submit jobs to the pool, and the workers handle them as they become available.
The easiest function to use is
[`Pool.map`](http://docs.python.org/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool.map),
which runs a given function for each of the arguments in the passed sequence,
and returns the result for each argument. If you don't need return values, you
could also use `apply_async` in a loop.

    
    
    def do_work(arg):
        pass # do whatever you actually want to do
    
    def run_battery(args):
        # args should be like [arg1, arg2, ...]
        pool = multiprocessing.Pool()
        ret_vals = pool.map(do_work, arg_tuples)
        pool.close()
        pool.join()
        return ret_vals
    

If you're trying to call external programs and not just Python functions, use
[`subprocess`](http://docs.python.org/library/subprocess.html). For example,
this will call `cmd_name` with the list of arguments passed, raise an
exception if the return code isn't 0, and return the output:

    
    
    def do_work(subproc_args):
        return subprocess.check_output(['cmd_name'] + list(subproc_args))
    

