Multiprocess Python 3
Question: I have a script that loops over an array of numbers, those numbers are passed
to a function which calls and API. It returns JSON data which is then written
to a CSV.

    
    
    for label_number in label_array:
        call_api(domain, api_call_1, api_call_2, label_number, api_key)
    

The list can be up to 7000 elements big, as the API takes a few seconds to
respond this can take hours to run the entire script. Multiprocessing seems
the way to go with this. I can't quite working out how to do this with the
above loop. The documentation I am looking at is

<https://docs.python.org/3.5/library/multiprocessing.html>

I found a similar article at

[Python Multiprocessing a for
loop](http://stackoverflow.com/questions/20190668/python-multiprocessing-a-
for-loop)

But manipulating it doesn't seem to work, I think I am buggering it up when it
comes to passing all the variables into the function.

Any help would be appreciated.


Answer: Multiprocessing could help but this sounds more like a threading problem. Any
IO implementation should be made asynchronous, which is what threading does.
Better, in `python3.4` onwards, you could do `asyncio`.
<https://docs.python.org/3.4/library/asyncio.html>

If you have `python3.5`, this will be useful:
<https://docs.python.org/3.5/library/asyncio-task.html#example-hello-world-
coroutine>

You can mix `asyncio` with `multiprocessing` to get the optimized result. I
use in addition `joblib`.

    
    
    import multiprocessing
    from joblib import Parallel, delayed 
    
    def parallelProcess(i):
        for index, label_number in enumerate(label_array):
            if index % i == 0:
                call_api_async(domain, api_call_1, api_call_2, label_number, api_key)
    
    if __name__=="__main__":
        num_cores_to_use = multiprocessing.cpu_count()
        inputs = range(num_cores_to_use)
        Parallel(n_jobs=num_cores_to_use)(delayed(parallelProcess)(i) for i in inputs)
    

