concurrent.futures ThreadPoolExecutor fails to wait?
Question: I am trying to get a script performance improved, using ThreadPoolExecutor
from concurrent.futures. I am launching some external python scripts via Popen
and encapsulating them as a future objects, but these objects enter the
callback function as finished, but I can see them running on my machine (they
run for quite some minutes). The code looks like this:

    
    
    with futures.ThreadPoolExecutor(max_workers=4) as executor: 
            p1 = executor.submit(subprocess.Popen([myotherPythonscript1], stdout = subprocess.PIPE))
            p1.add_done_callback(process_result)
            p2 = executor.submit(subprocess.Popen([myotherPythonscript2], stdout = subprocess.PIPE))
            p2.add_done_callback(process_result)
    
    def process_result( future ):
            logger.info( "Seeding process finished...")
    

I also tried different approaches with running() and wait() future functions,
but with the same results. Future objects are marked as already done, but in
fact they are still running. Am I missing something?

Thanks,


Answer: you can't just pass the result of `Popen` to your executor, you have to pass a
callable.

    
    
    >>> from concurrent.futures import *
    >>> from subprocess import *
    >>> executor = ThreadPoolExecutor(4)
    >>> future = executor.submit(Popen(['echo', 'test'], stdout=PIPE))
    >>> future.exception()
    TypeError("'Popen' object is not callable",)
    

this on the other hand works:

    
    
    from concurrent.futures import *
    from subprocess import *
    
    def make_call():
        process = Popen(['echo', 'test'], stdout=PIPE)
        return process.communicate()[0]
    
    executor = ThreadPoolExecutor(4)
    future = executor.submit(make_call)
    print(future.result())
    

