how to prevent failure of subprocess stopping the main process in python
Question: I wrote a python script to run a command called "gtdownload" on a bunch of
files with multiprocessing. The function "download" is where I am having
trouble with.

    
    
    #/usr/bin/env python
    
    import os, sys, subprocess
    from multiprocessing import Pool
    
    def cghub_dnld_file(file1, file2, file3, np):
        <open files>
        <read in lines>
        p = Pool(int(np))
        map_args = [(line.rstrip(),name_lines[i].rstrip(),bar_lines[i].rstrip()) for i, line in enumerate(id_lines)]
        p.map(download_wrapper,map_args)
    
    def download(id, name, bar):
        <check if file has been downloaded, if not download>
        <.....>
        link = "https://cghub.ucsc.edu/cghub/data/analysis/download/" + id
        dnld_cmd = "gtdownload -c ~/.cghub.key --max-children 4 -vv -d " + link + " > gt.out 2>gt.err"
        subprocess.call(dnld_cmd,shell=True)
    
    def download_wrapper(args):
        return download(*args)
    
    def main():
        <read in arguments>
        <...>
        cghub_dnld_file(file1,file2,file3,threads)
    
    if __name__ == "__main__":
       main()
    

If this file does not exist in the database, gtdownload would quit which also
kills my python job with the following error:

    
    
    Traceback (most recent call last):
      File "/rsrch1/rists/djiao/bin/cghub_dnld.py", line 102, in <module>
        main()
      File "/rsrch1/rists/djiao/bin/cghub_dnld.py", line 98, in main
        cghub_dnld_file(file1,file2,file3,threads)
      File "/rsrch1/rists/djiao/bin/cghub_dnld.py", line 22, in cghub_dnld_file
        p.map(download_wrapper,map_args)
      File "/rsrch1/rists/apps/x86_64-rhel6/anaconda/lib/python2.7/multiprocessing/pool.py", line 250, in map
        return self.map_async(func, iterable, chunksize).get()
      File "/rsrch1/rists/apps/x86_64-rhel6/anaconda/lib/python2.7/multiprocessing/pool.py", line 554, in get
        raise self._value
    OSError: [Errno 2] No such file or directory
    

The actual error message from gtdownload :

    
    
    Welcome to gtdownload-3.8.5a.
    Ready to download
    Communicating with GT Executive ...
    Headers received from the client:  'HTTP/1.1 100 Continue
    
    HTTP/1.1 404 Not Found
    Date: Tue, 29 Jul 2014 18:49:57 GMT
    Server: Apache/2.2.15 (Red Hat and CGHub)
    Strict-Transport-Security: max-age=31536000
    X-Powered-By: PHP/5.3.3
    Content-Length: 669
    Connection: close
    Content-Type: text/xml
    
    '
    
    Error:  You have requested to download a uuid which either does not exist within the system, or has not yet reached the 'live' state.  The requested action will not be performed.  Please double check the supplied uuid or contact thelpdesk for further assistance.
    

I would like the script to skip the one that does not exist and start
gtdownload on the next one. I tried to output the stderr of subprocess.call to
a pipe and see if there is the "error" keyword. But it seems it stops at the
exact subprocess.call command. Same thing with os.system.

I made a MCV case without the multiprocessing and subprocess did not kill the
main process at all. Looks like multiprocessing messes things up although I
had it run with 1 thread just for testing.

    
    
    #!/usr/bin/env python
    import subprocess
    #THis is the id that gtdownload had problem with
    id = "df1e073f-4485-4d5f-8659-cd8eaac17329"
    link = "https://cghub.ucsc.edu/cghub/data/analysis/download/" + id
    dlnd_cmd = "gtdownload -c ~/.cghub.key --max-children 4 -vv -d " + link + " > gt.out 2>gt.err"
    print dlnd_cmd
    subprocess.call(dlnd_cmd,shell=True)
    print "done"
    

Clearly multiprocessing conflicts subprocess.call but it is not clear to me
why.


Answer: > What is the best way to avoid the failure of subprocess killing the main
> process?

Handle the exception in some appropriate way and move on, of course.

    
    
    try:
        subprocess.call(dlnd_cmd)
    except OSError as e:
        print 'failed to download: {!r}'.format(e)
    

However, this may not be appropriate here. The kinds of exceptions that
`subprocess.call` raises are usually not transient things that you can just
log and work around; if it's not working now, it will continue to not work
forever until you fix the underlying problem (a bug in your script, or
`gtdownload` not being installed right, or whatever).

For example, if the code you showed us is your actual code:

    
    
    dlnd_cmd = "gtdownload -c ~/.cghub.key --max-children 4 -vv -d " + link + " > gt.out 2>gt.err"
    subprocess.call(dlnd_cmd)
    

… then this is guaranteed to raise an `OSError` for the reason explained in
dano's answer: `call` (without `shell=True`) will try to take that entire
string—spaces, shell-redirection, etc.—as the name of an executable program to
find on your `$PATH`. And there is no such program. So it will raise an
`OSError(errno.ENOENT)`. (Which is exactly what you're seeing.) Just logging
that doesn't do you any good; it's a good thing that your entire process is
exiting, so you can debug that problem.

