capture Python script output
Question: I have written very small script and try to capture output of the script. I
have written multiple time similar way but never had issue. Could I have
input. I think I am doing very silly mistake

    
    
    numpy_temp = """
    import numpy
    import sys
    a, b, c = numpy.polyfit(%s,%s, 2)
    print a, b, c""" %(x, y)
    fp_numpy = open("numpy_temp.py", "w")
    fp_numpy.write(numpy_temp)
    cmd = "/remote/Python-2.7.2/bin/python numpy_temp.py "
    proc = subprocess.Popen(cmd, stdout = subprocess.PIPE,
                stderr = subprocess.PIPE, shell = True)
    out, err = proc.communicate()
    print "out", out
    


Answer: You're never actually closing `fp_numpy`, so the script can be empty or
incomplete at the time you try to run it.

It isn't _guaranteed_ to be empty, but it's very _likely_. When I try this on
two different *nix computers with 7 different versions of Python, it's empty
every time… (The fact that, after your script finishes, the file gets closed,
and therefore flushed, makes the problem harder to debug.)

The best way to fix this is to use a `with` statement, so it's impossible to
forget to close the file:

    
    
    with open("numpy_temp.py", "w") as fp_numpy:
        fp_numpy.write(numpy_temp)
    

* * *

But beyond that, you've got another problem. If the generated script raises an
exception, it will print nothing to stdout, and will dump a traceback to
stderr, which you will read and ignore. It's very hard to debug problems when
you're ignoring the errors… I don't know what you're passing for `x` and `y`,
but if you're, say, passing a `numpy.array` instead of a string that evaluates
to one, you could easily get an exception and never see it. Either send stderr
to stdout, or `print "err", err` at the end.

And finally, you really shouldn't be using a command string and `shell=True`
here, because you end up with an extra level of indirection for no good
reason, which can _also_ make things harder to debug. Just do this:

    
    
    cmd = ["/remote/Python-2.7.2/bin/python", "numpy_temp.py"]
    proc = subprocess.Popen(cmd, stdout = subprocess.PIPE,
                            stderr = subprocess.PIPE)
    

