return a variable value from a subprocess in python
Question: I have a code workflow in which from a main script(level 0) I call another
script through `subprocess`. This `subprocess script` (level 1) in turn calls
another script as a `subprocess`. Now from this level 2 subprocess script I
want to return the value of a variable upto the main script(level 0). I have
tried `Popen.communicate()` but I am not able to return the value. My current
code is as:

**main_script.py**

    
    
    def func():
    
        para = ['a','b']
        result = subprocess.Popen([sys.executable,"first_subprocess.py"]+para,stdout=subprocess.PIPE)
        result.wait()
    
        return_code = result.returncode
        out, err = sub_result.communicate()
    
    
        if return_code == 1:
            return return_code
    
        else:
            if out is not None:
                print 'return value: ', out
    
    
    if __name__ == '__main__':
    
        func()
    

From above script is called `first_subprocess.py` which has:

    
    
    def some_func():
    
        # some other code and multiple print statements
    
        para = ['a','b']
        result = subprocess.Popen([sys.executable,"second_subprocess.py"]+para,stdout=subprocess.PIPE)
    
        result.wait()
        out, err = result.communicate()
        return_code = sub_result.returncode
        if return_code == 0:
            return out
    
    
    if __name__ == '__main__':
    
        some_func()
    

The `second_subprocess.py` returns a value like:

    
    
    def test():
        # some other code and multiple print statements
        val = 'time'
        print 'returning value'
        return val   
    
    if __name__ == '__main__':    
    
        test()
    

When I try above code I get all the `print` statements in the codes as an
output but not the return value. Even if try to `print` the variable value in
subprocess instead of doing a return it wont serve the purpose because there
are multiple print statements.

How can I return the variable value in this case?

**UPDATED VERSION:**

After @Anthons suggestion I have modifed my `first_subprocess.py` script and
`main_script.py` as follows:

`first_subprocess.py:`

    
    
    def some_func():
    
       try:
        key = None
        if not (key is None):
    
           para = ['a','b']
           result = subprocess.Popen([sys.executable,"second_subprocess.py"]+para,stdout=subprocess.PIPE)
    
           sub_result.wait()
           out, err = sub_result.communicate()
           return_code = sub_result.returncode
           if return_code == 0:
           for line in out.splitlines():
               if not line.startswith('>>>'):
                  continue
               print line
       else:
         sys.exit(0)
     except:
       return 1
    

**Main_script.py:**

    
    
    if out is not None:
       for line in out.splitlines():
           if not line.startswith('>>>'):
              continue
          value = line.split(':',1)[1].lstrip()
    
    print 'return value:',value`
    

When I execute above I get `UnboundLocalError: local variable 'value'
referenced before assignment` at the `print value` command. It seems if I do
not execute the code in level 1 script and do a sys.exit() then `out` in main
script is neither empty nor none but it has some undefined value and thus the
`value` variable doesn't get initialized and throws error


Answer: If you just want to return an integer value you can use the exit value. This
is not the same a returning from `some_func()`, you would have to do
`sys.exit(integer_val)`.

If you want to return a string like `time` you should print that ( or write to
`sys.stdout` ) and then in the calling process (level 1) parse the output from
the subprocess and print it to its own stdout for level 0 to see it.

In your case level two should do something like:

    
    
    def test():
        # some other code and multiple print statements
        val = 'time'
        print 'returning value:', val
    
    if __name__ == '__main__':    
        test()
    

And at level 1 you would do:

    
    
    def some_func():
    
        # some other code and multiple print statements
    
        para = ['a','b']
        result = subprocess.Popen([sys.executable,"second_subprocess.py"]+para,stdout=subprocess.PIPE)
    
        result.wait()
        out, err = result.communicate()
        return_code = sub_result.returncode
        if return_code == 0:
            print out
    
    if __name__ == '__main__':
        some_func()
    

With that main_script.py has something to read from the invocation of your
level 1 script.

I normally use `subprocess.check_output()` for these kind of passing on info.
That throws an exception if the called process has a non-zero exit status
(i.e. on error). I can also recommend that if the subprocess writes more info
than just the variable, you make the output lines easily parseable by
returning something unique at the beginning of the line (so you still can use
print statements for debugging the individual scripts **and** get the right
value from the output):

Level 2:

    
    
    def test():
        # some other code and multiple print statements
        print 'debug: Still going strong'
        val = 'time'
        print '>>>> returning value:', val
    
    if __name__ == '__main__':    
        test()
    

Level 1:

    
    
    ...
    out, err = result.communicate()
    for line in out.splitlines():
        if not line.startswith('>>>>'):
            continue
        print line
    ...
    

Level 0:

    
    
    ...
    out, err = result.communicate()
    for line in out.splitlines():
        if not line.startswith('>>>>'):
            continue
        try:
            value = line.split(':', 1)[1]
        except IndexError:
            print 'wrong input line', repr(line)
        print 'return value: ', value
    ...
    

* * *

The following files work together. Save them under their indicated names

**lvl2.py** :

    
    
    # lvl2
    import sys
    
    def test():
        # some other code and multiple print statements
        print >> sys.stderr, 'argv', sys.argv[1:]
        print 'debug: Still going strong'
        val = 'time'
        print '>>>> returning value:', val
        return 0
    
    if __name__ == '__main__':
        sys.exit(test())
    

**lvl1.py** :

    
    
    # lvl1.py
    import sys
    import subprocess
    
    def some_func():
        para = ['a','b']
        sub_result = subprocess.Popen(
            [sys.executable, "lvl2.py" ] + para,
            stdout=subprocess.PIPE)
        sub_result.wait()
        out, err = sub_result.communicate()
        return_code = sub_result.returncode
        if return_code == 0:
            for line in out.splitlines():
                if not line.startswith('>>>'):
                    continue
                print line
        else:
            print >> sys.stderr, 'level 2 exited with' + return_code
        sys.exit(0)
    
    if __name__ == '__main__':
        sys.exit(some_func())
    

**lvl0.py** :

    
    
    # lvl0
    import subprocess
    import sys
    
    def func():
        para = ['a','b']
        result = subprocess.Popen(
            [sys.executable, "lvl1.py"] + para,
            stdout=subprocess.PIPE)
        result.wait()
        return_code = result.returncode
        out, err = result.communicate()
    
        value = None
        if return_code == 0:
            for line in out.splitlines():
                if not line.startswith('>>>'):
                    continue
                value = line.split(':',1)[1].lstrip()
                print
        else:
            print 'non-zero exit', return_code
        print 'return value:', value
    
    if __name__ == '__main__':
        func()
    

Then run `python lvl0.py` to check the output to be

    
    
    argv ['a', 'b']
    
    return value: time
    

Now bring these under your revision control system and start making changes a
few lines at a time, every time running `python lvl0.py` to check what you
might have broken. Commit each revision so you can roll back to last "known
good" state and slowly bring in the rest of your code.

