Wrapping a commandline program with pstream
Question: I want to be able to read and write to a program from C++. It seems like
pstream can do the job, but I find the documentation difficult to understand
and have not yet find an example.

I have setup the following minimum working example. This opens python, which
in turn (1) prints `hello` (2) ask input, and (3) prints `hello2`:

    
    
    #include <iostream>
    #include <cstdio>
    #include "pstream.h"
    
    using namespace std;
    
    int main(){
        std::cout << "start";
        redi::pstream proc(R"(python -c "if 1:
            print 'hello'
            raw_input()
            print 'hello2'
            ")");
    
        std::string line;
        //std::cout.flush();
    
        while (std::getline(proc.out(), line)){
            std::cout << " " << "stdout: " << line << '\n';
        }
    
        std::cout << "end";
        return 0;
    }
    

If I run this with the "ask input" part commented out (i.e. `#raw_input()`), I
get as output:

    
    
    start stdout: hello
     stdout: hello2
    end
    

But if I leave the "ask input" part in (i.e. uncommented `raw_input()`), all I
get is blank, not even `start`, but rather what seems like a program waiting
for input.

My question is, how can one interact with this pstream, how can one establish
a little read-write-read-write session? Why does the program not even show
`start` or the first `hello`?

**EDIT:**  
I don't seem to be making much progress. I don't think I really grasp what is
going on. Here are some further attempts with commentary.

**1) It seems like I can successfully feed raw_input**  
I prove this by writing to the child's stderr:

    
    
    int main(){    
        cout << "start" <<endl;
        redi::pstream proc(R"(python -c "if 1:
            import sys
    
            print 'hello'
            sys.stdout.flush()
    
            a = raw_input()
            sys.stdin.flush()
    
            sys.stderr.write('hello2 '+ a)
            sys.stderr.flush()
    
            ")");
    
        string line;
        getline(proc.out(), line);
        cout << line << endl;
    
        proc.write("foo",3).flush();
    
        cout << "end" << endl;
        return 0;
    }
    

output:

    
    
    start
    hello
    end
    hello2 foo
    

But it locks if I try to read from the stdout again

    
    
    int main(){
            ...
            a = raw_input()
            sys.stdin.flush()
    
            print 'hello2', a
            sys.stdout.flush()
            ")");
    
        ...
        proc.write("foo",3).flush();
    
        std::getline(proc.out(), line);
        cout << line << endl;
        ...
    }
    

output

    
    
    start
    hello
    

**2) I can't get the readsome approach to work at all**

    
    
    int main(){
        cout << "start" <<endl;
        redi::pstream proc(R"(python -c "if 1:
            import sys
    
            print 'hello'
            sys.stdout.flush()
    
            a = raw_input()
            sys.stdin.flush()
            ")");
    
        std::streamsize n;
        char buf[1024];
        while ((n = proc.out().readsome(buf, sizeof(buf))) > 0)
                        std::cout.write(buf, n).flush();
    
        proc.write("foo",3).flush();
    
        cout << "end" << endl;
        return 0;
    }
    

output

    
    
    start
    end
    Traceback (most recent call last):
      File "<string>", line 5, in <module>
    IOError: [Errno 32] Broken pipe
    

The output contains a Python error, it seems like the C++ program finished
while the Python pipe were still open.

**Question:** Can anyone provide a working example of how this sequential
communication should be coded?


Answer: > But if I leave the "ask input" part in (i.e. uncommented raw_input()), all I
> get is blank, not even start, but rather what seems like a program waiting
> for input.

The Python process _is_ waiting for input, from its stdin, which is connected
to a pipe in your C++ program. If you don't write to the pstream then the
Python process will never receive anything.

The reason you don't see "start" is that Python thinks it's not connected to a
terminal, so it doesn't bother flushing every write to stdout. Try `import
sys` and then `sys.stdout.flush()` after printing in the Python program. If
you need it to be interactive then you need to flush regularly, or set stdout
to non-buffered mode (I don't know how to do that in Python).

You should also be aware that just using `getline` in a loop will block
waiting for more input, and if the Python process is _also_ blocking waiting
for input you have a deadlock. See the usage example on the [pstreams home
page](http://pstreams.sourceforge.net/#usage) showing how to use `readsome()`
for non-blocking reads. That will allow you to read as much as is available,
process it, then send a response back to the child process, so that it
produces more output.

**EDIT:**

> I don't think I really grasp what is going on.

Your problems are not really problems with pstreams or python, you're just not
thinking through the interactions between two communicating processes and what
each is waiting for.

Get a pen and paper and draw state diagrams or some kind of chart that shows
where the two processes have got to, and what they are waiting for.

> **1) It seems like I can successfully feed raw_input**

Yes, but you're doing it wrong. `raw_input()` reads a line, you aren't writing
a line, you're writing three characters, `"foo"`. That's not a line.

That means the python process keeps trying to read from its stdin. The parent
C++ process writes the three characters then exits, running the `pstream`
destructor which closes the pipes. Closing the pipes causes the Python process
so get EOF, so it stops reading (after only getting three characters not a
whole line). The Python process then prints to stderr, which is connected to
your terminal, because you didn't tell the `pstream` to attach a pipe to the
child's stderr, and so you see that output.

> But it locks if I try to read from the stdout again

Because now the parent C++ process doesn't exit, so doesn't close the pipes,
so the child Python process doesn't read EOF and keeps waiting for more input.
The parent C++ process is _also_ waiting for input, but that will never come.

If you want to send a line to be read by `raw_input()` then write a newline!

This works fine, because it sends a newline, which causes the Python process
to get past the `raw_input()` line:

    
    
    cout << "start" <<endl;
    redi::pstream proc(R"(python -c "if 1:
        import sys
    
        print 'hello'
        sys.stdout.flush()
    
        a = raw_input()
    
        print 'hello2', a
        sys.stdout.flush()
    
        ")");
    
    string line;
    getline(proc, line);
    cout << line << endl;
    
    proc << "foo" << endl; // write to child FOLLOWED BY NEWLINE!
    
    std::getline(proc, line); // read child's response
    cout << line << endl;
    
    cout << "end" << endl;
    

N.B. you don't need to use `proc.out()` because you haven't attached a pipe to
the process' stderr, so it always reads from `proc.out()`. You would only need
to use that when reading from _both_ stdout and stderr, where you would use
`proc.out()` and `proc.err()` to distinguish them.

> **2) I can't get the readsome approach to work at all**

Again, you have the same problem that you're only writing three characters and
so the Python processes waits forever. The C++ process is trying to read as
well, so it also waits forever. Deadlock.

If you fix that by sending a newline (as shown above) you have another
problem: the C++ program will run so fast that it will get to the `while` loop
that calls `readsome` before the Python process has even started. It will find
nothing to read in the pipe, and so the first `readsome` call returns 0 and
you exit the loop. Then the C++ program gets to the second `while` loop, and
the child python process _still_ hasn't started printing anything yet, so that
loop also reads nothing and exits. Then the whole C++ program exits, and
finally the Python child is ready to run and tries to print "hello" but by
then it's parent is gone and it can't write to the pipe.

You need `readsome` to keep trying if there's nothing to read the _first_ time
you call it_, so it waits long enough for the first data to be readable.

For your simple program you don't really need `readsome` because the Python
process only writes a single line at a time, so you can just read it with
`getline`. But if it might write more than one line you need to be able to
keep reading until there's no more data coming, which `readsome` can do (it
reads only if there's data available). But you also need some way to tell
whether more data is still going to come (maybe the child is busy doing some
calculations before it sends more data) or if it's really finished. There's no
general way to know that, it depends on what the child process is doing. Maybe
you need the child to send some sentinel value, like `"---END OF
RESPONSE---"`, which the parent can look for to know when to stop trying to
read more.

For the purposes of your simple example, let's just assume that if `readsome`
gets more than 4 bytes it received the whole response:

    
    
    cout << "start" <<endl;
    redi::pstream proc(R"(python -c "if 1:
        import sys
    
        print 'hello'
        sys.stdout.flush()
    
        a = raw_input()
        sys.stdin.flush()
    
        print 'hello2', a
        sys.stdout.flush()
        ")");
    
    string reply;
    streamsize n;
    char buf[1024];
    while ((n = proc.readsome(buf, sizeof(buf))) != -1)
    {
        if (n > 0)
            reply.append(buf, n);
        else
        {
            // Didn't read anything.  Is that a problem?
            // Need to try to process the content of 'reply' and see if
            // it's what we're expecting, or if it seems to be incomplete.
            //
            // Let's assume that if we've already read more than 4 characters
            // it's a complete response and there's no more to come:
            if (reply.length() > 3)
                break;
        }
    }
    cout << reply << std::flush;
    
    proc << "foo" << std::endl;
    
    while (getline(proc, reply))   // maybe use readsome again here
        cout << reply << std::endl;
    
    cout << "end" << endl;
    

This loops while `readsome() != -1`, so it keeps retrying if it reads nothing
and only stops the loop if there's an error. In the loop body it decides what
to do if nothing was read. You'll need to insert your own logic in here that
makes sense for whatever you're trying to do, but basically if `readsome()`
hasn't read _anything_ yet, then you should loop and retry. That makes the C++
program wait long enough for the Python program to print something.

You'd probably want to split out the `while` loop into a separate function
that reads a whole reply into a `std::string` and returns it, so that you can
re-use that function every time you want to read a response. If the child
sends some sentinel value that function would be easy to write, as it would
just stop every time it receives the sentinel string.

