Feasibility of using pipe for ruby-python communication
Question: Currently, I have two programs, one running on Ruby and the other in Python. I
need to read a file in Ruby but I need first a library written in Python to
parse the file. Currently, I use XMLRPC to have the two programs communicate.
Porting the Python library to Ruby is out of question. However, I find and
read that using XMLRPC has some performance overhead. Recently, I read that
another solution for the Ruby-Python conundrum is the use of pipes. So I tried
to experiment on that one. For example, I wrote this master script in ruby:

    
    
        (0..2).each do
          slave = IO.popen(['python','slave.py'],mode='r+')
          slave.write "master"
          slave.close_write
          line = slave.readline
          while line do
            sleep 1
            p eval line
            break if slave.eof
            line = slave.readline
          end
        end
    

The following is the Python slave:

    
    
        import sys
    
        cmd = sys.stdin.read()
        while cmd:
          x = cmd
          for i in range(0,5):
            print "{'%i'=>'%s'}" % (i, x)
            sys.stdout.flush()
            cmd = sys.stdin.read()
    

Everything seems to work fine:

    
    
        ~$ ruby master.rb
        {"0"=>"master"}
        {"1"=>"master"}
        {"2"=>"master"}
        {"3"=>"master"}
        {"4"=>"master"}
        {"0"=>"master"}
        {"1"=>"master"}
        {"2"=>"master"}
        {"3"=>"master"}
        {"4"=>"master"}
        {"0"=>"master"}
        {"1"=>"master"}
        {"2"=>"master"}
        {"3"=>"master"}
        {"4"=>"master"}
    

My question is, is it really feasible to implement the use of pipes for
working with objects between Ruby and Python? One consideration is that there
may be multiple instances of master.rb running. Will concurrency be an issue?
Can pipes handle extensive operations and objects to be passed in between? If
so, would it be a better alternative for RPC?


Answer: Yes. No. If you implement it, yes. Depends on what your application needs.

Basically if all you need is simple data passing pipes are fine, if you need
to be constantly calling functions on objects in your remote process then
you'll probably be better of using some form of existing RPC instead of
reinventing the wheel. Whether that should be XMLRPC or something else is
another matter.

Note that RPC will have to use some underlying IPC mechanism, which could well
be pipes. but might also be sockets, message queues, shared memory, whatever.

