Can a spawned process communicate with the "main" MPI communicator
Question: Is there a way using MPI to let spawned processes communicate with all other
actors in the MPI_WORLD and not only with the parent that spawned the process?

Now I have two main agents, the so-called master and slave that run the
following code (spawn.py):

    
    
    # Spawn test: master and first slave
    import sys
    
    from mpi4py import MPI
    comm = MPI.COMM_WORLD
    rank = comm.Get_rank()
    
    if rank == 0 : # master code
        print "i am the master on rank %i" % (rank)
        running = True
        while running :
            msg = comm.recv(source=MPI.ANY_SOURCE,tag=0)
            print "master received message: ", msg
            if msg == "Done" :
                running = False
        print "master is done"
    
    if rank == 1 : # slave code
        no_spawn = 1
        print "I am a slave on rank %i, about the spawn lower slaves" % (rank)
        icomm = MPI.COMM_SELF.Spawn(sys.executable,args=["Cpi.py","ben"],maxprocs=no_spawn)
        comm.send("Test_comm",dest=0,tag=0)
        icomm.send("Test_icomm",dest=0,tag=0)
        isize =  icomm.Get_size()
        print "on slave, isize= %i" % (isize)
        rec = 0
        while rec <= (no_spawn-1) :
            msg = icomm.recv(source=MPI.ANY_SOURCE,tag=20)
            print "slave received message: %s (rec=%i)" % (msg, rec)
            rec = rec +1
        import time
        print "slave going to sleep\n"
        time.sleep(1)
        for i in range(no_spawn) :
            message = ("To spawn from slave",)
            icomm.send(message,dest=i,tag=0)
        for i in range(no_spawn) :
            message = ("Done",)
            icomm.send(message,dest=i,tag=0)
    
        msg = comm.recv(source=MPI.ANY_SOURCE,tag=0)
        print "slave received message: ", msg
    
        comm.send("Done",dest=0,tag=0)
    
        MPI.Finalize()
    

The slave, in turn, spawns 1 more processes that runs the following code
(CPi.py, named after the mpi4py tutorial file):

    
    
    #!/usr/bin/env python
    
    import sys
    
    from mpi4py import MPI
    comm = MPI.COMM_WORLD
    icomm = MPI.Comm.Get_parent()
    irank = icomm.Get_rank()
    
    print "Spawn irank=%i" % (irank)
    message = "From_Spawn_%i"%(irank)
    icomm.send(message,dest=0,tag=20)
    
    running = True
    while running :
        msg = icomm.recv(source=MPI.ANY_SOURCE,tag=0)
        print "Spawn on irank %i received message: %s " %(irank,msg)
        if msg[0] == "Done" :
            running = False    
    
    print "spawn %i sending a last msg to the master and the slave" % (irank)
    comm.send(("To master from spawn",), dest=0,tag=0)
    comm.send(("To slave from spawn",), dest=0,tag=0)
    

Between the master and slave I can send messages by using the `comm`
communicator. Between the slave and the spawned process I can send messages
over the `icomm` communicator. But what I really want is to spawn a process
and that this process can communicate with both the master and slave over the
`comm` communicator; see the last two lines of the spawned process. Is that
possible? And would it be possible for the spawned process to listen as well
to the main `comm` used by the slave and the master? Which rank would it be
send to / listen to?

The provided code does not terminate because the last two messages send by the
spawned process are neither received by the slave or the master. (I run the
code with `mpiexec -n 2 python spawn.py`)


Answer: For the process spawned by the slave to talk to the master, it would need to
create another new communicator using something like MPI_CONNECT and
MPI_ACCEPT. It's possible to do, but you'd have to use the slave to transfer
the connection details between the two.

Before you go through all of that, be sure that you can't just start your job
with more process and arbitrarily assign different roles to different ranks.
It's a pain to use intercommunicators under the best of circumstances and it's
probably simpler to start out with the correct number of processes.

