Async Thread Deadlock in Julia When Used With PyCall
Question: I'm trying to implement a basic client-server socket structure using Python
and Julia, where the producer is in Python and the consumer is in Julia.

My code on the Python side looks like this:

    
    
    def startServer(host='127.0.0.1', port=4002):
      connected = False
      s = socket.socket()
      s.bind((host, port))
      s.listen(5)
      scon, addr = s.accept()
      print 'Got connection from', addr
      return scon, addr
    

On the Julia side, it looks like this:

    
    
    using PyCall
    
    @pyimport server as sdlib
    
    @async begin
      sleep(10)
      print("In the async thread\n")
      s,a = sdlib.startServer("127.0.0.1",4002)
      print("Server started\n")
    end
    
    print("After the async thread\n")
    print("Connecting...\n")
    connected = false
    while !connected
      try
        connected = true
        c = connect(4002)
        print("Connected = $(connected), $(c)\n")
      catch ex
        print("$(ex)\n")
        connected = false
        sleep(1)
      end
    end
    print("Connection established: $(c)\n")
    

The output looks like this:

    
    
    After the async thread
    Connecting...
    connect: connection refused (ECONNREFUSED)
    connect: connection refused (ECONNREFUSED)
    connect: connection refused (ECONNREFUSED)
    connect: connection refused (ECONNREFUSED)
    connect: connection refused (ECONNREFUSED)
    connect: connection refused (ECONNREFUSED)
    connect: connection refused (ECONNREFUSED)
    connect: connection refused (ECONNREFUSED)
    connect: connection refused (ECONNREFUSED)
    connect: connection refused (ECONNREFUSED)
    In the async thread
    

What seems to be happening is that as soon as the Python listener starts, the
thread locks waiting for a connect. Control never seems to pass back to the
main thread to allow the client to connect.

Appreciate any help I can get on this.

Thanks, Ravi


Answer: I am way outside my area of knowledge here, but as nobody answers in 10 hours,
I will make an attempt.

I think the problem is that julia does not have an aggressive scheduler. You
can use concurrency features, but they will be scheduled cooperatively in the
same process. Since PyCall does not return or call a (julia?) sleep, the
control will not return to the client. You might try addprocs or start julia
with -p 2, but I'm not sure it will help.

See also <http://docs.julialang.org/en/release-0.2/manual/parallel-computing/>
where this is much better explained.

