Simultaneously using multiple views in the iPython Notebook
Question: I've got a question I'm hoping someone can help me figure out. I'm trying to
construct two different parallel views in an iPython notebook. The first view
has the processor with ID 0, and the second has all the rest of the
processors. I associate a prefix with each of the views so I can easily run
different things on the different processors.

I launch a background thread that does a long calculation using the processors
in the second view. While that's running in the background, I try to run a
command using the first view, but it doesn't work. I get this error:
ValueError: '' is not in list.

So I'm wondering if there's a way to do what I'm trying to do here, or if this
is unsupported behavior. In short, I'd like to create two different views
using different processors. No processors will be shared between the views.
Then I'd like to be able to run a background task that uses one view, while
simultaneously using the other view for unrelated tasks.

Here's a small example script that results in the error. I'm not sure how to
post a notebook directly, so I've just copied and pasted the python script
generated from it.

    
    
    # <codecell>
    
    from IPython import parallel
    cli = parallel.Client()
    
    # <codecell>
    
    view1 = cli[0]
    view1.block = True
    view1.activate("_one")
    
    # <codecell>
    
    view2 = cli[1:]
    view2.block = True
    view2.activate("_two")
    
    # <codecell>
    
    %px_two import time
    def backFunc():
        for i in range(10):
            %px_two time.sleep(5)
            %px_two print "In bg thread"
    
    # <codecell>
    
    from IPython.lib import backgroundjobs as bg
    bgJob = bg.BackgroundJobManager()
    bgJob.new('backFunc()')
    
    # <codecell>
    
    %px_one import time
    def foreFunc():
        for i in range(10):
            %px_one time.sleep(1)
            %px_one print "In fg thread"
    
    
    # <codecell>
    
    foreFunc()
    

As soon as foreFunc() is run, it gives the error:

    
    
    ValueError: '<IDS|MSG>' is not in list
    

Any thoughts? I'd appreciate any ideas anyone has.


Answer: ## Short Answer

The sockets used by the Client are not threadsafe, so you cannot use them
simultaneously in multiple threads. You can use the _cluster_ simultaneously,
but you need to create a separate Client for the background task, which will
have its own set of sockets:

    
    
    rc = parallel.Client()
    rc2 = parallel.Client()
    
    view1 = rc[0]
    view2 = rc2[1:]
    

And the rest should work as expected.

## PS: What's going on

A Client object is mostly an API around a collection of sockets. Each Client
has its own set of sockets, and all Views on one Client use the same sockets.
When you share those sockets across threads, it's possible for one thread to
get part of a message intended for another thread, garbling the message.

Each message is actually a multi-part message zeromq message, sent or received
with
[zmq.Socket.send/recv_multipart](https://github.com/zeromq/pyzmq/blob/v14.4.1/zmq/sugar/socket.py#L275),
which amounts to:

    
    
    multipart = []
    for i in range(nparts):
        multipart.append(socket.send/recv())
    

If two threads are doing this at the same time on the same socket, it's
possible for messages to get interleaved, so instead of getting two messages:

    
    
    ['a1', 'a2', 'a3'], ['b1', 'b2', 'b3']
    

we get

    
    
    ['a1', 'a2', 'b1', 'b2', 'b3'], ['a3']
    

causing the problem you are seeing. The simplest fix is to use different
sockets in different threads. An alternate fix is to use
[locking](https://docs.python.org/2/library/threading.html#lock-objects) to
ensure that the multi-part messages are received atomically. Separating
sockets per-thread allows you to avoid the need for locking, but it does
increase the number of sockets you need to use proportionally to the number of
concurrent threads.

## PPS ...but IPython.parallel is async

I will finish by asking why you are using the Background job at all. You do
not need to use threads to accomplish the task you described, because the
Client normally doesn't wait for results from the engines. IPython.parallel is
async by nature, so you don't need to wait for jobs to finish in order to
submit new ones, or do work locally in your interactive session. I generally
do not recommend using `block=True` for anything other than debugging.

