Strange Queue.PriorityQueue behaviour with multiprocessing in Python 2.7.6
Question: As you know from the title, I'm trying to use PriorityQueue with
multiprocessing. More precisely, I wanted to make shared PriorityQueue, wrote
some code and it doesn't run as I expected.

Look at the code:

    
    
    import time
    from multiprocessing import Process, Lock
    from Queue import PriorityQueue
    
    
    def worker(queue):
        lock = Lock()
        with lock:
            for i in range(100):
                queue.put(i)
    
        print "worker", queue.qsize()
    
    
    pr_queue = PriorityQueue()
    worker_process = Process(target = worker, args = (pr_queue,))
    worker_process.start()
    
    time.sleep(5)    # nope, race condition, you shall not pass (probably)
    print "main", pr_queue.qsize()
    

Got the following output:

    
    
    worker 100
    main 0
    

What's happening and how to do what I want the right way? Thank you.


Answer: The problem isn't that it's not picklable in this case - if you're using a
Unix-like platform, the queue can be passed to the child without pickling. (On
Windows, I think you would get a pickling error here, though). The root
problem is that you're not using a process-safe queue. The only queues that
can be used between processes are [the `Queue`
objects](https://docs.python.org/2/library/multiprocessing.html#pipes-and-
queues) that live inside the `multiprocessing` module. Unfortunately, there is
no `PriorityQueue` implementation available. However, you can easily create
one by registering a `PriorityQueue` with a
[`multiprocessing.Manager`](https://docs.python.org/2/library/multiprocessing.html#managers)
class, like this:

    
    
    import time
    from multiprocessing import Process
    from multiprocessing.managers import SyncManager
    from Queue import PriorityQueue
    
    
    class MyManager(SyncManager):
        pass
    MyManager.register("PriorityQueue", PriorityQueue)  # Register a shared PriorityQueue
    
    def Manager():
        m = MyManager()
        m.start()
        return m
    
    def worker(queue):
        print(queue)
        for i in range(100):
            queue.put(i)
        print "worker", queue.qsize()
    
    
    m = Manager()
    pr_queue = m.PriorityQueue()  # This is process-safe
    worker_process = Process(target = worker, args = (pr_queue,))
    worker_process.start()
    
    time.sleep(5)    # nope, race condition, you shall not pass (probably)
    print "main", pr_queue.qsize()
    

Output:

    
    
    worker 100
    main 100
    

Note that this probably won't perform quite as well as it would if it was
standard `multiprocessing.Queue` subclass. The `Manager`-based `PriorityQueue`
is implemented by creating a `Manager` server process which actually contains
a regular `PriorityQueue`, and then providing your main and worker processes
with [`Proxy`](https://docs.python.org/2/library/multiprocessing.html#proxy-
objects) objects that use IPC to read/write to the queue in the server
process. Regular `multiprocessing.Queue`s just write/read data to/from a
`Pipe`. If that's a concern, you could try implementing your own
`multiprocessing.PriorityQueue` by subclassing or delegating from
`multiprocessing.Queue`. It may not be worth the effort, though.

