Consume Redis messages with a pool of workers
Question: I have a Redis list where a publisher pushes some messages (JSON serialized).

On the other side the subscriber can fetch each JSON blob and do something.
The simplest way is to do this serially. But I'd like to make it a little bit
faster; I'd like to maintain a pool of worker processes (multiple consumers)
and whenever a new message arrive check if there is a "free" process from the
pool that can start processing I'm looking for Pool-based version of the
following

    
    
    while not False:
        _, new_user = conn.blpop('queue:users')
        if not new_user:
            continue
        try:
            process_new_user(new_user, conn)
        except Exception as e:
            print e
        else:
            pass
    

However I can't translate this into code that uses pythons
multiprocessing.Pool class. The documentation doesn't help either


Answer: 
    from multiprocessing import Pool
    
    
    pool = Pool()
    
    while 1:
        _, new_user = conn.blpop('queue:users')
    
        if not new_user:
            continue
    
        pool.apply_async(process_new_user, args=(new_user, conn))
    

If you want to handle exceptions you need to collect the `AsyncResult` objects
returned by `pool.apply_async` and check their status.

If you can use Python 3,
[concurrent.futures](https://docs.python.org/3.4/library/concurrent.futures.html?highlight=concurrent.futures#module-
concurrent.futures) Pools allow to handle results in asynchronous callbacks
making easier to check the jobs exit status.

