Python worker pattern for multithreading
1. Introduction

From a practical point of view, the most important points are that Python has batteries included:


To run tasks in a pool of worker threads, use concurrent.futures.ThreadPoolExecutor.
But if what you really want to do is run external programs via the shell, as suggested by the use of os.system, then you don't need threads at all! Use subprocess.Popen instead.

(Note that on some operating systems, notably macOS, you can only have one call to os.system running at a time. Subsequent calls, even from other threads, have to wait for the first call to finish. On these systems you have to use subprocess.Popen if you want the subprocesses to run in parallel.)


Nonetheless I think it's a good exercise to have a go at writing your own thread pool, and so I'm go to look at how you might improve the code in the post. There's quite a lot of ground to cover, so I'm going to take it in stages, starting by reviewing improving the code in the post, and then identifying and fixing various problems with the design.

2. Initial review


There are no docstrings. How do I use this module? Which classes are public and which are private? Do I create my own Worker objects or do I let the Pool create them for me? What kind of object do I pass for queue? How should I specify the timeout?
The Pool constructor takes timeout and output arguments but these have no effect: they are passed to Worker.__call__ but then they are ignored.
The Pool class has only one method (other than __init__). When you have a class with just one method, then what you need is a function. Not everything needs to be a class! It would simplify the code if it were turned into a function with a specification like this:

def run_shell_commands_in_parallel(commands, max_threads):
    """Run shell commands in parallel. max_threads is the maximum
    number of threads that may run simultaneously.

    """

Running a shell command is not very general. It would make sense to generalize the code to do arbitrary function calls in parallel:

def apply_in_parallel(fun, args, max_threads=4):
    """Apply fun to each of the arguments in parallel. max_threads is the
    maximum number of threads that may run simultaneously.

    """


To try to run a bunch of shell commands in parallel, you'd call it like this:

apply_in_parallel(os.system, ['command1', 'command2', ...])


but now you can use it for other parallel tasks:

apply_in_parallel(print, range(10))

The main thread calls workers.pop(0) and the worker threads call workers.append(self), but the workers data structure is just an ordinary Python list, which is not thread-safe. Whenever you have a data structure that's shared between threads, access to the data structure needs to be synchronized, for example by using a lock to ensure that only one thread can be updating it at a time.
All the Worker objects are identical — their only attribute is self.pool, and that's the same for every worker, so instead of a list of workers all we really need is a count of how many workers are idle.


3. Revised code

Fixing the issues in §1 above yields the following, which does essentially the same thing as the code in the post, but in a simpler way:

def apply_in_parallel(fun, args, max_threads=4):
    """Apply fun to each of the arguments in parallel. max_threads is the
    maximum number of threads to run simultaneously.

    """
    available_workers = max_threads
    available_workers_lock = Lock()

    def worker(arg):
        nonlocal available_workers
        fun(arg)
        with available_workers_lock:
            available_workers += 1

    for arg in args:
        while True:
            with available_workers_lock:
                if available_workers:
                    available_workers -= 1
                    break
        Thread(target=worker, args=(arg,)).start()


4. Problem: busy-waiting

The while True: loop is busy-waiting. If it takes a long time for a worker to become available, the main thread will be wasting effort by repeatedly taking the lock and testing the condition if available_workers: which remains false.

What we'd like instead is to be able to suspend the main thread until a worker becomes available. For this use case what we need is a semaphore, and Python provides us with threading.Semaphore:

from threading import Semaphore, Thread

def apply_in_parallel(fun, args, max_threads=4):
    """Apply fun to each of the arguments in parallel. max_threads is the
    maximum number of threads that may run simultaneously.

    """
    available = Semaphore(max_threads)

    def worker(arg):
        fun(arg)
        available.release()

    for arg in args:
        available.acquire()
        Thread(target=worker, args=(arg,)).start()


5. Problem: can't tell when threads are done

The code in §4 starts the shell commands but does not wait for them all to complete. But in many use cases it is important to wait for workers to finish running before proceding.

To ensure that all threads have finished, we must join them before returning:

from threading import Semaphore, Thread

def apply_in_parallel(fun, args, max_threads=4):
    """Apply fun to each of the arguments in parallel and wait until all
    calls have completed. max_threads is the maximum number of threads
    that may run simultaneously.

    """
    available = Semaphore(max_threads)

    def worker(arg):
        fun(arg)
        available.release()

    threads = []
    for arg in args:
        available.acquire()
        thread = Thread(target=worker, args=(arg,))
        threads.append(thread)
        thread.start()

    for thread in threads:
        thread.join()


6. Problem: too many threads

The code in §5 creates a new thread for every task, and every thread exits after completing its task. This defeats one of the purposes of having a pool of threads, namely that it avoids some of the overhead of creating new threads by reusing the threads in the pool.

The usual approach to implementing a pool of threads is for each thread to execute another task when it is finished (so long as there are more tasks to run). That means that the main thread is going to need a safe way of sending the tasks to the running threads, and Python provides queue.Queue which is exactly what we want:

from queue import Queue
from threading import Thread

def apply_in_parallel(fun, args, max_threads=4):
    """Apply fun to each of the arguments in parallel and wait until all
    calls have completed. max_threads is the maximum number of threads
    that may run simultaneously.

    """
    queue = Queue()

    def worker():
        while True:
            arg = queue.get()
            fun(arg)
            queue.task_done()

    for _ in range(max_threads):
        Thread(target=worker).start()
    for arg in args:
        queue.put(arg)
    queue.join()


7. Problem: threads are left dangling

The problem with the implementation in §6 is that the worker threads never finish! When all the tasks are done, each worker thread is blocked in queue.get waiting for the next task, but it will never arrive. This is a dangerous resource leak because the number of threads that we can create is limited by the operating system, and so to ensure that we can continue to create new threads later in the program, we must clean up all the threads that we started.

So when all tasks are done, we must tell the threads to exit. A convenient way to do that is to pass in a special sentinel argument that is different from any argument that the caller could pass in:

from queue import Queue
from threading import Thread

def apply_in_parallel(fun, args, max_threads=4):
    """Apply fun to each of the arguments in parallel and wait until all
    calls have completed. max_threads is the maximum number of threads
    that may run simultaneously.

    """
    queue = Queue()
    sentinel = object()

    def worker():
        while True:
            arg = queue.get()
            if arg is sentinel:
                break
            fun(arg)

    threads = []
    for _ in range(max_threads):
        thread = Thread(target=worker)
        threads.append(thread)
        thread.start()
    for arg in args:
        queue.put(arg)
    for _ in range(max_threads):
        queue.put(sentinel)
    for thread in threads:
        thread.join()