TaskScheduler in python I have found the need to schedule a few simple functions in a first-come-first-served way so I developed a small task scheduler based on multiprocessing and I thought it would be a good candidate for my first question here.

I'm open to all kind of comments, regarding style, conventions, possible bugs and other things I'm not considering now. I added some comments for the most sensible mechanism but this is mostly a half-day code.

from multiprocessing import Process
from multiprocessing import JoinableQueue

def worker_process(in_queue, out_queue):
    run = True
    while run:
        # in the queue there should be a Job or None.
        # Latter case is the signal that we should stop.
        task = in_queue.get()
        if task:
            task.run()
            in_queue.task_done()
            # when done, we signal and we put the task id in the output queue
            out_queue.put(task.get_id())
        else:
            run = False

class Scheduler:
    def __init__(self, job_done_cb, cb_args=[], N=2):
        self.n = N
        self.processes = []
        self.job_queue = JoinableQueue()
        self.done_queue = JoinableQueue()
        self.job_done_cb = job_done_cb
        self.cb_args = cb_args
        for i in range(self.n):
            # these workers will get jobs from the queue and give us back the id when done
            p = Process(target=worker_process, args=(self.job_queue, self.done_queue))
            self.processes.append(p)
        # there process will get back the job of the id done and execute the callback
        self.cb_process = Process(target=self.on_job_done)

    def start(self):
        for p in self.processes:
            p.start()
        self.cb_process.start()

    def stop(self):
        # add the necessary None to the queue and wait for the processes to be done
        # this will make the queue execute the remaining task before finishing
        for i in range(self.n):
            self.job_queue.put(None)
        self.job_queue.close()
        for p in self.processes:
            p.join()
        self.done_queue.put(None)
        self.cb_process.join()

    def add_task(self, task_id, task, arg_list):
        job = Job(task, task_id, arg_list)
        self.job_queue.put(job)

    def wait_for_idle(self):
        self.job_queue.join()

    def on_job_done(self):
        run = True
        while run:
            # again, if the id is not None then we should go on waiting for stuff
            done = self.done_queue.get()
            if done:
                self.job_done_cb(self, done, *self.cb_args)
            else:
                run = False

class Job:
    def __init__(self, task, job_id, arg_list):
        self.task = task
        self.id = job_id
        self.arg_list = arg_list

    def run(self):
        self.task(*self.arg_list)

    def get_id(self):
        return self.id
Unused Loop Variables

for i in range(self.n):
    self.job_queue.put(None)


can be written like this

for _ in range(self.n):
    self.job_queue.put(None)


and any place where you don't use the loop variable. The _ lets you and other programmers know that that variable isn't used and should be ignored.

Naming

p should be process to be descriptive. While p in self.processes can be obvious, it's good to get in the practice of descriptive variable names.

Type Hints

This

def add_task(self, task_id, task, arg_list):


can instead be (assuming task is a str and arg_list is a list)

def add_task(self, task_id: int, task: str, arg_list: list) -&gt; None:


Using type hints allows you to see/show what types of parameters are accepted, and what type(s) is/are returned from the function.