Tasks are distributed to producer in multiple worker process. Results from workers are sent through Queue to a Consumer at the main process. The consumer uses less time per task than the worker.The manual counting of tasks seems unsatisfying ... Can imap or anything else make that simpler?By the way, I guess my rough representation of time for hard disk reading is not quite correct because hard disk reads are buffered.import multiprocessing as mpimport timefrom datetime import datetimedef log(msg):    d = datetime.utcnow().strftime('%M:%S.%f')[:-3]    print(d + ' %s' % msg)def main():    n_proc = 3    p = mp.Pool(n_proc)    m = mp.Manager()    q = m.Queue()    size = 20    c = Consumer(q, size, n_proc)    for i in range(size):        log(Main Loop %d % i)        # 0.01 sec for (buffered) reading of hard-disk for arrays.        # values are not 0, 1, 2 ... in the actual program.        hd_arr = i  # array from hard disk        time.sleep(0.1)        p.apply_async(producer, args=(hd_arr, q))        c.invoke()    log(Processsing remaining tasks)    p.close()    p.join()    c.flush()    log(c.data)def producer(x, q):    time.sleep(1)    result = (x, x*x)    q.put(result)class Consumer():    def __init__(self, queue, size, n_proc):        self.queue = queue        # 1.0 / (0.1 + 0.1) = 5        self.lb_factor = 5 * n_proc        self.data = [None] * size        self.task_count = 0    def invoke(self, task_increment = 1):        log(  Consumer checks:)        self.task_count += task_increment        if self.task_count == self.lb_factor:            self.flush()    def flush(self):        log(    # Consumer flushes #)        for i in range(self.task_count):            x, x2 = self.queue.get()            self.data[x] = x2            time.sleep(0.1)        self.task_count = 0if __name__ == __main__:    main()Result:32:45.796 Main Loop 032:45.896   Consumer checks:32:45.897 Main Loop 132:45.997   Consumer checks:32:45.997 Main Loop 232:46.098   Consumer checks:32:46.098 Main Loop 332:46.198   Consumer checks:32:46.199 Main Loop 432:46.299   Consumer checks:32:46.299 Main Loop 532:46.400   Consumer checks:32:46.400 Main Loop 632:46.500   Consumer checks:32:46.500 Main Loop 732:46.601   Consumer checks:32:46.601 Main Loop 832:46.701   Consumer checks:32:46.701 Main Loop 932:46.802   Consumer checks:32:46.802 Main Loop 1032:46.903   Consumer checks:32:46.903 Main Loop 1132:47.003   Consumer checks:32:47.003 Main Loop 1232:47.104   Consumer checks:32:47.104 Main Loop 1332:47.204   Consumer checks:32:47.205 Main Loop 1432:47.305   Consumer checks:32:47.305     # Consumer flushes #32:51.243 Main Loop 1532:51.343   Consumer checks:32:51.344 Main Loop 1632:51.444   Consumer checks:32:51.444 Main Loop 1732:51.545   Consumer checks:32:51.545 Main Loop 1832:51.645   Consumer checks:32:51.646 Main Loop 1932:51.746   Consumer checks:32:51.746 Processsing remaining tasks32:53.497     # Consumer flushes #32:54.001 [0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256, 289, 324, 361]Simulation of Multiple Producer Consumer: Setting the boundary between consumer, producer, and monitor.
Load balancing consumer and producer with multiprocessing
python;queue;producer consumer;multiprocessing