Multiprocessing with python3 only runs once
Question: I have a problem running multiple processes in python3 .

My program does the following: 1\. Takes entries from an sqllite database and
passes them to an input_queue 2\. Create multiple processes that take items
off the input_queue, run it through a function and output the result to the
output queue. 3\. Create a thread that takes items off the output_queue and
prints them (This thread is obviously started before the first 2 steps)

My problem is that currently the 'function' in step 2 is only run as many
times as the number of processes set, so for example if you set the number of
processes to 8, it only runs 8 times then stops. I assumed it would keep
running until it took all items off the input_queue.

Do I need to rewrite the function that takes the entries out of the database
(step 1) into another process and then pass its output queue as an input queue
for step 2?

Edit: Here is an example of the code, I used a list of numbers as a substitute
for the database entries as it still performs the same way. I have 300 items
on the list and I would like it to process all 300 items, but at the moment it
just processes 10 (the number of processes I have assigned)

    
    
    #!/usr/bin/python3
    from multiprocessing import Process,Queue
    import multiprocessing
    from threading import Thread
    
    
    ## This is the class that would be passed to the multi_processing function
    class Processor:
        def __init__(self,out_queue):
            self.out_queue = out_queue
        def __call__(self,in_queue):
            data_entry = in_queue.get()
            result = data_entry*2
            self.out_queue.put(result)
    
    
    
    #Performs the multiprocessing
    def perform_distributed_processing(dbList,threads,processor_factory,output_queue):
        input_queue = Queue()
    
    
        # Create the Data processors.
        for i in range(threads):
            processor  = processor_factory(output_queue)
            data_proc = Process(target = processor,
                                args   = (input_queue,))
    
            data_proc.start()
    
        # Push entries to the queue.
    
        for entry in dbList:
            input_queue.put(entry)
    
    
        # Push stop markers to the queue, one for each thread.
    
        for i in range(threads):
            input_queue.put(None)
    
        data_proc.join()
        output_queue.put(None)
    
    
    if __name__ == '__main__':
        output_results   = Queue()
    
        def output_results_reader(queue):
            while True:
                item = queue.get()
                if item is None:
                    break
                print(item)
    
    
        # Establish results collecting thread.
        results_process = Thread(target = output_results_reader,args   = (output_results,))
        results_process.start()
    
        # Use this as a substitute for the database in the example
        dbList = [i for i in range(300)]
    
        # Perform multi processing
        perform_distributed_processing(dbList,10,Processor,output_results)
    
        # Wait for it all to finish.
        results_process.join()
    


Answer: Don't try to rewrite the whole multiprocessing library again. I think you can
use any of `multiprocessing.Pool` methods depending on your needs - if this is
a batch job you can even use the synchronous `multiprocessing.Pool.map()` \-
only instead of pushing to input queue, you need to write a generator that
yields input to the threads.

