Compute the outputs of a generator in parallel
Here's a condensed illustration of how to achieve your stated purpose, namely
to compute the outputs of a generator in parallel. I offer it because
I could not understand the purpose of most of the complexity in your current
code. I suspect there are issues that you have not explained to us or that I
failed to infer (if so, this answer might be off the mark). In any case,
sometimes it helps to look at a problem in a simplified form to see
whether your actual use case might work just fine within those parameters.
import multiprocessing as mp

def main():
    for chunk in process_data(1000, 50):
        print()
        print(len(chunk), chunk)
    
def process_data(n, chunksize):
    # Set up the Pool using a context manager.
    # This relieves you of the hassle of join/close.
    with mp.Pool() as p:
        s = set()
        # Just iterate over the results as they come in.
        # No need to check for empty queues, etc.
        for res in p.imap(worker, source_gen(n)):
            s.update(res)
            if len(s) &gt;= chunksize:
                yield s
                s = set()

def source_gen(n):
    # A data source that will yield N values.
    # To get the data in this example:
    # curl 'https://www.gutenberg.org/files/2600/2600-0.txt' -o war-peace.txt
    with open('war-peace.txt') as fh:
        for i, line in enumerate(fh):
            yield line
            if i &gt;= n:
                break

def worker(line):
    # A single-argument worker function.
    # If you need multiple args, bundle them in tuple/dict/etc.
    return [word.lower() for word in line.split()]

if __name__ == &quot;__main__&quot;:
    main()

This example illustrates a garden-variety type of parallelization:

The data source is running in a single process (the parent).

Downstream computation is
running in multiple child processes.

And final aggregation/reporting
is running in a single process (also the parent).


Behind the scenes, Pool.imap() is
using pickling and queues to shuttle data between processes. If your real
use case requires parallelizing data generation and/or reporting,
a different approach is needed.