Persist Read-Only Data Across Jobs via Python Multiprocessing Process Subclass
Question: I am using the [Python multiprocessing
module](https://docs.python.org/2/library/multiprocessing.html) and am looking
for a way to attach read only data once when the process is constructed. I
want this data to persist across multiple jobs.

I planned to subclass Process and attach data to the class, something like
this:

    
    
    import multiprocessing
    
    class Worker(multiprocessing.Process):
        _lotsofdata = LotsOfDataHolder()
    
        def run(self, arg):
            do something with _lotsofdata
            return value
    
    if __name__ == '__main__':
        jobs = []
        for i in range(5):
            p = Worker()
            jobs.append(p)
            p.start()
        for j in jobs:
            j.join()
    

However, the number of jobs is on the order of 500k so I would rather use the
Pool construct and I don't see a way to tell Pool to use a subclass of
process.

Is there a way to tell Pool to use a subclass of Process or is there another
way to persist data on a worker for multiple jobs that works with Pool?

Note: There is along explanation
[here](http://stackoverflow.com/questions/659865/python-multiprocessing-
sharing-a-large-read-only-object-between-processes), but subclassing process
was not specifically discussed.

*I see now that the args are passed to the process constructor. This makes my approach all the more unlikely. 


Answer: As [explained in this answer](http://stackoverflow.com/a/659888/1595865),
**multiple processes don't share the same memory space**. This makes
statements like `persist data on a worker for multiple jobs` meaningless:
there's no way for a worker to access any other worker's data.

What multiprocessing _can_ do is **copying** the same initial data over
workers. This happens auto-magically:

    
    
    import multiprocessing
    
    _lotsofdata = [0]*1000
    def run(arg):
        return arg+_lotsofdata[0]
    
    pool= multiprocessing.Pool()
    l=[1,2,3]
    print pool.map(run, l)
    

* * *

If you don't want to copy memory, you're left to implement your own (OS
dependent) mechanism for sharing state between processes. There's several
approaches for that outlined in the linked answer.

Realistically, unless you're trying to do supercomputations on a cluster with
dozens of CPUs, I'd think twice before going down that path.

