Python3 filling a dictionary concurrently
Question: I want to fill a dictionary in a loop. Iterations in the loop are independent
from each other. I want to perform this on a cluster with thousands of
processors. Here is a simplified version of what I tried and need to do.

    
    
    import multiprocessing
    
    class Worker(multiprocessing.Process):
       def setName(self,name):
           self.name=name
       def run(self):
           print ('In %s' % self.name)
           return
    
    if __name__ == '__main__':
       jobs = []
       names=dict()
       for i in range(10000):
           p = Worker()
           p.setName(str(i))
           names[str(i)]=i
           jobs.append(p)
           p.start()
       for j in jobs:
           j.join()
    

I tried this one in python3 on my own computer and received the following
error:

    
    
        ..
        In 249
        Traceback (most recent call last):
          File "test.py", line 16, in <module>
            p.start()
          File         "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/process.py", line 105, in start
        In 250
            self._popen = self._Popen(self)
          File         "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/context.py", line 212, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
          File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/context.py", line 267, in _Popen
    return Popen(process_obj)
          File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
          File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/popen_fork.py", line 66, in _launch
    parent_r, child_w = os.pipe()
        OSError: [Errno 24] Too many open files
    

Is there any better way to do this?


Answer: `multiprocessing` talks to its subprocesses via pipes. Each subprocesses
requires two open file descriptors, one for read and one for write. If you
launch 10000 workers, you'll end opening 20000 file descriptors which exceeds
the default limit on OS X (which your paths indicate you're using).

You can fix the issue by raising the limit. See
<http://superuser.com/questions/433746/is-there-a-fix-for-the-too-many-open-
files-in-system-error-on-os-x-10-7-1> for details - basically, it amounts to
setting two sysctl knobs and upping your shell's ulimit setting.

