title: python multiprocessing.pool.map, passing arguments to spawned processes
body: <pre><code>def content_generator(applications, dict):
      for app in applications:
          yield(app, dict[app])
  
  with open('abc.pickle', 'r') as f:
      very_large_dict = pickle.load(f)
  all_applications = set(very_large_dict.keys())
  
  pool = multiprocessing.Pool()
  for result in pool.imap_unordered(func_process_application, content_generator(all_applications, very_large_dict)):
      do some aggregation on result
  </code></pre>
  
  <p>I have a really large dictionary whose keys are strings (application names), values are information concerning the application. Since applications are independent, I want to use multiprocessing to process them in parallel. Parallelization works when the dictionary is not that big but all the python processes were killed when the dictionary is too big. I used <code>dmesg</code> to check what went wrong and found they were killed since the machine ran out of memory. I did <code>top</code> when the pool processes are running and found that they all occupy the same amount of resident memory(RES), which is all 3.4G. This confuses me since it seems to have copied the whole dictionaries into the spawned processes. I thought I broke up the dictionary and passing only what is relevant to the spawned process by yielding only <code>dict[app]</code> instead of <code>dict</code>. Any thoughts on what I did wrong?</p>