title: Sharing many queues among processes in Python
body: <p>I am aware of <code>multiprocessing.Manager()</code> and how it can be used to create shared objects, in particular queues which can be shared between workers. There is <a href="https://stackoverflow.com/questions/9908781/sharing-a-queue-among-several-processes">this question</a>, <a href="https://stackoverflow.com/questions/3217002/how-do-you-pass-a-queue-reference-to-a-function-managed-by-pool-map-async">this question</a>, <a href="https://stackoverflow.com/questions/6832554/python-multiprocessing-how-do-i-share-a-dict-among-multiple-processes?rq=1">this question</a> and even <a href="https://stackoverflow.com/questions/29571671/basic-multiprocessing-with-while-loop">one of my own questions</a>.</p>
  
  <p>However, I need to define a great many queues, each of which is linking a specific pair of processes. Say that each pair of processes and its linking queue is identified by the variable <code>key</code>.</p>
  
  <p>I want to use a dictionary to access my queues when I need to put and get data. I cannot make this work. I've tried a number of things. With <code>multiprocessing</code> imported as <code>mp</code>:</p>
  
  <p>Defining a dict like <code>for key in all_keys: DICT[key] = mp.Queue</code> in a config file which is imported by the multiprocessing module (call it <code>multi.py</code>) does not return errors, but the queue <code>DICT[key]</code> is not shared between the processes, each one seems to have their own copy of the queue and thus no communication happens. </p>
  
  <p>If I try to define the <code>DICT</code> at the beginning of the main multiprocessing function that defines the processes and starts them, like</p>
  
  <pre><code>DICT = mp.Manager().dict()    
  for key in all_keys:
      DICT[key] = mp.Queue()
  </code></pre>
  
  <p>I get the error</p>
  
  <pre><code>RuntimeError: Queue objects should only be shared between processes through
   inheritance
  </code></pre>
  
  <p>Changing to </p>
  
  <pre><code>DICT = mp.Manager().dict()    
  for key in all_keys:
      DICT[key] = mp.Manager().Queue()
  </code></pre>
  
  <p>only makes everything worse. Trying similar definitions at the head of <code>multi.py</code> rather than inside the main function returns similar errors. </p>
  
  <p>There must be a way to share many queues between processes without explicitly naming each one in the code. Any ideas?</p>
  
  <p><strong>Edit</strong></p>
  
  <p>Here is a basic schema of the program: </p>
  
  <p>1- load the first module, which defines some variables, imports <code>multi</code>, launches <code>multi.main()</code>, and loads another module which starts a cascade of module loads and code execution. Meanwhile... </p>
  
  <p>2- <code>multi.main</code> looks like this:</p>
  
  <pre><code>def main():
      manager = mp.Manager()
      pool = mp.Pool()
      DICT2 = manager.dict()
  
      for key in all_keys:
          DICT2[key] = manager.Queue()
          proc_1 = pool.apply_async(targ1,(DICT1[key],) ) #DICT1 is defined in the config file
          proc_2 =  pool.apply_async(targ2,(DICT2[key], otherargs,) 
  </code></pre>
  
  <p>Rather than use <code>pool</code> and <code>manager</code>, I was also launching processes with the following:</p>
  
  <pre><code>mp.Process(target=targ1, args=(DICT[key],))
  </code></pre>
  
  <p>3 - The function <code>targ1</code> takes input data that is coming in (sorted by <code>key</code>) from the main process. It is meant to pass the result to <code>DICT[key]</code> so <code>targ2</code> can do its work. This is the part that is not working. There are an arbitrary number of <code>targ1</code>s, <code>targ2</code>s, etc. and therefore an arbitrary number of queues. </p>
  
  <p>4 - The results of some of these processes will be sent to a bunch of different arrays / pandas dataframes which are also indexed by <code>key</code>, and which I would like to be accessible from arbitrary processes, even ones launched in a different module. I have yet to write this part and it might be a different question. (I mention it here because the answer to 3 above might also solve 4 nicely.)</p>