title: Dynamically updating a nested dictionary with multiprocessing.pool (speed issue)
body: <p>I have written a simple code to understand how lack of communication between the child processes leads to a random result when using <code>multiprocessing.Pool</code>. I input a nested dictionary as a dictproxy object made by <code>multiprocessing.Manager</code>:</p>
  
  <pre><code>manager = Manager()
  my_dict = manager.dict()
  my_dict['nested'] = nested
  </code></pre>
  
  <p>into a pool embedding 16 open processes. The nested dictionary is defined below. The function <code>my_function</code> simply generates the second power of each number stored in the elements of the nested dictionary. </p>
  
  <p>As expected because of the shared memory in multithreading, I get the correct result when I use <code>multiprocessing.dummy</code> </p>
  
  <pre><code>{0: 1, 1: 4, 2: 9, 3: 16}
  {0: 4, 1: 9, 2: 16, 3: 25}
  {0: 9, 1: 16, 2: 25, 3: 36}
  {0: 16, 1: 25, 2: 36, 3: 49}
  {0: 25, 1: 36, 2: 49, 3: 64}
  </code></pre>
  
  <p>but when I use <code>multiprocessing</code>, the result is incorrect and completely random in each run. One example of the incorrect result is:</p>
  
  <pre><code>{0: 1, 1: 2, 2: 3, 3: 4}
  {0: 4, 1: 9, 2: 16, 3: 25}
  {0: 3, 1: 4, 2: 5, 3: 6}
  {0: 16, 1: 25, 2: 36, 3: 49}
  {0: 25, 1: 36, 2: 49, 3: 64}
  </code></pre>
  
  <p>In this particular run, the <code>'data'</code> in <code>'element'</code> 1 and 3 was not updated. I understand that this happens due to the lack of communication between the child processes which prohibits the "updated" nested dictionary in each child process to be properly sent to the others. However, can someone help me use <code>Manager.Queue</code> to organize this inter-child communication and get the correct results possibly with minimal runtime?</p>
  
  <p>Code (Python 3.5)</p>
  
  <pre><code>from multiprocessing import Pool, Manager
  import numpy as np
  
  def my_function(A):
      arg1 = A[0]
      my_dict = A[1]
  
      temporary_dict = my_dict['nested']
      for arg2 in np.arange(len(my_dict['nested']['elements'][arg1]['data'])):
          temporary_dict['elements'][arg1]['data'][arg2] = temporary_dict['elements'][arg1]['data'][arg2] ** 2 
  
      my_dict['nested'] = temporary_dict
  
  
  if __name__ == '__main__':
  
  
      # nested dictionary definition
      strs1 = {}
      strs2 = {}
      strs3 = {}
      strs4 = {}
      strs5 = {}
      strs1['data'] = {}
      strs2['data'] = {}
      strs3['data'] = {}
      strs4['data'] = {}
      strs5['data'] = {}
  
      for i in [0,1,2,3]:
          strs1['data'][i] = i + 1
          strs2['data'][i] = i + 2
          strs3['data'][i] = i + 3
          strs4['data'][i] = i + 4
          strs5['data'][i] = i + 5
  
      nested = {}
      nested['elements'] = [strs1, strs2, strs3, strs4, strs5]
      nested['names'] = ['series1', 'series2', 'series3', 'series4', 'series5']
  
      # parallel processing
      pool = Pool(processes = 16)
      manager = Manager()
      my_dict = manager.dict()
      my_dict['nested'] = nested
  
      sequence = np.arange(len(my_dict['nested']['elements']))
      pool.map(my_function, ([seq,my_dict] for seq in sequence))
  
      pool.close()
      pool.join()
  
      # printing the data in all elements of the nested dictionary
      print(my_dict['nested']['elements'][0]['data'])
      print(my_dict['nested']['elements'][1]['data'])
      print(my_dict['nested']['elements'][2]['data'])
      print(my_dict['nested']['elements'][3]['data'])
      print(my_dict['nested']['elements'][4]['data'])
  </code></pre>
  
  <p>One way to go around this and get correct results would be using <code>multiprocessing.Lock</code>, but that kills the speed:</p>
  
  <pre><code>from multiprocessing import Pool, Manager, Lock
  import numpy as np
  
  def init(l):
      global lock
      lock = l
  
  def my_function(A):
      arg1 = A[0]
      my_dict = A[1]
  
      with lock:
          temporary_dict = my_dict['nested']
          for arg2 in np.arange(len(my_dict['nested']['elements'][arg1]['data'])):
              temporary_dict['elements'][arg1]['data'][arg2] = temporary_dict['elements'][arg1]['data'][arg2] ** 2 
  
          my_dict['nested'] = temporary_dict
  
  
  if __name__ == '__main__':
  
  
      # nested dictionary definition
      strs1 = {}
      strs2 = {}
      strs3 = {}
      strs4 = {}
      strs5 = {}
      strs1['data'] = {}
      strs2['data'] = {}
      strs3['data'] = {}
      strs4['data'] = {}
      strs5['data'] = {}
  
      for i in [0,1,2,3]:
          strs1['data'][i] = i + 1
          strs2['data'][i] = i + 2
          strs3['data'][i] = i + 3
          strs4['data'][i] = i + 4
          strs5['data'][i] = i + 5
  
      nested = {}
      nested['elements'] = [strs1, strs2, strs3, strs4, strs5]
      nested['names'] = ['series1', 'series2', 'series3', 'series4', 'series5']
  
      # parallel processing
  
      manager = Manager()
      l =  Lock()
      my_dict = manager.dict()
      my_dict['nested'] = nested
      pool = Pool(processes = 16, initializer=init, initargs=(l,))
  
      sequence = np.arange(len(my_dict['nested']['elements']))
      pool.map(my_function, ([seq,my_dict] for seq in sequence))
  
      pool.close()
      pool.join()
  
      # printing the data in all elements of the nested dictionary
      print(my_dict['nested']['elements'][0]['data'])
      print(my_dict['nested']['elements'][1]['data'])
      print(my_dict['nested']['elements'][2]['data'])
      print(my_dict['nested']['elements'][3]['data'])
      print(my_dict['nested']['elements'][4]['data'])
  </code></pre>