title: Python - multiprocessing a nested dictionary
body: <p>I have tried using <a href="https://stackoverflow.com/questions/70279344/multiprocessing-with-nested-dictionary">this</a> question to answer my problem, but I haven't had any success.</p>
  <p>I'm using <strong>Python 3.10</strong>.</p>
  <p>My <code>dictionary</code> is structured like this <em>(where each list of string is a review of the product)</em>:</p>
  <pre><code>{storeNameA : {productA : 0 [string, string, ..., string]
                            1 [string, string, ..., string]
                            2 [string, string, ..., string]
                            ...
                            n [string, string, ..., string], 
                 productB : 0 [string, string, ..., string]
                            1 [string, string, ..., string]
                            2 [string, string, ..., string]
                            ...
                            n [string, string, ..., string],
                 ...,
                 product_n : 0 [string, string, ..., string]
                             1 [string, string, ..., string]
                             2 [string, string, ..., string]
                             ...
                             n [string, string, ..., string]},
   storeNameB : {productA : 0 [string, string, ..., string]
                            1 [string, string, ..., string]
                            2 [string, string, ..., string]
                            ...
                            n [string, string, ..., string], 
                 productB : 0 [string, string, ..., string]
                            1 [string, string, ..., string]
                            2 [string, string, ..., string]
                            ...
                            n [string, string, ..., string],
                 ...,
                 product_n : 0 [string, string, ..., string]
                             1 [string, string, ..., string]
                             2 [string, string, ..., string]
                             ...
                             n [string, string, ..., string]}}
  </code></pre>
  <p>So I would access a single 'review' like <code>dictionary['storeNameA']['productB'][0]</code><br />
  or <code>dictionary['storeNameB']['productB'][2]</code>. Each <code>product</code> is the same in each store.</p>
  <p>I am trying to perform a process on each review across the entire dictionary.
  I can perform this sucessfuly in an iterative manner with this code:</p>
  <pre class="lang-py prettyprint-override"><code>def mapAllValues(nestedDict, func):
      return {storeName: {product: func(prodFile) for product, prodFile in storeDict.items()} for storeName, storeDict in nestedDict.items()}
  
  new_dictionary = mapAllValues(dictionary, lambda reviews: reviews.apply(processFunction))
  # processFunction takes a list of string and returns a list of tuples.
  # So I end up with a new dictionary where there is now a list of tuples, where there was a list of string.
  # {storeName : {product : 0 [(str, str), (str, str), ..., (str, str)]    and so on...
  </code></pre>
  <p>It's a pretty long dictionary, and takes <strong>~606</strong> seconds to complete.<br />
  So, I have tried to implement a way to run this in parallel, but it's obviously not working as I expect it to because that runs in <strong>~2170</strong> seconds. I do get the right output though.</p>
  <p>My question is, what am I doing wrong in the following code please?
  Can anyone provide me a solution to this problem?</p>
  <pre class="lang-py prettyprint-override"><code>manager = multiprocessing.Manager()
  container = manager.dict()
      d = manager.dict(dictionary)
      container = manager.dict()
      for key in d:
          container[key] = manager.dict()
      for key in d['storeNameA']:
          container['storeNameA'][key] = manager.dict()
      for key in d['storeNameB']:
          container['storeNameB'][key] = manager.dict()
      
      with multiprocessing.Pool() as pool:
          pool.starmap(processFunction, [('storeNameA', product, d, container) for product in d['storeNameA']], chunksize=round(42739 / multiprocessing.cpu_count()))
          pool.starmap(processFunction, [('storeNameB', product, d, container) for product in d['storeNameB']], chunksize=round(198560 / multiprocessing.cpu_count()))
      
  new_dictionary = dict(container)
  </code></pre>
  <p>I'm sure I'm misunderstanding how this is actually working, but as I see it it should be chunking each product from each store and parellising those?</p>
  <p>Anyway, I think I've explained it as well as I can. If I need to clarify anything, please let me know.<br />
  Thank you in advance!</p>