title: multiprocessing in python - sharing large object (e.g. pandas dataframe) between multiple processes
body: <p>I am using Python multiprocessing, more precisely</p>
  
  <pre><code>from multiprocessing import Pool
  p = Pool(15)
  
  args = [(df, config1), (df, config2), ...] #list of args - df is the same object in each tuple
  res = p.map_async(func, args) #func is some arbitrary function
  p.close()
  p.join()
  </code></pre>
  
  <p>This approach has a huge memory consumption; eating up pretty much all my RAM (at which point it gets extremely slow, hence making the multiprocessing pretty useless). I assume the problem is that <code>df</code> is a huge object (a large pandas dataframe) and it gets copied for each process. I have tried using <code>multiprocessing.Value</code> to share the dataframe without copying</p>
  
  <pre><code>shared_df = multiprocessing.Value(pandas.DataFrame, df)
  args = [(shared_df, config1), (shared_df, config2), ...] 
  </code></pre>
  
  <p>(as suggested in <a href="https://stackoverflow.com/q/14124588/2565842">Python multiprocessing shared memory</a>), but that gives me <code>TypeError: this type has no size</code> (same as <a href="https://stackoverflow.com/q/3671666/2565842">Sharing a complex object between Python processes?</a>, to which I unfortunately don't understand the answer). </p>
  
  <p>I am using multiprocessing for the first time and maybe my understanding is not (yet) good enough. Is <code>multiprocessing.Value</code> actually even the right thing to use in this case? I have seen other suggestions (e.g. queue) but am by now a bit confused. What options are there to share memory, and which one would be best in this case?</p>