title: python multiprocessing - OverflowError('cannot serialize a bytes object larger than 4GiB')
body: <p>We are running a script using the <code>multiprocessing</code> library (<code>python 3.6</code>), where a big <code>pd.DataFrame</code> is passed as an argument to a function :</p>
  
  <pre><code>from multiprocessing import Pool
  import time 
  
  def my_function(big_df):
      # do something time consuming
      time.sleep(50)
  
  if __name__ == '__main__':
      with Pool(10) as p:
          res = {}
          output = {}
          for id, big_df in some_dict_of_big_dfs:
              res[id] = p.apply_async(my_function,(big_df ,))
          output = {u : res[id].get() for id in id_list}
  </code></pre>
  
  <p>The problem is that we are getting an error from the <code>pickle</code> library.</p>
  
  <blockquote>
    <p>Reason: 'OverflowError('cannot serialize a bytes objects larger than
    4GiB',)'</p>
  </blockquote>
  
  <p>We are aware than <code>pickle v4</code> can serialize larger objects <a href="https://stackoverflow.com/questions/29704139/pickle-in-python3-doesnt-work-for-large-data-saving">question related</a>, <a href="https://www.python.org/dev/peps/pep-3154/" rel="nofollow noreferrer">link</a>, but we don't know how to modify the protocol that <code>multiprocessing</code> is using. </p>
  
  <p>does anybody know what to do? 
  Thanks !!</p>