title: MemoryError with Pickle in Python
body: <p>I am processing some data and I have stored the results in three dictionaries, and I have saved them to the disk with Pickle. Each dictionary has 500-1000MB. </p>
  
  <p>Now I am loading them with:</p>
  
  <pre><code>import pickle
  with open('dict1.txt', "rb") as myFile:
      dict1 = pickle.load(myFile)
  </code></pre>
  
  <p>However, already at loading the first dictionary I get:</p>
  
  <pre><code>*** set a breakpoint in malloc_error_break to debug
  python(3716,0xa08ed1d4) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
  *** error: can't allocate region securely
  *** set a breakpoint in malloc_error_break to debug
  Traceback (most recent call last):
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 858, in load
      dispatch[key](self)
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1019, in load_empty_dictionary
      self.stack.append({})
  MemoryError
  </code></pre>
  
  <p>How to solve this? My computer has 16GB of RAM so I find it unusual that loading a 800MB dictionary crashes. What I also find unusual is that there were no problems while saving the dictionaries.</p>
  
  <p>Further, in future I plan to process more data resulting in larger dictionaries (3-4GB on the disk), so any advice how to improve the efficiency is appreciated.</p>