title: Load a Single Large Python Dictionary Encoded as Json Without Killing Memory Usage?
body: <p>I've seen a lot of similar questions to this, but nothing that really matched. Most other questions seemed to relate to speed. What I'm experiencing is a single json dictionary that sits in a 1.1gig file on my local box taking up all of my 16 gigabytes of memory when I try to load it using anything along the lines of:</p>
  
  <pre><code>f = open(some_file, "rb")
  new_dictionary = json.load(f)
  </code></pre>
  
  <p>This happens regardless of what json library I use (I've tried ujson, json, yajl), and regardless of whether I read things in as a byte stream or not. This makes absolutely no sense to me. What's with the crazy memory usage, and how do I get around it?</p>
  
  <p>In case it helps, the dictionary is just a bunch of nested dictionaries all having ints point to other ints. A sample looks like:</p>
  
  <pre><code>{"0":{"3":82,"4":503,"15":456},"956":{"56":823,"678":50673,"35":1232}...}
  </code></pre>
  
  <p><strong>UPDATE:</strong> When I run this with simplejson, it actually only takes up 8 gigs. No idea why that one takes up so much less than all the others.</p>
  
  <p><strong>UPDATE 2:</strong> So I did some more investigation. I loaded up my dictionary with simplejson, and tried converting all the keys to ints (per Liori's suggestion that strings might take up more space). Space stayed the same at 8 gigs. Then I tried Winston Ewert's suggestion of running a gc.collect(). Space still remained at 8 gigs. Finally, annoyed and curious, I pickled my new data structure, exited Python, and reloaded. Lo and behold, it still takes up 8 gigs. I guess Python just wants that much space for a big 2d dictionary. Frustrating, for sure, but at least now I know it's not a JSON problem so long as I use simplejson to load it.</p>