title: How to read a large JSON file using Python ijson?
body: <p>I am trying to parse a big json file (hundreds of gigs) to extract information from its keys. For simplicity, consider the following example:</p>
  
  <pre><code>import random, string
  
  # To create a random key 
  def random_string(length):
          return "".join(random.choice(string.lowercase) for i in range(length))
  
  # Create the dicitonary 
  dummy = {random_string(10): random.sample(range(1, 1000), 10) for times in range(15)}
  
  # Dump the dictionary into a json file 
  with open("dummy.json", "w") as fp:
          json.dump(dummy, fp)
  </code></pre>
  
  <p>Then, I use ijson in python 2.7 to parse the file:</p>
  
  <pre><code>file_name = "dummy.json"
  
  with open(file_name, "r") as fp:
  
      for key in dummy.keys():
  
          print "key: ", key 
  
          parser = ijson.items(fp, str(key) + ".item")
  
          for number in parser:
              print number,
  </code></pre>
  
  <p>I was expecting to retrieve all the numbers in the lists corresponding to the keys of the dic. However, I got </p>
  
  <blockquote>
    <p>IncompleteJSONError: Incomplete JSON data</p>
  </blockquote>
  
  <p>I am aware of this post: <a href="https://stackoverflow.com/questions/37200302/using-python-ijson-to-read-a-large-json-file-with-multiple-json-objects">Using python ijson to read a large json file with multiple json objects</a>, but in my case I have a single json file, that is well formed, with a relative simple schema. Any ideas on how can I parse it? Thank you. </p>