title: Fastest Method To Read Thousands of JSON Files in Python
body: <p>I have a number of JSON files I need to analyze. I am using iPython (<code>Python 3.5.2 | IPython 5.0.0</code>), reading in the files to a dictionary and appending each dictionary to a list.</p>
  
  <p>My main bottleneck is reading in the files. Some files are smaller, and are read quickly, but the larger files are slowing me down.</p>
  
  <p>Here is some example code (sorry, I cannot provide the actual data files):</p>
  
  <pre><code>import json
  import glob
  
  def read_json_files(path_to_file):
      with open(path_to_file) as p:
          data = json.load(p)
          p.close()
      return data
  
  def giant_list(json_files):
      data_list = []
      for f in json_files:
          data_list.append(read_json_files(f))
      return data_list
  
  support_files = glob.glob('/Users/path/to/support_tickets_*.json')
  small_file_test = giant_list(support_files)
  
  event_files = glob.glob('/Users/path/to/google_analytics_data_*.json')
  large_file_test = giant_list(event_files)
  </code></pre>
  
  <p>The support tickets are very small in size--largest I've seen is 6KB. So, this code runs pretty fast:</p>
  
  <pre><code>In [3]: len(support_files)
  Out[3]: 5278
  
  In [5]: %timeit giant_list(support_files)
  1 loop, best of 3: 557 ms per loop
  </code></pre>
  
  <p>But larger files definitely are slowing me down...these event files can reach ~2.5MB each:</p>
  
  <pre><code>In [7]: len(event_files) # there will be a lot more of these soon :-/
  Out[7]: 397
  
  In [8]: %timeit giant_list(event_files)
  1 loop, best of 3: 14.2 s per loop
  </code></pre>
  
  <p>I've researched how to speed up the process and came across <a href="https://stackoverflow.com/questions/706101/python-json-decoding-performance">this post</a>, however, when using UltraJSON the timing was just slightly worse:</p>
  
  <pre><code>In [3]: %timeit giant_list(traffic_files)
  1 loop, best of 3: 16.3 s per loop
  </code></pre>
  
  <p>SimpleJSON did not do much better: </p>
  
  <pre><code>In [4]: %timeit giant_list(traffic_files)
  1 loop, best of 3: 16.3 s per loop
  </code></pre>
  
  <p>Any tips on how to optimize this code and more efficiently read a lot of JSON files into Python is much appreciated.</p>
  
  <p>Finally, <a href="https://stackoverflow.com/questions/39781716/loading-large-twitter-json-data-7gb-into-python">this post</a> is the closest I've found to my question, but deals with one giant JSON file, not many smaller sized ones.</p>