title: Optimization of json.load() to reduce in-memory usage and time in Python
body: <p>I have 10K folders each with 200 records in 200 JSON format files.
  Trying to compile all records into one data frame then finally into a CSV (other format suggestions welcome)</p>
  <p>Here is my working solution which takes around 8.3hrs just for the dataframe building process. (Not converting into CSV)</p>
  <pre><code>%%time
  finalDf = pd.DataFrame()
  rootdir ='/path/foldername'
  all_files = Path(rootdir).rglob('*.json')
  for filename in all_files:
      with open(filename, 'r+') as f:
          data = json.load(f)
          df = pd.json_normalize(data).drop(columns=[A]).rename(columns={'B': 'Date'})
          finalDf = finalDf.append(df, ignore_index=True)
  </code></pre>
  <p>Any suggestions to optimize this and bring the time down.</p>