title: How to optimize writing a list of dictionaries using json dump into a file?
body: <p>Currently I'm having the following implementation to write a <code>list</code> of <code>dictionaries</code> into a file. The <code>file_limit_counter</code> below would be initialized to 0 whereas the <code>file_limit</code> will be initialized to let's say 50 for now. So whenever the counter becomes equal to the file limit, it'll start writing the output into a new file. The counter approach is to break the output into multiple files:</p>
  <pre><code>self.file = self.generate_new_micro_file_name()
  for response_dict in all_response_dict_list:
      if file_limit_counter == file_limit:
          self.propagate_log_msg('wrote {} records '.format(file_limit))
          self.file = self.generate_new_micro_file_name()
          file_limit_counter = 0
      with open(self.file, 'a', encoding=&quot;utf8&quot;) as open_out_file:
          json.dump(response_dict, open_out_file)
          open_out_file.write('\n')
      file_limit_counter += 1
  </code></pre>
  <p>The list <code>all_response_dict_list</code> would contain something like this:</p>
  <pre><code>somelist = [{&quot;Name&quot;:&quot;a1&quot;,&quot;Age&quot;:&quot;24&quot;},{&quot;Name&quot;:&quot;a2&quot;,&quot;Age&quot;:&quot;26&quot;}]
  </code></pre>
  <p>and my intention is to have something like this on my output file:</p>
  <pre><code>{&quot;Name&quot;:&quot;a1&quot;,&quot;Age&quot;:&quot;24&quot;}
  {&quot;Name&quot;:&quot;a2&quot;,&quot;Age&quot;:&quot;26&quot;}
  ...
  </code></pre>
  <p>This above approach works fine. But when it comes to a large set of dictionaries for example 5000 it tends to slow down a bit (it takes approximately 10 mins). So it'll be helpful if someone have already come across this kind of scenario. I think it would take less time if we can do the same thing above in parallel i.e. writing multiple dictionaries at the same time into the same file rather than writing one by one.</p>