title: How do I Convert Python Dict to JSON in a Multi-Threaded Fashion
body: <p>I have a number of large files with many thousands of lines in python dict format. I'm converting them with json.dumps to json strings.</p>
  
  <pre><code>import json
  import ast
  
  mydict = open('input', 'r')
  output = open('output.json', "a")
  
  for line in mydict:
          line = ast.literal_eval(line)
          line = json.dumps(line)
          output.write(line)
          output.write("\n")
  </code></pre>
  
  <p>This works flawlessly, however, it does so in a single threaded fashion. Is there an easy way to utilize the remaining cores in my system to speed things up?</p>
  
  <p>Edit:</p>
  
  <p>Based on the suggestions I've started here with the multiprocessing library:</p>
  
  <pre><code>import os
  import json
  import ast
  from multiprocessing import Process, Pool
  
  mydict = open('twosec.in', 'r')
  
  def info(title):
          print title
          print 'module name:', __name__
          print 'parent process: ', os.getppid()
          print 'process id:', os.getpid()
  
  def converter(name):
          info('converter function')
          output = open('twosec.out', "a")
          for line in mydict:
                  line = ast.literal_eval(line)
                  line = json.dumps(line)
                  output.write(line)
                  output.write("\n")
  
  if __name__ == '__main__':
          info('main line')
          p = Process(target=converter, args=(mydict))
          p.start()
          p.join()
  </code></pre>
  
  <p>I don't quite understand where Pool comes into play, can you explain more?</p>