title: How to Reduce the time taken to load a pickle file in python
body: <p>I have created a dictionary in python and dumped into pickle. Its size went to 300MB. Now, I want to load the same pickle.</p>
  
  <pre><code>output = open('myfile.pkl', 'rb')
  mydict = pickle.load(output)
  </code></pre>
  
  <p>Loading this pickle takes around <strong>15 seconds</strong>. <strong>How can I reduce this time?</strong></p>
  
  <p>Hardware Specification: Ubuntu 14.04, 4GB RAM</p>
  
  <p>The code bellow shows how much time takes to dump or load a file using json, pickle, cPickle.</p>
  
  <p>After dumping, file size would be around 300MB. </p>
  
  <pre><code>import json, pickle, cPickle
  import os, timeit
  import json
  
  mydict= {all values to be added}
  
  def dump_json():    
      output = open('myfile1.json', 'wb')
      json.dump(mydict, output)
      output.close()    
  
  def dump_pickle():    
      output = open('myfile2.pkl', 'wb')
      pickle.dump(mydict, output,protocol=cPickle.HIGHEST_PROTOCOL)
      output.close()
  
  def dump_cpickle():    
      output = open('myfile3.pkl', 'wb')
      cPickle.dump(mydict, output,protocol=cPickle.HIGHEST_PROTOCOL)
      output.close()
  
  def load_json():
      output = open('myfile1.json', 'rb')
      mydict = json.load(output)
      output.close()
  
  def load_pickle():
      output = open('myfile2.pkl', 'rb')
      mydict = pickle.load(output)
      output.close()
  
  def load_cpickle():
      output = open('myfile3.pkl', 'rb')
      mydict = pickle.load(output)
      output.close()
  
  
  if __name__ == '__main__':
      print "Json dump: "
      t = timeit.Timer(stmt="pickle_wr.dump_json()", setup="import pickle_wr")  
      print t.timeit(1),'\n'
  
      print "Pickle dump: "
      t = timeit.Timer(stmt="pickle_wr.dump_pickle()", setup="import pickle_wr")  
      print t.timeit(1),'\n'
  
      print "cPickle dump: "
      t = timeit.Timer(stmt="pickle_wr.dump_cpickle()", setup="import pickle_wr")  
      print t.timeit(1),'\n'
  
      print "Json load: "
      t = timeit.Timer(stmt="pickle_wr.load_json()", setup="import pickle_wr")  
      print t.timeit(1),'\n'
  
      print "pickle load: "
      t = timeit.Timer(stmt="pickle_wr.load_pickle()", setup="import pickle_wr")  
      print t.timeit(1),'\n'
  
      print "cPickle load: "
      t = timeit.Timer(stmt="pickle_wr.load_cpickle()", setup="import pickle_wr")  
      print t.timeit(1),'\n'
  </code></pre>
  
  <p><strong>Output :</strong></p>
  
  <pre><code>Json dump: 
  42.5809804916 
  
  Pickle dump: 
  52.87407804489 
  
  cPickle dump: 
  1.1903790187836 
  
  Json load: 
  12.240660209656 
  
  pickle load: 
  24.48748306274 
  
  cPickle load: 
  24.4888298893
  </code></pre>
  
  <p>I have seen that cPickle takes less time to dump and load but <strong>loading a file still takes a long time</strong>. </p>