title: Sequentually unpickle large files asynchronously
body: <p>I have a directory of Pickled lists which I would like to load sequentially, use as part of an operation, and then discard. The files are around 0.75 - 2GB each when pickled and I can load a number in memory at any one time, although nowhere near all of them. Each pickled file represents one day of data.</p>
  <p>Currently, the unpickling process consumes a substantial proportion of the runtime of the program. My proposed solution is to load the first file and, whilst the operation is running on this file, asynchronously load the next file in the list.</p>
  <p>I have thought of two ways I could do this: 1) Threading and 2) Asyncio. I have tried both of these but neither has seemed to work. Below is my (attempted) implementation of a Threading-based solution.</p>
  <pre><code>import os
  import threading
  import pickle
  
  class DataSource:
      def __init__(self, folder):
          self.folder = folder
          self.next_file = None
  
      def get(self):
          if self.next_file is None:
              self.load_file()
          data = self.next_file
      
          io_thread = threading.Thread(target=self.load_file, daemon=True)
          io_thread.start()
  
          return data
  
      def get_next_file(self):
          for filename in sorted(os.listdir(self.folder)):
              yield self.folder + filename
  
      def load_file(self):
          self.next_file = pickle.load(open(next(self.get_next_file()), &quot;rb&quot;))
  </code></pre>
  <p>The main program will call <em>DataSource().get()</em> to retrieve each file. The first time it is loaded, <em>load_file()</em> will load the file into <em>next_file</em> where it will be stored. Then, the thread <em>io_thread</em> should load each successive file into <em>next_file</em> to be returned via <em>get()</em> as needed.</p>
  <p>The thread that is launched does appear to do some work (it consumes a vast amount of RAM, ~60GB) however it does not appear to update <em>next_file</em>.</p>
  <p>Could someone suggest why this doesn't work? And, additionally, if there is a better way to achieve this result?</p>
  <p>Thanks</p>