title: How to keep track of the files I read into a database in python?
body: <h2>Situation</h2>
  
  <p>I get a ton of json files from a remote data source. I organize these files into an archive, then read them into a database. The archive exists to rebuild the database, if necessary.</p>
  
  <p>The json files are generated remotely and sent to my server periodically and the reading-in process happens continuously. One more than one occasion, we had a power loss to our severs overnight or over the weekend, this was a huge problem for database loading, since the processes halted and I didn't know what had been loaded and what hadn't so I had to roll back to some previously known state and rebuild out of the archive.</p>
  
  <p>To fix this problem, my master loader daemon (written in python) now uses the <code>logging</code> package to track what files it has loaded. The basic workflow of the loader daemon is </p>
  
  <ul>
  <li><code>cp</code> json file to archive </li>
  <li>`rm' original</li>
  <li>insert archived copy to database (its MariaDB)</li>
  <li>commit to database </li>
  <li>log filename of loaded json file</li>
  </ul>
  
  <p>I'm not so much worried about duplicates in the database, but I don't want gaps; that is, things in the archive that are not in the database. This methods has so far and seems guaranteed to prevent any gaps.</p>
  
  <p>For my logging, it basically looks like this. When the daemon starts up on a set of received files' names, it checks for duplicates that have already been loaded to the destination database and then loads all the non-duplicates. It is possible to get duplicates from my remote data source. </p>
  
  <pre><code>def initialize_logs(filenames, destination)
     try:
        with open("/data/dblogs/{0}.log".format(destination), 'r') as already_used:
           seen = set([line.rstrip("\n") for line in already_used])
     except FileNotFoundError:
        print("Log file for {0} not found. Repair database".format(destination))
        quit()
  
     fnamelog = logging.getLogger('filename.log')
     fnamelog.setLevel(logging.INFO)
     fh = logging.FileHandler("/data/dblogs/{0}.log".format(destination))
     fh.setLevel(logging.INFO)
     fnamelog.addHandler(fh)
  </code></pre>
  
  <p>Then, as I process the jsonfiles, I log each file added using</p>
  
  <pre><code>fnamelog.info(filename)
  </code></pre>
  
  <p>The database loader is run parallelized, so I originally chose the <code>logging</code> package for its built in concurrency protections. There are a variety of databases; not every database pulls all data from the json files. Some databases with more information are shorter in time, usually one to two months. In this case, it is nice to have a log file with all json files in a given database, so if I want to add some on to it, I don't have to worry about what is already in there, the log file is keeping track.</p>
  
  <hr>
  
  <h2>Problem</h2>
  
  <p>A year has passed. I have kept getting json files. I am now getting around a million files per month. The text logging of each filename as it is processed in is clumsy, but it still works...for now. There are multiple databases, but for the largest ones, the log file is over half a GB. I feel like this logging solution will not work well for much longer.</p>
  
  <p><strong>What options are available in python to track which filenames have been inserted into a database, when there are over 10 million filenames per database, and rising?</strong></p>