title: how to efficiently make airflow dag definitions database-driven
body: <p><strong>Background</strong></p>
  
  <p>I have some dags that pull data from an 3rd-party api. </p>
  
  <p>The accounts we need to pull can change over time.  To determine which accounts to pull, depending on the process we may need to query a database or make an HTTP request.  </p>
  
  <p>Before airflow, we would just get the account list at the start of the python script.  Then we would iterate through the account list and pull each account to file or whatever it was we needed to do.  </p>
  
  <p>But now, using airflow, it makes sense to define tasks at the account level and let airflow handle retry functionality and date range and parallel execution etc.  </p>
  
  <p>Thus my dag might look something like this:</p>
  
  <p><a href="https://i.stack.imgur.com/2WeFg.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/2WeFg.png" alt="Tree view of dag.  Each account gets its one task for data pull."></a></p>
  
  <p><strong>Problem</strong></p>
  
  <p>Since each account is a task, the account list needs to be accessed with every dag parse.  But since dag files are parsed frequently, you don't necessarily want to query the database or wait for a REST call with every dag parse from every machine all day long.  This could be resource intensive, and could cost money.</p>
  
  <p><strong>Question</strong></p>
  
  <p>Is there a good way to cache this type of config information in a local file, ideally with a specified time-to-live?</p>
  
  <p><strong>Thoughts</strong></p>
  
  <p>I have thought about a couple different approaches:</p>
  
  <ol>
  <li>write to csv or pickle file and use mtime to expire.  
  
  <ul>
  <li>the concern with this is that i might get collisions if two processes try to expire the file at the same time. i don't know how likely this is or what the consequences would be but probably nothing terrible.</li>
  </ul></li>
  <li>create a common sqlite DB for all such processes.  should be auto created first time a variable is accessed.  each config variable gets a row in table.  use last_modified_datetime column to tell when to expire.
  
  <ul>
  <li>requires more elaborate code &amp; dependencies. </li>
  </ul></li>
  <li>use airflow variables
  
  <ul>
  <li>nice thing about this would be that it uses existing DB, so would be no $ per query and reasonable network lag, but it still requires network round trip.  </li>
  <li>has benefit of being identical across all nodes in a multi-node setup.</li>
  <li>determining when to expire would probably be problematic so would probably create config manager dag to update the config variables periodically.  </li>
  <li>but then this would add complexity to deployment and devolpment process -- the variables need to be populated in order to define the DAGs properly -- all developers would need to manage this locally too, as opposed to a more create-on-read cacheing approach.</li>
  </ul></li>
  <li>Subdags?  
  
  <ul>
  <li>never used them, but I have a suspicion they could be used here.  But the community seems to discourage their use anyway...</li>
  </ul></li>
  </ol>
  
  <p>Have you dealt with this problem?  Did you arrive at a good solution?  None of these seems very good.</p>