title: Airflow: Proper way to run DAG for each file
body: <p>I have the following task to solve:</p>
  
  <blockquote>
    <p>Files are being sent at irregular times through an endpoint and stored locally. I need to trigger a DAG run for each of these files. For each file the same tasks will be performed</p>
  </blockquote>
  
  <p>Overall the flows looks as follows: For each file, run tasks A->B->C->D</p>
  
  <p>Files are being processed in batch. While this task seemed trivial to me, I have found several ways to do this and I am confused about which one is the "proper" one (if any).</p>
  
  <h2>First pattern: Use experimental REST API to trigger dag.</h2>
  
  <p>That is, expose a web service which ingests the request and the file, stores it to a folder, and uses the <a href="https://airflow.apache.org/docs/stable/api.html" rel="noreferrer">experimental REST api</a> to trigger the DAG, by passing the file_id as conf</p>
  
  <p><strong>Cons</strong>: REST apis are still <em>experimental</em>, not sure how Airflow can handle a load test with many requests coming at one point (which shouldn't happen, but, what if it does?)</p>
  
  <h2>Second pattern: 2 dags. One senses and triggers with TriggerDagOperator, one processes.</h2>
  
  <p>Always using the same ws as described before, but this time it justs stores the file. Then we have:</p>
  
  <ul>
  <li>First dag: Uses a FileSensor along with the TriggerDagOperator to trigger N dags given N files</li>
  <li>Second dag: Task A->B->C </li>
  </ul>
  
  <p><strong>Cons</strong>: Need to avoid that the same files are being sent to two different DAG runs. 
  Example:</p>
  
  <p>Files in folder x.json
  Sensor finds x, triggers DAG (1)</p>
  
  <p>Sensor goes back and scheduled again. If DAG (1) did not process/move the file, the sensor DAG might reschedule a new DAG run with the same file. Which is unwanted.</p>
  
  <h2>Third pattern: for file in files, task A->B->C</h2>
  
  <p>As seen in <a href="https://stackoverflow.com/questions/55973680/how-to-implement-airflow-dag-in-a-loop">this question</a>.</p>
  
  <p><strong>Cons</strong>: This could work, however what I dislike is that the UI will probably get messed up because every DAG run will not look the same but it will change with the number of files being processed. Also if there are 1000 files to be processed the run would probably be very difficult to read</p>
  
  <h2>Fourth pattern: Use subdags</h2>
  
  <p>I am not yet sure how they completely work as I have seen  <a href="https://www.astronomer.io/guides/subdags/" rel="noreferrer">they are not encouraged</a> (at the end), however it should be possible to spawn a subdag for each file and have it running. Similar to <a href="https://stackoverflow.com/questions/48947867/how-to-dynamically-create-subdags-in-airflow">this question</a>.</p>
  
  <p><strong>Cons</strong>: Seems like subdags can only be used with the sequential executor.</p>
  
  <hr>
  
  <p>Am I missing something and over-thinking something that should be (in my mind) quite straight-forward? Thanks</p>