title: Wiring top-level DAGs together
body: <p>I need to have several identical (differing only in arguments) <strong>top-level <code>DAG</code>s</strong> that can also be triggered together with following constraints / assumptions:</p>
  
  <ul>
  <li>Individual top-level DAGs will have <code>schedule_interval=None</code> as they will only need occasional <em>manual triggering</em></li>
  <li>The series of DAGs, however, needs to run daily</li>
  <li><em>Order</em> and <em>number</em> of DAGs in series is fixed (known ahead of writing code) and changes rarely (once in a few months)</li>
  <li>Irrespective of whether a DAG fails or succeeds, the chain of triggering must not break</li>
  <li>Currently they must be run together in series; in future they may require parallel triggering</li>
  </ul>
  
  <hr>
  
  <p>So I created one file for each DAG in my <code>dags</code> directory and now I must <strong>wire them up for sequential execution</strong>. I have identified two ways this could be done:</p>
  
  <ol>
  <li><p><strong><code>SubDagOperator</code></strong></p>
  
  <ul>
  <li>Works without a glitch in <a href="https://github.com/y2k-shubham/docker-airflow/blob/subdag/dags/subdag_simplified.py" rel="noreferrer">my demo</a></li>
  <li>Can <a href="https://medium.com/bluecore-engineering/airflow-why-is-nothing-working-f705eb6b7b04" rel="noreferrer">lead to <em>deadlocks</em></a> but there are <a href="https://medium.com/@team_24989/fixing-subdagoperator-deadlock-in-airflow-6c64312ebb10" rel="noreferrer">easy solutions</a>; still there's a lot of <a href="https://stackoverflow.com/questions/51301763">haze around using them</a></li>
  <li>SubDag's <code>dag_id</code> <a href="http://airflow.incubator.apache.org/concepts.html#subdags" rel="noreferrer">must be prefixed</a> by it's parent's, that would force <em>absurd</em> IDs on <em>top-level</em> DAGs that are supposed to be functional independently too</li>
  </ul></li>
  <li><p><strong><code>TriggerDagRunOperator</code></strong></p>
  
  <ul>
  <li>Works in <a href="https://github.com/y2k-shubham/docker-airflow/blob/trigger-dag/dags/trigger_dag.py" rel="noreferrer">my demo</a> but runs in <em>parallel</em> (not <em>sequentially</em>) as it doesn't <em>wait for triggered DAG</em> to finish before moving onto next one</li>
  <li><code>ExternalTaskSensor</code> might help overcome above limitation but it would make things very messy</li>
  </ul></li>
  </ol>
  
  <hr>
  
  <p>My questions are</p>
  
  <ul>
  <li>How to overcome limitation of <code>parent_id</code> prefix in <code>dag_id</code> of <code>SubDag</code>s?</li>
  <li>How to force <code>TriggerDagRunOperator</code>s to <strong>await completion of DAG</strong>?</li>
  <li>Any alternate / better way to <em>wire-up</em> independent (top-level) DAGs together?</li>
  <li>Is there a workaround for my approach of <em>creating separate files</em> (for DAGs that differ only in input) for each top-level DAG?</li>
  </ul>
  
  <hr>
  
  <p>I'm using <a href="https://github.com/puckel/docker-airflow" rel="noreferrer">puckel/docker-airflow</a> with</p>
  
  <ul>
  <li><code>Airflow 1.9.0-4</code></li>
  <li><code>Python 3.6-slim</code></li>
  <li><code>CeleryExecutor</code> with <code>redis:3.2.7</code></li>
  </ul>
  
  <hr>
  
  <p><strong>EDIT-1</strong></p>
  
  <p>Clarifying <strong>@Viraj Parekh</strong>'s <a href="https://stackoverflow.com/a/51328562/3679900">queries</a></p>
  
  <blockquote>
    <p>Can you give some more detail on what you mean by awaiting completion
    of the DAG before getting triggered?</p>
  </blockquote>
  
  <p>When I trigger the <code>import_parent_v1</code> DAG, all the 3 external DAGs that it is supposed to fire using <code>TriggerDagRunOperator</code> start running parallely even when I chain them sequentially. Actually the <em>logs</em> indicate that while they are fired one-after another, the execution moves onto next DAG (<code>TriggerDagRunOperator</code>) before the previous one has finished.
  <a href="https://i.stack.imgur.com/EQQTP.png" rel="noreferrer"><img src="https://i.stack.imgur.com/EQQTP.png" alt="enter image description here"></a>
  <a href="https://i.stack.imgur.com/ih3yW.png" rel="noreferrer"><img src="https://i.stack.imgur.com/ih3yW.png" alt="enter image description here"></a>
  <em>NOTE: In this example, the top-level DAGs are named as <code>importer_child_v1_db_X</code> and their corresponding <code>task_id</code>s (for <code>TriggerDagRunOperator</code>) are named as <code>importer_v1_db_X</code></em></p>
  
  <blockquote>
    <p>Would it be possible to just have the TriggerDagRunOperator be the
    last task in a DAG?</p>
  </blockquote>
  
  <p>I have to chain several similar (differing only in arguments) DAGs together in a <strong>workflow that triggers them one-by-one</strong>. So there isn't just <em>one</em> <code>TriggerDagRunOperator</code> that I could put at last, there are many (here 3, but would be upto 15 in production)</p>