<p dir="auto"><strong>Apache Airflow version</strong>:</p>
<p dir="auto">2.0.0</p>
<p dir="auto"><strong>Kubernetes version (if you are using kubernetes)</strong> (use <code class="notranslate">kubectl version</code>):</p>
<p dir="auto">1.15.x</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: AWS</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">Apparently <code class="notranslate">removed</code> task state causes Airflow to consider <code class="notranslate">dag_run</code> as active, due to <code class="notranslate">removed</code> not being part of <code class="notranslate">finished</code> frozenset:</p>
<p dir="auto"><a href="https://github.com/apache/airflow/blob/v2-0-stable/airflow/utils/state.py#L103">https://github.com/apache/airflow/blob/v2-0-stable/airflow/utils/state.py#L103</a></p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">Removed state is counted as finished</p>
<p dir="auto"><strong>How to reproduce it</strong>:</p>
<p dir="auto">Create a dag run, backfill with success (-m) and then deploy a change removing some tasks. Airflow will now count it as active</p>
<p dir="auto"><strong>Apache Airflow version</strong>: 2.0.0<br>
<strong>Kubernetes version (if you are using kubernetes)</strong> (use <code class="notranslate">kubectl version</code>):<br>
<strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>OS</strong> (e.g. from /etc/os-release): Debian GNU/Linux 10 (buster)</li>
<li><strong>Kernel</strong> (e.g. <code class="notranslate">uname -a</code>): Linux 6ae65b86e112 5.4.0-52-generic <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="90004785" data-permission-text="Title is private" data-url="https://github.com/apache/airflow/issues/57" data-hovercard-type="issue" data-hovercard-url="/apache/airflow/issues/57/hovercard" href="https://github.com/apache/airflow/issues/57">#57</a>-Ubuntu SMP Thu Oct 15 10:57:00 UTC 2020 x86_64 GNU/Linux</li>
<li><strong>Others</strong>: Python 3.8</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">After migrating one of our development Airflow instances from 1.10.14 to 2.0.0, the scheduler started to refuse to schedule tasks for a DAG that did not actually exceed its <code class="notranslate">max_active_runs</code>.</p>
<p dir="auto">When it did this the following error would be logged:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="DAG &lt;dag_name&gt; already has 577 active runs, not queuing any tasks for run 2020-12-17 08:05:00+00:00"><pre class="notranslate"><code class="notranslate">DAG &lt;dag_name&gt; already has 577 active runs, not queuing any tasks for run 2020-12-17 08:05:00+00:00
</code></pre></div>
<p dir="auto">A bit of digging revealed that this DAG had task instances associated with it that are in the <code class="notranslate">removed</code> state. As soon as I forced the task instances that are in the <code class="notranslate">removed</code> state into the <code class="notranslate">failed</code> state, the tasks would be scheduled.</p>
<p dir="auto">I believe the root cause of the issue is that <a href="https://github.com/apache/airflow/blob/master/airflow/jobs/scheduler_job.py#L1506">this filter</a> does not filter out tasks that are in the <code class="notranslate">removed</code> state.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">I expected the task instances in the DAG to be scheduled, because the DAG did not actually exceed the number of <code class="notranslate">max_active_runs</code>.</p>
<p dir="auto"><strong>How to reproduce it</strong>:</p>
<p dir="auto">I think the best approach to reproduce it is as follows:</p>
<ol dir="auto">
<li>Create a DAG and set <code class="notranslate">max_active_runs</code> to 1.</li>
<li>Ensure the DAG has ran successfully a number of times, such that it has some history associated with it.</li>
<li>Set one historical task instance to the <code class="notranslate">removed</code> state (either by directly updating it in the DB, or deleting a task from a DAG before it has been able to execute).</li>
</ol>
<p dir="auto"><strong>Anything else we need to know</strong>:</p>
<p dir="auto">The Airflow instance that I ran into this issue with contains about 3 years of task history, which means that we actually had quite a few task instances that are in the <code class="notranslate">removed</code> state, but there is no easy way to delete those from the Web UI.</p>
<p dir="auto">A work around is to set the tasks to <code class="notranslate">failed</code>, which will allow the scheduler to proceed.</p>