<h3 dir="auto">Apache Airflow version</h3>
<p dir="auto">Other Airflow 2 version</p>
<h3 dir="auto">What happened</h3>
<p dir="auto">We have a DAG running with a cronjob schedule every 4 minutes that checks whether there is a new file present in a file system or not. If there is one available, it lists its path and passes it as a dynamic task via XCOM (one task = one file).</p>
<p dir="auto">We have experienced a couple of times already that this task is never started, nor does it appear in the logs (not in scheduler logs, nor in AKS logs). It simply disappears and the status of the task is removed.</p>
<p dir="auto">The DAG has following configurations:<br>
Schedule interval: "*/4 5-22 * * 1-5"<br>
Catchup: False<br>
End Date: None<br>
Max Active Runs: 1<br>
Concurrency: 1<br>
DAG Run Timeout: 0:10:00</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/94971679/188836813-64e8e89b-619e-4270-b1a3-60583cf4a46a.jpg"><img src="https://user-images.githubusercontent.com/94971679/188836813-64e8e89b-619e-4270-b1a3-60583cf4a46a.jpg" alt="list_metadata success" style="max-width: 100%;"></a><br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/94971679/188836817-ebe79b8d-a49f-494c-9981-8c546f115a8d.jpg"><img src="https://user-images.githubusercontent.com/94971679/188836817-ebe79b8d-a49f-494c-9981-8c546f115a8d.jpg" alt="removed_task" style="max-width: 100%;"></a><br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/94971679/188836821-6dcb6c82-5fab-45da-b5be-29793c79d53f.jpg"><img src="https://user-images.githubusercontent.com/94971679/188836821-6dcb6c82-5fab-45da-b5be-29793c79d53f.jpg" alt="xcom" style="max-width: 100%;"></a></p>
<h3 dir="auto">What you think should happen instead</h3>
<p dir="auto">The task should be triggered and started. Instead, it gets the status removed without any logs or trace</p>
<h3 dir="auto">How to reproduce</h3>
<p dir="auto">This happens in random moments, so exact steps to reproduce are hard to determine</p>
<h3 dir="auto">Operating System</h3>
<p dir="auto">Airflow base docker image</p>
<h3 dir="auto">Versions of Apache Airflow Providers</h3>
<ul dir="auto">
<li>apache-airflow-providers-microsoft-azure: 4.2.0</li>
</ul>
<h3 dir="auto">Deployment</h3>
<p dir="auto">Official Apache Airflow Helm Chart</p>
<h3 dir="auto">Deployment details</h3>
<p dir="auto">We are running on AKS with a postgresql backend, with the airflow helm chart of 2.3.3. Docker image is the airflow base image.<br>
Logs are being sent to an azure file share.</p>
<h3 dir="auto">Anything else</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>
<h3 dir="auto">Apache Airflow version</h3>
<p dir="auto">2.3.3</p>
<h3 dir="auto">What happened</h3>
<p dir="auto">If you try to <code class="notranslate">backfill</code> a DAG that uses <a href="https://airflow.apache.org/docs/apache-airflow/2.3.0/concepts/dynamic-task-mapping.html#repeated-mapping" rel="nofollow">repeated dynamic task mapping</a>, some of the repeated mapped tasks will be marked as "removed".  Any tasks downstream of the removed tasks will refuse to start:</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/7269927/184906229-f0b1130a-5fa3-41cd-a117-81a054b57386.png"><img src="https://user-images.githubusercontent.com/7269927/184906229-f0b1130a-5fa3-41cd-a117-81a054b57386.png" alt="Screenshot from 2022-08-16 08-51-35" style="max-width: 100%;"></a></p>
<p dir="auto">Or, if there is nothing downstream of those tasks, the DAG Run will claim it was successful, even though some of its tasks never ran:</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/7269927/184906238-2ae31619-d77c-4781-acd2-52b0f2d54b0a.png"><img src="https://user-images.githubusercontent.com/7269927/184906238-2ae31619-d77c-4781-acd2-52b0f2d54b0a.png" alt="Screenshot from 2022-08-16 09-32-57" style="max-width: 100%;"></a></p>
<h3 dir="auto">What you think should happen instead</h3>
<p dir="auto">All mapped tasks should run as normal during a backfill.</p>
<h3 dir="auto">How to reproduce</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="#!/usr/bin/env python3
import datetime
import itertools
import logging
import string

from airflow.decorators import dag, task


logger = logging.getLogger(__name__)

TASK_COUNT = 25


@dag(
    schedule_interval='@daily',
    start_date=datetime.datetime(2022, 8, 16),
    default_args={
        'retries': 0,
    },
)
def test_backfill_removed():
    @task
    def get_tasks():
        return list(itertools.islice(itertools.cycle(string.ascii_letters), TASK_COUNT))

    @task
    def t1(arg):
        logger.info(f'{arg=}')
        return arg

    t1.expand(arg=t1.expand(arg=get_tasks()))
    # t1.expand(arg=t1.expand(arg=t1.expand(arg=get_tasks())))


dag = test_backfill_removed()


if __name__ == '__main__':
    dag.cli()"><pre class="notranslate"><code class="notranslate">#!/usr/bin/env python3
import datetime
import itertools
import logging
import string

from airflow.decorators import dag, task


logger = logging.getLogger(__name__)

TASK_COUNT = 25


@dag(
    schedule_interval='@daily',
    start_date=datetime.datetime(2022, 8, 16),
    default_args={
        'retries': 0,
    },
)
def test_backfill_removed():
    @task
    def get_tasks():
        return list(itertools.islice(itertools.cycle(string.ascii_letters), TASK_COUNT))

    @task
    def t1(arg):
        logger.info(f'{arg=}')
        return arg

    t1.expand(arg=t1.expand(arg=get_tasks()))
    # t1.expand(arg=t1.expand(arg=t1.expand(arg=get_tasks())))


dag = test_backfill_removed()


if __name__ == '__main__':
    dag.cli()
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="airflow dags backfill test_backfill_removed -s 2022-08-01 -e 2022-08-05"><pre class="notranslate"><code class="notranslate">airflow dags backfill test_backfill_removed -s 2022-08-01 -e 2022-08-05
</code></pre></div>
<h3 dir="auto">Operating System</h3>
<p dir="auto">CentOS Stream 8</p>
<h3 dir="auto">Versions of Apache Airflow Providers</h3>
<p dir="auto">None</p>
<h3 dir="auto">Deployment</h3>
<p dir="auto">Other</p>
<h3 dir="auto">Deployment details</h3>
<p dir="auto">Self-hosted + standalone</p>
<h3 dir="auto">Anything else</h3>
<p dir="auto">I've reproduced this with SQLite + <code class="notranslate">SequentialExecutor</code> (standalone), Postgres + <code class="notranslate">LocalExecutor</code> (standalone), and Postgres + <code class="notranslate">CeleryExecutor</code> (production deployment, self-hosted).</p>
<p dir="auto">I don't see any errors or warnings in the logs.  In fact, even with <code class="notranslate">logging_level = DEBUG</code>, I don't see <em>any</em> log lines that reference the tasks that are listed as "removed".</p>
<p dir="auto">This error is sometimes inconsistent to reproduce - if you are unlucky, you might instead get the deadlock mentioned in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1339550696" data-permission-text="Title is private" data-url="https://github.com/apache/airflow/issues/25730" data-hovercard-type="issue" data-hovercard-url="/apache/airflow/issues/25730/hovercard" href="https://github.com/apache/airflow/issues/25730">#25730</a>.</p>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>