<p dir="auto">I have a problem with scheduler memory that continue to increase when i am using docker image.<br>
I am using Celery Executor<br>
<a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/14239278/121222326-bf902580-c886-11eb-851f-dd78ea5fa43d.png"><img src="https://user-images.githubusercontent.com/14239278/121222326-bf902580-c886-11eb-851f-dd78ea5fa43d.png" alt="image" style="max-width: 100%;"></a><br>
Any solution ??<br>
Thanks in advance</p>
<p dir="auto"><strong>Apache Airflow version</strong>: 2.0.1</p>
<p dir="auto"><strong>Kubernetes version (if you are using kubernetes)</strong> (use <code class="notranslate">kubectl version</code>): v1.17.4</p>
<p dir="auto"><strong>Environment</strong>: Dev</p>
<ul dir="auto">
<li><strong>OS</strong> (e.g. from /etc/os-release): RHEL7</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">After running fine for some time my airflow tasks got stuck in scheduled state with below error in Task Instance Details:<br>
"All dependencies are met but the task instance is not running. In most cases this just means that the task will probably be scheduled soon unless: - The scheduler is down or under heavy load If this task instance does not start soon please contact your Airflow administrator for assistance."</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">I restarted the scheduler then it started working fine. When i checked my metrics i realized the scheduler has a memory leak and over past 4 days it has reached up to 6GB of memory utilization</p>
<p dir="auto">In version &gt;2.0 we don't even have the run_duration config option to restart scheduler periodically to avoid this issue until it is resolved.</p>
<p dir="auto"><strong>How to reproduce it</strong>:<br>
I saw this issue in multiple dev instances of mine all running Airflow 2.0.1 on kubernetes with KubernetesExecutor.<br>
Below are the configs that i changed from the default config.<br>
max_active_dag_runs_per_dag=32<br>
parallelism=64<br>
dag_concurrency=32<br>
sql_Alchemy_pool_size=50<br>
sql_Alchemy_max_overflow=30</p>
<p dir="auto"><strong>Anything else we need to know</strong>:</p>
<p dir="auto">The scheduler memory leaks occurs consistently in all instances i have been running. The memory utilization keeps growing for scheduler.</p>