<h3 dir="auto">Apache Airflow version</h3>
<p dir="auto">2.5.0</p>
<h3 dir="auto">What happened</h3>
<p dir="auto">I am trying to set up Airflow Celery to use sentinel. However, we have multiple sentinel instances up and running. So referring to the examples on the internet, I tried to pass multiple sentinel URL as below in ENV VAR.</p>
<p dir="auto"><code class="notranslate">AIRFLOW__CELERY__BROKER_URL='sentinel://redis_sentinel1:26379;sentinel://redis_sentinel2:26379'</code> (tried with/without quote)</p>
<p dir="auto">After executing, the worker keeps throwing errors as below:</p>
<blockquote>
<p dir="auto">Traceback (most recent call last):<br>
File "", line 1, in <br>
File "/usr/local/lib/python3.8/urllib/parse.py", line 177, in port<br>
raise ValueError(message) from None<br>
<strong>ValueError: Port could not be cast to integer value as '26379;sentinel:'</strong></p>
</blockquote>
<p dir="auto">From the error, it looks like the parser is NOT separating sentinel URLs with a semicolon separator. Is there any workaround for it?</p>
<h3 dir="auto">What you think should happen instead</h3>
<p dir="auto">Using a semicolon separator, sentinel URLs should be separated, and the Redis connection pool should make use of all the sentinel URLs.</p>
<h3 dir="auto">How to reproduce</h3>
<p dir="auto">Try to pass multiple sentinel URLs to <code class="notranslate">AIRFLOW__CELERY__BROKER_URL</code> environment variables and then try to bring up the airflow workers</p>
<h3 dir="auto">Operating System</h3>
<p dir="auto">ubuntu</p>
<h3 dir="auto">Versions of Apache Airflow Providers</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Deployment</h3>
<p dir="auto">Official Apache Airflow Helm Chart</p>
<h3 dir="auto">Deployment details</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Anything else</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>
<h3 dir="auto">Apache Airflow version</h3>
<p dir="auto">2.4.3</p>
<h3 dir="auto">What happened</h3>
<p dir="auto">When configuring Airflow/Celery to use Redis Sentinel as a broker, the following pops up:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="airflow.exceptions.AirflowException: The broker you configured does not support SSL_ACTIVE to be True. Please use RabbitMQ or Redis if you would like to use SSL for broker."><pre class="notranslate"><code class="notranslate">airflow.exceptions.AirflowException: The broker you configured does not support SSL_ACTIVE to be True. Please use RabbitMQ or Redis if you would like to use SSL for broker.
</code></pre></div>
<h3 dir="auto">What you think should happen instead</h3>
<p dir="auto">Celery has supported TLS on Redis Sentinel <a href="https://docs.celeryq.dev/en/latest/history/whatsnew-5.1.html#support-redis-sentinel-with-ssl" rel="nofollow">for a while</a> now.</p>
<p dir="auto">It looks like <a href="https://github.com/apache/airflow/blob/main/airflow/config_templates/default_celery.py#L68-L88">this piece of code</a> explicitly prohibits from passing a valid Redis Sentinel TLS configuration through to Celery. (Since Sentinel broker URL's are prefixed with <code class="notranslate">sentinel://</code> instead of <code class="notranslate">redis://</code>.)</p>
<h3 dir="auto">How to reproduce</h3>
<p dir="auto">This problem can be reproduced by deploying Airflow using Docker with the following environment variables:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="AIRFLOW__CELERY__BROKER_URL=sentinel://sentinel1:26379;sentinel://sentinel2:26379;sentinel://sentinel3:26379
AIRFLOW__CELERY__SSL_ACTIVE=true
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__MASTER_NAME='some-master-name'
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__PASSWORD='some-password'
AIRFLOW__LOGGING__LOGGING_LEVEL=DEBUG"><pre class="notranslate"><code class="notranslate">AIRFLOW__CELERY__BROKER_URL=sentinel://sentinel1:26379;sentinel://sentinel2:26379;sentinel://sentinel3:26379
AIRFLOW__CELERY__SSL_ACTIVE=true
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__MASTER_NAME='some-master-name'
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__PASSWORD='some-password'
AIRFLOW__LOGGING__LOGGING_LEVEL=DEBUG
</code></pre></div>
<p dir="auto">Note that I'm not 100% certain of the syntax for the password environment var. I can't get to the point of testing this because without TLS connections to our internal brokers are denied (because they require TLS), and with TLS it doesn't attempt a connection because of the earlier linked code.</p>
<p dir="auto">I've verified with the reference <code class="notranslate">redis-cli</code> that the settings we use for <code class="notranslate">master-name</code> does result in a valid response and the Sentinel set-up works as expected.</p>
<h3 dir="auto">Operating System</h3>
<p dir="auto">Docker (apache/airflow:2.4.3-python3.10)</p>
<h3 dir="auto">Versions of Apache Airflow Providers</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Deployment</h3>
<p dir="auto">Other Docker-based deployment</p>
<h3 dir="auto">Deployment details</h3>
<p dir="auto">Deployed using Nomad.</p>
<h3 dir="auto">Anything else</h3>
<p dir="auto">This is my first issue with this open source project. Please let me know if there's more relevant information I can provide to follow through on this issue.</p>
<p dir="auto">I will try to make some time available soon to see if a simple code change in the earlier mentioned file would work, but as this is my first issue here I would still have to set-up a full development environment.</p>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Yes I am willing to submit a PR!</li>
</ul>
<p dir="auto">If this is indeed a simple fix I'd be willing to look into making a PR. I would like some feedback on the problem first though if possible!</p>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>