<p dir="auto"><strong>Apache Airflow version</strong>:<br>
1.10.10</p>
<p dir="auto"><strong>Kubernetes version (if you are using kubernetes)</strong> (use <code class="notranslate">kubectl version</code>):<br>
1.18.3</p>
<p dir="auto"><strong>Environment</strong>:<br>
Kubernetes cluser</p>
<p dir="auto"><strong>What happened</strong>:<br>
During dag execution I am not able to see logs</p>
<blockquote>
<p dir="auto">*** Log file does not exist: /opt/airflow/logs/example_dag/python_print_date_152/2019-01-01T00:00:00+00:00/1.log<br>
*** Fetching from: htttp://fi-airflow-worker-97bcfb76-jdwj7:8793/log/example_dag/python_print_date_152/2019-01-01T00:00:00+00:00/1.log<br>
*** Failed to fetch log file from worker. HTTPConnectionPool(host='fi-airflow-worker-97bcfb76-jdwj7', port=8793): Max retries exceeded with url: /log/example_dag/python_print_date_152/2019-01-01T00:00:00+00:00/1.log (Caused by NewConnectionError('&lt;urllib3.connection.HTTPConnection object at 0x7f6409dfcd10&gt;: Failed to establish a new connection: [Errno -2] Name or service not known'))</p>
</blockquote>

<p dir="auto"><strong>What you expected to happen</strong>:<br>
See logs from dag execution in UI no only from command line</p>
<p dir="auto"><strong>Apache Airflow version</strong>: 2.0.2</p>
<p dir="auto"><strong>Kubernetes version (if you are using kubernetes)</strong>:</p>
<ul dir="auto">
<li>Client Version: v1.17.4</li>
<li>Server Version: v1.18.14</li>
</ul>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: Azure</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">Using KEDA, the logs are not accessible.</p>
<p dir="auto">Trying to access the logs on a task gives an error like the following:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="*** Log file does not exist: /opt/airflow/logs/tutorial_taskflow_api_etl/load/2021-04-28T09:58:43.050614+00:00/1.log
*** Fetching from: http://airflow2-worker-0.airflow2-worker.airflow2.svc.cluster.local:8793/log/tutorial_taskflow_api_etl/load/2021-04-28T09:58:43.050614+00:00/1.log
*** Failed to fetch log file from worker. HTTPConnectionPool(host='airflow2-worker-0.airflow2-worker.airflow2.svc.cluster.local', port=8793): Max retries exceeded with url: /log/tutorial_taskflow_api_etl/load/2021-04-28T09:58:43.050614+00:00/1.log (Caused by NewConnectionError('&lt;urllib3.connection.HTTPConnection object at 0x7f761829eda0&gt;: Failed to establish a new connection: [Errno -2] Name or service not known',))"><pre class="notranslate"><span class="pl-k">***</span> Log file does not exist: /opt/airflow/logs/tutorial_taskflow_api_etl/load/2021-04-28T09:58:43.050614+00:00/1.log
<span class="pl-k">***</span> Fetching from: http://airflow2-worker-0.airflow2-worker.airflow2.svc.cluster.local:8793/log/tutorial_taskflow_api_etl/load/2021-04-28T09:58:43.050614+00:00/1.log
<span class="pl-k">***</span> Failed to fetch log file from worker. HTTPConnectionPool(host=<span class="pl-s"><span class="pl-pds">'</span>airflow2-worker-0.airflow2-worker.airflow2.svc.cluster.local<span class="pl-pds">'</span></span>, port=8793): Max retries exceeded with url: /log/tutorial_taskflow_api_etl/load/2021-04-28T09:58:43.050614+00:00/1.log (Caused by NewConnectionError(<span class="pl-s"><span class="pl-pds">'</span>&lt;urllib3.connection.HTTPConnection object at 0x7f761829eda0&gt;: Failed to establish a new connection: [Errno -2] Name or service not known<span class="pl-pds">'</span></span>,))</pre></div>
<p dir="auto">Accessing the logs when the worker is executing a task (hence running) works as expected.</p>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">To be able to access the logs even when the worker is not running</p>
<p dir="auto"><strong>How to reproduce it</strong>:</p>
<p dir="auto">Install Airflow using the Helm chart with the values</p>
<div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="executor: &quot;CeleryExecutor&quot;
workers:
  keda:
    enabled: true
dags:
  persistence:
    enabled: true
    existingClaim: dags"><pre class="notranslate"><span class="pl-ent">executor</span>: <span class="pl-s"><span class="pl-pds">"</span>CeleryExecutor<span class="pl-pds">"</span></span>
<span class="pl-ent">workers</span>:
  <span class="pl-ent">keda</span>:
    <span class="pl-ent">enabled</span>: <span class="pl-c1">true</span>
<span class="pl-ent">dags</span>:
  <span class="pl-ent">persistence</span>:
    <span class="pl-ent">enabled</span>: <span class="pl-c1">true</span>
    <span class="pl-ent">existingClaim</span>: <span class="pl-s">dags</span></pre></div>
<p dir="auto">where the persistent volume claim for the dags in</p>
<div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: dags
  labels:
    tier: airflow
    release: airflow2
    heritage: &quot;Helm&quot;
spec:
  accessModes:
    - &quot;ReadWriteMany&quot;
  resources:
    requests:
      storage: &quot;1Gi&quot;
  storageClassName: &quot;azurefile&quot;"><pre class="notranslate"><span class="pl-ent">kind</span>: <span class="pl-s">PersistentVolumeClaim</span>
<span class="pl-ent">apiVersion</span>: <span class="pl-c1">v1</span>
<span class="pl-ent">metadata</span>:
  <span class="pl-ent">name</span>: <span class="pl-s">dags</span>
  <span class="pl-ent">labels</span>:
    <span class="pl-ent">tier</span>: <span class="pl-s">airflow</span>
    <span class="pl-ent">release</span>: <span class="pl-s">airflow2</span>
    <span class="pl-ent">heritage</span>: <span class="pl-s"><span class="pl-pds">"</span>Helm<span class="pl-pds">"</span></span>
<span class="pl-ent">spec</span>:
  <span class="pl-ent">accessModes</span>:
    - <span class="pl-s"><span class="pl-pds">"</span>ReadWriteMany<span class="pl-pds">"</span></span>
  <span class="pl-ent">resources</span>:
    <span class="pl-ent">requests</span>:
      <span class="pl-ent">storage</span>: <span class="pl-s"><span class="pl-pds">"</span>1Gi<span class="pl-pds">"</span></span>
  <span class="pl-ent">storageClassName</span>: <span class="pl-s"><span class="pl-pds">"</span>azurefile<span class="pl-pds">"</span></span></pre></div>