<p dir="auto"><strong>Apache Airflow version</strong>: 2.0.0</p>
<p dir="auto"><strong>Kubernetes version (if you are using kubernetes)</strong> (use <code class="notranslate">kubectl version</code>): 1.19.4</p>
<p dir="auto"><strong>Environment</strong>:</p>
<ul dir="auto">
<li><strong>Cloud provider or hardware configuration</strong>: Google Cloud Platform/GKE</li>
</ul>
<p dir="auto"><strong>What happened</strong>:</p>
<p dir="auto">I successfully cleared the state of a failed task using the graph view UI, but when I attempted to re-run the cleared task instance in graph view manually by selecting the task instance and clicking "Run", I received the following error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Something bad has happened.
Please consider letting us know by creating a bug report using GitHub.

Python version: 3.8.7
Airflow version: 2.0.0
Node: mr_node
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/flask/app.py&quot;, line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/flask/app.py&quot;, line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/flask/app.py&quot;, line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/flask/_compat.py&quot;, line 39, in reraise
    raise value
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/flask/app.py&quot;, line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/flask/app.py&quot;, line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/airflow/www/auth.py&quot;, line 34, in decorated
    return func(*args, **kwargs)
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/airflow/www/decorators.py&quot;, line 60, in wrapper
    return f(*args, **kwargs)
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/airflow/www/views.py&quot;, line 1366, in run
    executor.start()
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py&quot;, line 493, in start
    raise AirflowException(&quot;Could not get scheduler_job_id&quot;)
airflow.exceptions.AirflowException: Could not get scheduler_job_id"><pre class="notranslate"><code class="notranslate">Something bad has happened.
Please consider letting us know by creating a bug report using GitHub.

Python version: 3.8.7
Airflow version: 2.0.0
Node: mr_node
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/airflow/.local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/airflow/.local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/auth.py", line 34, in decorated
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/decorators.py", line 60, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/www/views.py", line 1366, in run
    executor.start()
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 493, in start
    raise AirflowException("Could not get scheduler_job_id")
airflow.exceptions.AirflowException: Could not get scheduler_job_id
</code></pre></div>
<p dir="auto"><strong>What you expected to happen</strong>:</p>
<p dir="auto">I expected the task instance to be scheduled and begin running again.</p>
<p dir="auto"><strong>How to reproduce it</strong>:</p>
<p dir="auto">Configure Airflow 2.0.0 to run on GCP, clear the state of a finished task instance using the UI (I was able to reproduce the error on a task instance maked "Success" as well), and again use the Web UI to "Run" the task.</p>
<p dir="auto"><strong>Anything else we need to know</strong>:</p>
<p dir="auto">One important item to note is that when I <em>only</em> clear task instance and do not attempt to run it manually using the UI, the task does queue and is placing in a <code class="notranslate">running</code> state, but quickly fails with the following error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2021-01-08 21:08:16,140] {taskinstance.py:1396} ERROR - (0)
Reason: Handshake status 500 Internal Server Error
Traceback (most recent call last):
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/kubernetes/stream/ws_client.py&quot;, line 296, in websocket_call
    client = WSClient(configuration, get_websocket_url(url), headers, capture_all)
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/kubernetes/stream/ws_client.py&quot;, line 94, in __init__
    self.sock.connect(url, header=header)
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/websocket/_core.py&quot;, line 226, in connect
    self.handshake_response = handshake(self.sock, *addrs, **options)
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/websocket/_handshake.py&quot;, line 80, in handshake
    status, resp = _get_resp_headers(sock)
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/websocket/_handshake.py&quot;, line 165, in _get_resp_headers
    raise WebSocketBadStatusException(&quot;Handshake status %d %s&quot;, status, status_message, resp_headers)
websocket._exceptions.WebSocketBadStatusException: Handshake status 500 Internal Server Error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py&quot;, line 1086, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py&quot;, line 1260, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py&quot;, line 1300, in _execute_task
    result = task_copy.execute(context=context)
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py&quot;, line 335, in execute
    final_state, result = self.handle_pod_overlap(
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py&quot;, line 375, in handle_pod_overlap
    final_state, result = self.monitor_launched_pod(launcher, pod)
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py&quot;, line 513, in monitor_launched_pod
    (final_state, result) = launcher.monitor_pod(pod, get_logs=self.get_logs)
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/airflow/kubernetes/pod_launcher.py&quot;, line 151, in monitor_pod
    result = self._extract_xcom(pod)
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/airflow/kubernetes/pod_launcher.py&quot;, line 246, in _extract_xcom
    resp = kubernetes_stream(
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/kubernetes/stream/stream.py&quot;, line 35, in stream
    return func(*args, **kwargs)
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py&quot;, line 841, in connect_get_namespaced_pod_exec
    (data) = self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs)  # noqa: E501
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py&quot;, line 927, in connect_get_namespaced_pod_exec_with_http_info
    return self.api_client.call_api(
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/kubernetes/client/api_client.py&quot;, line 340, in call_api
    return self.__call_api(resource_path, method,
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/kubernetes/client/api_client.py&quot;, line 172, in __call_api
    response_data = self.request(
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/kubernetes/stream/stream.py&quot;, line 30, in _intercept_request_call
    return ws_client.websocket_call(config, *args, **kwargs)
  File &quot;/home/airflow/.local/lib/python3.8/site-packages/kubernetes/stream/ws_client.py&quot;, line 302, in websocket_call
    raise ApiException(status=0, reason=str(e))
kubernetes.client.rest.ApiException: (0)
Reason: Handshake status 500 Internal Server Error"><pre class="notranslate"><code class="notranslate">[2021-01-08 21:08:16,140] {taskinstance.py:1396} ERROR - (0)
Reason: Handshake status 500 Internal Server Error
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/stream/ws_client.py", line 296, in websocket_call
    client = WSClient(configuration, get_websocket_url(url), headers, capture_all)
  File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/stream/ws_client.py", line 94, in __init__
    self.sock.connect(url, header=header)
  File "/home/airflow/.local/lib/python3.8/site-packages/websocket/_core.py", line 226, in connect
    self.handshake_response = handshake(self.sock, *addrs, **options)
  File "/home/airflow/.local/lib/python3.8/site-packages/websocket/_handshake.py", line 80, in handshake
    status, resp = _get_resp_headers(sock)
  File "/home/airflow/.local/lib/python3.8/site-packages/websocket/_handshake.py", line 165, in _get_resp_headers
    raise WebSocketBadStatusException("Handshake status %d %s", status, status_message, resp_headers)
websocket._exceptions.WebSocketBadStatusException: Handshake status 500 Internal Server Error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1086, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1260, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1300, in _execute_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 335, in execute
    final_state, result = self.handle_pod_overlap(
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 375, in handle_pod_overlap
    final_state, result = self.monitor_launched_pod(launcher, pod)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 513, in monitor_launched_pod
    (final_state, result) = launcher.monitor_pod(pod, get_logs=self.get_logs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/kubernetes/pod_launcher.py", line 151, in monitor_pod
    result = self._extract_xcom(pod)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/kubernetes/pod_launcher.py", line 246, in _extract_xcom
    resp = kubernetes_stream(
  File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/stream/stream.py", line 35, in stream
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 841, in connect_get_namespaced_pod_exec
    (data) = self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs)  # noqa: E501
  File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 927, in connect_get_namespaced_pod_exec_with_http_info
    return self.api_client.call_api(
  File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 340, in call_api
    return self.__call_api(resource_path, method,
  File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 172, in __call_api
    response_data = self.request(
  File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/stream/stream.py", line 30, in _intercept_request_call
    return ws_client.websocket_call(config, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/kubernetes/stream/ws_client.py", line 302, in websocket_call
    raise ApiException(status=0, reason=str(e))
kubernetes.client.rest.ApiException: (0)
Reason: Handshake status 500 Internal Server Error
</code></pre></div>
<p dir="auto">Thanks!</p>
<p dir="auto"><strong>Apache Airflow version:</strong></p>
<p dir="auto">2.0.0</p>
<p dir="auto"><strong>Kubernetes version (if you are using kubernetes) (use kubectl version):</strong></p>
<p dir="auto">1.18.3</p>
<p dir="auto"><strong>Environment:</strong></p>
<p dir="auto">Cloud provider or hardware configuration: AWS</p>
<p dir="auto"><strong>What happened:</strong></p>
<p dir="auto">When trying to run a DAG, it gets scheduled, but task is never run. When attempting to run task manually, it shows an error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Something bad has happened.
Please consider letting us know by creating a bug report using GitHub.

Python version: 3.8.7
Airflow version: 2.0.0
Node: airflow-web-ffdd89d6-h98vj
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File &quot;/usr/local/lib/python3.8/site-packages/flask/app.py&quot;, line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File &quot;/usr/local/lib/python3.8/site-packages/flask/app.py&quot;, line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File &quot;/usr/local/lib/python3.8/site-packages/flask/app.py&quot;, line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File &quot;/usr/local/lib/python3.8/site-packages/flask/_compat.py&quot;, line 39, in reraise
    raise value
  File &quot;/usr/local/lib/python3.8/site-packages/flask/app.py&quot;, line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File &quot;/usr/local/lib/python3.8/site-packages/flask/app.py&quot;, line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/www/auth.py&quot;, line 34, in decorated
    return func(*args, **kwargs)
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/www/decorators.py&quot;, line 60, in wrapper
    return f(*args, **kwargs)
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/www/views.py&quot;, line 1366, in run
    executor.start()
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py&quot;, line 493, in start
    raise AirflowException(&quot;Could not get scheduler_job_id&quot;)
airflow.exceptions.AirflowException: Could not get scheduler_job_id"><pre class="notranslate"><code class="notranslate">Something bad has happened.
Please consider letting us know by creating a bug report using GitHub.

Python version: 3.8.7
Airflow version: 2.0.0
Node: airflow-web-ffdd89d6-h98vj
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/auth.py", line 34, in decorated
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/decorators.py", line 60, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/views.py", line 1366, in run
    executor.start()
  File "/usr/local/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 493, in start
    raise AirflowException("Could not get scheduler_job_id")
airflow.exceptions.AirflowException: Could not get scheduler_job_id
</code></pre></div>
<p dir="auto"><strong>What you expected to happen:</strong></p>
<p dir="auto">The task to be run successfully without</p>
<p dir="auto"><strong>How to reproduce it:</strong></p>
<p dir="auto">Haven't pinpointed what causes the issue, besides an attempted upgrade from Airflow 1.10.14 to Airflow 2.0.0</p>
<p dir="auto"><strong>Anything else we need to know:</strong></p>
<p dir="auto">This error is encountered in an upgrade of Airflow from 1.10.14 to Airflow 2.0.0</p>
<p dir="auto">EDIT: Formatted to fit the issue template</p>