<h3 dir="auto">Apache Airflow version</h3>
<p dir="auto">2.2.0 (latest released)</p>
<h3 dir="auto">Operating System</h3>
<p dir="auto">Debian GNU/Linux 10 (buster)</p>
<h3 dir="auto">Versions of Apache Airflow Providers</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apache-airflow-providers-celery==2.1.0
apache-airflow-providers-mysql==2.1.1
apache-airflow-providers-postgres==2.3.0
apache-airflow-providers-sqlite==2.0.1"><pre class="notranslate"><code class="notranslate">apache-airflow-providers-celery==2.1.0
apache-airflow-providers-mysql==2.1.1
apache-airflow-providers-postgres==2.3.0
apache-airflow-providers-sqlite==2.0.1
</code></pre></div>
<h3 dir="auto">Deployment</h3>
<p dir="auto">Docker-Compose</p>
<h3 dir="auto">Deployment details</h3>
<p dir="auto">docker-compose file:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="version: &quot;2&quot;

services:
  airflow-webserver:
    build: .
    image: airflow
    command: airflow webserver
    ports:
      - &quot;8080:8080&quot;

  airflow-scheduler:
    image: airflow
    command: airflow scheduler

  airflow-flower:
    image: airflow
    command: airflow celery flower
    ports:
      - &quot;5555:5555&quot;
    depends_on:
      - airflow-celery
      - airflow-webserver
      - airflow-scheduler
      - airflow-worker
      - airflow-broker

  airflow-worker:
    image: airflow
    command: airflow celery worker

  airflow-celery:
    image: mysql:8.0.19
    environment:
      MYSQL_PASSWORD: ...
      MYSQL_USER: ...
      MYSQL_DATABASE: airflow
      MYSQL_HOST: airflow-celery

  airflow-broker:
    image: redis:5.0.7-alpine

volumes:
  dbdata:"><pre class="notranslate"><code class="notranslate">version: "2"

services:
  airflow-webserver:
    build: .
    image: airflow
    command: airflow webserver
    ports:
      - "8080:8080"

  airflow-scheduler:
    image: airflow
    command: airflow scheduler

  airflow-flower:
    image: airflow
    command: airflow celery flower
    ports:
      - "5555:5555"
    depends_on:
      - airflow-celery
      - airflow-webserver
      - airflow-scheduler
      - airflow-worker
      - airflow-broker

  airflow-worker:
    image: airflow
    command: airflow celery worker

  airflow-celery:
    image: mysql:8.0.19
    environment:
      MYSQL_PASSWORD: ...
      MYSQL_USER: ...
      MYSQL_DATABASE: airflow
      MYSQL_HOST: airflow-celery

  airflow-broker:
    image: redis:5.0.7-alpine

volumes:
  dbdata:
</code></pre></div>
<p dir="auto">Dockerfile:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="FROM python:3.8

COPY requirements.txt .
RUN pip install -U pip
RUN pip install -r requirements.txt"><pre class="notranslate"><code class="notranslate">FROM python:3.8

COPY requirements.txt .
RUN pip install -U pip
RUN pip install -r requirements.txt
</code></pre></div>
<p dir="auto">requirements.txt:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apache-airflow[celery,postgres,slack,docker,redis,mysql,http]==2.2.0
kombu==4.6.10
python-dotenv
psycopg2-binary
..."><pre class="notranslate"><code class="notranslate">apache-airflow[celery,postgres,slack,docker,redis,mysql,http]==2.2.0
kombu==4.6.10
python-dotenv
psycopg2-binary
...
</code></pre></div>
<h3 dir="auto">What happened</h3>
<p dir="auto">After updating <code class="notranslate">requirements.txt</code> file to use Airflow <code class="notranslate">2.2.0</code> instead of <code class="notranslate">2.1.4</code>, I ran:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="~/airflow $ docker-compose build --no-cache
~/airflow $ docker-compose up -d --force
~/airflow $ docker exec -it airflow_airflow-webserver_1 airflow db upgrade"><pre class="notranslate"><code class="notranslate">~/airflow $ docker-compose build --no-cache
~/airflow $ docker-compose up -d --force
~/airflow $ docker exec -it airflow_airflow-webserver_1 airflow db upgrade
</code></pre></div>
<p dir="auto">Which throws this exception:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="DB: mysql://airflow:***@airflow-celery/airflow
[2021-10-13 12:22:57,699] {db.py:823} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 142555e44c17 -&gt; 7b2661a43ba3, TaskInstance keyed to DagRun
Traceback (most recent call last):
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py&quot;, line 1276, in _execute_context
    self.dialect.do_execute(
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py&quot;, line 608, in do_execute
    cursor.execute(statement, parameters)
  File &quot;/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py&quot;, line 206, in execute
    res = self._query(query)
  File &quot;/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py&quot;, line 319, in _query
    db.query(q)
  File &quot;/usr/local/lib/python3.8/site-packages/MySQLdb/connections.py&quot;, line 259, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1091, &quot;Can't DROP 'dag_id'; check that column/key exists&quot;)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File &quot;/usr/local/bin/airflow&quot;, line 8, in &lt;module&gt;
    sys.exit(main())
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/__main__.py&quot;, line 40, in main
    args.func(args)
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/cli/cli_parser.py&quot;, line 48, in command
    return func(*args, **kwargs)
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/utils/cli.py&quot;, line 92, in wrapper
    return f(*args, **kwargs)
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/cli/commands/db_command.py&quot;, line 48, in upgradedb
    db.upgradedb()
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/utils/session.py&quot;, line 70, in wrapper
    return func(*args, session=session, **kwargs)
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/utils/db.py&quot;, line 824, in upgradedb
    command.upgrade(config, 'heads')
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/command.py&quot;, line 320, in upgrade
    script.run_env()
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/script/base.py&quot;, line 563, in run_env
    util.load_python_file(self.dir, &quot;env.py&quot;)
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py&quot;, line 92, in load_python_file
    module = load_module_py(module_id, path)
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py&quot;, line 108, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File &quot;&lt;frozen importlib._bootstrap_external&gt;&quot;, line 848, in exec_module
  File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 219, in _call_with_frames_removed
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/migrations/env.py&quot;, line 107, in &lt;module&gt;
    run_migrations_online()
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/migrations/env.py&quot;, line 101, in run_migrations_online
    context.run_migrations()
  File &quot;&lt;string&gt;&quot;, line 8, in run_migrations
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/runtime/environment.py&quot;, line 851, in run_migrations
    self.get_context().run_migrations(**kw)
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/runtime/migration.py&quot;, line 620, in run_migrations
    step.migration_fn(**kw)
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/migrations/versions/7b2661a43ba3_taskinstance_keyed_to_dagrun.py&quot;, line 140, in upgrade
    batch_op.create_unique_constraint('dag_run_dag_id_run_id_key', ['dag_id', 'run_id'])
  File &quot;/usr/local/lib/python3.8/contextlib.py&quot;, line 120, in __exit__
    next(self.gen)
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/operations/base.py&quot;, line 374, in batch_alter_table
    impl.flush()
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/operations/batch.py&quot;, line 107, in flush
    fn(*arg, **kw)
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/ddl/mysql.py&quot;, line 150, in drop_constraint
    super(MySQLImpl, self).drop_constraint(const)
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/ddl/impl.py&quot;, line 340, in drop_constraint
    self._exec(schema.DropConstraint(const))
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/ddl/impl.py&quot;, line 197, in _exec
    return conn.execute(construct, multiparams)
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py&quot;, line 1011, in execute
    return meth(self, multiparams, params)
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/ddl.py&quot;, line 72, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py&quot;, line 1068, in _execute_ddl
    ret = self._execute_context(
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py&quot;, line 1316, in _execute_context
    self._handle_dbapi_exception(
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py&quot;, line 1510, in _handle_dbapi_exception
    util.raise_(
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py&quot;, line 182, in raise_
    raise exception
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py&quot;, line 1276, in _execute_context
    self.dialect.do_execute(
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py&quot;, line 608, in do_execute
    cursor.execute(statement, parameters)
  File &quot;/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py&quot;, line 206, in execute
    res = self._query(query)
  File &quot;/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py&quot;, line 319, in _query
    db.query(q)
  File &quot;/usr/local/lib/python3.8/site-packages/MySQLdb/connections.py&quot;, line 259, in query
    _mysql.connection.query(self, query)
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1091, &quot;Can't DROP 'dag_id'; check that column/key exists&quot;)
[SQL: ALTER TABLE dag_run DROP INDEX dag_id]
(Background on this error at: http://sqlalche.me/e/13/e3q8)"><pre class="notranslate"><code class="notranslate">DB: mysql://airflow:***@airflow-celery/airflow
[2021-10-13 12:22:57,699] {db.py:823} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 142555e44c17 -&gt; 7b2661a43ba3, TaskInstance keyed to DagRun
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1091, "Can't DROP 'dag_id'; check that column/key exists")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 8, in &lt;module&gt;
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/usr/local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/cli.py", line 92, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/db_command.py", line 48, in upgradedb
    db.upgradedb()
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper
    return func(*args, session=session, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/db.py", line 824, in upgradedb
    command.upgrade(config, 'heads')
  File "/usr/local/lib/python3.8/site-packages/alembic/command.py", line 320, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.8/site-packages/alembic/script/base.py", line 563, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "&lt;frozen importlib._bootstrap_external&gt;", line 848, in exec_module
  File "&lt;frozen importlib._bootstrap&gt;", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/site-packages/airflow/migrations/env.py", line 107, in &lt;module&gt;
    run_migrations_online()
  File "/usr/local/lib/python3.8/site-packages/airflow/migrations/env.py", line 101, in run_migrations_online
    context.run_migrations()
  File "&lt;string&gt;", line 8, in run_migrations
  File "/usr/local/lib/python3.8/site-packages/alembic/runtime/environment.py", line 851, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.8/site-packages/alembic/runtime/migration.py", line 620, in run_migrations
    step.migration_fn(**kw)
  File "/usr/local/lib/python3.8/site-packages/airflow/migrations/versions/7b2661a43ba3_taskinstance_keyed_to_dagrun.py", line 140, in upgrade
    batch_op.create_unique_constraint('dag_run_dag_id_run_id_key', ['dag_id', 'run_id'])
  File "/usr/local/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.8/site-packages/alembic/operations/base.py", line 374, in batch_alter_table
    impl.flush()
  File "/usr/local/lib/python3.8/site-packages/alembic/operations/batch.py", line 107, in flush
    fn(*arg, **kw)
  File "/usr/local/lib/python3.8/site-packages/alembic/ddl/mysql.py", line 150, in drop_constraint
    super(MySQLImpl, self).drop_constraint(const)
  File "/usr/local/lib/python3.8/site-packages/alembic/ddl/impl.py", line 340, in drop_constraint
    self._exec(schema.DropConstraint(const))
  File "/usr/local/lib/python3.8/site-packages/alembic/ddl/impl.py", line 197, in _exec
    return conn.execute(construct, multiparams)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/ddl.py", line 72, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1068, in _execute_ddl
    ret = self._execute_context(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
    util.raise_(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
    _mysql.connection.query(self, query)
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1091, "Can't DROP 'dag_id'; check that column/key exists")
[SQL: ALTER TABLE dag_run DROP INDEX dag_id]
(Background on this error at: http://sqlalche.me/e/13/e3q8)
</code></pre></div>
<p dir="auto">Trying to drop the index manually, gives the same output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="~/airflow $ docker exec -it airflow_airflow-celery_1 mysql
mysql&gt; use airflow;
mysql&gt; ALTER TABLE airflow.dag_run DROP INDEX dag_id;
ERROR 1091 (42000): Can't DROP 'dag_id'; check that column/key exists"><pre class="notranslate"><code class="notranslate">~/airflow $ docker exec -it airflow_airflow-celery_1 mysql
mysql&gt; use airflow;
mysql&gt; ALTER TABLE airflow.dag_run DROP INDEX dag_id;
ERROR 1091 (42000): Can't DROP 'dag_id'; check that column/key exists
</code></pre></div>
<h3 dir="auto">What you expected to happen</h3>
<p dir="auto"><code class="notranslate">airflow db upgrade</code> to not fail</p>
<h3 dir="auto">How to reproduce</h3>
<ul dir="auto">
<li>Copy the provided <code class="notranslate">docker-compose.yml</code> file content in conjunction with <code class="notranslate">Dockerfile</code> &amp; <code class="notranslate">requirements.txt</code> with Airflow <code class="notranslate">2.1.4</code></li>
<li>Init db</li>
<li>build docker containers</li>
<li>all services should to be up &amp; running</li>
<li>now update <code class="notranslate">requirements.txt</code> to use <code class="notranslate">2.2.0</code></li>
<li>build docker containers again</li>
<li>Run <code class="notranslate">airflow db upgrade</code> command</li>
<li>You would see error in stdout as well as <code class="notranslate">worker</code> service fails to run</li>
</ul>
<h3 dir="auto">Anything else</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>
<h3 dir="auto">Apache Airflow version</h3>
<p dir="auto">2.2.0 (latest released)</p>
<h3 dir="auto">Operating System</h3>
<p dir="auto">Debian GNU/Linux 10 (buster)</p>
<h3 dir="auto">Versions of Apache Airflow Providers</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apache-airflow-providers-celery==2.1.0
apache-airflow-providers-mysql==2.1.1
apache-airflow-providers-postgres==2.3.0
apache-airflow-providers-sqlite==2.0.1"><pre class="notranslate"><code class="notranslate">apache-airflow-providers-celery==2.1.0
apache-airflow-providers-mysql==2.1.1
apache-airflow-providers-postgres==2.3.0
apache-airflow-providers-sqlite==2.0.1
</code></pre></div>
<h3 dir="auto">Deployment</h3>
<p dir="auto">Docker-Compose</p>
<h3 dir="auto">Deployment details</h3>
<p dir="auto">docker-compose file:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="version: &quot;2&quot;

services:
  airflow-webserver:
    build: .
    image: airflow
    command: airflow webserver
    ports:
      - &quot;8080:8080&quot;

  airflow-scheduler:
    image: airflow
    command: airflow scheduler

  airflow-flower:
    image: airflow
    command: airflow celery flower
    ports:
      - &quot;5555:5555&quot;
    depends_on:
      - airflow-celery
      - airflow-webserver
      - airflow-scheduler
      - airflow-worker
      - airflow-broker

  airflow-worker:
    image: airflow
    command: airflow celery worker

  airflow-celery:
    image: mysql:8.0.19
    environment:
      MYSQL_PASSWORD: ...
      MYSQL_USER: ...
      MYSQL_DATABASE: airflow
      MYSQL_HOST: airflow-celery

  airflow-broker:
    image: redis:5.0.7-alpine

volumes:
  dbdata:"><pre class="notranslate"><code class="notranslate">version: "2"

services:
  airflow-webserver:
    build: .
    image: airflow
    command: airflow webserver
    ports:
      - "8080:8080"

  airflow-scheduler:
    image: airflow
    command: airflow scheduler

  airflow-flower:
    image: airflow
    command: airflow celery flower
    ports:
      - "5555:5555"
    depends_on:
      - airflow-celery
      - airflow-webserver
      - airflow-scheduler
      - airflow-worker
      - airflow-broker

  airflow-worker:
    image: airflow
    command: airflow celery worker

  airflow-celery:
    image: mysql:8.0.19
    environment:
      MYSQL_PASSWORD: ...
      MYSQL_USER: ...
      MYSQL_DATABASE: airflow
      MYSQL_HOST: airflow-celery

  airflow-broker:
    image: redis:5.0.7-alpine

volumes:
  dbdata:
</code></pre></div>
<p dir="auto">Dockerfile:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="FROM python:3.8

COPY requirements.txt .
RUN pip install -U pip
RUN pip install -r requirements.txt"><pre class="notranslate"><code class="notranslate">FROM python:3.8

COPY requirements.txt .
RUN pip install -U pip
RUN pip install -r requirements.txt
</code></pre></div>
<p dir="auto">requirements.txt:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="apache-airflow[celery,postgres,slack,docker,redis,mysql,http]==2.2.0
kombu==4.6.10
python-dotenv
psycopg2-binary
..."><pre class="notranslate"><code class="notranslate">apache-airflow[celery,postgres,slack,docker,redis,mysql,http]==2.2.0
kombu==4.6.10
python-dotenv
psycopg2-binary
...
</code></pre></div>
<h3 dir="auto">What happened</h3>
<p dir="auto">After updating <code class="notranslate">requirements.txt</code> file to use Airflow <code class="notranslate">2.2.0</code> instead of <code class="notranslate">2.1.4</code>, I ran:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="~/airflow $ docker-compose build --no-cache
~/airflow $ docker-compose up -d --force
~/airflow $ docker exec -it airflow_airflow-webserver_1 airflow db upgrade"><pre class="notranslate"><code class="notranslate">~/airflow $ docker-compose build --no-cache
~/airflow $ docker-compose up -d --force
~/airflow $ docker exec -it airflow_airflow-webserver_1 airflow db upgrade
</code></pre></div>
<p dir="auto">Which throws this exception:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="DB: mysql://airflow:***@airflow-celery/airflow
[2021-10-13 12:22:57,699] {db.py:823} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 142555e44c17 -&gt; 7b2661a43ba3, TaskInstance keyed to DagRun
Traceback (most recent call last):
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py&quot;, line 1276, in _execute_context
    self.dialect.do_execute(
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py&quot;, line 608, in do_execute
    cursor.execute(statement, parameters)
  File &quot;/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py&quot;, line 206, in execute
    res = self._query(query)
  File &quot;/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py&quot;, line 319, in _query
    db.query(q)
  File &quot;/usr/local/lib/python3.8/site-packages/MySQLdb/connections.py&quot;, line 259, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1091, &quot;Can't DROP 'dag_id'; check that column/key exists&quot;)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File &quot;/usr/local/bin/airflow&quot;, line 8, in &lt;module&gt;
    sys.exit(main())
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/__main__.py&quot;, line 40, in main
    args.func(args)
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/cli/cli_parser.py&quot;, line 48, in command
    return func(*args, **kwargs)
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/utils/cli.py&quot;, line 92, in wrapper
    return f(*args, **kwargs)
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/cli/commands/db_command.py&quot;, line 48, in upgradedb
    db.upgradedb()
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/utils/session.py&quot;, line 70, in wrapper
    return func(*args, session=session, **kwargs)
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/utils/db.py&quot;, line 824, in upgradedb
    command.upgrade(config, 'heads')
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/command.py&quot;, line 320, in upgrade
    script.run_env()
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/script/base.py&quot;, line 563, in run_env
    util.load_python_file(self.dir, &quot;env.py&quot;)
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py&quot;, line 92, in load_python_file
    module = load_module_py(module_id, path)
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py&quot;, line 108, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File &quot;&lt;frozen importlib._bootstrap_external&gt;&quot;, line 848, in exec_module
  File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 219, in _call_with_frames_removed
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/migrations/env.py&quot;, line 107, in &lt;module&gt;
    run_migrations_online()
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/migrations/env.py&quot;, line 101, in run_migrations_online
    context.run_migrations()
  File &quot;&lt;string&gt;&quot;, line 8, in run_migrations
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/runtime/environment.py&quot;, line 851, in run_migrations
    self.get_context().run_migrations(**kw)
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/runtime/migration.py&quot;, line 620, in run_migrations
    step.migration_fn(**kw)
  File &quot;/usr/local/lib/python3.8/site-packages/airflow/migrations/versions/7b2661a43ba3_taskinstance_keyed_to_dagrun.py&quot;, line 140, in upgrade
    batch_op.create_unique_constraint('dag_run_dag_id_run_id_key', ['dag_id', 'run_id'])
  File &quot;/usr/local/lib/python3.8/contextlib.py&quot;, line 120, in __exit__
    next(self.gen)
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/operations/base.py&quot;, line 374, in batch_alter_table
    impl.flush()
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/operations/batch.py&quot;, line 107, in flush
    fn(*arg, **kw)
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/ddl/mysql.py&quot;, line 150, in drop_constraint
    super(MySQLImpl, self).drop_constraint(const)
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/ddl/impl.py&quot;, line 340, in drop_constraint
    self._exec(schema.DropConstraint(const))
  File &quot;/usr/local/lib/python3.8/site-packages/alembic/ddl/impl.py&quot;, line 197, in _exec
    return conn.execute(construct, multiparams)
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py&quot;, line 1011, in execute
    return meth(self, multiparams, params)
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/ddl.py&quot;, line 72, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py&quot;, line 1068, in _execute_ddl
    ret = self._execute_context(
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py&quot;, line 1316, in _execute_context
    self._handle_dbapi_exception(
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py&quot;, line 1510, in _handle_dbapi_exception
    util.raise_(
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py&quot;, line 182, in raise_
    raise exception
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py&quot;, line 1276, in _execute_context
    self.dialect.do_execute(
  File &quot;/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py&quot;, line 608, in do_execute
    cursor.execute(statement, parameters)
  File &quot;/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py&quot;, line 206, in execute
    res = self._query(query)
  File &quot;/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py&quot;, line 319, in _query
    db.query(q)
  File &quot;/usr/local/lib/python3.8/site-packages/MySQLdb/connections.py&quot;, line 259, in query
    _mysql.connection.query(self, query)
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1091, &quot;Can't DROP 'dag_id'; check that column/key exists&quot;)
[SQL: ALTER TABLE dag_run DROP INDEX dag_id]
(Background on this error at: http://sqlalche.me/e/13/e3q8)"><pre class="notranslate"><code class="notranslate">DB: mysql://airflow:***@airflow-celery/airflow
[2021-10-13 12:22:57,699] {db.py:823} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl MySQLImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade 142555e44c17 -&gt; 7b2661a43ba3, TaskInstance keyed to DagRun
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1091, "Can't DROP 'dag_id'; check that column/key exists")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 8, in &lt;module&gt;
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/usr/local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/cli.py", line 92, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/db_command.py", line 48, in upgradedb
    db.upgradedb()
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper
    return func(*args, session=session, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/db.py", line 824, in upgradedb
    command.upgrade(config, 'heads')
  File "/usr/local/lib/python3.8/site-packages/alembic/command.py", line 320, in upgrade
    script.run_env()
  File "/usr/local/lib/python3.8/site-packages/alembic/script/base.py", line 563, in run_env
    util.load_python_file(self.dir, "env.py")
  File "/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 92, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 108, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
  File "&lt;frozen importlib._bootstrap_external&gt;", line 848, in exec_module
  File "&lt;frozen importlib._bootstrap&gt;", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/site-packages/airflow/migrations/env.py", line 107, in &lt;module&gt;
    run_migrations_online()
  File "/usr/local/lib/python3.8/site-packages/airflow/migrations/env.py", line 101, in run_migrations_online
    context.run_migrations()
  File "&lt;string&gt;", line 8, in run_migrations
  File "/usr/local/lib/python3.8/site-packages/alembic/runtime/environment.py", line 851, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/local/lib/python3.8/site-packages/alembic/runtime/migration.py", line 620, in run_migrations
    step.migration_fn(**kw)
  File "/usr/local/lib/python3.8/site-packages/airflow/migrations/versions/7b2661a43ba3_taskinstance_keyed_to_dagrun.py", line 140, in upgrade
    batch_op.create_unique_constraint('dag_run_dag_id_run_id_key', ['dag_id', 'run_id'])
  File "/usr/local/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.8/site-packages/alembic/operations/base.py", line 374, in batch_alter_table
    impl.flush()
  File "/usr/local/lib/python3.8/site-packages/alembic/operations/batch.py", line 107, in flush
    fn(*arg, **kw)
  File "/usr/local/lib/python3.8/site-packages/alembic/ddl/mysql.py", line 150, in drop_constraint
    super(MySQLImpl, self).drop_constraint(const)
  File "/usr/local/lib/python3.8/site-packages/alembic/ddl/impl.py", line 340, in drop_constraint
    self._exec(schema.DropConstraint(const))
  File "/usr/local/lib/python3.8/site-packages/alembic/ddl/impl.py", line 197, in _exec
    return conn.execute(construct, multiparams)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/ddl.py", line 72, in _execute_on_connection
    return connection._execute_ddl(self, multiparams, params)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1068, in _execute_ddl
    ret = self._execute_context(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
    util.raise_(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
    _mysql.connection.query(self, query)
sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1091, "Can't DROP 'dag_id'; check that column/key exists")
[SQL: ALTER TABLE dag_run DROP INDEX dag_id]
(Background on this error at: http://sqlalche.me/e/13/e3q8)
</code></pre></div>
<p dir="auto">Trying to drop the index manually, gives the same output:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="~/airflow $ docker exec -it airflow_airflow-celery_1 mysql
mysql&gt; use airflow;
mysql&gt; ALTER TABLE airflow.dag_run DROP INDEX dag_id;
ERROR 1091 (42000): Can't DROP 'dag_id'; check that column/key exists"><pre class="notranslate"><code class="notranslate">~/airflow $ docker exec -it airflow_airflow-celery_1 mysql
mysql&gt; use airflow;
mysql&gt; ALTER TABLE airflow.dag_run DROP INDEX dag_id;
ERROR 1091 (42000): Can't DROP 'dag_id'; check that column/key exists
</code></pre></div>
<h3 dir="auto">What you expected to happen</h3>
<p dir="auto"><code class="notranslate">airflow db upgrade</code> to not fail</p>
<h3 dir="auto">How to reproduce</h3>
<ul dir="auto">
<li>Copy the provided <code class="notranslate">docker-compose.yml</code> file content in conjunction with <code class="notranslate">Dockerfile</code> &amp; <code class="notranslate">requirements.txt</code> with Airflow <code class="notranslate">2.1.4</code></li>
<li>Init db</li>
<li>build docker containers</li>
<li>all services should to be up &amp; running</li>
<li>now update <code class="notranslate">requirements.txt</code> to use <code class="notranslate">2.2.0</code></li>
<li>build docker containers again</li>
<li>Run <code class="notranslate">airflow db upgrade</code> command</li>
<li>You would see error in stdout as well as <code class="notranslate">worker</code> service fails to run</li>
</ul>
<h3 dir="auto">Anything else</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Are you willing to submit PR?</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Yes I am willing to submit a PR!</li>
</ul>
<h3 dir="auto">Code of Conduct</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I agree to follow this project's <a href="https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md">Code of Conduct</a></li>
</ul>