<p dir="auto"><strong>Apache Airflow version</strong>: 1.10.12<br>
<strong>Environment</strong>: VMware VM<br>
<strong>Cloud provider or hardware configuration</strong>: 8 G RAM, 50G HDD, 4Cores<br>
<strong>OS</strong>:<br>
NAME="CentOS Linux"<br>
VERSION="7 (Core)"<br>
ID="centos"<br>
ID_LIKE="rhel fedora"<br>
VERSION_ID="7"<br>
PRETTY_NAME="CentOS Linux 7 (Core)"<br>
ANSI_COLOR="0;31"<br>
CPE_NAME="cpe:/o:centos:centos:7"<br>
HOME_URL="<a href="https://www.centos.org/" rel="nofollow">https://www.centos.org/</a>"<br>
BUG_REPORT_URL="<a href="https://bugs.centos.org/" rel="nofollow">https://bugs.centos.org/</a>"</p>
<p dir="auto">CENTOS_MANTISBT_PROJECT="CentOS-7"<br>
CENTOS_MANTISBT_PROJECT_VERSION="7"<br>
REDHAT_SUPPORT_PRODUCT="centos"<br>
REDHAT_SUPPORT_PRODUCT_VERSION="7"<br>
<strong>Kernel</strong>:<br>
Linux mycentosvm 3.10.0-1160.6.1.el7.x86_64 <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="69689814" data-permission-text="Title is private" data-url="https://github.com/apache/airflow/issues/1" data-hovercard-type="pull_request" data-hovercard-url="/apache/airflow/pull/1/hovercard" href="https://github.com/apache/airflow/pull/1">#1</a> SMP Tue Nov 17 13:59:11 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux<br>
<strong>Install tools</strong>:<br>
yum groupinstall "Development tools"<br>
yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel python3-devel wget gcc gcc-c++ libffi-devel  cyrus-sasl-devel<br>
yum install mysql-server [version8]<br>
[set global_explicit, create airflow database and user then give permissions]<br>
alternative python 3.7 [pip 20.2.4  setuptools 47.1.0 ]</p>
<p dir="auto"><strong>What happened</strong>:<br>
All kinds of errors:<br>
I copied <code class="notranslate">https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt</code> to a file called <code class="notranslate">req.txt</code> to be able to change the dependencies</p>
<ol dir="auto">
<li>Error1:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ERROR: Could not satisfy constraints for 'pyhive': installation from path or url cannot be constrained to a version"><pre class="notranslate"><code class="notranslate">ERROR: Could not satisfy constraints for 'pyhive': installation from path or url cannot be constrained to a version
</code></pre></div>
<p dir="auto">My solution1:<br>
Install specified pyhive manually</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install PyHive==0.6.3"><pre class="notranslate"><code class="notranslate">pip install PyHive==0.6.3
</code></pre></div>
<ol start="2" dir="auto">
<li>Error2:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ERROR: Cannot install apache-airflow, apache-airflow and apache-airflow because these package versions have conflicting dependencies.

The conflict is caused by:
    email-validator 1.1.1 depends on idna&gt;=2.0.0
    requests 2.24.0 depends on idna&lt;3 and &gt;=2.5
    moto 1.3.14 depends on idna&lt;2.9 and &gt;=2.5

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies"><pre class="notranslate"><code class="notranslate">ERROR: Cannot install apache-airflow, apache-airflow and apache-airflow because these package versions have conflicting dependencies.

The conflict is caused by:
    email-validator 1.1.1 depends on idna&gt;=2.0.0
    requests 2.24.0 depends on idna&lt;3 and &gt;=2.5
    moto 1.3.14 depends on idna&lt;2.9 and &gt;=2.5

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
</code></pre></div>
<p dir="auto">My solution2:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install idna==2.9"><pre class="notranslate"><code class="notranslate">pip install idna==2.9
</code></pre></div>
<ol start="3" dir="auto">
<li>Error3:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ERROR: Cannot install apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh] 1.10.12, apache-airflow and apache-airflow because these package versions have conflicting dependencies.

The conflict is caused by:
    apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh] 1.10.12 depends on pytest&lt;6.0.0; extra == &quot;devel_hadoop&quot;
    pytest-rerunfailures 9.0 depends on pytest&gt;=5.0
    pytest-xdist 2.0.0 depends on pytest&gt;=6.0.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies"><pre class="notranslate"><code class="notranslate">ERROR: Cannot install apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh] 1.10.12, apache-airflow and apache-airflow because these package versions have conflicting dependencies.

The conflict is caused by:
    apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh] 1.10.12 depends on pytest&lt;6.0.0; extra == "devel_hadoop"
    pytest-rerunfailures 9.0 depends on pytest&gt;=5.0
    pytest-xdist 2.0.0 depends on pytest&gt;=6.0.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
</code></pre></div>
<p dir="auto">My solution3:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install pytest==6.0.0"><pre class="notranslate"><code class="notranslate">pip install pytest==6.0.0
</code></pre></div>
<ol start="4" dir="auto">
<li>Error4:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  Downloading eventlet-0.14.0.tar.gz (279 kB)
     |████████████████████████████████| 279 kB 404 kB/s
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '&quot;'&quot;'/tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/setup.py'&quot;'&quot;'; __file__='&quot;'&quot;'/tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/setup.py'&quot;'&quot;';f=getattr(tokenize, '&quot;'&quot;'open'&quot;'&quot;', open)(__file__);code=f.read().replace('&quot;'&quot;'\r\n'&quot;'&quot;', '&quot;'&quot;'\n'&quot;'&quot;');f.close();exec(compile(code, __file__, '&quot;'&quot;'exec'&quot;'&quot;'))' egg_info --egg-base /tmp/pip-pip-egg-info-aw4qpa8h
         cwd: /tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/
    Complete output (8 lines):
    Traceback (most recent call last):
      File &quot;&lt;string&gt;&quot;, line 1, in &lt;module&gt;
      File &quot;/tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/setup.py&quot;, line 3, in &lt;module&gt;
        from eventlet import __version__
      File &quot;/tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/eventlet/__init__.py&quot;, line 42
        except ImportError, e:
                          ^
    SyntaxError: invalid syntax
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output."><pre class="notranslate"><code class="notranslate">  Downloading eventlet-0.14.0.tar.gz (279 kB)
     |████████████████████████████████| 279 kB 404 kB/s
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-aw4qpa8h
         cwd: /tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/
    Complete output (8 lines):
    Traceback (most recent call last):
      File "&lt;string&gt;", line 1, in &lt;module&gt;
      File "/tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/setup.py", line 3, in &lt;module&gt;
        from eventlet import __version__
      File "/tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/eventlet/__init__.py", line 42
        except ImportError, e:
                          ^
    SyntaxError: invalid syntax
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
</code></pre></div>
<p dir="auto">My solution4:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" pip install eventlet==0.29.1 "><pre class="notranslate"><code class="notranslate"> pip install eventlet==0.29.1 
</code></pre></div>
<ol start="5" dir="auto">
<li>Error5:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="bitmq,ssh]==1.10.12) (0.4.17)
ERROR: Could not find a version that satisfies the requirement pytest&lt;6.0.0; extra == &quot;devel_hadoop&quot; (from apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh])
ERROR: No matching distribution found for pytest&lt;6.0.0; extra == &quot;devel_hadoop&quot;"><pre class="notranslate"><code class="notranslate">bitmq,ssh]==1.10.12) (0.4.17)
ERROR: Could not find a version that satisfies the requirement pytest&lt;6.0.0; extra == "devel_hadoop" (from apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh])
ERROR: No matching distribution found for pytest&lt;6.0.0; extra == "devel_hadoop"
</code></pre></div>
<p dir="auto">then install older version and i get this error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="  Downloading paramiko-2.7.1-py2.py3-none-any.whl (206 kB)
     |████████████████████████████████| 206 kB 377 kB/s
Collecting pytest-xdist
  Using cached pytest_xdist-2.0.0-py2.py3-none-any.whl (36 kB)
ERROR: Cannot install apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh] 1.10.12 and apache-airflow because these package versions have conflicting dependencies.

The conflict is caused by:
    apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh] 1.10.12 depends on pytest&lt;6.0.0; extra == &quot;devel_hadoop&quot;
    pytest-xdist 2.0.0 depends on pytest&gt;=6.0.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies"><pre class="notranslate"><code class="notranslate">  Downloading paramiko-2.7.1-py2.py3-none-any.whl (206 kB)
     |████████████████████████████████| 206 kB 377 kB/s
Collecting pytest-xdist
  Using cached pytest_xdist-2.0.0-py2.py3-none-any.whl (36 kB)
ERROR: Cannot install apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh] 1.10.12 and apache-airflow because these package versions have conflicting dependencies.

The conflict is caused by:
    apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh] 1.10.12 depends on pytest&lt;6.0.0; extra == "devel_hadoop"
    pytest-xdist 2.0.0 depends on pytest&gt;=6.0.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
</code></pre></div>
<p dir="auto">I even don't know any of my solutions even work!</p>
<p dir="auto"><strong>How to reproduce it</strong>:<br>
Run this command</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="pip install \
 apache-airflow[async,celery,devel,devel_hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh]==1.10.12 \
 --constraint &quot;https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt&quot; \
 --use-feature=2020-resolver"><pre class="notranslate">pip install \
 apache-airflow[async,celery,devel,devel_hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh]==1.10.12 \
 --constraint <span class="pl-s"><span class="pl-pds">"</span>https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt<span class="pl-pds">"</span></span> \
 --use-feature=2020-resolver</pre></div>
<p dir="auto"><strong>Anything else we need to know</strong>:<br>
According to the announcement linked below, pip will introduce a new dependency resolver in October 2020, which will be more robust but might break some existing setups. Therefore they are suggesting users to try running their pip install scripts at least once (in dev mode) with this option: --use-feature=2020-resolver to anticipate any potential issue before the new resolver becomes the default in October 2020 with pip version 20.3.<br>
<a href="https://discuss.python.org/t/announcement-pip-20-2-release/4863" rel="nofollow"></a></p>
<p dir="auto">How often does this problem occur? Once? Every time etc?<br>
Everytime, the dependency problem is a nightmare, especially with adding hadoop extras</p>
<p dir="auto"><strong>Description</strong></p>
<p dir="auto">We should consider adding <code class="notranslate">pip check</code> command in our CI system to verify that dependencies that we use (together with constrains) are fully compatible. This should be run for biggest possible user-installation (<code class="notranslate">apache-airflow[all]</code>).</p>
<p dir="auto"><strong>Use case / motivation</strong></p>
<p dir="auto">Successful <code class="notranslate">pip check</code> is often a business requirement for releasing new version of Airflow. It assure us that all dependencies we use to build and run Airflow are compatible and we won't face any unexpected errors.</p>
<p dir="auto"><strong>Related Issues</strong></p>
<p dir="auto">N/A</p>