title: spark-submit with external python packages
body: <p>I am using the python textblob package to calculate the sentiment in a spark job. In order to make the textblob package available on the spark cluster, I created a virtual environment, installed the textblob package in there and packed it using venv-pack. The resulting archive is then used as a parameter for the <code>--archives</code> flag of <code>spark-submit</code>. Eventually, I set the environment variables according to the python package management section of the spark documentation (<a href="https://spark.apache.org/docs/latest/api/python/user_guide/python_packaging.html#using-virtualenv" rel="nofollow noreferrer">https://spark.apache.org/docs/latest/api/python/user_guide/python_packaging.html#using-virtualenv</a>):</p>
  <pre><code>export PYSPARK_DRIVER_PYTHON=python # Do not set in cluster modes.
  export PYSPARK_PYTHON=./environment/bin/python
  spark-submit --archives pyspark_venv.tar.gz#environment app.py 
  </code></pre>
  <p>When I execute the spark-submit command, the application tells me that it could not find the textblob package.</p>
  <p>I am trying to make this setup work for days. I already tried using conda, pex and the --py-files option but none of it worked. Am I missing something?</p>
  <p>Eventually, I ended up installing textblob directly on the spark-master machine which seems to work, but I'd rather stick to the documented way.</p>