title: Import error occurs while using pyspark udf
body: <p>I'm trying to run spark application using spark-submit.
  I've created the followig udf:</p>
  <pre><code>from pyspark.sql.functions import udf
  from pyspark.sql.types import StringType
  from tldextract import tldextract
  
  @udf(StringType())
  def get_domain(url):
      ext = tldextract.extract(url)
      return ext.domain
  </code></pre>
  <p>Then I used it like this:</p>
  <pre><code>df = df.withColumn('domain', col=get_domain(df['url']))
  </code></pre>
  <p>And got the following error:</p>
  <pre><code>Driver stacktrace:
  21/01/03 16:53:41 INFO DAGScheduler: Job 1 failed: showString at NativeMethodAccessorImpl.java:0, took 2.842401 s
  Traceback (most recent call last):
    File &quot;/home/michal/dv-etl/main.py&quot;, line 54, in &lt;module&gt;
      main()
    File &quot;/home/michal/dv-etl/main.py&quot;, line 48, in main
      df.show(truncate=False)
    File &quot;/opt/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py&quot;, line 442, in show
    File &quot;/opt/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py&quot;, line 1304, in __call__
    File &quot;/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py&quot;, line 134, in deco
    File &quot;&lt;string&gt;&quot;, line 3, in raise_from
  pyspark.sql.utils.PythonException: 
    An exception was thrown from the Python worker. Please see the stack trace below.
  Traceback (most recent call last):
    File &quot;/opt/spark/python/lib/pyspark.zip/pyspark/worker.py&quot;, line 589, in main
      func, profiler, deserializer, serializer = read_udfs(pickleSer, infile, eval_type)
    File &quot;/opt/spark/python/lib/pyspark.zip/pyspark/worker.py&quot;, line 447, in read_udfs
      udfs.append(read_single_udf(pickleSer, infile, eval_type, runner_conf, udf_index=i))
    File &quot;/opt/spark/python/lib/pyspark.zip/pyspark/worker.py&quot;, line 254, in read_single_udf
      f, return_type = read_command(pickleSer, infile)
    File &quot;/opt/spark/python/lib/pyspark.zip/pyspark/worker.py&quot;, line 74, in read_command
      command = serializer._read_with_length(file)
    File &quot;/opt/spark/python/lib/pyspark.zip/pyspark/serializers.py&quot;, line 172, in _read_with_length
      return self.loads(obj)
    File &quot;/opt/spark/python/lib/pyspark.zip/pyspark/serializers.py&quot;, line 458, in loads
      return pickle.loads(obj, encoding=encoding)
    File &quot;/opt/spark/python/lib/pyspark.zip/pyspark/cloudpickle.py&quot;, line 1110, in subimport
      __import__(name)
  ModuleNotFoundError: No module named 'tldextract'
  </code></pre>
  <p>I zipped the dependencies and used an egg file and it still doesn't work.
  My spark cluser consists from a master and one worker on the same server.
  I suspect it is the udf, because when I use <code>tldextract</code> in a regular function and not in the udf it works.</p>
  <p>This is my <code>spark-submit</code> command:</p>
  <pre><code>spark-submit --master spark://spark-server:7077 main.py --py-files dist/app-0.0.1-py3.8.egg requirements.zip
  </code></pre>
  <p>Thank you!</p>