title: Lemmatizing with Spacy and Pyspark
body: <p>I have created a function to lemmatize a Pyspark DataFrame column.
  Here is the code:</p>
  
  <pre><code>def custom_lemmatizer(df,input_col, output_col) -&gt; DataFrame:
    """
        Lemmatizes the df's using input_column and returns
        a df with lemmatized words as output_column
    """
    input_col = udf(lambda x: " ".join(x))(df[input_col])
    def spacy_lemma(input_col)-&gt;list:
      nlp = spacy.load('en', disable=['parser', 'ner'])
      doc = nlp(input_col)
      return [token.lemma_ for token in doc]
  
    spacy_udf = udf(lambda string: spacy_lemma(string), ArrayType(StringType()))
    return df.withColumn(output_col, spacy_udf(input_col))
  </code></pre>
  
  <p>The problem is that when I test it, it works after around two minutes whereas if I run it in the main function with other operations such as Vectorization, Stopword removal etc; The main runs forever. It seems like it disturbs the whole process. I am confident that the main is perfectly fine as it executes other tasks efficiently. How can I solve this problem?</p>