title: pyspark: StopWordsRemover with user defined functions (UDF)
body: <p>I have a dataframe consisting of text and languages</p>
  <pre><code>sf = spark.createDataFrame([
      ('eng', &quot;I saw the red balloon&quot;),
      ('eng', 'She was drinking tea from a black mug'),
      ('ger','Er ging heute sehr weit'),
      ('ger','Ich habe dich seit hundert Jahren nicht mehr gesehen')
  ], [&quot;lang&quot;, &quot;text&quot;])
  display(sf)
  </code></pre>
  <p>Output:</p>
  <pre><code>+----+--------------------+
  |lang|                text|
  +----+--------------------+
  | eng|I saw the red bal...|
  | eng|She was drinking ...|
  | ger|Er ging heute seh...|
  | ger|Ich habe dich sei...|
  +----+--------------------+
  </code></pre>
  <p>I want to remove the stop word for each text, for this I create a dictionary:</p>
  <pre><code>from pyspark.ml.feature import StopWordsRemover
  
  ger_stopwords = StopWordsRemover.loadDefaultStopWords(&quot;german&quot;)
  eng_stopwords = StopWordsRemover.loadDefaultStopWords(&quot;english&quot;)
  stopwords = {'eng':eng_stopwords,
              'ger':ger_stopwords}
  </code></pre>
  <p>And now I don't understand how can I apply stop words to a col('text') using <strong>udf</strong> ? Because transform() will not suit me in this case</p>