title: How do I remove words numerics pyspark
body: <p>I'm trying to remove only words that are numerical from my words array, but the function I created is not working correctly. When I try to view the information from my dataframe the following error message appears</p>
  
  <p>First I converted my string and word tokens</p>
  
  
  
  <pre class="lang-py prettyprint-override"><code>from pyspark.ml.feature import RegexTokenizer
  regexTokenizer = RegexTokenizer(
      inputCol="description",
      outputCol="words_withnumber",
      pattern="\\W"
  )
  
  data = regexTokenizer.transform(data)
  </code></pre>
  
  <p>I created the function to remove only the numbers</p>
  
  <pre class="lang-py prettyprint-override"><code>from pyspark.sql.functions import when,udf
  from pyspark.sql.types import BooleanType
  
  def is_digit(value):
      if value:
          return value.isdigit()
      else:
          return False
  
  is_digit_udf = udf(is_digit, BooleanType())
  </code></pre>
  
  <p>Call function</p>
  
  <pre class="lang-py prettyprint-override"><code>data = data.withColumn(
      'words_withoutnumber', 
      when(~is_digit_udf(data['words_withnumber']), data['words_withnumber'])
  )
  </code></pre>
  
  <p>Error:</p>
  
  <blockquote>
    <p>org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 5.0 failed 4 times, most recent failure: Lost task 0.3 in stage 5.0 (TID 14, 10.139.64.4, executor 0): org.apache.spark.api.python.PythonException: Traceback (most recent call last):</p>
  </blockquote>
  
  <p>Sample Dataframe</p>
  
  <pre class="lang-py prettyprint-override"><code>+-----------+-----------------------------------------------------------+
  |categoryid |description                                                |
  +-----------+-----------------------------------------------------------+
  |      33004|["short","sarja", "40567","detalhe","couro"]               | 
  |      22033|["multipane","6768686868686867868888","220v","branco"]     | 
  +-----------+-----------------------------------------------------------+
  </code></pre>
  
  <p>expected result</p>
  
  <pre class="lang-py prettyprint-override"><code>+-----------+-----------------------------------------------------------+
  |categoryid |description                                                |
  +-----------+-----------------------------------------------------------+
  |      33004|["short","sarja","detalhe","couro"]                        | 
  |      22033|["multipane","220v","branco"]                              |
  +-----------+-----------------------------------------------------------+
  </code></pre>