title: Removing specific character from text in spark
body: <p>My Spark dataframe column has some weird character in there.  I wanted to remove that.When I select that particular column and do .show() I see it as below</p>
  
  <p><code>Dominant technology firm seeks ambitious, assertive, confident, headstrong salesperson to lead our organization into the next era! If you are ready to thrive in a highly competitive environment, this is the job for you. ¥ Superior oral and written communication skills¥ Extensive experience with negotiating and closing sales ¥ Outspoken ¥ Thrives in competitive environment¥ Self-reliant and able to succeed in an independent setting  ¥ Manage portfolio of clients ¥ Aggressively close sales to exceed quarterly quotas ¥ Deliver expertise to clients as needed ¥ Lead the company into new markets</code>            |</p>
  
  <p>The character as you see is ¥. </p>
  
  <p>I wrote the following code to remove this from the 'description' column of data frame</p>
  
  <pre><code>from pyspark.sql.functions import udf
  
  charReplace=udf(lambda x: x.replace('¥',''))
  
  train_cleaned=train_triLabel.withColumn('dsescription',charReplace('description'))
  train_cleaned.show(2,truncate=False)
  </code></pre>
  
  <p>It however throws an error: </p>
  
  <pre><code>File "/Users/i854319/spark/python/lib/pyspark.zip/pyspark/serializers.py", line 263, in dump_stream
      vs = list(itertools.islice(iterator, batch))
    File "/Users/i854319/spark/python/pyspark/sql/functions.py", line 1563, in &lt;lambda&gt;
      func = lambda _, it: map(lambda x: returnType.toInternal(f(*x)), it)
    File "&lt;ipython-input-32-864efe6f3257&gt;", line 3, in &lt;lambda&gt;
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 0: ordinal not in range(128)
  </code></pre>
  
  <p>However when I do it on a test string the character is recognized by the replace method. </p>
  
  <pre><code>s='hello ¥'
  print s
  s.replace('¥','')
  ​
  hello ¥
  Out[37]:
  'hello '
  </code></pre>
  
  <p>Any idea where I am going wrong?</p>