title: PySpark DF and RDD encoding of latin letters
body: <p>I have to import <code>CSV</code> file in <code>Spark</code> and convert it in <code>DF</code> and then in <code>RDD</code> before process it.</p>
  
  <p>First I import the full <code>CSV</code> file as a <code>DF</code></p>
  
  <pre><code>stopwords_df = (
      sqlc
      .read
      .format('csv')
      .option('header', True)
      .option('delimiter', ';')
      .option('encoding', 'latin1')
      .load('/mnt/sparkdata/stopwords.csv', schema = stopSchema)
      .repartition(72)
  )
  </code></pre>
  
  <p>Then I select the suitable words only and convert this into a set</p>
  
  <pre><code>stopwords_set = set(
      stopwords_df
      .filter(f.col('retain').isNull())
      .rdd
      .map(lambda x: x[0].encode('latin1')) # the [0] is to extract strings from Rows
      .collect()
  )
  </code></pre>
  
  <p>I'm messing up with the encoding and I don't know how to fix this.</p>
  
  <p>If I "show" the <code>DF</code>, the Latin letters will be shown correctly (sperò)</p>
  
  <pre><code>stopwords_df.show(100, truncate = False)
  
  +--------------+--------+------+----------+------+
  |word          |language|type  |doubletype|retain|
  +--------------+--------+------+----------+------+
  |informava     |IT      |verbo |null      |null  |
  |sperò         |IT      |verbo |null      |null  |
  |four          |EN      |null  |null      |null  |
  </code></pre>
  
  <p>But this won't happen if I show the RDD </p>
  
  <pre><code>(
      stopwords_df
      .filter(f.col('word') == r'sperò')
      .rdd
      .first()
  )
  
  Row(word=u'sper\xf2', language=u'IT', type=u'verbo', doubletype=None, retain=None)
  </code></pre>
  
  <p>With <code>UTF-8 encoding</code> it will also go worst</p>
  
  <pre><code>+--------------+--------+------+----------+------+
  |word          |language|type  |doubletype|retain|
  +--------------+--------+------+----------+------+
  |thanks        |EN      |saluto|null      |null  |
  |fossero       |IT      |verbo |null      |null  |
  |sper�         |IT      |verbo |null      |null  |
  </code></pre>
  
  <p>Could you please suggest me how to fix this? </p>