title: pyspark generate row hash of specific columns and add it as a new column
body: <p>I am working with spark 2.2.0 and pyspark2.</p>
  
  <p>I have created a DataFrame <code>df</code> and now trying to add a new column <code>"rowhash"</code> that is the sha2 hash of specific columns in the DataFrame.</p>
  
  <p>For example, say that <code>df</code> has the columns: <code>(column1, column2, ..., column10)</code></p>
  
  <p>I require <code>sha2((column2||column3||column4||...... column8), 256)</code> in a new column <code>"rowhash"</code>.</p>
  
  <p>For now, I tried using below methods:</p>
  
  <p>1) Used <code>hash()</code> function but since it gives an integer output it is of not much use</p>
  
  <p>2) Tried using <code>sha2()</code> function but it is failing.</p>
  
  <p>Say <code>columnarray</code> has array of columns I need.</p>
  
  <pre class="lang-python prettyprint-override"><code>def concat(columnarray):
      concat_str = ''
      for val in columnarray:
          concat_str = concat_str + '||' + str(val) 
      concat_str = concat_str[2:] 
      return concat_str 
  </code></pre>
  
  <p>and then </p>
  
  <pre class="lang-python prettyprint-override"><code>df1 = df1.withColumn("row_sha2", sha2(concat(columnarray),256))
  </code></pre>
  
  <p>This is failing with "cannot resolve" error.</p>
  
  <p>Thanks gaw for your answer. Since I have to hash only specific columns, I created a list of those column names (in hash_col) and changed your function as :</p>
  
  <pre class="lang-python prettyprint-override"><code> def sha_concat(row, columnarray):
     row_dict = row.asDict()      #transform row to a dict
     concat_str = '' 
     for v in columnarray: 
         concat_str = concat_str + '||' + str(row_dict.get(v)) 
     concat_str = concat_str[2:] 
     #preserve concatenated value for testing (this can be removed later)
     row_dict["sha_values"] = concat_str  
     row_dict["sha_hash"] = hashlib.sha256(concat_str).hexdigest()
     return Row(**row_dict) 
  </code></pre>
  
  <p>Then passed as : </p>
  
  <pre class="lang-python prettyprint-override"><code>    df1.rdd.map(lambda row: sha_concat(row,hash_col)).toDF().show(truncate=False)
  </code></pre>
  
  <p>It is now however failing with error: </p>
  
  <pre class="lang-python prettyprint-override"><code>    UnicodeEncodeError: 'ascii' codec can't encode character u'\ufffd' in position 8: ordinal not in range(128)
  </code></pre>
  
  <p>I can see value of \ufffd in one of the column so I am unsure if there is a way to handle this ?</p>