title: How can I get a distinct RDD of dicts in PySpark?
body: <p>I have an RDD of dictionaries, and I'd like to get an RDD of just the distinct elements.  However, when I try to call</p>
  
  <pre><code>rdd.distinct()
  </code></pre>
  
  <p>PySpark gives me the following error </p>
  
  <pre><code>TypeError: unhashable type: 'dict'
  
      at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:166)
      at org.apache.spark.api.python.PythonRunner$$anon$1.&lt;init&gt;(PythonRDD.scala:207)
      at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125)
      at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
      at org.apache.spark.api.python.PairwiseRDD.compute(PythonRDD.scala:342)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
      at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
      at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
      at org.apache.spark.scheduler.Task.run(Task.scala:89)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
  16/02/19 16:55:56 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
    File "/usr/local/Cellar/apache-spark/1.6.0/libexec/python/lib/pyspark.zip/pyspark/worker.py", line 111, in main
      process()
    File "/usr/local/Cellar/apache-spark/1.6.0/libexec/python/lib/pyspark.zip/pyspark/worker.py", line 106, in process
      serializer.dump_stream(func(split_index, iterator), outfile)
    File "/usr/local/Cellar/apache-spark/1.6.0/libexec/python/lib/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
    File "/usr/local/Cellar/apache-spark/1.6.0/libexec/python/lib/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
    File "/usr/local/Cellar/apache-spark/1.6.0/libexec/python/lib/pyspark.zip/pyspark/rdd.py", line 317, in func
    File "/usr/local/Cellar/apache-spark/1.6.0/libexec/python/lib/pyspark.zip/pyspark/rdd.py", line 1776, in combineLocally
    File "/usr/local/Cellar/apache-spark/1.6.0/libexec/python/lib/pyspark.zip/pyspark/shuffle.py", line 238, in mergeValues
      d[k] = comb(d[k], v) if k in d else creator(v)
  TypeError: unhashable type: 'dict'
  </code></pre>
  
  <p>I do have a key inside of the dict that I could use as the distinct element, but the documentation doesn't give any clues on how to solve this problem.</p>
  
  <p><strong>EDIT:</strong>  The content is made up of strings, arrays of strings, and a dictionary of numbers</p>
  
  <p><strong>EDIT 2:</strong>    Example of a dictionary... I'd like dicts with equal "data_fingerprint" keys to be considered equal:</p>
  
  <pre><code>{"id":"4eece341","data_fingerprint":"1707db7bddf011ad884d132bf80baf3c"}
  </code></pre>
  
  <p>Thanks</p>