title: Aggregate JSON files in Spark RDD
body: <p>I have a series of files that look similar to this:</p>
  
  <pre><code>[
   {
    'id':1,
    'transactions': [
     {
      'date': '2019-01-01',
      'amount': 50.50
     },
     {
      'date': '2019-01-02',
      'amount': 10.20
     },
    ]
   },
   {
    'id':2,
    'transactions': [
     {
      'date': '2019-01-01',
      'amount': 10.20
     },
     {
      'date': '2019-01-02',
      'amount': 0.50
     },
    ]
   }
  ]
  </code></pre>
  
  <p>I load these files to Spark using the following code  </p>
  
  <p><code>users= spark.read.option("multiline", "true").json(file_location)</code>  </p>
  
  <p>The result is a SparkData Frame with two columns <code>id</code> and <code>transactions</code> where <code>transactions</code> is a StructType.  </p>
  
  <p>I want to be able to "map" the <code>transactions</code> per user to aggregate them.  </p>
  
  <p>Currently I am using rdd and a function that looks like this:  </p>
  
  <pre><code>users.rdd.map(lambda a: summarize_transactions(a.transactions))
  </code></pre>
  
  <p>The summarize function can be of two types:<br>
  a) Turn the list of objects into a Pandas Dataframe to summarize it.<br>
  b) Iterate over the list of objects to summarize it.  </p>
  
  <p>However I find out that <code>a.transactions</code> is a list of <code>pyspark.sql.types.Row</code>. Instead of actual dictionaries.  </p>
  
  <p>1) Is this the best way to accomplish my goal?<br>
  2) How can I turn the list of Spark Rows into the original list of Dictionaries?</p>