title: How to use PySpark to run SQL aggregations on Cassandra row values?
body: <p>I am currently using issuing 1000s of queries in python to cassandra. The row values looks something like this</p>
  
  <pre><code>{'company_id': 1234, 
   'locations': ['sf', 'sj'...], 
   'attributes':[{'action': 'page_visit', 'timestamp': 123, 'address': reddit.com/asdf}, 
                 {'action': 'email', 'timestamp': 125', 'subject': 'stuff},
                 {'action': 'update', 'timestamp': 127, 'email': 'new_email@gmail.com'}]
    ...}
  </code></pre>
  
  <p>I process the data by picking out specific fields and subfields such as company_id, page visits from the attributes field and then running aggregations.  The rest of the table has deeper nestings.</p>
  
  <p>I want to use PySpark to parallelize the queries and issue SQL (not CQL) queries on the row values.  The rows are nested json so I would need to write a custom mapping from the json to a columnar object. Is this possible without writing to a temporary table?</p>