title: Processing json strings in a Spark dataframe column
body: <p>When I look for ways to parse json within a string column of a dataframe, I keep running into results that more simply read json file sources. My source is actually a hive ORC table with some strings in one of the columns which is in a json format. I'd really like to convert that to something parsed like a map.</p>
  
  <p>I'm having trouble finding a way to do this:</p>
  
  <pre><code>import java.util.Date
  import org.apache.spark.sql.Row
  import scala.util.parsing.json.JSON
  
  val items = sql("select * from db.items limit 10")
  //items.printSchema
  val internal = items.map {
    case Row(externalid: Long, itemid: Long, locale: String,
             internalitem: String, version: Long,
             createdat: Date, modifiedat: Date)
         =&gt; JSON.parseFull(internalitem)
  }
  </code></pre>
  
  <p>I thought this should work, but maybe there's a more <strong>Spark</strong> way of doing this instead because I get the following error:</p>
  
  <blockquote>
  <pre><code>java.lang.ClassNotFoundException: scala.Any
   at scala.reflect.internal.util.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:62)
  </code></pre>
  </blockquote>
  
  <p>Specifically, my input data looks approximately like this:</p>
  
  <pre><code>externalid, itemid, locale, internalitem, version, createdat, modifiedat
         123,    321, "en_us", "{'name':'thing','attr':{
                                 21:{'attrname':'size','attrval':'big'},
                                 42:{'attrname':'color','attrval':'red'}
                                }}",              1, 2017-05-05…, 2017-05-06…
  </code></pre>
  
  <p>Yes it's not RFC 7158 exactly.</p>
  
  <p>The <code>attr</code> keys can be 5 to 30 of any 80,000 values, so I wanted get to something like this instead:</p>
  
  <pre><code>externalid, itemid, locale, internalitem, version, createdat, modifiedat
         123,    321, "en_us", "{"name':"thing","attr":[
                                 {"id":21,"attrname':"size","attrval":"big"},
                                 {"id":42,"attrname":"color","attrval":"red"}
                                ]}",              1, 2017-05-05…, 2017-05-06…
  </code></pre>
  
  <p>Then flatten the <code>internalitem</code> to fields and explode the <code>attr</code> array:</p>
  
  <pre><code>externalid, itemid, locale, name, attrid, attrname attrval, version, createdat, modifiedat
         123,    321, "en_us", "thing", 21,   "size",  "big",       1, 2017-05-05…, 2017-05-06…
         123,    321, "en_us", "thing", 21,  "color",  "red",       1, 2017-05-05…, 2017-05-06…
  </code></pre>