How to map structured data to schemaRDD in Spark?
Question: I've asked this question differently before but there are some changes so I
thought asking it again as a new question. I have a structured data which only
part of it is in json format but I need to map the entire data to an
schemaRDD. The data looks like this:

> 03052015 04:13:20
> {"recordType":"NEW","data":{"keycol":"val1","col2":"val2","col3":"val3"}

Each line starts with date followed by time and a json formatted text. I need
to map not only the text in json but also the date and time into the same
structure.

I tried it in Python but obviously it doesn't work because Row does not take
an RDD (jsonRDD in this case).

    
    
        from pyspark.sql import SQLContext, Row
        sqlContext = SQLContext(sc)
        orderFile  = sc.textFile(myfile)
        orderLine  = orderFile.map(lambda line: line.split(" ", 2))
        anotherOrderLine = orderLine.map(lambda p: Row(date=p[0], time=p[1], content=sqlContext.jsonRDD(p[3])))
        schemaOrder = sqlContext.inferSchema(anotherOrderLine)
        schemaOrder.printSchema()
        for x in schemaOrder.collect():
            print x
    

The goal is to be able to run a query like this against the schemaRDD:

    
    
    select date, time, data.keycol, data.val1, data.val2, data.val3 from myOrder
    

How can I map the entire line to a schemaRDD?

Any help is appreciated?


Answer: The sqlContext.jsonRDD creates a schema rdd from an RDD containing strings
where each string contains a JSON representation. This code sample is from the
SparkSQL documentation (<https://spark.apache.org/docs/1.2.0/sql-programming-
guide.html>):

    
    
    val anotherPeopleRDD = sc.parallelize("""{"name":"Yin","address":{"city":"Columbus","state":"Ohio"}}""" :: Nil)
    val anotherPeople = sqlContext.jsonRDD(anotherPeopleRDD)
    

One of the cool things about jsonRDD is the fact that you can provide and
additional parameter stating the JSONs schema, which should improve
performance your performance. This can be don by creating an schemaRDD (just
load a sample document) and then call the schemaRDD.schema method to get the
schema.

