title: How to specify only particular fields using read.schema in JSON : SPARK Scala
body: <p>I am trying to programmatically enforce schema(json) on textFile which looks like json. I tried with jsonFile but the issue is for creating a dataframe from a list of json files, spark has to do a 1 pass through the data to create a schema for the dataframe. So it needs to parse all the data which is taking longer time (4 hours since my data is zipped and of size TBs). So I want to try reading it as textFile and enforce schema to get interested fields alone to later query on the resulting data frame. But I am not sure how do I map it to the input. Can some give me some reference on how do I map schema to json like input.</p>
  
  <p>input :</p>
  
  <p>This is the full schema :</p>
  
  <pre><code>records: org.apache.spark.sql.DataFrame = [country: string, countryFeatures: string, customerId: string, homeCountry: string, homeCountryFeatures: string, places: array&lt;struct&lt;freeTrial:boolean,placeId:string,placeRating:bigint&gt;&gt;, siteName: string, siteId: string, siteTypeId: string, Timestamp: bigint, Timezone: string, countryId: string, pageId: string, homeId: string, pageType: string, model: string, requestId: string, sessionId: string, inputs: array&lt;struct&lt;inputName:string,inputType:string,inputId:string,offerType:string,originalRating:bigint,processed:boolean,rating:bigint,score:double,methodId:string&gt;&gt;] 
  </code></pre>
  
  <p>But I am only interested in few fields like :</p>
  
  <pre><code>res45: Array[String] = Array({"requestId":"bnjinmm","siteName":"bueller","pageType":"ad","model":"prepare","inputs":[{"methodId":"436136582","inputType":"US","processed":true,"rating":0,"originalRating":1},{"methodId":"23232322","inputType":"UK","processed":falase,"rating":0,"originalRating":1}]
  
  
   val  records = sc.textFile("s3://testData/sample.json.gz")
  
    val schema = StructType(Array(StructField("requestId",StringType,true),
                            StructField("siteName",StringType,true),
                            StructField("model",StringType,true),
                            StructField("pageType",StringType,true),
                            StructField("inputs", ArrayType(
                                  StructType(
                                              StructField("inputType",StringType,true), 
                                              StructField("originalRating",LongType,true), 
                                              StructField("processed",BooleanType,true), 
                                              StructField("rating",LongType,true), 
                                              StructField("methodId",StringType,true)
                                              ),true),true)))
  
      val rowRDD = ?? 
  
      val inputRDD = sqlContext.applySchema(rowRDD, schema)
      inputRDD.registerTempTable("input")
  
       sql("select * from input").foreach(println)
  </code></pre>
  
  <p>Is there any way to map this ? Or do I need to use son parser or something. I want to use textFile only because of the constraints. </p>
  
  <p>Tried with  :</p>
  
  <pre><code>val  records =sqlContext.read.schema(schema).json("s3://testData/test2.gz")
  </code></pre>
  
  <p>But keeping getting the error :</p>
  
  <pre><code>&lt;console&gt;:37: error: overloaded method value apply with alternatives:
       (fields: Array[org.apache.spark.sql.types.StructField])org.apache.spark.sql.types.StructType &lt;and&gt;
        (fields: java.util.List[org.apache.spark.sql.types.StructField])org.apache.spark.sql.types.StructType &lt;and&gt;
        (fields: Seq[org.apache.spark.sql.types.StructField])org.apache.spark.sql.types.StructType
       cannot be applied to (org.apache.spark.sql.types.StructField, org.apache.spark.sql.types.StructField, org.apache.spark.sql.types.StructField, org.apache.spark.sql.types.StructField, org.apache.spark.sql.types.StructField, org.apache.spark.sql.types.StructField)
             StructField("inputs",ArrayType(StructType(StructField("inputType",StringType,true), StructField("originalRating",LongType,true), StructField("processed",BooleanType,true), StructField("rating",LongType,true), StructField("score",DoubleType,true), StructField("methodId",StringType,true)),true),true)))
                                                ^
  </code></pre>