title: Importing schema from json with optional value
body: <p>I'm trying to create a table from a json datasource. </p>
  
  <p>The problem is that there is a field in the json data that is not always present for every entry and looks like this. </p>
  
  <pre><code>[ { "k1" : "someValue",
      "optK" : { "nestedK" : true } },
    { "k1" : "someOtherValue" }
  ]
  </code></pre>
  
  <p>When I try to specify the optional field in the schema, all the entries without that field have all null value in the table:</p>
  
  <pre><code>columns:  k1        |  optK
  row1:   "someValue"    [true]
  row2:    null           null
  </code></pre>
  
  <p>is it possible to write a schema such that I would have null only in the column where the value is missing?</p>
  
  <p>Like this:</p>
  
  <pre><code>columns:  k1            |  optK
  row1:   "someValue"       "optV"
  row2:   "someOtherValue"   null
  </code></pre>
  
  <p>My current code:</p>
  
  <pre><code>import org.apache.spark.sql.expressions.scalalang._
  import org.apache.spark.sql.types._
  import org.apache.spark.sql.{DataFrame, Dataset, SparkSession}
  
  val session = SparkSession.builder().enableHiveSupport().getOrCreate()
  
  val schema = StructType(Seq(
    StructField("k1", StringType, false),
    StructField("optK", StructType(Seq(StructField("nestedK", BooleanType, false))), false)
  ))
  
  val df = session.read.schema(schema).json("data.json")
  df.registerTempTable("Mr_Table")
  </code></pre>