title: SPARK dataframe returning null when trying to apply schema to JSON data
body: <p>I'm using the SPARK Java API to read a text file, convert it to JSON, and then apply a schema to it. The schema can vary based on a mapping table in the database, which is why I need to first convert the file to JSON so the schema mapping does not have to be in column order. Here is what I've done:</p>
  
  <pre><code>// Defined the schema (basic representation)
  StructType myschema = new StructType().add("a", DataTypes.StringType, true)
                                        .add("b", DataTypes.StringType, true)
                                        .add("x", DataTypes.StringType, true)
                                        .add("y", DataTypes.IntegerType, true)
                                        .add("z", DataTypes.BooleanType, true);
  
  //Reading a pipe delimited text file as JSON, the file has less columns than myschema
  Dataset&lt;String&gt; data = spark.read().option("delimiter","|").option("header","true").csv(myFile).toJSON();
  </code></pre>
  
  <p>The above table returns something like this:</p>
  
  <pre><code>data.show(false);
  
  |value|
  +----------------------------------------+
  |      {"x":"name1","z":"true","y":"1234"}|
  |      {"x":"name2","z":"false","y":"1445"}|
  |      {"x":"name3","z":"true",:y":"1212"}|
  </code></pre>
  
  <p>My issue comes when I run this:</p>
  
  <pre><code>Dataset&lt;Row&gt; data_with_schema = spark.read().schema(myschema).json(data);
  </code></pre>
  
  <p>Because my result turns into this:</p>
  
  <pre><code>data_with_schema.show(false);
  |x|y|z|
  +-------+-------+-------+
  |null  |null  |null  |
  |null  |null  |null  |
  |null  |null  |null  |
  </code></pre>
  
  <p>I read on stackoverflow that this might be because I'm trying to cast json strings as integers. However, I tried to define the data variable as a Row Dataset instead of String Dataset but there was an Incompatible Types error. I'm not sure what the workaround is or what the real issue is. </p>