title: spark dataframes : reading json having duplicate column names but different datatypes
body: <p>I have json data like below where version field is the differentiator -</p>
  <p><code>file_1  = {&quot;version&quot;: 1, &quot;stats&quot;: {&quot;hits&quot;:20}}</code></p>
  <p><code>file_2  = {&quot;version&quot;: 2, &quot;stats&quot;: [{&quot;hour&quot;:1,&quot;hits&quot;:10},{&quot;hour&quot;:2,&quot;hits&quot;:12}]}</code></p>
  <p>In the new format, stats column is now <code>Arraytype(StructType)</code>.</p>
  <p>Earlier only file_1 was needed so I was using</p>
  <pre><code>spark.read.schema(schema_def_v1).json(path)
  </code></pre>
  <p>Now I need to read both these type of multiple json files which come together. I cannot define stats as string in schema_def as that would affect the <code>corruptrecord</code> feature(for stats column) which checks malformed json and schema compliance of all the fields.</p>
  <p><strong>Example df output required in 1 read only -</strong></p>
  <pre><code>version | hour | hits
  1       | null | 20
  2       | 1    | 10
  2       | 2    | 12
  </code></pre>
  <p>I have tried to read with <code>mergeSchema</code> option but that makes stats field String type.</p>
  <p>Also, I have tried making two dataframes   by filtering on the version field, and applying <code>spark.read.schema(schema_def_v1).json(df_v1.toJSON)</code>. Here also stats column becomes <code>String</code> type.</p>
  <p>I was thinking if while reading, I could parse the df column headers as <code>stats_v1</code> and <code>stats_v2</code> on basis of data-types can solve the problem. Please help with any possible solutions.</p>