title: Processing JSON in spark - different schemas in different files
body: <p>I have a large amount of JSON files obtained from 3rd party. They all have the same schema, except when a nested element is empty, it is represented as empty array.</p>
  
  <p>1st example</p>
  
  <pre><code>{
  ....
  "survey_data":
      {
          "2": { "question":"....", "answer_id":"....", .... },
          "3": { "question":"....", "answer_id":"....", .... },
      }
   }
  </code></pre>
  
  <p>So this is a valid JSON, the "survey_data" element is a struct_type, but with pretty complicated nested structure (with more sub-elements than in this simplified example)</p>
  
  <p>However, when the survey_data has no nested elements, it is represented as empty array:</p>
  
  <pre><code>{
  ....
  "survey_data": []
   }
  </code></pre>
  
  <p>which is obviously schematically incompatible, but I cannot affect this since the data is from 3rd party.</p>
  
  <p>When I want to load these JSON files in spark as single dataframe, spark infers the survey_data type as string, and escapes all the characters:</p>
  
  <pre><code>"survey_data":"{\"2\":{\"question\": ...
  </code></pre>
  
  <p>This is obviously not good for me, I see 2 approaches how to deal with this:</p>
  
  <ol>
  <li>somehow pre-process the files as pure text and remove [] characters ?</li>
  <li>use spark to remove the array chars, or to tell spark that the column should be a struct type ?</li>
  </ol>
  
  <p>Anybody able to hint me with a solution for this problem?</p>