title: How to read a value with nested fields in Spark?
body: <p>The dataset I have is riddled with nested fields. For instance the output of  <code>data.take(1)</code> gives 9 columns in which the 4th column (c4) has 3 sub-fields and the 1st column of c4 has 3 sub-fields and so on.</p>
  
  <p>The format looks a bit like so</p>
  
  <pre><code>[A,B,C,[[d1,d2,d3],D2,D3],E,[F1,[f1,[f21,f22,f23],f3,f4],F3,F4],G,H,I]
  </code></pre>
  
  <p>I would like an array of array data structure (which can be then unrolled to a single array).</p>
  
  <p>Just to make the data look clearer:</p>
  
  <pre><code>A
  B
  C
  D
    -D1
      -d1
      -d2
      -d3
    -D2
    -D3
  E
  F
    -F1
    -F2
      -f1
      -f2
        -f21
        -f22
        -f23
      -f3
      -f4
    -F3
    -F4
  G
  H
  I
  </code></pre>
  
  <p>Of course, I could write a parsing program that would recursively search for sub-fields given a record and generate this tree structure (as an array of arrays). However, I'm hoping there would be a simpler and more efficient pre-built routine in Spark that would handle this in a straight-forward manner.</p>
  
  <p>Any answer in either Spark-Scala or PySpark would be appreciated. </p>