title: Convert Spark Dataframe Column with Seq of String to Nested Json with Dynamic Json Schema
body: <p>I have a problem statement where in,I have to create a nested json which will be stored in redis and then served to ui via rest service where it will be used to plot sunburst chart.</p>
  <p><em><strong>Json example attached below</strong></em></p>
  <p>Issue with creating nested json is that,for a json object (<strong>where category is cat_a</strong>) inside given json array I can have depth with size <strong>n</strong> which is not known before hand and for the same key (<strong>children in my case</strong>) in different json object (<strong>where category is cat_b</strong>) of the array the depth could be <strong>m</strong>.</p>
  <p>My Dataframe looks like following.</p>
  <pre><code>+----------------+----------------+----------------+
  |Category        |EventSeq        |Count           |
  +----------------+----------------+----------------+
  |cat_a           |[A]             |30              |
  |cat_a           |[A,B]           |20              |
  |cat_a           |[A,B,C]         |5               |
  |cat_a           |[A,W]           |4               |
  |cat_a           |[C]             |30              |
  |cat_a           |[C,A]           |20              |
  |cat_a           |[C,A,B]         |5               |
  |cat_b           |[X]             |30              |
  |cat_b           |[X,Y]           |20              |
  |cat_b           |[X,Y,Z]         |5               |
  |cat_b           |[Z]             |30              |
  |cat_b           |[Z,X]           |20              |
  +----------------+----------------+----------------+
  </code></pre>
  <p>Now I want to convert this dataframe to Nested Json as follows:</p>
  <pre><code>[
     {
        &quot;category&quot;:&quot;cat_a&quot;,
        &quot;children&quot;:[
           {
              &quot;name&quot;:&quot;A&quot;,
              &quot;count&quot;:30,
              &quot;children&quot;:[
                 {
                    &quot;name&quot;:&quot;B&quot;,
                    &quot;count&quot;:20,
                    &quot;children&quot;:[
                       {
                          &quot;name&quot;:&quot;C&quot;,
                          &quot;count&quot;:5
                       }
                    ]
                 },
                 {
                    &quot;name&quot;:&quot;W&quot;,
                    &quot;count&quot;:4
                 }
              ]
           },
           {
              &quot;name&quot;:&quot;C&quot;,
              &quot;count&quot;:30,
              &quot;children&quot;:[
                 {
                    &quot;name&quot;:&quot;A&quot;,
                    &quot;count&quot;:20,
                    &quot;children&quot;:[
                       {
                          &quot;name&quot;:&quot;B&quot;,
                          &quot;count&quot;:5
                       }
                    ]
                 }
              ]
           }
        ]
     },
     {
        &quot;category&quot;:&quot;cat_b&quot;,
        &quot;children&quot;:[
           {
              &quot;name&quot;:&quot;X&quot;,
              &quot;count&quot;:30,
              &quot;children&quot;:[
                 {
                    &quot;name&quot;:&quot;Y&quot;,
                    &quot;count&quot;:20,
                    &quot;children&quot;:[
                       {
                          &quot;name&quot;:&quot;Z&quot;,
                          &quot;count&quot;:5
                       }
                    ]
                 }
              ]
           },
           {
              &quot;name&quot;:&quot;Z&quot;,
              &quot;count&quot;:30,
              &quot;children&quot;:[
                 {
                    &quot;name&quot;:&quot;X&quot;,
                    &quot;count&quot;:20
                 }
              ]
           }
        ]
     }
  ]
  </code></pre>
  <p>I don't know if it is even possible in spark with scala or not.
  If not then could it be done in Rest Service layer?</p>